Single-core FreeRTOS, ISR stops running on the non-FreeRTOS core

slimhazard commented

2024-06-28 02:08:33 -05:00

(Migrated from github.com)

This problem was discussed on the SDK forum here, where @kilograham thought it sounds like a bug.

The issue arises with single-core FreeRTOS (configNUMBER_OF_CORES=1), starting the task scheduler on core0, and an ISR running on core1. The same code was also tested with threadsafe background mode, poll mode, and SMP FreeRTOS, and the problem doesn't happen in any of those versions.

On core1, the ADC is set to free-running mode, and this ISR is initiated to trigger on ADC_IRQ_FIFO. It saves the value from adc_fifo_get() in a static variable:

static volatile uint16_t temp_adc_raw;
static volatile unsigned long isr_runs = 0;
static volatile absolute_time_t last_isr_t;

static void
__time_critical_func(temp_isr)(void)
{
	uint16_t val = adc_fifo_get();
	critical_section_enter_blocking(&temp_critsec);
	temp_adc_raw = val;
	critical_section_exit(&temp_critsec);
	isr_runs++;
	last_isr_t = get_absolute_time();
}

isr_runs and last_isr_t are debugging variables, to count the number of ISR runs and the time of the last run. The critical section is initialized in main().

On core0, the FreeRTOS task scheduler runs a task that gets a WiFi connection and initializes lwIP callbacks. The lwIP calls are then managed by the task scheduler. The app uses this function to retrieve the latest ADC value, called from the lwIP callbacks on core0:

uint32_t
get_temp(void)
{
	uint16_t raw;

	critical_section_enter_blocking(&temp_critsec);
	raw = temp_adc_raw;
	critical_section_exit(&temp_critsec);
	printf("isr_runs=%u last_isr_t=%llu\n", isr_runs,
	       to_us_since_boot(last_isr_t));

	/* ... do some computation and return a value ... */
}

The printf output shows how often the ISR ran, and when it ran last in µs since boot.

In the problematic version of the app, when a network client makes a request and the lwIP callback calls get_temp(), the output shows that the ISR ran a few thousand times, then stopped:

isr_runs=12133 last_isr_t=5549347

These values never increase. The effect is that the network client gets back the value that was saved on the last ISR run, rather than continuously updated values derived from the ADC.

When I run the app in the other modes (threadsafe background, poll, and SMP FreeRTOS), the debug outputs always increase, and the network client gets the most recent value from the ADC.

This has been tested with SDK 1.5.1 and both of FreeRTOS V11.1.0 (current latest) and V10.6.2, which was the last version before the smp branch was merged to main. The same thing happened with both FreeRTOS versions.

In the same app, I have an async_context at-time worker that is also started on core1. It does the intercore synchronization in the same way as with the ISR -- the recurring worker function saves a static value, and the lwIP calls from core0 retrieve it, protected by a critical section. The network client always gets an updated value, including in the single-core FreeRTOS version. So although the ISR has stopped, the at-time worker continues running.

There is evidence that the ISR may have stopped around the time that the WiFi connection reaches the linkup state (which is done by the initializer task run by FreeRTOS on core0). I'm also saving a timestamp right after cyw43_tcpip_link_status() returns CYW43_LINK_UP, whose us_since_boot value is shown here as t:

t=5549470
[...]
isr_runs=12133 last_isr_t=5549347

The difference between the LINK_UP timestamp and the last ISR timestamp was 123µs in this case. The deltas I've seen here are usually in the 100s of µs range, always <= 1ms.

So to summarize:

An ISR triggered on ADC_IRQ_FIFO for a free-running ADC stops for some reason after a few thousand runs.
This happens when the app runs with single-core FreeRTOS, but not with the non-FreeRTOS choices for cyw43_arch, and not with SMP FreeRTOS.
The ISR is stopped at a time that is apparently close to the time when the WiFi connection, executed in a FreeRTOS task, reaches the linkup state.

This problem was discussed on the SDK forum [here](https://forums.raspberrypi.com/viewtopic.php?p=2232662), where @kilograham thought it sounds like a bug. The issue arises with single-core FreeRTOS (`configNUMBER_OF_CORES=1`), starting the task scheduler on core0, and an ISR running on core1. The same code was also tested with threadsafe background mode, poll mode, and SMP FreeRTOS, and the problem doesn't happen in any of those versions. On core1, the ADC is set to free-running mode, and this ISR is initiated to trigger on `ADC_IRQ_FIFO`. It saves the value from `adc_fifo_get()` in a static variable: ``` static volatile uint16_t temp_adc_raw; static volatile unsigned long isr_runs = 0; static volatile absolute_time_t last_isr_t; static void __time_critical_func(temp_isr)(void) { uint16_t val = adc_fifo_get(); critical_section_enter_blocking(&temp_critsec); temp_adc_raw = val; critical_section_exit(&temp_critsec); isr_runs++; last_isr_t = get_absolute_time(); } ``` `isr_runs` and `last_isr_t` are debugging variables, to count the number of ISR runs and the time of the last run. The critical section is initialized in `main()`. On core0, the FreeRTOS task scheduler runs a task that gets a WiFi connection and initializes lwIP callbacks. The lwIP calls are then managed by the task scheduler. The app uses this function to retrieve the latest ADC value, called from the lwIP callbacks on core0: ``` uint32_t get_temp(void) { uint16_t raw; critical_section_enter_blocking(&temp_critsec); raw = temp_adc_raw; critical_section_exit(&temp_critsec); printf("isr_runs=%u last_isr_t=%llu\n", isr_runs, to_us_since_boot(last_isr_t)); /* ... do some computation and return a value ... */ } ``` The printf output shows how often the ISR ran, and when it ran last in µs since boot. In the problematic version of the app, when a network client makes a request and the lwIP callback calls `get_temp()`, the output shows that the ISR ran a few thousand times, then stopped: ``` isr_runs=12133 last_isr_t=5549347 ``` These values never increase. The effect is that the network client gets back the value that was saved on the last ISR run, rather than continuously updated values derived from the ADC. When I run the app in the other modes (threadsafe background, poll, and SMP FreeRTOS), the debug outputs always increase, and the network client gets the most recent value from the ADC. This has been tested with SDK 1.5.1 and both of FreeRTOS V11.1.0 (current latest) and V10.6.2, which was the last version before the smp branch was merged to main. The same thing happened with both FreeRTOS versions. In the same app, I have an `async_context` at-time worker that is also started on core1. It does the intercore synchronization in the same way as with the ISR -- the recurring worker function saves a static value, and the lwIP calls from core0 retrieve it, protected by a critical section. The network client always gets an updated value, including in the single-core FreeRTOS version. So although the ISR has stopped, the at-time worker continues running. There is evidence that the ISR may have stopped around the time that the WiFi connection reaches the linkup state (which is done by the initializer task run by FreeRTOS on core0). I'm also saving a timestamp right after `cyw43_tcpip_link_status()` returns `CYW43_LINK_UP`, whose us_since_boot value is shown here as `t`: ``` t=5549470 [...] isr_runs=12133 last_isr_t=5549347 ``` The difference between the `LINK_UP` timestamp and the last ISR timestamp was 123µs in this case. The deltas I've seen here are usually in the 100s of µs range, always <= 1ms. So to summarize: - An ISR triggered on `ADC_IRQ_FIFO` for a free-running ADC stops for some reason after a few thousand runs. - This happens when the app runs with single-core FreeRTOS, but not with the non-FreeRTOS choices for cyw43_arch, and not with SMP FreeRTOS. - The ISR is stopped at a time that is apparently close to the time when the WiFi connection, executed in a FreeRTOS task, reaches the linkup state.

slimhazard commented

2024-06-28 02:22:50 -05:00

(Migrated from github.com)

For background, this is how the ADC and ISR are initiated on core1 -- nothing surprising, I think:

	adc_init();
	adc_set_temp_sensor_enabled(true);
	adc_select_input(ADC_TEMP_CH);
	/* Write to FIFO length 1, and retain the ERR bit. */
	adc_fifo_setup(true, false, 1, true, false);
	adc_set_clkdiv(ADC_MAX_CLKDIV);

	irq_set_exclusive_handler(ADC_IRQ_FIFO, temp_isr);
	adc_irq_set_enabled(true);
	irq_set_enabled(ADC_IRQ_FIFO, true);
	adc_run(true);

ADC_MAX_CLKDIV is set to the largest possible divider for the ADC (65535.f + 255.f/256.f) -- run as slowly as possible.

In case you're wondering what it's all for, I'm developing the FreeRTOS apps for this project:

https://gitlab.com/slimhazard/picow-http-example

Which is just sample applications to demonstrate the HTTP library in this project:

https://gitlab.com/slimhazard/picow_http

So the lwIP callbacks implement the http server. The idea being that the server can be used with any choice of a cyw43_arch, and the sample apps show you how. Up to now, I've been saying that the HTTP lib doesn't support FreeRTOS, but that limitation can be removed now.

The problem may be a bug of my own, of course, but I suspect that it has less to do with the specifics of the HTTP library, and more about how to initiate WiFi and lwIP with FreeRTOS on one core, and run an ISR without FreeRTOS on the other core.

For background, this is how the ADC and ISR are initiated on core1 -- nothing surprising, I think: ``` adc_init(); adc_set_temp_sensor_enabled(true); adc_select_input(ADC_TEMP_CH); /* Write to FIFO length 1, and retain the ERR bit. */ adc_fifo_setup(true, false, 1, true, false); adc_set_clkdiv(ADC_MAX_CLKDIV); irq_set_exclusive_handler(ADC_IRQ_FIFO, temp_isr); adc_irq_set_enabled(true); irq_set_enabled(ADC_IRQ_FIFO, true); adc_run(true); ``` `ADC_MAX_CLKDIV` is set to the largest possible divider for the ADC (65535.f + 255.f/256.f) -- run as slowly as possible. In case you're wondering what it's all for, I'm developing the FreeRTOS apps for this project: https://gitlab.com/slimhazard/picow-http-example Which is just sample applications to demonstrate the HTTP library in this project: https://gitlab.com/slimhazard/picow_http So the lwIP callbacks implement the http server. The idea being that the server can be used with any choice of a cyw43_arch, and the sample apps show you how. Up to now, I've been saying that the HTTP lib doesn't support FreeRTOS, but that limitation can be removed now. The problem may be a bug of my own, of course, but I suspect that it has less to do with the specifics of the HTTP library, and more about how to initiate WiFi and lwIP with FreeRTOS on one core, and run an ISR without FreeRTOS on the other core.

slimhazard commented

2024-06-28 04:21:24 -05:00

(Migrated from github.com)

I've found a workaround, or a fix, depending on how you look at it.

I've added a semaphore that ADC initialization (as shown in the last comment) now waits for before doing anything else:

	sem_acquire_blocking(up);
	adc_init();
	/* ... and so forth as above ... */

The semaphore is signaled from the core0 initialization task after the network reaches the linkup state. So the ADC/IRQ initialization does not proceed until after that happens.

With that, everything works as expected. The debugging output shows that the number of ISR runs and the time of the last run increase on every get_temp() call, and the network client gets up-to-date values that originated from the ADC.

So the solution appears to be: don't start an ISR on a non-FreeRTOS core until the network is brought up by a task running on the FreeRTOS core. That is evidently only necessary in the single-core FreeRTOS configuration.

(I suspect that the ADC init code can run before waiting, and only the ISR enable has to wait for the semaphore.)

I'd say this is either a workaround for a bug in FreeRTOS or in the RP2040 port, or a fix for a bug in my own code, depending on whether this is considered works as designed for FreeRTOS and/or the port.

I personally would say that it's worth looking into why this happens in the port, since the wait is only necessary in the one specific configuration. I wouldn't have expected that an ISR on one core gets stopped unless it waits until after the link is brought up on the other core.

I've found a workaround, or a fix, depending on how you look at it. I've added a semaphore that ADC initialization (as shown in the last comment) now waits for before doing anything else: ``` sem_acquire_blocking(up); adc_init(); /* ... and so forth as above ... */ ``` The semaphore is signaled from the core0 initialization task after the network reaches the linkup state. So the ADC/IRQ initialization does not proceed until after that happens. With that, everything works as expected. The debugging output shows that the number of ISR runs and the time of the last run increase on every `get_temp()` call, and the network client gets up-to-date values that originated from the ADC. So the solution appears to be: don't start an ISR on a non-FreeRTOS core until the network is brought up by a task running on the FreeRTOS core. That is evidently only necessary in the single-core FreeRTOS configuration. (I suspect that the ADC init code can run before waiting, and only the ISR enable has to wait for the semaphore.) I'd say this is either a workaround for a bug in FreeRTOS or in the RP2040 port, or a fix for a bug in my own code, depending on whether this is considered works as designed for FreeRTOS and/or the port. I personally would say that it's worth looking into why this happens in the port, since the wait is only necessary in the one specific configuration. I wouldn't have expected that an ISR on one core gets stopped unless it waits until after the link is brought up on the other core.

kilograham commented

2024-06-28 08:04:56 -05:00

(Migrated from github.com)

It is surprising that this works on SMP and not otherwise; the most common issue is with IRQ setup on SMP if you don't pin the setup task, as it may migrate from one core to the other at inconvenient times when not pinned.

Can you attach a ZIP for repro?

It is surprising that this works on SMP and not otherwise; the most common issue is with IRQ setup on SMP if you don't pin the setup task, as it may migrate from one core to the other at inconvenient times when not pinned. Can you attach a ZIP for repro?

slimhazard commented

2024-06-28 12:10:04 -05:00

(Migrated from github.com)

The workaround doesn't work as well as I thought. The code sample from my previous comment left out debugging output that I believed to be irrelevant:

	sem_acquire_blocking(up);
	puts("Doesn't work without this output");

But as the message says, if I leave out the puts() line, the fix doesn't work. I tried a few things to fix that, but can't explain it.

@kilograham I pushed a branch just for this issue, can you use that to try to reproduce?

https://gitlab.com/slimhazard/picow-http-example/-/tree/issue-1751

If so, there are some build requirements:

https://gitlab.com/slimhazard/picow_http/-/wikis/required-software

At minimum you need gperf and the four Python classes. I don't think that's too much of a PITA, but I guess I'm biased.

I think you'll be all right with the Quick Start:

https://gitlab.com/slimhazard/picow-http-example/-/tree/issue-1751#quick-start

cmake -DWIFI_SSID and -DWIFI_PASSWORD are just like in pico-examples. You only need -DHOSTNAME to test the TLS examples, and I wouldn't bother, that has nothing to do with the issue. -DCMAKE_BUILD_TYPE=Debug helps with some extra debug output.

The four relevant test apps are:

picow-http-example-freertos-sys-single (this was the problem case)
picow-http-example-freertos-sys-smp
picow-http-example-background
picow-http-example-poll

When you deploy one of those and if all goes well, you first see progress getting the wifi connection in the console, and eventually "http started". Then you can point a browser to http://picow (or whatever host name you chose), and you should see something like this:

https://gitlab.com/slimhazard/picow-http-example/-/wikis/Screenshots

The temperature value gets updated by Javascript every few seconds, or you can invoke it directly with curl http://picow/temp. The issue here was that the value should change occasionally; when the ISR stopped, it never changed. As you know, the on-die sensor doesn't change especially rapidly, even when all is working well. If I want it change faster, I put my finger on the RP2040 or blow a fan on it to make it warmer or cooler.

Let me know if you have any questions.

The workaround doesn't work as well as I thought. The code sample from my previous comment left out debugging output that I believed to be irrelevant: ``` sem_acquire_blocking(up); puts("Doesn't work without this output"); ``` But as the message says, if I leave out the `puts()` line, the fix doesn't work. I tried a few things to fix that, but can't explain it. @kilograham I pushed a branch just for this issue, can you use that to try to reproduce? https://gitlab.com/slimhazard/picow-http-example/-/tree/issue-1751 If so, there are some build requirements: https://gitlab.com/slimhazard/picow_http/-/wikis/required-software At minimum you need gperf and the four Python classes. I don't think that's too much of a PITA, but I guess I'm biased. I think you'll be all right with the Quick Start: https://gitlab.com/slimhazard/picow-http-example/-/tree/issue-1751#quick-start cmake `-DWIFI_SSID` and `-DWIFI_PASSWORD` are just like in pico-examples. You only need `-DHOSTNAME` to test the TLS examples, and I wouldn't bother, that has nothing to do with the issue. `-DCMAKE_BUILD_TYPE=Debug` helps with some extra debug output. The four relevant test apps are: - picow-http-example-freertos-sys-single (this was the problem case) - picow-http-example-freertos-sys-smp - picow-http-example-background - picow-http-example-poll When you deploy one of those and if all goes well, you first see progress getting the wifi connection in the console, and eventually "http started". Then you can point a browser to http://picow (or whatever host name you chose), and you should see something like this: https://gitlab.com/slimhazard/picow-http-example/-/wikis/Screenshots The temperature value gets updated by Javascript every few seconds, or you can invoke it directly with `curl http://picow/temp`. The issue here was that the value should change occasionally; when the ISR stopped, it never changed. As you know, the on-die sensor doesn't change especially rapidly, even when all is working well. If I want it change faster, I put my finger on the RP2040 or blow a fan on it to make it warmer or cooler. Let me know if you have any questions.

kilograham commented

2024-08-25 13:26:30 -05:00

(Migrated from github.com)

Might be worth retrying with the latest FreeRTOS here https://github.com/raspberrypi/FreeRTOS-Kernel

slimhazard commented

2024-08-30 16:17:51 -05:00

(Migrated from github.com)

Might be worth retrying with the latest FreeRTOS here https://github.com/raspberrypi/FreeRTOS-Kernel

Do I need to use SDK 2.0.0 to build with that version? I tried it with version 1.5.1, and the app doesn't appear to be happy with it at all. Not just the single-core variant, but also the SMP version that I previously had working is now not working.

I'll have to investigate more to find out why. But if SDK 2.0.0 is required, then that explains it.

> Might be worth retrying with the latest FreeRTOS here https://github.com/raspberrypi/FreeRTOS-Kernel Do I need to use SDK 2.0.0 to build with that version? I tried it with version 1.5.1, and the app doesn't appear to be happy with it at all. Not just the single-core variant, but also the SMP version that I previously had working is now not working. I'll have to investigate more to find out why. But if SDK 2.0.0 is required, then that explains it.

slimhazard commented

2024-09-06 15:20:22 -05:00

(Migrated from github.com)

To correct my previous comment, I can run the SMP version successfully. I'm using SDK 1.5.1, and the FreeRTOS version from the raspberrypi repo as @kilograham advised above, at commit 4f7299d.

The single-core version is now failing configASSERT( pxTCB == pxCurrentTCB ) in xTaskPriorityDisinherit(), line 6700 of tasks.c. The call stack includes async_context_freertos_release_lock(), called by async_context_freertos_add_at_time_worker() which in turn was called by async_context_add_at_time_worker_in_ms() from my own code. Full stack trace below.

However, the async_context code called is called on the core that isn't running the FreeRTOS task scheduler. Is that the fundamental conceptual problem? My hypothesis: in a build that includes pico_cyw43_arch_lwip_sys_freertos in target_link_libraries, every async_context worker is assumed to be a FreeRTOS task. With configNUMBER_OF_CORES=1, if it's launched from a core that isn't running the scheduler, it's bound to fail.

Here's what my app does:

main() calls multicore_launch_core1(core1_main)
On core1, which is not running the scheduler, async_context workers are started.
On core0, main() goes on to create a FreeRTOS task and launches the scheduler.

Now I'm thinking that that's doomed to fail, because in the second step the workers are assumed to be FreeRTOS tasks. Is that right?

The assertion has something to do with holding a mutex, and might be bypassed with configUSE_MUTEXES != 1, but if my guess is right that's kind of beside the point. If FreeRTOS tasks are running around that the scheduler doesn't know about, something is bound to fail.

If this explanation is correct, then it means that single-core FreeRTOS can only succeed if the non-FreeRTOS core does not use async_context workers -- right?

Stack trace:

#0  _exit (status=status@entry=1)
    at /home/me/Develop/pico/pico-sdk/src/rp2_common/pico_runtime/runtime.c:187
#1  0x1001148a in __assert_func (
    file=file@entry=0x1005c760 "/home/geoff/Develop/pico/picow-http-example/lib/FreeRTOS-Kernel/tasks.c", line=line@entry=6700, 
    func=func@entry=0x1005c6b8 <__func__.6232> "xTaskPriorityDisinherit", 
    failedexpr=failedexpr@entry=0x1005c9ac "pxTCB == pxCurrentTCB")
    at /home/me/Develop/pico/pico-sdk/src/rp2_common/pico_runtime/runtime.c:271
#2  0x1001aa80 in xTaskPriorityDisinherit (
    pxMutexHolder=0x200031c8 <ucHeap+5304>)
    at /home/me/Develop/pico/picow-http-example/lib/FreeRTOS-Kernel/tasks.c:6700
#3  0x1001904c in prvCopyDataToQueue (
    pxQueue=pxQueue@entry=0x200043c0 <ucHeap+9904>, 
    pvItemToQueue=pvItemToQueue@entry=0x0, xPosition=xPosition@entry=0)
    at /home/me/Develop/pico/picow-http-example/lib/FreeRTOS-Kernel/queue.c:2408
#4  0x10019440 in xQueueGenericSend (
    xQueue=xQueue@entry=0x200043c0 <ucHeap+9904>, 
    pvItemToQueue=pvItemToQueue@entry=0x0, xTicksToWait=<optimized out>, 
    xTicksToWait@entry=0, xCopyPosition=0)
    at /home/me/Develop/pico/picow-http-example/lib/FreeRTOS-Kernel/queue.c:975
#5  0x100195c4 in xQueueGiveMutexRecursive (xMutex=0x200043c0 <ucHeap+9904>)
    at /home/me/Develop/pico/picow-http-example/lib/FreeRTOS-Kernel/queue.c:787
#6  0x10017fb2 in async_context_freertos_release_lock (
    self_base=self_base@entry=0x20001030 <cyw43_async_context_freertos>)
    at /home/me/Develop/pico/pico-sdk/src/rp2_common/pico_async_context/async_context_freertos.c:226
#7  0x1001816c in async_context_freertos_add_at_time_worker (
    self_base=0x20001030 <cyw43_async_context_freertos>, 
    worker=<optimized out>)
    at /home/me/Develop/pico/pico-sdk/src/rp2_common/pico_async_context/async_context_freertos.c:235
#8  0x1000063c in async_context_add_at_time_worker_in_ms (ms=0, 
    worker=0x20000470 <rssi_worker>, 
    context=0x20001030 <cyw43_async_context_freertos>)
    at /home/me/Develop/pico/pico-sdk/src/rp2_common/pico_async_context/include/pico/async_context.h:317
#9  initiate_rssi (params=params@entry=0x20021e28 <linkup>)
    at /home/me/Develop/pico/picow-http-example/src/tasks.c:215
#10 0x10000364 in core1_main ()
    at /home/me/Develop/pico/picow-http-example/src/main-freertos.c:40
#11 0x100182e4 in core1_wrapper (entry=0x10000355 <core1_main>, 
    stack_base=<optimized out>)
    at /home/me/Develop/pico/pico-sdk/src/rp2_common/pico_multicore/multicore.c:97
#12 0x0000017e in ?? ()

To correct my previous comment, I can run the SMP version successfully. I'm using SDK 1.5.1, and the FreeRTOS version from the raspberrypi repo as @kilograham advised above, at commit 4f7299d. The single-core version is now failing `configASSERT( pxTCB == pxCurrentTCB )` in `xTaskPriorityDisinherit()`, [line 6700 of tasks.c](https://github.com/raspberrypi/FreeRTOS-Kernel/blob/main/tasks.c#L6700). The call stack includes `async_context_freertos_release_lock()`, called by `async_context_freertos_add_at_time_worker()` which in turn was called by `async_context_add_at_time_worker_in_ms()` from my own code. Full stack trace below. However, the async_context code called is called on the core that isn't running the FreeRTOS task scheduler. Is that the fundamental conceptual problem? My hypothesis: in a build that includes `pico_cyw43_arch_lwip_sys_freertos` in `target_link_libraries`, every async_context worker is assumed to be a FreeRTOS task. With `configNUMBER_OF_CORES=1`, if it's launched from a core that isn't running the scheduler, it's bound to fail. Here's what my app does: - `main()` calls `multicore_launch_core1(core1_main)` - On core1, which is not running the scheduler, async_context workers are started. - On core0, `main()` goes on to create a FreeRTOS task and launches the scheduler. Now I'm thinking that that's doomed to fail, because in the second step the workers are assumed to be FreeRTOS tasks. Is that right? The assertion has something to do with holding a mutex, and might be bypassed with `configUSE_MUTEXES != 1`, but if my guess is right that's kind of beside the point. If FreeRTOS tasks are running around that the scheduler doesn't know about, something is bound to fail. If this explanation is correct, then it means that single-core FreeRTOS can only succeed if the non-FreeRTOS core does not use async_context workers -- right? Stack trace: ``` #0 _exit (status=status@entry=1) at /home/me/Develop/pico/pico-sdk/src/rp2_common/pico_runtime/runtime.c:187 #1 0x1001148a in __assert_func ( file=file@entry=0x1005c760 "/home/geoff/Develop/pico/picow-http-example/lib/FreeRTOS-Kernel/tasks.c", line=line@entry=6700, func=func@entry=0x1005c6b8 <__func__.6232> "xTaskPriorityDisinherit", failedexpr=failedexpr@entry=0x1005c9ac "pxTCB == pxCurrentTCB") at /home/me/Develop/pico/pico-sdk/src/rp2_common/pico_runtime/runtime.c:271 #2 0x1001aa80 in xTaskPriorityDisinherit ( pxMutexHolder=0x200031c8 <ucHeap+5304>) at /home/me/Develop/pico/picow-http-example/lib/FreeRTOS-Kernel/tasks.c:6700 #3 0x1001904c in prvCopyDataToQueue ( pxQueue=pxQueue@entry=0x200043c0 <ucHeap+9904>, pvItemToQueue=pvItemToQueue@entry=0x0, xPosition=xPosition@entry=0) at /home/me/Develop/pico/picow-http-example/lib/FreeRTOS-Kernel/queue.c:2408 #4 0x10019440 in xQueueGenericSend ( xQueue=xQueue@entry=0x200043c0 <ucHeap+9904>, pvItemToQueue=pvItemToQueue@entry=0x0, xTicksToWait=<optimized out>, xTicksToWait@entry=0, xCopyPosition=0) at /home/me/Develop/pico/picow-http-example/lib/FreeRTOS-Kernel/queue.c:975 #5 0x100195c4 in xQueueGiveMutexRecursive (xMutex=0x200043c0 <ucHeap+9904>) at /home/me/Develop/pico/picow-http-example/lib/FreeRTOS-Kernel/queue.c:787 #6 0x10017fb2 in async_context_freertos_release_lock ( self_base=self_base@entry=0x20001030 <cyw43_async_context_freertos>) at /home/me/Develop/pico/pico-sdk/src/rp2_common/pico_async_context/async_context_freertos.c:226 #7 0x1001816c in async_context_freertos_add_at_time_worker ( self_base=0x20001030 <cyw43_async_context_freertos>, worker=<optimized out>) at /home/me/Develop/pico/pico-sdk/src/rp2_common/pico_async_context/async_context_freertos.c:235 #8 0x1000063c in async_context_add_at_time_worker_in_ms (ms=0, worker=0x20000470 <rssi_worker>, context=0x20001030 <cyw43_async_context_freertos>) at /home/me/Develop/pico/pico-sdk/src/rp2_common/pico_async_context/include/pico/async_context.h:317 #9 initiate_rssi (params=params@entry=0x20021e28 <linkup>) at /home/me/Develop/pico/picow-http-example/src/tasks.c:215 #10 0x10000364 in core1_main () at /home/me/Develop/pico/picow-http-example/src/main-freertos.c:40 #11 0x100182e4 in core1_wrapper (entry=0x10000355 <core1_main>, stack_base=<optimized out>) at /home/me/Develop/pico/pico-sdk/src/rp2_common/pico_multicore/multicore.c:97 #12 0x0000017e in ?? () ```

slimhazard commented

2024-09-07 15:18:55 -05:00

(Migrated from github.com)

I think I have this figured out. So that this isn't just a blog about me debugging my app, I think the relevance for the SDK is that, if FreeRTOS is running on one core, then there are constraints about what can run on the other core that hadn't been clear to me. If an async_context is initialized on the core running FreeRTOS, in this case via pico_cyw43_arch, then async_context_freertos will be assumed as the default for workers running on the non-FreeRTOS core. They will execute code as though they are FreeRTOS tasks, but not under the control of the task scheduler, and hence will almost certainly fail.

So I'm writing this down in case someone goes searching for a similar issue someday. Maybe something about these restrictions could be mentioned in a future edition of the SDK docs.

I had implemented work on the non-FreeRTOS core that was calling a function from cyw43_driver (cyw43_wifi_get_rssi()), after the pico_cyw43_arch had been initialized on the core running FreeRTOS. The cyw43_driver code on the non-FreeRTOS core makes use of the default context, which turns out to be async_context_freertos, and then bad things happen -- for example it tries to get the FreeRTOS mutex, or it checks the core number against that of the task scheduler, all of which throw assertions.

The solution was just to implement the work that involves pico_cyw43_arch as a FreeRTOS task, and run it on the FreeRTOS core. Other work that doesn't involve any async_context (in my case IRQ-driven ADC reads) can run on the non-FreeRTOS core.

I have it working with V11.1.0 of the FreeRTOS kernel again. But I think the newer version in the raspberrypi repo did a better job of revealing what the problem was, mainly by failing assertions that showed the call stack in FreeRTOS code where I hadn't expected it.

I'll update the forum thread where this got started. Unless you think I've got the explanation all wrong, I think this issue can be closed now.

I think I have this figured out. So that this isn't just a blog about me debugging my app, I think the relevance for the SDK is that, if FreeRTOS is running on one core, then there are constraints about what can run on the other core that hadn't been clear to me. If an async_context is initialized on the core running FreeRTOS, in this case via pico_cyw43_arch, then async_context_freertos will be assumed as the default for workers running on the non-FreeRTOS core. They will execute code as though they are FreeRTOS tasks, but not under the control of the task scheduler, and hence will almost certainly fail. So I'm writing this down in case someone goes searching for a similar issue someday. Maybe something about these restrictions could be mentioned in a future edition of the SDK docs. I had implemented work on the non-FreeRTOS core that was calling a function from cyw43_driver (cyw43_wifi_get_rssi()), after the pico_cyw43_arch had been initialized on the core running FreeRTOS. The cyw43_driver code on the non-FreeRTOS core makes use of the default context, which turns out to be async_context_freertos, and then bad things happen -- for example it tries to get the FreeRTOS mutex, or it checks the core number against that of the task scheduler, all of which throw assertions. The solution was just to implement the work that involves pico_cyw43_arch as a FreeRTOS task, and run it on the FreeRTOS core. Other work that doesn't involve any async_context (in my case IRQ-driven ADC reads) can run on the non-FreeRTOS core. I have it working with V11.1.0 of the FreeRTOS kernel again. But I think the newer version in the raspberrypi repo did a better job of revealing what the problem was, mainly by failing assertions that showed the call stack in FreeRTOS code where I hadn't expected it. I'll update the forum thread where this got started. Unless you think I've got the explanation all wrong, I think this issue can be closed now.

peterharperuk commented

2024-09-09 05:12:42 -05:00

(Migrated from github.com)

Yes, it's a bit non-obvious. Maybe we should have an example for this.

slimhazard commented

2024-09-09 08:15:34 -05:00

(Migrated from github.com)

I just noticed that #1540 discusses a similar initial misunderstanding (with a similar response by @peterharperuk).

peterharperuk commented

2024-11-12 07:58:02 -06:00

(Migrated from github.com)

I just noticed that https://github.com/raspberrypi/pico-sdk/issues/1540 discusses a similar initial misunderstanding (with a similar response by @peterharperuk).

Doesn't look the same to me.

> I just noticed that https://github.com/raspberrypi/pico-sdk/issues/1540 discusses a similar initial misunderstanding (with a similar response by @peterharperuk). Doesn't look the same to me.

Rows
Columns

Single-core FreeRTOS, ISR stops running on the non-FreeRTOS core #1751