When using flash_safe_execute, if the victim core does not respond within the timeout, it will be locked out forever #2454

New issue

Closed

opened 2025-05-03 03:32:55 -05:00 by jwhitham · 1 comment

jwhitham commented

2025-05-03 03:32:55 -05:00

(Migrated from github.com)

My Pico program uses two cores. Here is a sequence of events leading to the bug

bootup: core 0: call flash_safe_execute_core_init
core 0: start code on core 1
...normal operation for a while...
core 0: enter long-running interrupt handler
core 1: call flash_safe_execute with short timeout parameter
core 1: call multicore_lockout_start_block_until
core 1: send LOCKOUT_MAGIC_START and wait for an acknowledgment
core 1: timeout reached
core 1: multicore_lockout_handler returns PICO_ERROR_TIMEOUT is returned
core 0: leave long-running interrupt handler and enter multicore_lockout_handler

Observed behaviour:

core 1 continues execution as usual
core 0 stays within multicore_lockout_handler forever, in the following loop:

            while (multicore_fifo_pop_blocking_inline() != LOCKOUT_MAGIC_END) {
                tight_loop_contents(); // not tight but endless potentially
            }

Interrupts are disabled, as core 0 is waiting for multicore_lockout_end_block_until to be called.
Pico stops responding to USB and any other interrupts which are serviced on core 0.

Expected behaviour:

core 0: multicore_lockout_handler exits quickly as the lockout request has been abandoned

Analysis:
The timeout caused flash_safe_execute to abandon the attempt to lock out core 0, so LOCKOUT_MAGIC_END is never sent. Core 0 is not aware that the lockout request was abandoned and waits forever.

Workarounds:
Core 0 can be recovered by a second attempt to use flash_safe_execute. Alternatively, using a very long timeout will avoid the problem.

Steps to reproduce
The attached project flash-lockout-timeout-bug.tar.gz contains a small program which reproduces the problem.

The two cores run in loops, with cycle counts in global variables for easier debugging. One core runs flash_safe_execute periodically, while the other disables interrupts periodically. When a call to flash_safe_execute on core Y coincides with disabled interrupts on core X, flash_safe_execute returns PICO_ERROR_TIMEOUT (-2). Unfortunately this also causes core X to get stuck in multicore_lockout_handler. If core X = core 1, then USB continues to work, and the status can be seen on the serial line (the cycle count for core X stops, while the cycle count for core Y continues to increment). If core X = core 0, then USB stops working (no interrupts) but a hardware debugger can still be used to observe the cycle counts and see where core 0 has become stuck.

My Pico program uses two cores. Here is a sequence of events leading to the bug - bootup: core 0: call `flash_safe_execute_core_init` - core 0: start code on core 1 - ...normal operation for a while... - core 0: enter long-running interrupt handler - core 1: call `flash_safe_execute` with short timeout parameter - core 1: call `multicore_lockout_start_block_until` - core 1: send `LOCKOUT_MAGIC_START` and wait for an acknowledgment - core 1: timeout reached - core 1: `multicore_lockout_handler` returns `PICO_ERROR_TIMEOUT` is returned - core 0: leave long-running interrupt handler and enter `multicore_lockout_handler` **Observed behaviour**: - core 1 continues execution as usual - core 0 stays within `multicore_lockout_handler` forever, in the following loop: ``` while (multicore_fifo_pop_blocking_inline() != LOCKOUT_MAGIC_END) { tight_loop_contents(); // not tight but endless potentially } ``` - Interrupts are disabled, as core 0 is waiting for `multicore_lockout_end_block_until` to be called. - Pico stops responding to USB and any other interrupts which are serviced on core 0. **Expected behaviour**: - core 0: `multicore_lockout_handler` exits quickly as the lockout request has been abandoned **Analysis**: The timeout caused `flash_safe_execute` to abandon the attempt to lock out core 0, so `LOCKOUT_MAGIC_END` is never sent. Core 0 is not aware that the lockout request was abandoned and waits forever. **Workarounds:** Core 0 can be recovered by a second attempt to use `flash_safe_execute`. Alternatively, using a very long timeout will avoid the problem. **Steps to reproduce** The attached project flash-lockout-timeout-bug.tar.gz contains a small program which reproduces the problem. The two cores run in loops, with cycle counts in global variables for easier debugging. One core runs `flash_safe_execute` periodically, while the other disables interrupts periodically. When a call to `flash_safe_execute` on core Y coincides with disabled interrupts on core X, `flash_safe_execute` returns `PICO_ERROR_TIMEOUT` (-2). Unfortunately this also causes core X to get stuck in `multicore_lockout_handler`. If core X = core 1, then USB continues to work, and the status can be seen on the serial line (the cycle count for core X stops, while the cycle count for core Y continues to increment). If core X = core 0, then USB stops working (no interrupts) but a hardware debugger can still be used to observe the cycle counts and see where core 0 has become stuck.

kilograham commented

2025-05-20 15:43:18 -05:00

(Migrated from github.com)