stdio_usb: getchar_timeout_us(0) can block for PICO_STDIO_DEADLOCK_TIMEOUT_MS (default 1000 ms) while the tud_task alarm-IRQ holds stdio_usb_mutex — starves app watchdog loops and turns USB stack wedges into watchdog resets #3201
Labels
No labels
bazel
bluetooth
board settings
boot_stage2
bug
build
c++
can't reproduce
clang
clib-interface
cmsis
code cleanliness
code size
compiler version issues
cyw43_driver
debug
discussion
documentation
duplicate
dynamic_clock
elf2uf2
enhancement
errata
filesystem
flash layout
floating point
FreeRTOS
hardware_adc
hardware_claim
hardware_clocks
hardware_dma
hardware_exception
hardware_flash
hardware_gpio
hardware_i2c
hardware_irq
hardware_pio
hardware_pwm
hardware_rcp
hardware_rtc
hardware_spi
hardware_sync
hardware_uart
hardware_vreg
hardware_watchdog
hardware_xosc
help wanted
host
IAR
installation
invalid
linker scripts
lto
lwip
mbedtls
missing API
needs reporter detail
networking
newlib
newlib-multicore
non-c
non-cmake
non-gcc
openocd
orphan PR
OS integration
pico_async_context
pico_base
pico_binary_info
pico_flash
picolibc
pico_lwip
pico_malloc
pico_multicore
pico_printf
pico_runtime
pico_stdio
pico_stdio_usb
pico_sync
pico_time
pico_util
pioasm
port
question
review
review2
rtos interop
runtime init
static library
svd
test
tinyusb
vscode
wifi integration
wontfix
No milestone
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
gh-raspberrypi/pico-sdk#3201
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Environment:
pico_enable_stdio_usb(1),pico_enable_stdio_uart(0)boards/metrotech_xerxes_rp2040.hwatchdog_enable(1000, 1),getchar_timeout_us(0)poll + ASCII replies (printf), ~1 kHz loop rateGET/SETlines)PICO_RP2040_USB_DEVICE_UFRAME_FIX=1)Symptom:
Under sustained CDC traffic the board intermittently drops off the bus: host writes stop being answered for ~1–2 s, then
Errno 5(device gone), then the device re-enumerates (fresh tty, ~0.45 s). Frequency: between 1/min and 1/40 min under load; idle stretches of 10–20 min are clean. Kerneludevevents show only a device remove/add (no hub over-current).Evidence (on-hardware instrumentation):
Reset forensics that survive warm resets (
.uninitialized_data+ watchdog scratch registers) plus a customisr_hardfaulthook:WATCHDOG->REASON= 1 (watchdog timer) on every reset — never 0 (no power/brownout)getcharwindow of the loop (phase markers written toWATCHDOG->SCRATCH[0]); iteration counters 450k–2.2M (≈8–37 min of healthy looping)PC=0,LR=0x5001FDE4,CFSR=0x6AA8A5E1— the CFSR value is constant across separate resets and bit-pattern-wise not a plausible live fault status, i.e. the stack frame itself is suspectSuspected mechanism:
The 1 ms background worker (
timer_task→low_priority_worker_irq) takesstdio_usb_mutexand callstud_task()inside IRQ context. We believetud_task()occasionally does not return (dcd/RP2040 USB wedge) while holding the mutex.The main loop's next
getchar_timeout_us(0)reachesstdio_usb_in_chars, sees data available, and waits on the mutex:PICO_STDIO_DEADLOCK_TIMEOUT_MSdefaults to 1000 ms — exactly the typical watchdog period. Agetchar_timeout_us(0)(documented as non-blocking!) can therefore block for the full watchdog window;watchdog_update()(called at loop top ~1 ms earlier) expires mid-block → reset,REASON=1.This explains all observations: loop gap stays small (the fatal call never returns, so it is never recorded), death phase stays in the
getchar/command window, resets only under traffic, USB comes back clean after the chip reboots.Suggested fixes:
1. Don't let a non-blocking stdio poll wait 1000 ms on the mutex (contract fix):
(If a blocking variant is wanted for blocking-read users, gate it on the caller's deadline instead of a fixed 1000 ms.)
2. Escape the write loop in
stdio_usb_out_charswhen the driver accepts nothing (defensive):3. Document/config: call out that
PICO_STDIO_DEADLOCK_TIMEOUT_MS(default 1000) interacts badly with watchdog-fed apps and suggest setting it well below the app watchdog period (verified:-DPICO_STDIO_DEADLOCK_TIMEOUT_MS=50builds and is a viable mitigation).Open question for maintainers:
We could not yet prove where
tud_task()wedges — the hardfault frames we captured are themselves corrupted (PC=0, constantCFSR=0x6AA8A5E1across resets), pointing to memory corruption during the wedge. Is adcd_rp2040hang under CDC control-transfer/buffer races known, even withPICO_RP2040_USB_DEVICE_UFRAME_FIX=1? We can provide instrumented builds (phase markers in watchdog scratch + a counter formutex_try_enter_block_untiltiming out) if that helps.Workaround we ship meanwhile:
Replies bypass
stdio(tud_cdc_writedirectly under an IRQ-off critical section) soout_charsis unreachable, and the board treats the watchdog reset as self-recovery (~2 s outage; the USB device re-enumerates and the host application re-scans). It works, but the underlying wedge is real.Root cause identified — correcting the diagnosis in the OP. The resets are
deterministic stack overflow on RP2040, not the stdio mutex wait. Evidence and
consequences for the suggested fixes below.
Mechanism (RP2040, pico-sdk 2.3.1, newlib / arm-none-eabi 15.2)
SCRATCH_Y(4 KB at 0x20041000) and is hard-cappedat 4 KB by the linker script.
PICO_STACK_SIZE=16384fails withsection .stack_dummy will not fit in region SCRATCH_Y/region SCRATCH_Y overflowed by 12288 bytes(= 16384−4096).SCRATCH_X(4 KB at 0x20040000) sits directly below.
flash_get_unique_id()runs an XIP-off SSI command sequence(
__no_inline_not_in_flash_func(flash_do_cmd_cs)) on the caller's stack.hardware_flash's docs cover the IRQ/XIP hazards ("entering a flash-residentinterrupt handler ... will be fatal") but say nothing about stack depth.
printf/snprintfframes) while thetud_taskalarm IRQ stacks onto the same MSP. WithPICO_USE_STACK_GUARDS=1the overflow trips the RP2040 PMSAv6-M MPU guard at
__StackLimit(32-bytesubregion granularity,
runtime_init_stack_guard.c); ARMv6-M escalatesMemManage to HardFault, the default handler cannot recover, and the watchdog
resets the chip one period later. Because M0+ exception entry pushes 8
registers (32 bytes) in one step against a 32-byte guard subregion, the
stacking write itself can cross the guard — which matches our otherwise
inexplicable fault frames (
PC=0, constantCFSRacross resets).Without the guard the same overflow silently corrupts adjacent state
(
SCRATCH_Xand beyond) — observed as a second mode where bulk endpointsstop transferring while the app loops normally.
Evidence (single-variable)
Repro: 1 ms loop,
watchdog_enable(1000, 1),getchar_timeout_us(0), ASCIIreplies over
stdio_usb. Per host session: burst of 1000 empty lines, burst of500 unknown-command lines, burst of 50
GETlines (each write newline-terminated, fenced with one
SET). TheGETburst resets the board 12/12 at1.0–1.2 s on every configuration below:
PICO_STDIO_DEADLOCK_TIMEOUT_MSprintfchain,flash_get_unique_id()per replyprintfchain, ID per replysnprintf+ directtud_cdc_write, ID per replyflash_get_unique_id()moved to bootRemoving the per-reply ROM call is the single change that fixes it. The 1.0–1.2 s
timing is invariant across all stdio parameters and equals one 1000 ms watchdog
period after burst onset — i.e. a fault/unrecoverable stop within the first
moments of the burst. Note
WATCHDOG_REASON(=TIMER) is ambiguous evidence: adefault
isr_hardfaultthat spins also ends in a watchdog reset.Consequences for the suggested fixes in this issue
PICO_STDIO_DEADLOCK_TIMEOUT_MS=50is measurably not a mitigation forthis class: 50 ms and 1000 ms give identical outcomes.
in_charsnon-blocking change and theout_charsloop escape remainworth landing as hardening (they close secondary routes), but they do not
affect the crash above.
PICO_USE_STACK_GUARDSis real on RP2040 (MPU present) and was enabled inthe failing builds — it converts silent corruption into a deterministic
HardFault, but the 32-byte stacking-vs-32-byte-subregion granularity limits
what it can catch, and nothing recovers from the fault.
Suggested SDK changes
hardware_flashROM wrappers (XIP-off coderuns on the caller's stack;
SCRATCH_Ycaps rp2040 stacks at 4 KB) — theexisting warnings cover IRQs and the second core only.
PICO_STACK_SIZE > 4096onrp2040 instead of a raw linker overflow.
exception-entry stacking) next to
PICO_USE_STACK_GUARDS; a canary word at__StackLimitwould complement it.flash_get_unique_id()costs an XIP-off sequence per call; a cached/lazyaccessor or a doc note would defuse its use in hot paths.
TinyUSB notes
ad7acc849("re-issue in-flight transfer on clear-halt toggle reset") is inno release tag — 0.19.0/0.20.0/0.21.0 lack it. (The
active+pendingstaterework
78d34d5b6/087ceb741did ship in 0.21.0.)PICO_RP2040_USB_DEVICE_UFRAME_FIXname but drives arewritten errata regime (
CFG_TUSB_RP2_ERRATA_E15), whose documentedsymptom — "USB Device controller will hang if certain bus errors occur during
an IN transfer" — matches our second failure mode. Against pico-sdk 2.3.1
glue, identical firmware wedges on 0.21.0 but is clean on 0.18.0 and 0.20.0.
A rename or
#warningwhen the old flag is consumed by the new regime wouldprevent glue-version traps (cf. pico-sdk#1260 on this flag's confusing
semantics).
Can you ask your LLM to summarise the issue and remove all irrelevant details - I don't want to read all that.
Sorry @peterharperuk
The issue was basically a combination of problems. The TLDR: the usb/serial interface +
printf(...)caused stack overflow when flooded with requests WHILE callingflash_get_unique_id()in each cycle, with stack basically limited to only 4kB and no clear warning on trying to enlarge stack (overflowing SCRATCH_Y bank i guess).These indirectly caused memory corruption, ghost USB hangs, starvation of watchdog and finally SW reset.
the details got indeed digged up by LLM, unfortunately I am not senior enough in this field to confirm all details, I provided them because they MIGHT be helpful.
Still a huge fan of community work, we have been using rp2040 in our sensors for 5 years now.
Thank you for understanding
Stanley
The most relevant paragraph to consider is :
Suggested SDK changes
runs on the caller's stack;
SCRATCH_Ycaps rp2040 stacks at 4 KB) — theexisting warnings cover IRQs and the second core only.
PICO_STACK_SIZE > 4096onrp2040 instead of a raw linker overflow.
exception-entry stacking) next to
PICO_USE_STACK_GUARDS; a canary word at__StackLimit would complement it.
flash_get_unique_id()costs an XIP-off sequence per call; a cached/lazyaccessor or a doc note would defuse its use in hot paths.