improve non-SWD quality-of-life with fault handler + postmortem support

swetland commented

2021-03-04 19:25:03 -06:00

(Migrated from github.com)

For folks developing using picoboot or U2F and debug chatter over ACM CDC, hard faults are pretty painful -- the device stops responding, host side drivers may wedge up, figuring out what blew up is hard.

One could provide in the SDK:

default fatal exception / unsupported IRQ handlers which pack up essential information somewhere safe (exception, registers, etc) and watchdog reboot.
startup code that looks for a marker for "hard fault" reboot that then jumps to a debug stub that minimally makes the key fault state (registers, fault registers, etc) readable / printable over UART or ACM CDC
maximally some more extensive debugging (a little interactive debugger? gdb serial protocol over ACM CDC, etc)

Just a minimal dump like this would probably be insightful in many cases:

Unhandled Exception ### at PC 0x00011eaa

r0 00000000 r4 00000000 r8 00000010 ip 000142cd psr 21000000
r1 20000bb8 r5 40004000 r9 0000002a sp 20007f58 msp 20007f58
r2 0000000d r6 4001f000 10 00000040 lr 00011e87 psp 00000000
r3 e000e100 r7 00000001 11 00000005 pc 00011eaa

20007f58: 40007000 d1000002 20000aac 00000000
20007f68: 00000000 00011f7f 00000000 000110c1
20007f78: 00000054 000000f1 000000d5 00c20117
20007f88: abf9ffcb 00208108 fb73fffb 02720200
20007f98: 6e5a7fef 9a400269 50000000 00000000
20007fa8: 00000000 000005c3 3bdeffbb 00000000
20007fb8: 00000000 00000000 00000000 0001115b
20007fc8: f93eb75f a02d12c9 e5ef75fb dc03a0e1

For folks developing using picoboot or U2F and debug chatter over ACM CDC, hard faults are pretty painful -- the device stops responding, host side drivers may wedge up, figuring out what blew up is hard. One could provide in the SDK: 1. default fatal exception / unsupported IRQ handlers which pack up essential information somewhere safe (exception, registers, etc) and watchdog reboot. 2. startup code that looks for a marker for "hard fault" reboot that then jumps to a debug stub that minimally makes the key fault state (registers, fault registers, etc) readable / printable over UART or ACM CDC 3. maximally some more extensive debugging (a little interactive debugger? gdb serial protocol over ACM CDC, etc) Just a minimal dump like this would probably be insightful in many cases: ``` Unhandled Exception ### at PC 0x00011eaa r0 00000000 r4 00000000 r8 00000010 ip 000142cd psr 21000000 r1 20000bb8 r5 40004000 r9 0000002a sp 20007f58 msp 20007f58 r2 0000000d r6 4001f000 10 00000040 lr 00011e87 psp 00000000 r3 e000e100 r7 00000001 11 00000005 pc 00011eaa 20007f58: 40007000 d1000002 20000aac 00000000 20007f68: 00000000 00011f7f 00000000 000110c1 20007f78: 00000054 000000f1 000000d5 00c20117 20007f88: abf9ffcb 00208108 fb73fffb 02720200 20007f98: 6e5a7fef 9a400269 50000000 00000000 20007fa8: 00000000 000005c3 3bdeffbb 00000000 20007fb8: 00000000 00000000 00000000 0001115b 20007fc8: f93eb75f a02d12c9 e5ef75fb dc03a0e1 ```

lurch commented

2021-03-04 20:21:26 -06:00

(Migrated from github.com)

improve non-SWD quality-of-life

OTOH, getting access to SWD only involves an additional $4 outlay for a second Raspberry Pi Pico and the usage of picoprobe 😉

> improve non-SWD quality-of-life OTOH, getting access to SWD only involves an additional $4 outlay for a second Raspberry Pi Pico and the usage of picoprobe :wink:

lurch commented

2021-03-04 20:25:07 -06:00

(Migrated from github.com)

startup code that looks for a marker for "hard fault" reboot that then jumps to a debug stub

I guess https://github.com/raspberrypi/pico-sdk/tree/develop/src/rp2_common/pico_bootsel_via_double_reset is a good template for how something like that could work (i.e. the looking for a special marker after reboot).

> startup code that looks for a marker for "hard fault" reboot that then jumps to a debug stub I guess https://github.com/raspberrypi/pico-sdk/tree/develop/src/rp2_common/pico_bootsel_via_double_reset is a good template for how something like that could work (i.e. the looking for a special marker after reboot).

swetland commented

2021-03-04 20:26:59 -06:00

(Migrated from github.com)

OTOH, getting access to SWD only involves an additional $4 outlay for a second Raspberry Pi Pico and the usage of picoprobe

Indeed. Certainly it's what I prefer. But some people seem really allergic to it and then get very frustrated when they can't figure out why their program is failing. A little bit of effort in the SDK could brighten their days.

> OTOH, getting access to SWD only involves an additional $4 outlay for a second Raspberry Pi Pico and the usage of picoprobe Indeed. Certainly it's what I prefer. But some people seem really allergic to it and then get very frustrated when they can't figure out why their program is failing. A little bit of effort in the SDK could brighten their days.

👍 1

lurch commented

2021-03-04 20:39:16 -06:00

(Migrated from github.com)

Indeed. Certainly it's what I prefer. But some people seem really allergic to it

As somebody outside of the RPi organisation, can you see anything in the documentation that might be putting people off from using picoprobe? (i.e. are there any parts that could be made friendlier or more helpful?)

> Indeed. Certainly it's what I prefer. But some people seem really allergic to it As somebody outside of the RPi organisation, can you see anything in the documentation that might be putting people off from using picoprobe? (i.e. are there any parts that could be made friendlier or more helpful?)

swetland commented

2021-03-05 01:12:46 -06:00

(Migrated from github.com)

I maintain my own debugger an am aware that I'm pretty far from a "typical target user" -- personally I've always found openocd a little clunky and gdb even more clunky for the basic "attach + reset-and-halt + flash + reset" cycle with the occasional "dump registers and memory and look at backtraces".

I've seen some folks grumble about needing additional hardware and wires and software and whatnot -- perhaps feeling it's a extra burden to "just write some programs" (though you're right that $4 for a pico in picoprobe mode is probably the most affordable SWD interface available).

I can imagine that some people, looking at the pitch of "oh you don't need any special software, just drag your app on to the virtual usb drive" might find "oh and get a second board to run a debugger" to be an annoying increase in complexity.

I maintain my own debugger an am aware that I'm pretty far from a "typical target user" -- personally I've always found openocd a little clunky and gdb even more clunky for the basic "attach + reset-and-halt + flash + reset" cycle with the occasional "dump registers and memory and look at backtraces". I've seen some folks grumble about needing additional hardware and wires and software and whatnot -- perhaps feeling it's a extra burden to "just write some programs" (though you're right that $4 for a pico in picoprobe mode is probably the most affordable SWD interface available). I can imagine that some people, looking at the pitch of "oh you don't need any special software, just drag your app on to the virtual usb drive" might find "oh and get a second board to run a debugger" to be an annoying increase in complexity.

Wren6991 commented

2021-03-05 06:31:11 -06:00

(Migrated from github.com)

This does sound really handy. I guess it should be in a known location (or at least one that can be found from a table in a known location, like the bininfo table) so that the debugger can also dump the state of the most recent crash, even if the system has already watchdogged.

As for printing out the last-exception state on boot, I can think of worse ideas than (optionally) hooking stdio_init_all(), since this is the earliest point where stdout is known to work, and just printing the dump to all available stdouts.

This does sound really handy. I guess it should be in a known location (or at least one that can be found from a table in a known location, like the bininfo table) so that the debugger can also dump the state of the most recent crash, even if the system has already watchdogged. As for printing out the last-exception state on boot, I can think of worse ideas than (optionally) hooking `stdio_init_all()`, since this is the earliest point where stdout is known to work, and just printing the dump to all available stdouts.

lurch commented

2021-03-05 07:26:21 -06:00

(Migrated from github.com)

hooking stdio_init_all(), since this is the earliest point where stdout is known to work, and just printing the dump to all available stdouts.

That's a great idea, but it seems that it takes a while for the USB serial console to connect, so I think if you were using USB CDC for stdio then this would already be 'gone' by the time e.g. mincom connected?
(A while ago I modified the hello_world example to print to both UART and USB CDC, and to print an auto-incrementing number; and IIRC whilst the UART-connected minicom always caught every single print, the USB-connected minicom always missed the first couple)

> hooking stdio_init_all(), since this is the earliest point where stdout is known to work, and just printing the dump to all available stdouts. That's a great idea, but it seems that it takes a while for the USB serial console to connect, so I think if you were using USB CDC for stdio then this would already be 'gone' by the time e.g. mincom connected? (A while ago I modified the `hello_world` example to print to both UART **and** USB CDC, and to print an auto-incrementing number; and IIRC whilst the UART-connected minicom always caught every single print, the USB-connected minicom always missed the first couple)

davidgiven commented

2021-03-05 15:41:17 -06:00

(Migrated from github.com)

SWD is definitely a faff. On Linux, I have to have three different xterms open: one for openocd, one for gdb, and one for the terminal I'm actually developing on. Flashing via SWD is slow, so if you're working with large images it's actually faster to wire up a reset button and use the mass storage boot loader. (OTOH just being able to type load into gdb and have it work is so nice.) SWD also doesn't help if you're doing anything with USB, because as soon as the chip halts the host punts the device.

Re postmortem support: I like the idea, although I reckon it should be an optional feature; it should be up to the user to check for a postmortem report and register the handler, so that if they don't use it they don't pay for it.

SWD is definitely a faff. On Linux, I have to have three different xterms open: one for openocd, one for gdb, and one for the terminal I'm actually developing on. Flashing via SWD is slow, so if you're working with large images it's actually faster to wire up a reset button and use the mass storage boot loader. (OTOH just being able to type _load_ into gdb and have it work is _so nice_.) SWD also doesn't help if you're doing anything with USB, because as soon as the chip halts the host punts the device. Re postmortem support: I like the idea, although I reckon it should be an optional feature; it should be up to the user to check for a postmortem report and register the handler, so that if they don't use it they don't pay for it.

swetland commented

2021-03-05 16:48:28 -06:00

(Migrated from github.com)

I think a postmortem crash handler should stop after displaying the crash report rather than just continuing to run (and possibly crashing and rebooting over and over again) -- it could spit out its message via USB serial console on connect.

I also agree that it should be a build option. Not everyone will want or need it on every build.

It should certainly be possible to improve SWD flash speed -- my solution involves shooting down a little flash agent to sram to handle the actual heavy lifting (which on rp2040 is just calling the ROM routines) -- I expect if I adjusted the agent to spin up the clock first it could be competitive with USB. I certainly can download to RAM on most targets at 400KB/s or better. I've been porting my SWD backend to Pico and am still investigating performance differences vs LPCLINKII (the hardware I've traditionally used -- but that critter is $20 vs $4 for a spare Pico)

I think a postmortem crash handler should stop after displaying the crash report rather than just continuing to run (and possibly crashing and rebooting over and over again) -- it could spit out its message via USB serial console on connect. I also agree that it should be a build option. Not everyone will want or need it on every build. It should certainly be possible to improve SWD flash speed -- my solution involves shooting down a little flash agent to sram to handle the actual heavy lifting (which on rp2040 is just calling the ROM routines) -- I expect if I adjusted the agent to spin up the clock first it could be competitive with USB. I certainly can download to RAM on most targets at 400KB/s or better. I've been porting my SWD backend to Pico and am still investigating performance differences vs LPCLINKII (the hardware I've traditionally used -- but that critter is $20 vs $4 for a spare Pico)

kilograham commented

2021-03-05 19:32:27 -06:00

(Migrated from github.com)

I don't know where your bottlenecks are but you could certainly set things up (you may already) such that you are writing to flash while downloading the next data

kilograham commented

2023-02-09 09:18:32 -06:00

(Migrated from github.com)

fixed by #1026

Rows
Columns

improve non-SWD quality-of-life with fault handler + postmortem support #228