Adafruit qtpy rp2040 target starts CDC/ACM USB only once - suspect flash timing #401
No reviewers
Labels
No labels
bazel
bluetooth
board settings
boot_stage2
bug
build
c++
can't reproduce
clang
clib-interface
cmsis
code cleanliness
code size
compiler version issues
cyw43_driver
debug
discussion
documentation
duplicate
dynamic_clock
elf2uf2
enhancement
errata
filesystem
flash layout
floating point
FreeRTOS
hardware_adc
hardware_claim
hardware_clocks
hardware_dma
hardware_exception
hardware_flash
hardware_gpio
hardware_i2c
hardware_irq
hardware_pio
hardware_pwm
hardware_rcp
hardware_rtc
hardware_spi
hardware_sync
hardware_uart
hardware_vreg
hardware_watchdog
hardware_xosc
help wanted
host
IAR
installation
invalid
linker scripts
lto
lwip
mbedtls
missing API
needs reporter detail
networking
newlib
newlib-multicore
non-c
non-cmake
non-gcc
openocd
orphan PR
OS integration
pico_async_context
pico_base
pico_binary_info
pico_flash
picolibc
pico_lwip
pico_malloc
pico_multicore
pico_printf
pico_runtime
pico_stdio
pico_stdio_usb
pico_sync
pico_time
pico_util
pioasm
port
question
review
review2
rtos interop
runtime init
static library
svd
test
tinyusb
vscode
wifi integration
wontfix
No milestone
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
gh-raspberrypi/pico-sdk!401
Loading…
Reference in a new issue
No description provided.
Delete branch "target-qtpy_clkdiv-b"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Changed:
Reference system Linux host PC will not enumerate /dev/ttyACM0
(or related device names) except upon UF2 firmware upload.
UF2 upload is fine and program will run once, until power is removed.
Then, never again.
This patch is meant to allow the (end-user authored) firmware/program
to run repeatedly, including cycling power to the target board.
First (SPI clock divisor) value tried was '4' (from '2').
No other experiments done, in search of the optimal clock divisor.
Not at all sure what the clock divisor 'does'.
Simple guess:
Going from 2, to 4 would 'halve' the clock frequency (divide it
by 2).
Halving the frequency would cause the subsystem to evolve
all events more slowly, which apparently helps in enumeration
of the CDC/ACM device to the host PC.
ping @tannewt and @dhalbert and @ladyada for comment?
@lurch the QT Py 2040 uses the winbond 25Q64JVXGIQ chip recommended by the pi foundation
ItsyBitsy RP2040 enumerates on my Linux host PC with a clock divisor of 2.
We have one other user who has trouble getting a QT Py RP2040 running CircuitPython to respond to the reset button, but the Itsy works. This sounds similar: https://forums.adafruit.com/viewtopic.php?f=60&t=178640. We will investigate that. Are you running a pico-sdk C program?
"This ran fine until I hit the reset button. Then I had to hit it 6 times to get it running again."
@dhalbert Yeah, it does sound like there might be some marginal timing somewhere that works with the flash chip on some QT Py RP2040 boards but not with the flash chip on other QT Py RP2040 boards? 🤷
Hi @dhalbert - Yes, in a pico-sdk C program.
The issue manifests in pico-examples 'hello_world' but I usually test in CamelForth .
Since it happens with pico-examples hello world (serial USB) I didn't investigate further wrt the code being run.
Same code runs fine on Adafruit Feather RP2040, ItsyBitsy RP2040, Raspberry Pi Pico RP2040.
Host PC Dell Optiplex about 10 yrs old, Debian Linux amd64. Suspect it may have slight trouble enumerating USB, as a second device doesn't get enumerated, if two CDC/ACM devices are inserted into the USB jack array on the PC chassis. Only /dev/ttyACM0 presents (not /dev/ttyACM1, for example).
I don't remember this being a problem in the past, with (for example) the Arduino IDE.
No issue, this morning, with enumerating QTPY RP2040 (/dev/ttyACM0) running pico-sdk C program, concurrent with enumeration (and conversation with) CP2104 Friend (/dev/ttyUSB0) (which is talking to STM32F40x Black Pill on the Black Pill's USART - concurrently.
Both conversations CDC/ACM based (text interpreters running Forth).
This prebuilt UF2 exhibits the issue (does not survive removal of power; runs fine upon UF2 upload, one time only):
wa1tnr: CamelForth
The same flash chip (maybe even from the same reel) is being used on the QT Py and the ItsyBitsy. I don't see any board configuration build differences, so this may be electrical, but it sounds like we might need to adjust some clock speed.
See Errata RP2040-E5 in the RP2040 datasheet. Easiest fix is to plug the two different RP2040 devices into separate USB hubs.
Thanks!
The content below is optional reading; tangental to this PR. ;)
I was able to
38 int main(void) {39 stdio_init_all();40 sleep_ms(1200);41 rp2040_usb_device_enumeration_fix();and flash to both targets (Adafruit Feather RP2040, QTPy RP2040).
There doesn't seem to be a preference for one over the other.
Both will enumerate, and I'll have both /dev/ttyACM0 and /dev/ttyACM1 available for interactive sessions (minicom, seyon, hyperterm &c.)
The one thing I can't do is 'claim' the interface (by invoking minicom, seyon, hyperterm &c.) and then try to enumerate the other target board (by plugging in its USB cable).
Both cables must be plugged in, and enumeration verified, before proceeding further.
Under those circumstances, I can have concurrent sessions on the two /dev/ttyACM devices, without a special USB hub (I don't own any external USB hubs).
@hathach has been maintaining these board defs. I found 4 was a reliable divisor for CircuitPython. It needs to account for all of the different command speed limits (not just the read speed) and 62.5 mhz can end up over the common limit of 60mhz for commands.
Yeah, I also noticed this issue as well when using qtpy as picoprobe. I have also tried to change the
PICO_FLASH_SPI_CLKDIV=4but it also does not solve the issue either. Furthermore, I have pulled and compiled the latest circuitpythongithub.com/adafruit/circuitpython@35ee4add63, it also has the same issues. To sum upCurrently I have no ideas why it failed to run, further investigation is needed.
Something to look into is the default output driver strength. The flash part on my QT Py RP2040 is a Winbond Q64JVXGIQ, which according to Winbond's Rev K datasheet defaults to 25% driver strength on read operations (see pg. 17, section 7.1.6). I'm going to give kicking it up to 50, 75, and 100% a try and will report back.
Something else to note, I can get my QT Py RP2040 to rock solid operation by switching to boot2_generic. That's not ideal as there's a pretty big performance hit, but it does point to a problem in boot2_w25q080. Since I can't bond a probe to SWCLK/SWDIO (those are some small pins), I'm resorting to using the LED to find where its hanging up.
Well that's weird. I was able to get it to work reliably with boot2_generic_03, but after attempting to set read driver strength I seem to have bricked the flash. Guess I'll try bonding to those tiny pins...
There are a lot of moving parts between changing the SPI speed and USB operations failing. It could be some flash signal integrity issue, or it could be the different cache miss delay bumping against some hidden timing issue in the USB stack or elsewhere.
Are you able to reproduce this with any simpler applications (like
blinkfrompico-examples)?Like @tannewt said 62.5 MHz is quite high for some flash operations (particularly
03htends to have a lower frequency limit) but the ROM programming routines use a fixed divisor of 6 (see here) when used with the stockhardware/flashcode in the SDK, independently of what is used for XIP, so I wouldn't expect that to be affected by changing the second stage.A closer reading of the Winbond datasheet reveals that certain SR bits are marked somewhat cryptically "Volatile / Non-Volatile Writable". What this means in practice is that on reset, these bits are copied from flash into the SR flip-flops. Using two different instructions, Write Enable (06h) or Write Enable for Volatile SR (50h), the programmer can permanently alter the SR bit in flash or temporarily (until the next reset) alter the bit in the flip-flop it's been copied into, respectively.
I don't know who 'you' is. ;)
hello_world in pico-examples exhibits the behavior - that's how I knew not to tear my own code apart looking for a flaw.
i.e. pico-examples/hello_world/usb/hello_usb.c
Sweet success! By setting flash read driver strength to 75% in non-volatile Status Register 3 bits
DRV1andDRV2, I'm able to run withPICO_BOOT_STAGE2_CHOOSE_W25Q080 1andPICO_FLASH_SPI_CLKDIV 4. If I attempt to kick the divider down to 2, it fails.Here is the utility I cobbled together to update flash read driver strength:
https://github.com/eightycc/fix_qtpy_rp2040
We have been seeing some problems with long crystal oscillator startup time on a few samples of Qt Py RP2040. I'm not sure that's related to the problem you're seeing, but try changing this line:
github.com/raspberrypi/pico-sdk@afc10f3599/src/rp2_common/hardware_xosc/xosc.c (L23)to
If you are willing to set at least one Qt Py RP2040 that was acting up back to the stock drive strength, and then trying the above, that would be an interesting test. But I don't know why the drive strength should have anything to do with flaky clocking.
Will do.
Could be we've got more than one problem in play. What I'm seeing looks more like a signal integrity problem in XIP mode, so adding drive (25% -> 75%) makes sense as a remedy.
What we actually saw on some Saleae traces was the the SCK signal to the flash was irregular and too fast after the xosc was started and used. So we surmised that the crystal oscillator was having trouble starting up and lengthened the startup delay experimentally. We haven't yet looked at analog traces.
Stranger and stranger. The QT Py that was failing reliably for me now works with flash read drive at 25% and no additional delay in
xosc_init(). Likewise, any combination of 75% drive and additional xosc delay also works. That's withclock divider = 4and w25q80 second stage boot. Withclock divider = 2all combinations of drive and delay fail. I'll go over it again tomorrow to be sure I didn't miss anything.Hmm! I hadn't looked in the bootrom code and
assumed thatEDIT: I realized that you must have the xosc for USB to work. See https://github.com/raspberrypi/pico-sdk/pull/401#issuecomment-845953208.xosc_init()was only being called later, because theoretically you could run the chip without a crystal.(EDIT) Maybe it's simply the extra delay that's being added in
xosc_init()that is helping for some reason. I first tried small delays like 2x, 6x, etc. Even 16x was not consistently reliable - I had to go to 32x.We are using a clock divider of 4, and we have our own stage2 boot, templatized and written in C:
https://github.com/adafruit/circuitpython/blob/main/ports/raspberrypi/stage2.c.jinja
I'd be very interested in a simple program that just dumped all the NVM parameters in the flash chip. I have several boards that work fine with one date code on the Winbond chip, and one that does not with another date code. I wonder if they have different factory settings.
(I think there is also a difference in the datecodes of the RP2040 chips, but they are both B1)
Date code on QT Py and other boards without problems:
Date code on QT Py with problems:
Fascinating info on the date codes. I have QT Py's stuffed with 2048 and 2051 date coded flash on hand.
I'll get on it later today.
I realized that the bootrom must have the xosc on for USB to work. But the bootrom code does not even set the startup delay for the xosc that I can see. I asked a q about this: https://github.com/raspberrypi/pico-feedback/issues/158.
Remember that the board must be power-cycled to test the xosc startup changes (and maybe your changes?). Things always work fine for me after the initial UF2 load regardless of changes. It's after a power-cycle, where the bootloader goes directly to the load program, where things fall over.
Good point about the power cycling. This also may have some bearing on the problem: bootrom will initialize xosc when
bootselis pressed or when_flash_boot()fails to find a valid second stage boot in flash. So, if the QT Py is power cycled,xosc_init()will run with xosc uninitialized, otherwise it will already be running.I'm away from my lab until later today. I'm raring to do some more debug.
Here are dumps of flash registers from my QT Py modules:
Flash date code 2048:
Flash date Code 2051:
As you can see, both are identical except for unique_id.
So just the unique id is different - oh well! Thanks for checking, in any case!
I've gone over bootrom and SDK initialization code and have found that initial
clk_sysfrequency will differ, depending on how the part booted. If bootrom succeeds loading a valid stage2 boot out of power on, thenclk_syswill be ~12MHz sourced fromrosc; otherwise, if loaded out of a uf2 drop, it will be 48MHz sourced fromxoscvia its PLL.Also significant is that by the time we're running
clock_init(), code is executing from flash via XIP. So, we're changingclk_perifrequency in code we're executing via XIP. I wonder why this works at all?Regarding lack of initialization of
xosc_hw->startupin bootrom, the RP2040 defaults to 47, the same value that is calculated byuint32_t startup_delay = (((12 * MHZ) / 1000) + 128) / 256;.@Wren6991 you may have comments on initial clocking
One more data point:
I have a QtPy RP2040 which was working well on initial program, but would neither reset properly nor start properly when connected to power. I could not get it to restart in these cases, even inconsistently.
The lot code on the Winbond memory is similar to the lot code specified above as 'good' (so these lot codes may be a red herring):
Q64JVXGIQ
2051-6036
7B600ZY
When I adjusted the CLK_DIV to 4 as suggested by @wa1tnr , I was able to consistently reset the board once power was applied, but initial startup from USB power was inconsistent. Applying applying the startup_delay *32 as suggested by @dhalbert , the initial power-on seems to have been corrected as well.
Let me know if there if there is any more needed information I can help to provide.
Did you find this out by reading the register? (Since it's not in the datasheet.)
This makes sense given the symptoms. Our Saleae traces on the SCK of the the flash chip show it clocked quite slowly for a while, I assume while stage2 is being read. Then the xosc is turned on in the pre-
main()code, and the SCK clocks become too fast and very irregular.If instead the bootloader goes into USB mode, either because the boot button ws pressed or there was no valid stage, then xosc is turned on and the execution remains in the boot ROM. Any xosc startup issue would probably just delay USB operation briefly.
See section datasheet 2.16.3, "The 1ms default is sufficient for the RP2040 reference design..." which I verified by setting up a GPIO as a trigger and measuring it using my trusty Rigol.
An interesting characteristic of
xoscis thatSTARTUP_DELAYcounts crystal pulses, so if the crystal is misbehaving the delay can be unpredictable.I've brought
sys_clkout on a GPIO so I can scope it, and observed that in the failing case thexoscstartup is truncated andxoscappears to be running much faster than 12MHz. Once the PLL VCO thinks it's stabilized, it too is running way too fast and produces nothing but a hot mess (term of art) on its output. I can't be precise about the frequencies as they exceed my scope's bandwidth.The way that clock initialization works between
bootromand SDK initialization is not as well structured as I'd wish, but it does seem to work in most cases. I rewroteclocks_init()andxosc_init()to setup clocks correctly to the best of my understanding, and the failure still occurs.Yes, I believe that helps drain more mystery out of this bug.
The bottom line is that some QT Py's have a defect. It may be a wonky crystal or a layout problem. Your fix (increase
STARTUP_DELAY) works by delaying long enough for the crystal to finally stabilize. At this point I'd go with that and move on.One more thing, I'm convinced that the Winbond flash part is not part of the problem. What's coming out of the PLL is such a mess that it's unreasonable to expect anything using it as a clock to work.
github.com/raspberrypi/pico-sdk@afc10f3599/src/rp2_common/boot_stage2/boot2_is25lp080.S (L100-L102)https://www.infineon.com/dgdl/Infineon-AP56002_Crystal_Oscillator_Basics-AN-v01_00-EN.pdf?fileId=5546d46269bda8df0169c9ee19ed23ac
https://www.nxp.com/docs/en/application-note/AN2907.pdf
@eightycc Figure 117
@daveythacher Looked at your references, but I'm not grokking what your driving at?
Figure 117 of the rp2040 datasheet shows a 2:1 ratio (
SCKDV == 2) ofssi_clk(akaperi_clk) andsclk_outillustrating why that is the maximum allowable ratio, i.e., a ratio of 1:1 is not supported.In the Infineon reference, the pertinent reference for the problem at hand is section 6.3.1 Parasitic RC or LC Oscillation which matches the problem I've observed with the failing QT Py board closely enough that Adafruit engineers should take note.
Just in case it gets lost, the original pull should also be incorporated. WithSCKDV == 2, the resultingsclk_outfrequency, assuming aclk_perifrequency of 125MHz, exceeds the Winbond W25Q64JVfRrating of 50MHz (see section 9.6 AC Electrical Characteristics in its datasheet). Since only evenSCKDVvalues are valid,SCKDVneeds to be set to 4.So, this pull is needed for all boards with a Winbond W25Q64JV. Additionally, the pull by @dahlbert (https://github.com/raspberrypi/pico-sdk/pull/457) is needed for QT Py RP2040 parts to work around itsxoscstarting problem.When I sent in the PR, I just figured it was early in development - and I got very lucky on a wild guess.
@tannewt mentioned [the subject matter of] what you just wrote, it as well:
https://github.com/raspberrypi/pico-sdk/pull/401#issuecomment-840716826
Thanks!
#457 modifies all of the Adafruit board configs - should this PR do the same? 🤷
Yes, an excellent suggestion. @wa1tnr, could you do that?https://github.com/raspberrypi/pico-sdk/pull/401#issuecomment-842174569
@daveythacher Mea culpa. After carefully examining the SSI initialization code, I can see that the SSI baud rate will be properly handled through a transition from SDK code to bootrom code and back again. So, why was I seeing a failure with a divider of 2 vs. 4? Bad test hygiene on my part, i.e., building with a locally patched copy of pico-sdk that was picking up the generic stage 2 boot.
Going back to a clean checkout of pico-sdk, I can confirm that the QT Py RP2040 works with (1) https://github.com/raspberrypi/pico-sdk/pull/457, (2)
PICO_BOOT_STAGE2_CHOOSE_W25Q080 1, and (3)PICO_FLASH_SPI_CLKDIV 2.Humbly, I conclude that this pull is not necessary. Apologies to everyone.
Pull request closed