Possible lockup-issue with `i2c_write_blocking` versus `i2c_write_timeout_us` (maybe conflicting with USB serial?)

ZodiusInfuser commented

2023-08-10 09:33:18 -05:00

(Migrated from github.com)

Hi, I am working on a Pimoroni RP2040 product that involves regular communication with some onboard I2C devices at 400KHz.

During my testing I was encountering frequent lock-ups after a call to i2c_write_blocking had completed. I checked this by setting a pin high before the call and to low afterwards. The pin would go low, suggesting the function had completed, then the lockup would occur before reaching my next line of code.

This was very strange to me, as I had successfully got the product working with a custom build of CircuitPython, which I had written C level drivers for. Checking how they implement I2C, I saw that i2c_write_timeout_us was used, with a 1 second timeout.

Switching to this call in my code the lock-ups went away, without any timeouts actually getting triggered. Checking the pico-sdk, I see that both functions call i2c_write_blocking_internal, with the latter having a callback that results in the below if statement getting executed.

github.com/raspberrypi/pico-sdk@6a7db34ff6/src/rp2_common/hardware_i2c/i2c.c (L166-L172)

Suspecting that the longer loop time the timeout call was having a positive effect, I added a sleep_us(10) as an else condition, and sure enough the lock-ups from i2c_write_blocking_internal go away. Switching to a debug build also made the issue go away, thus ruling out SWD debugging.

Also, sending CTRL-C via the USB terminal would break the board out of the lock-up, though devices like PIO would not resume.

I am completely baffled by this!

For now I can switch my code to using the timeout function, and carry on. I just thought I would raise it in case there is an actual issue that needs resolving.

P.S: It may be inconsequential, but I have observed that my "debug" pin goes low before I2C comms complete, for the i2c_write_blocking code path:

Whereas my "debug" pin goes low after the I2C comms complete, for the i2c_write_timeout_us code path:

Hi, I am working on a Pimoroni RP2040 product that involves regular communication with some onboard I2C devices at 400KHz. During my testing I was encountering frequent lock-ups after a call to `i2c_write_blocking` had completed. I checked this by setting a pin high before the call and to low afterwards. The pin would go low, suggesting the function had completed, then the lockup would occur before reaching my next line of code. This was very strange to me, as I had successfully got the product working with a custom build of CircuitPython, which I had written C level drivers for. Checking how they implement I2C, I saw that `i2c_write_timeout_us` was used, with a 1 second timeout. Switching to this call in my code the lock-ups went away, without any timeouts actually getting triggered. Checking the pico-sdk, I see that both functions call `i2c_write_blocking_internal`, with the latter having a callback that results in the below if statement getting executed. https://github.com/raspberrypi/pico-sdk/blob/6a7db34ff63345a7badec79ebea3aaef1712f374/src/rp2_common/hardware_i2c/i2c.c#L166-L172 Suspecting that the longer loop time the timeout call was having a positive effect, I added a `sleep_us(10)` as an else condition, and sure enough the lock-ups from `i2c_write_blocking_internal` go away. Switching to a debug build also made the issue go away, thus ruling out SWD debugging. Also, sending CTRL-C via the USB terminal would break the board out of the lock-up, though devices like PIO would not resume. I am completely baffled by this! For now I can switch my code to using the timeout function, and carry on. I just thought I would raise it in case there is an actual issue that needs resolving. P.S: It may be inconsequential, but I have observed that my "debug" pin goes low before I2C comms complete, for the `i2c_write_blocking` code path: ![image](https://github.com/raspberrypi/pico-sdk/assets/7437269/62c711e8-e8be-4015-bbdb-26f5b2d061c2) Whereas my "debug" pin goes low after the I2C comms complete, for the `i2c_write_timeout_us` code path: ![image](https://github.com/raspberrypi/pico-sdk/assets/7437269/9ce6fb94-b152-4be5-87eb-399a5da37301)

ZodiusInfuser commented

2023-08-10 11:41:05 -05:00

(Migrated from github.com)

Okay, me just switching to i2c_write_timeout_us isn't enough to stop the lock-ups. Seems adding sleep_us(10) into both of those do...while loops works though. Putting a sleep after the loop (or even after the function call) has no effect.

I'm suspecting I2C may be a red herring though, as the lock-ups (without those added sleeps) seem more prevalent when there is more USB serial output, which is related to changes from the I2C devices. Just tested now and I don't get any lock-ups if I turn off the printouts after removing those sleeps.

Okay, me just switching to `i2c_write_timeout_us` isn't enough to stop the lock-ups. Seems adding `sleep_us(10)` into both of those do...while loops works though. Putting a sleep after the loop (or even after the function call) has no effect. I'm suspecting I2C may be a red herring though, as the lock-ups (without those added sleeps) seem more prevalent when there is more USB serial output, which is related to changes from the I2C devices. Just tested now and I don't get any lock-ups if I turn off the printouts after removing those sleeps.

peterharperuk commented

2023-09-05 08:45:27 -05:00

(Migrated from github.com)

Does the problem go away if you use a uart instead of usb? Do you have an example that demonstrates the issue?

DominusFulguris commented

2024-02-25 20:31:17 -06:00

(Migrated from github.com)

I am also seeing frequent lockups on the i2c bus (both instances).

On i2c1, my code is polling four colour sensors, each on their own pair of SDA/SCL pins (as the sensors are fixed with a single i2c address). Disabling the i2c, reconfiguring the bus pins, and reenabling the i2c was often leaving the SDA pin latched low. Adding a sleep_us(100) between bus switches appears to have worked around the issue. Power cycling the colour sensors clears the bus which indicates it was latching the SDA line.

On the i2c0, I am talking to a single MPU6050. In this case again the SDA line is being latched low by the slave.

The logic analyser trace would indicate that under some circumstances the i2c may not be sending the clock cycle required to clock in the slave device ACK, which leaves the slave holding the SDA line low.

Bit-banging a single clock cycle on SDA causes the slave to release the SDA line.

I am also seeing frequent lockups on the i2c bus (both instances). On i2c1, my code is polling four colour sensors, each on their own pair of SDA/SCL pins (as the sensors are fixed with a single i2c address). Disabling the i2c, reconfiguring the bus pins, and reenabling the i2c was often leaving the SDA pin latched low. Adding a sleep_us(100) between bus switches appears to have worked around the issue. Power cycling the colour sensors clears the bus which indicates it was latching the SDA line. On the i2c0, I am talking to a single MPU6050. In this case again the SDA line is being latched low by the slave. The logic analyser trace would indicate that under some circumstances the i2c may not be sending the clock cycle required to clock in the slave device ACK, which leaves the slave holding the SDA line low. Bit-banging a single clock cycle on SDA causes the slave to release the SDA line.

CoolNamesAllTaken commented

2024-06-02 03:44:11 -05:00

(Migrated from github.com)

Wow! I was encountering I2C lockups, and it took me a full half day to start considering that something might be wrong other than my own hardware or code. Thank you all very much for bringing up this thread, it's been a big help. I'm attempting to work around the issue with the i2c_write_timeout_us technique mentioned by @ZodiusInfuser for now. Hopefully this will be fixed soon.

How does such a fundamental part of the pico-sdk get broken? I'm surprised that there hasn't been a bigger stink about I2C failing in a fatal way like this. Does this bug only impact certain devices under very particular conditions?

In case it helps, I'm using a M24C02 EEPROM at 100kHz, right up next to GPIO 2-3 on the RP2040. I have 2.2kOhm pullup resistors on both SDA and SCL. I have another device (MCP4017 digipot) on the same net, a few millimeters further from the RP2040 than the EEPROM, and haven't had any lockup issues with that device when writing to it over I2C.

Wow! I was encountering I2C lockups, and it took me a full half day to start considering that something might be wrong other than my own hardware or code. Thank you all very much for bringing up this thread, it's been a big help. I'm attempting to work around the issue with the `i2c_write_timeout_us` technique mentioned by @ZodiusInfuser for now. Hopefully this will be fixed soon. How does such a fundamental part of the pico-sdk get broken? I'm surprised that there hasn't been a bigger stink about I2C failing in a fatal way like this. Does this bug only impact certain devices under very particular conditions? In case it helps, I'm using a [M24C02 EEPROM](https://www.st.com/content/ccc/resource/technical/document/datasheet/b0/d8/50/40/5a/85/49/6f/DM00071904.pdf/files/DM00071904.pdf/jcr:content/translations/en.DM00071904.pdf) at 100kHz, right up next to GPIO 2-3 on the RP2040. I have 2.2kOhm pullup resistors on both SDA and SCL. I have another device (MCP4017 digipot) on the same net, a few millimeters further from the RP2040 than the EEPROM, and haven't had any lockup issues with that device when writing to it over I2C.

mitchellcairns commented

2025-02-22 14:46:19 -06:00

(Migrated from github.com)

Wow! I was encountering I2C lockups, and it took me a full half day to start considering that something might be wrong other than my own hardware or code. Thank you all very much for bringing up this thread, it's been a big help. I'm attempting to work around the issue with the i2c_write_timeout_us technique mentioned by @ZodiusInfuser for now. Hopefully this will be fixed soon.

How does such a fundamental part of the pico-sdk get broken? I'm surprised that there hasn't been a bigger stink about I2C failing in a fatal way like this. Does this bug only impact certain devices under very particular conditions?

In case it helps, I'm using a M24C02 EEPROM at 100kHz, right up next to GPIO 2-3 on the RP2040. I have 2.2kOhm pullup resistors on both SDA and SCL. I have another device (MCP4017 digipot) on the same net, a few millimeters further from the RP2040 than the EEPROM, and haven't had any lockup issues with that device when writing to it over I2C.

I just updated my project to SDK 2.1.1 and began having these same issues. I implemented a wrapper function so I can use the SDK 1.5.1 version of the read/write I2C functions and the lockup issue went away completely.

> Wow! I was encountering I2C lockups, and it took me a full half day to start considering that something might be wrong other than my own hardware or code. Thank you all very much for bringing up this thread, it's been a big help. I'm attempting to work around the issue with the `i2c_write_timeout_us` technique mentioned by [@ZodiusInfuser](https://github.com/ZodiusInfuser) for now. Hopefully this will be fixed soon. > > How does such a fundamental part of the pico-sdk get broken? I'm surprised that there hasn't been a bigger stink about I2C failing in a fatal way like this. Does this bug only impact certain devices under very particular conditions? > > In case it helps, I'm using a [M24C02 EEPROM](https://www.st.com/content/ccc/resource/technical/document/datasheet/b0/d8/50/40/5a/85/49/6f/DM00071904.pdf/files/DM00071904.pdf/jcr:content/translations/en.DM00071904.pdf) at 100kHz, right up next to GPIO 2-3 on the RP2040. I have 2.2kOhm pullup resistors on both SDA and SCL. I have another device (MCP4017 digipot) on the same net, a few millimeters further from the RP2040 than the EEPROM, and haven't had any lockup issues with that device when writing to it over I2C. I just updated my project to SDK 2.1.1 and began having these same issues. I implemented a wrapper function so I can use the SDK 1.5.1 version of the read/write I2C functions and the lockup issue went away completely.

👀 1

kilograham commented

2025-02-23 14:14:56 -06:00

(Migrated from github.com)

Note that @ZodiusInfuser reported this problem prior to SDK 2.0.0 - I suspect it is entirely timing related

arntsonl commented

2025-06-17 22:16:44 -05:00

(Migrated from github.com)

We're now seeing this on the GP2040-CE game controller project (https://github.com/OpenStickCommunity/GP2040-CE) after updating to SDK 2.1.1 from SDK 1.5.1. I was battling updating our TinyUSB / PICO-PIO-USB code as there are a few bugs in those libraries, but I finally got everything working as intended.

Weirdly it only happens when we use the rndis net driver, otherwise i2c works fine with standard TinyUSB tud_task code. The i2c read is getting locked in the exact same spot as others have reported.

@peterharperuk I have a GP2040-CE dev board I could ship you, I'm not sure if shipping would be crazy expensive from the USA but I can 100% replicate it on my end. Feel free to ping me on the GP2040-CE discord https://discord.gg/k2pxhke7q8

We're now seeing this on the GP2040-CE game controller project (https://github.com/OpenStickCommunity/GP2040-CE) after updating to SDK 2.1.1 from SDK 1.5.1. I was battling updating our TinyUSB / PICO-PIO-USB code as there are a few bugs in those libraries, but I finally got everything working as intended. Weirdly it only happens when we use the rndis net driver, otherwise i2c works fine with standard TinyUSB tud_task code. The i2c read is getting locked in the exact same spot as others have reported. @peterharperuk I have a GP2040-CE dev board I could ship you, I'm not sure if shipping would be crazy expensive from the USA but I can 100% replicate it on my end. Feel free to ping me on the GP2040-CE discord https://discord.gg/k2pxhke7q8

peterharperuk commented

2025-06-18 04:07:33 -05:00

(Migrated from github.com)

@arntsonl I'm not a discord user but I'm interested in getting some hardware if I can use it to reproduce the problem. My user name is peterharper_75037 and I've accepted your invite.

I made this speculative change you could try https://github.com/raspberrypi/pico-sdk/pull/2493
There's also this which you could try reverting https://github.com/raspberrypi/pico-sdk/pull/2026

@arntsonl I'm not a discord user but I'm interested in getting some hardware if I can use it to reproduce the problem. My user name is peterharper_75037 and I've accepted your invite. I made this speculative change you could try https://github.com/raspberrypi/pico-sdk/pull/2493 There's also this which you could try reverting https://github.com/raspberrypi/pico-sdk/pull/2026

peterharperuk commented

2025-07-08 09:52:29 -05:00

(Migrated from github.com)

Unfortunately the GP2040-CE i2c issue seems to be a different problem https://github.com/OpenStickCommunity/GP2040-CE/pull/1420
Still looking for a way to reproduce the "i2c is completely broken" problem.

Unfortunately the GP2040-CE i2c issue seems to be a different problem https://github.com/OpenStickCommunity/GP2040-CE/pull/1420 Still looking for a way to reproduce the "i2c is completely broken" problem.

siegmund-beckmann-gmbh commented

2026-02-05 09:19:16 -06:00

(Migrated from github.com)

I am still experiencing lockups of the i2c_write_timeout_per_char_us() in my commercial RP2040 project and I think I found the reason in that routine calling the i2c_write_blocking_internal().

The i2c_write_blocking_internal() is called with init_per_iteration_timeout_us() from timeout_helper.c as a parameter.
Within the i2c_write_blocking_internal() it is referenced as timeout_check() that should initialise the timeout for every character sent and also check the timeout after writing to the i2c hardware.

The timeout will never occur !

The timeout_check() should ONLY call the check_per_iteration_timeout_us() from timeout_helper.c instead of init_per_iteration_timeout_us()

I am still experiencing lockups of the `i2c_write_timeout_per_char_us()` in my commercial RP2040 project and I think I found the reason in that routine calling the `i2c_write_blocking_internal()`. <img width="1015" height="156" alt="Image" src="https://github.com/user-attachments/assets/50b8eb64-5866-48c2-9c9d-df9acee3fe5a" /> The `i2c_write_blocking_internal()` is called with `init_per_iteration_timeout_us()` from `timeout_helper.c` as a parameter. Within the `i2c_write_blocking_internal()` it is referenced as `timeout_check()` that should initialise the timeout for every character sent and also check the timeout after writing to the i2c hardware. <img width="886" height="522" alt="Image" src="https://github.com/user-attachments/assets/be81c5c5-a81a-4e2f-85d5-ac1a39d42d07" /> The timeout will never occur ! The `timeout_check()` should ONLY call the `check_per_iteration_timeout_us()` from `timeout_helper.c` instead of `init_per_iteration_timeout_us()` <img width="968" height="391" alt="Image" src="https://github.com/user-attachments/assets/8af1dd5b-cfe3-4784-a6e3-c5349990d1d9" />

peterharperuk commented

2026-02-05 10:15:33 -06:00

(Migrated from github.com)

init_per_iteration_timeout_us returns check_per_iteration_timeout_us?

siegmund-beckmann-gmbh commented

2026-02-06 01:17:07 -06:00

(Migrated from github.com)

Yes, you are right... I did not see, that after the first call of init_per_iteration_timeout_us, it returns the check_timeout_fn-type, so it will call that pointer instead of init_per_iteration_timeout_us in succession.

I'm sorry... must have been blind.

I found the real reason for my problem. My development board is working with 2.2.0 where everything is fine, but in production, the project was still built with 1.5.1, that does the timeout reset every call and sometimes leads to hanging i2c.

You can delete my comment.

Yes, you are right... I did not see, that after the first call of `init_per_iteration_timeout_us`, it returns the `check_timeout_fn`-type, so it will call that pointer instead of `init_per_iteration_timeout_us` in succession. I'm sorry... must have been blind. I found the real reason for my problem. My development board is working with 2.2.0 where everything is fine, but in production, the project was still built with 1.5.1, that does the timeout reset every call and sometimes leads to hanging i2c. You can delete my comment.

peterharperuk commented

2026-02-06 03:41:10 -06:00

(Migrated from github.com)

No worries. Glad you found your problem.

Rows
Columns

Possible lockup-issue with i2c_write_blocking versus i2c_write_timeout_us (maybe conflicting with USB serial?) #1471

Possible lockup-issue with `i2c_write_blocking` versus `i2c_write_timeout_us` (maybe conflicting with USB serial?) #1471