clang+LTO makes USB Serial unacceptably slow #2464

New issue

Open

opened 2025-05-12 18:42:10 -05:00 by Ivan-Voronyuk · 4 comments

Ivan-Voronyuk commented

2025-05-12 18:42:10 -05:00

(Migrated from github.com)

TL;DR

Enabling -flto (or CMAKE_INTERPROCEDURAL_OPTIMIZATION ON) causes USB Serial to output only one "block" per second in Release mode. The issue disappears in Debug builds.

Detailed Description

Minimal Repro Code

#include <pico/stdio.h>
int main() {
  stdio_init_all();
  uint32_t cnt = 0;
  while (1) stdio_printf("!!! %d\n", cnt++);
}

Behavior with LTO:
- Outputs exactly one line per second (vs. ~1.8k lines/s without LTO).
- Replacing stdio_printf with printf doesn't make a difference
- std::cout << "!!! " << cnt++ << std::endl; emits exactly one symbol per second.
Issue only occurs with CMAKE_BUILD_TYPE=Release. Debug builds work normally.

How to Reproduce

Code: Use the main.cpp above.
CMakeLists.txt:

cmake_minimum_required(VERSION 3.13...3.27)
set(CMAKE_INTERPROCEDURAL_OPTIMIZATION TRUE)  # <-- Triggers the bug
set(PICO_SDK_PATH "<your_sdk_path>")
set(PICO_TOOLCHAIN_PATH "<llvm_toolchain_path>")  # Tested with clang-18, LLVM-ET-Arm-18.1.3-Linux-x86_64 toolchain
set(PICO_COMPILER pico_arm_cortex_m0plus_clang)
include(.../pico_sdk_import.cmake)

project(SLOW)
pico_sdk_init()

# Configure stdio
set(PICO_STDIO_UART OFF)
set(PICO_STDIO_USB ON)

add_executable(main main.cpp)
target_link_libraries(main PRIVATE pico_stdlib)
pico_enable_stdio_usb(main 1)
pico_enable_stdio_uart(main 0)
pico_set_float_implementation(main pico)
pico_add_extra_outputs(main)

This issue on tinyusb repo: hathach/tinyusb#3120

### **TL;DR** Enabling `-flto` (or `CMAKE_INTERPROCEDURAL_OPTIMIZATION ON`) causes USB Serial to output only **one "block" per second** in `Release` mode. The issue disappears in `Debug` builds. --- ### **Detailed Description** #### **Minimal Repro Code** ```cpp #include <pico/stdio.h> int main() { stdio_init_all(); uint32_t cnt = 0; while (1) stdio_printf("!!! %d\n", cnt++); } ``` - **Behavior with LTO**: - Outputs **exactly one line per second** (vs. ~1.8k lines/s without LTO). - Replacing `stdio_printf` with `printf` doesn't make a difference - `std::cout << "!!! " << cnt++ << std::endl;` emits exactly one symbol per second. Issue only occurs with `CMAKE_BUILD_TYPE=Release`. Debug builds work normally. --- ### **How to Reproduce** 1. **Code**: Use the `main.cpp` above. 2. **CMakeLists.txt**: ```cmake cmake_minimum_required(VERSION 3.13...3.27) set(CMAKE_INTERPROCEDURAL_OPTIMIZATION TRUE) # <-- Triggers the bug set(PICO_SDK_PATH "<your_sdk_path>") set(PICO_TOOLCHAIN_PATH "<llvm_toolchain_path>") # Tested with clang-18, LLVM-ET-Arm-18.1.3-Linux-x86_64 toolchain set(PICO_COMPILER pico_arm_cortex_m0plus_clang) include(.../pico_sdk_import.cmake) project(SLOW) pico_sdk_init() # Configure stdio set(PICO_STDIO_UART OFF) set(PICO_STDIO_USB ON) add_executable(main main.cpp) target_link_libraries(main PRIVATE pico_stdlib) pico_enable_stdio_usb(main 1) pico_enable_stdio_uart(main 0) pico_set_float_implementation(main pico) pico_add_extra_outputs(main) ``` --- This issue on tinyusb repo: hathach/tinyusb#3120

geurtv commented

2025-05-13 08:02:14 -05:00

(Migrated from github.com)

Don't know if this should be considered a clang(*) or a pico-sdk bug: with LTO enabled the linker discards all the auto_init_mutex mutexes; section ".mutex_array" is empty. In your case it's print_mutex in stdio.c that causes problems. I don't know what memory print_mutex now occupies but mutex_try_enter_block_until() in function stdout_serialize_begin() now times-out every time (1000 ms timeout). Removing static from macro auto_init_mutex or adding __attribute__((used)) to it fixes the issue ... but I wouldn't be surprised if there will be other (hidden) issues ...

Note that LTO with gcc also fails. -flto + -Wl,--wrap doesn't work due to a one or two decade old linker bug.

Maybe pico-sdk can detect whether LTO is enabled and then abort with an error instead.

(*) I used clang 20.1.0-ATfl

Don't know if this should be considered a clang(*) or a pico-sdk bug: with LTO enabled the linker discards all the `auto_init_mutex` mutexes; section ".mutex_array" is empty. In your case it's `print_mutex` in stdio.c that causes problems. I don't know what memory `print_mutex` now occupies but `mutex_try_enter_block_until()` in function `stdout_serialize_begin()` now times-out every time (1000 ms timeout). Removing `static` from macro `auto_init_mutex` or adding `__attribute__((used))` to it fixes the issue ... but I wouldn't be surprised if there will be other (hidden) issues ... Note that LTO with gcc also fails. -flto + -Wl,--wrap doesn't work due to a one or two decade old linker bug. Maybe pico-sdk can detect whether LTO is enabled and then abort with an error instead. (*) I used clang 20.1.0-ATfl

Ivan-Voronyuk commented

2025-05-13 08:49:43 -05:00

(Migrated from github.com)

Declaring mutex_t as volatile in mutex.h also fixes the issue.
It seems to me that mutexes should be volatile as they exist in order to sync code across different cores so it might be a good solution.

PS: I use clang-18.1.3 and LLVM-ET-Arm-18.1.3 toolchain.

Declaring `mutex_t` as `volatile` in `mutex.h` also fixes the issue. It seems to me that mutexes _should_ be volatile as they exist in order to sync code across different cores so it might be a good solution. PS: I use `clang-18.1.3` and `LLVM-ET-Arm-18.1.3` toolchain.

lurch commented

2025-05-13 09:08:04 -05:00

(Migrated from github.com)