Add flexible file system and POSIX file API support #1715
No reviewers
Labels
No labels
bazel
bluetooth
board settings
boot_stage2
bug
build
c++
can't reproduce
clang
clib-interface
cmsis
code cleanliness
code size
compiler version issues
cyw43_driver
debug
discussion
documentation
duplicate
dynamic_clock
elf2uf2
enhancement
errata
filesystem
flash layout
floating point
FreeRTOS
hardware_adc
hardware_claim
hardware_clocks
hardware_dma
hardware_exception
hardware_flash
hardware_gpio
hardware_i2c
hardware_irq
hardware_pio
hardware_pwm
hardware_rcp
hardware_rtc
hardware_spi
hardware_sync
hardware_uart
hardware_vreg
hardware_watchdog
hardware_xosc
help wanted
host
IAR
installation
invalid
linker scripts
lto
lwip
mbedtls
missing API
needs reporter detail
networking
newlib
newlib-multicore
non-c
non-cmake
non-gcc
openocd
orphan PR
OS integration
pico_async_context
pico_base
pico_binary_info
pico_flash
picolibc
pico_lwip
pico_malloc
pico_multicore
pico_printf
pico_runtime
pico_stdio
pico_stdio_usb
pico_sync
pico_time
pico_util
pioasm
port
question
review
review2
rtos interop
runtime init
static library
svd
test
tinyusb
vscode
wifi integration
wontfix
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
gh-raspberrypi/pico-sdk!1715
Loading…
Reference in a new issue
No description provided.
Delete branch "oyama/filesystem_support"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
This pull request adds flexible file system and POSIX file API support to pico-sdk. This PR fixes the following Issue:
#531
#767
#1663
The file system consists of several libraries, starting with the pico_filesystem library. Users are free to build the file system they need for their embedded projects using the familiar POSIX and C standard file API.
The pico_filesystem library contains two block devices with on-board flash memory and SPI connection SD card support. The file systems available are FAT and littlefs. The pico_filesystem library is enabled by simply appending the following function to
CMakeLists.txtin the user project:Users can programmatically
#include <pico/filesystem.h>and callfs_init()to manipulate files with the familiar POSIX file API:In the default configuration, littlefs with onboard flash memory as block device is mounted in
/. I have prepared a sample project pico-filesystem-examples repository that uses this actual diff, see also here.Libraries to be added
The pico_filesystem consists of the block device abstraction library, the file system abstraction library and the virtual file system library pico_filesystem, which integrates these as Newlib system calls.
Any block device can be combined with any file system. Users can select only the libraries they need and add them to their projects.
Configuration filesystem
Block devices and file systems are each abstracted and can be freely combined. For example, the following combinations are possible:.
/for littlefs on flash and/sdfor FAT on SD cards.To set a non-default file system, write
CMakeLists.txtas follows:.You can freely design your own file system with your own fs_init() function.
Of course, blockdevice_t and/or filesystem_t objects can also be implemented by users themselves and added to the system.
Program size per library combination
Supported POSIX and C standard file API
The pico_filesystem library implements basic Newlib "system calls", allowing users to
#include <stdio.h>and use a wide range of POSIX and C standard file APIs. The familiar success is set to 0, failure to -1 and errno. A list of tested APIs is available.Testing
The pico-filesystem-examples project contains unit and integration tests for pico_filesystem.
This library is BSD-3-Clause licensed. See also the pico-vfs repository for more information.
Regards,
It is worth referencing the discussion: https://forums.raspberrypi.com/viewtopic.php?t=368898
this looks great, thank you.
one initial question from a quick skip read; how does overriding
_writeplay with STDIN/STDOUT/STDERR handles already handled by SDK override of the same? I'm assuming it just breaks it?I'm putting this in 1.7.0 for now as we are looking at reworking our C library API (so we don't just support newlib), and we should think about how to "inject" mutliple FILE back ends (though arguably once you have a VFS, perhaps it is only the stdin/stdout/stderr that you care about, so the "replacement" APIs could just call back for those
Thank you for your comment,
pico_filesystem VFS linked
write(fileno(stdout), ...);returns a9: Bad file numbererror or write to a user opened file. the process is passed to the VFS of the pico_filesystem, which is certainly an unintended behaviour of the user. However, callingprintf("foo");does not seem to have any negative effect, as the process is sent to the UART and USB without being_write.A bad point of the current pico_filesystem VFS is that file descriptors are issued from
0when a file is opened. This can conflict withSTDIN_FILENO/STDOUT_FILENO/STDERR_FINENOof the standard OS and should probably be avoided.I think
pico_stdio_semihostingis more susceptible. However, this also needs to be verified.This problem has been fixed and the VFS file descriptor for pico_filesystem is now issued from
3. If the user callswrite(fileno(stdout), ...)orread(fileno(stdin), ...), returns-1anderrno == 9.To avoid inhibiting pico_stdio, file descriptors
0to2are passed to pico_stdio instead of being processed by pico_filesystem.This is the
pico_clib_interfacestuff in SDK 2.0.0 ... we should now have a good place to inject FILE supportThanks for the 2.0.0 release! This is a good hook point. I'll modify it to fit this API and test it.
I finally got my hands on Pico 2 and have made the following fixes and verifications
pico-vfs, the core of the PR implementation, works with Pico/Pico 2 Arm/Pico 2 RISC-V coresOnboard Flash and SPI SD card operation is also no longer a problem.
This awesome stuff and I'm looking forward using it. Thanks for sharing.
How do you define the range of flash being used by the file system? The end of the flash is notably used by btstack to store paired devices' keys, we don't want those to be overwritten. Do you check and assert if the firmware overlaps with the flash file system specified? Can I check how much space is currently used and how much is available on the file system?
So it looks like the default configuration is not taking care of leaving the Bluetooth storage alone.
https://github.com/oyama/pico-vfs/blob/main/src/filesystem/fs_init.c#L14
It also uses a large chunk of flash and does not try to check if the firmware is small enough.
It is possible to change where bluetooth stores its keys by defining PICO_FLASH_BANK_STORAGE_OFFSET and PICO_FLASH_BANK_TOTAL_SIZE.
Does each flash file use at least a 4K sector or is it somehow managed at the flash page level?
Answer:
This is more a question about littlefs really and from the testing I did it looks like littlefs can pack multiple files per sector thankfully.
See: https://github.com/oyama/pico-vfs/issues/60#issuecomment-2440125408
I'm planning to do it the other way around. I'll define a FS configuration that leaves enough space for my firmware and BT storage.
I got it working fine it seems. For those interested on how it looks for real take a look at that: https://github.com/oyama/pico-vfs/issues/59
Basically the above samples forgot to mention you need to format your file system before it can be mounted and used.
Why do we need the littlefs submodule in pico-sdk when it is already present in pico-vfs?
Worth noting that this also works through C++ fstream.
Thank you for your comment. My response is delayed as I was on a long business trip.
The reason for registering the littlefs submodule in both pico-vfs and pico-sdk is to avoid modifying the pico-sdk installation instructions. As you pointed out, a more natural layout would be to register littlefs as a submodule only once under pico-vfs. However, since git does not clone submodules recursively by default, we would need to adjust the pico-sdk installation process. To avoid this, I added littlefs directly as a submodule in pico-sdk as well, creating a flat submodule structure for the pull request.
This library is great, but I'd like to mention another library that does the same plus more. In addition to providing a VFS, it also supports concurrency (threads via FreeRTOS, a thread-safe VFS,
select/pollfunctions), tries to emulate as much of the POSIX API as possible (not just filesystem-related stuff), integrates with lwIP for the socket API, provides a driver model for TTY devices (so you can also virtualize where stdin/stdout go), even does dynamic linking of firmware binaries. I'm not sure what the goals are for Pico SDK, but you might want to consider this library. (P.S. I am the author of the library.)View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.Merge
Merge the changes and update on Forgejo.Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.