Add flexible file system and POSIX file API support #1715

Open
oyama wants to merge 17 commits from oyama/filesystem_support into develop
oyama commented 2024-05-27 04:41:17 -05:00 (Migrated from github.com)

This pull request adds flexible file system and POSIX file API support to pico-sdk. This PR fixes the following Issue:
#531
#767
#1663

The file system consists of several libraries, starting with the pico_filesystem library. Users are free to build the file system they need for their embedded projects using the familiar POSIX and C standard file API.

The pico_filesystem library contains two block devices with on-board flash memory and SPI connection SD card support. The file systems available are FAT and littlefs. The pico_filesystem library is enabled by simply appending the following function to CMakeLists.txt in the user project:

pico_enable_filesystem(${CMAKE_PROJECT_NAME})

Users can programmatically #include <pico/filesystem.h> and call fs_init() to manipulate files with the familiar POSIX file API:

#include <stdio.h>
#include <pico/stdlib.h>
#include <pico/filesystem.h>

int main(void) {
    stdio_init_all();
    fs_init();

    FILE *fp = fopen("/HELLO.TXT", "w");
    fprintf(fp, "Hello World!\n");
    fclose(fp);
}

In the default configuration, littlefs with onboard flash memory as block device is mounted in /. I have prepared a sample project pico-filesystem-examples repository that uses this actual diff, see also here.

Libraries to be added

The pico_filesystem consists of the block device abstraction library, the file system abstraction library and the virtual file system library pico_filesystem, which integrates these as Newlib system calls.

  • pico_filesystem: POSIX and C standard file API, and File system management API libraries
  • pico_filesystem_deafult: Convenience library retaining default file system configuration
  • pico_filesystem_blockdevice_flash: On-board flash memory block device library
  • pico_filesystem_blockdevice_sd: SPI-connected SD card block device library
  • pico_filesystem_blockdevice_heap: Heap memory block device library
  • pico_filesystem_filesystem_fat: FAT file system library (using FatFs)
  • pico_filesystem_filesystem_littlefs: Littlefs file system library (using littlefs)

Any block device can be combined with any file system. Users can select only the libraries they need and add them to their projects.

Configuration filesystem

Block devices and file systems are each abstracted and can be freely combined. For example, the following combinations are possible:.

  • / for littlefs on flash and /sd for FAT on SD cards.
  • Format and use the flash with 512 KB littlefs and 512 KB FAT.
  • Use SD card formatted with littlefs

To set a non-default file system, write CMakeLists.txt as follows:.

pico_enable_filesystem(${CMAKE_PROJECT_NAME} FS_INIT my_fs_init.c)

You can freely design your own file system with your own fs_init() function.

#include <stdio.h>
#include <string.h>
#include <hardware/clocks.h>
#include <hardware/flash.h>
#include <pico/filesystem.h>
#include <pico/filesystem/blockdevice/flash.h>
#include <pico/filesystem/blockdevice/sd.h>
#include <pico/filesystem/filesystem/fat.h>
#include <pico/filesystem/filesystem/littlefs.h>

bool fs_init(void) {
    blockdevice_t *flash = blockdevice_flash_create(PICO_FLASH_SIZE_BYTES - PICO_FS_DEFAULT_SIZE, 0);
    blockdevice_t *sd = blockdevice_sd_create(spi0,
                                              PICO_DEFAULT_SPI_TX_PIN,
                                              PICO_DEFAULT_SPI_RX_PIN,
                                              PICO_DEFAULT_SPI_SCK_PIN,
                                              PICO_DEFAULT_SPI_CSN_PIN,
                                              24 * MHZ,
                                              true);
    filesystem_t *lfs = filesystem_littlefs_create(500, 16);
    filesystem_t *fat = filesystem_fat_create();

    fs_mount("/", lfs, flash);
    fs_mount("/sd", fat, sd);
    return true;
}

Of course, blockdevice_t and/or filesystem_t objects can also be implemented by users themselves and added to the system.

Program size per library combination

Library Combination Used size Region Size %age Used
SD + Flash + FAT + littlefs 211648 B 2 MB 10.09%
SD + FAT 177776 B 2 MB 8.48%
Flash + FAT 172168 B 2 MB 8.21%
SD + Flash + littlefs 89104 B 2 MB 4.25%
SD + littlefs 88312 B 2 MB 4.21%
Flash + littlefs 82720 B 2 MB 3.94%

Supported POSIX and C standard file API

The pico_filesystem library implements basic Newlib "system calls", allowing users to #include <stdio.h> and use a wide range of POSIX and C standard file APIs. The familiar success is set to 0, failure to -1 and errno. A list of tested APIs is available.

Testing

The pico-filesystem-examples project contains unit and integration tests for pico_filesystem.

This library is BSD-3-Clause licensed. See also the pico-vfs repository for more information.

Regards,

This pull request adds flexible file system and POSIX file API support to pico-sdk. This PR fixes the following Issue: #531 #767 #1663 The file system consists of several libraries, starting with the _pico_filesystem_ library. Users are free to build the file system they need for their embedded projects using the familiar POSIX and C standard file API. The pico_filesystem library contains two block devices with on-board flash memory and SPI connection SD card support. The file systems available are FAT and littlefs. The pico_filesystem library is enabled by simply appending the following function to `CMakeLists.txt` in the user project: ```CMakeLists.txt pico_enable_filesystem(${CMAKE_PROJECT_NAME}) ``` Users can programmatically `#include <pico/filesystem.h>` and call `fs_init()` to manipulate files with the familiar POSIX file API: ```main.c #include <stdio.h> #include <pico/stdlib.h> #include <pico/filesystem.h> int main(void) { stdio_init_all(); fs_init(); FILE *fp = fopen("/HELLO.TXT", "w"); fprintf(fp, "Hello World!\n"); fclose(fp); } ``` In the default configuration, littlefs with onboard flash memory as block device is mounted in `/`. I have prepared a sample project [pico-filesystem-examples](https://github.com/oyama/pico-filesystem-examples) repository that uses this actual diff, see also here. ## Libraries to be added The pico_filesystem consists of the block device abstraction library, the file system abstraction library and the virtual file system library pico_filesystem, which integrates these as Newlib system calls. - **pico_filesystem**: POSIX and C standard file API, and File system management API libraries - **pico_filesystem_deafult**: Convenience library retaining default file system configuration - **pico_filesystem_blockdevice_flash**: On-board flash memory block device library - **pico_filesystem_blockdevice_sd**: SPI-connected SD card block device library - **pico_filesystem_blockdevice_heap**: Heap memory block device library - **pico_filesystem_filesystem_fat**: FAT file system library (using FatFs) - **pico_filesystem_filesystem_littlefs**: Littlefs file system library (using littlefs) Any block device can be combined with any file system. Users can select only the libraries they need and add them to their projects. ## Configuration filesystem __Block devices__ and __file systems__ are each abstracted and can be freely combined. For example, the following combinations are possible:. - `/` for littlefs on flash and `/sd` for FAT on SD cards. - Format and use the flash with 512 KB littlefs and 512 KB FAT. - Use SD card formatted with littlefs To set a non-default file system, write `CMakeLists.txt` as follows:. ```CMakeLists.txt pico_enable_filesystem(${CMAKE_PROJECT_NAME} FS_INIT my_fs_init.c) ``` You can freely design your own file system with your own fs_init() function. ```my_fs_init.c #include <stdio.h> #include <string.h> #include <hardware/clocks.h> #include <hardware/flash.h> #include <pico/filesystem.h> #include <pico/filesystem/blockdevice/flash.h> #include <pico/filesystem/blockdevice/sd.h> #include <pico/filesystem/filesystem/fat.h> #include <pico/filesystem/filesystem/littlefs.h> bool fs_init(void) { blockdevice_t *flash = blockdevice_flash_create(PICO_FLASH_SIZE_BYTES - PICO_FS_DEFAULT_SIZE, 0); blockdevice_t *sd = blockdevice_sd_create(spi0, PICO_DEFAULT_SPI_TX_PIN, PICO_DEFAULT_SPI_RX_PIN, PICO_DEFAULT_SPI_SCK_PIN, PICO_DEFAULT_SPI_CSN_PIN, 24 * MHZ, true); filesystem_t *lfs = filesystem_littlefs_create(500, 16); filesystem_t *fat = filesystem_fat_create(); fs_mount("/", lfs, flash); fs_mount("/sd", fat, sd); return true; } ``` Of course, blockdevice_t and/or filesystem_t objects can also be implemented by users themselves and added to the system. ### Program size per library combination | Library Combination | Used size |Region Size | %age Used | |---------------------------|-----------|------------|-----------| |SD + Flash + FAT + littlefs| 211648 B | 2 MB | 10.09% | | SD + FAT | 177776 B | 2 MB | 8.48% | | Flash + FAT | 172168 B | 2 MB | 8.21% | | SD + Flash + littlefs | 89104 B | 2 MB | 4.25% | | SD + littlefs | 88312 B | 2 MB | 4.21% | | Flash + littlefs | 82720 B | 2 MB | 3.94% | ## Supported POSIX and C standard file API The pico_filesystem library implements basic Newlib "system calls", allowing users to `#include <stdio.h>` and use a wide range of POSIX and C standard file APIs. The familiar success is set to 0, failure to -1 and errno. A [list of tested APIs](https://github.com/oyama/pico-vfs/blob/main/STANDARD.md) is available. ## Testing The [pico-filesystem-examples](https://github.com/oyama/pico-filesystem-examples) project contains unit and integration tests for pico_filesystem. This library is BSD-3-Clause licensed. See also the [pico-vfs repository](https://github.com/oyama/pico-vfs) for more information. Regards,
Memotech-Bill commented 2024-05-27 08:33:01 -05:00 (Migrated from github.com)

It is worth referencing the discussion: https://forums.raspberrypi.com/viewtopic.php?t=368898

It is worth referencing the discussion: https://forums.raspberrypi.com/viewtopic.php?t=368898
kilograham commented 2024-05-27 10:11:40 -05:00 (Migrated from github.com)

this looks great, thank you.

one initial question from a quick skip read; how does overriding _write play with STDIN/STDOUT/STDERR handles already handled by SDK override of the same? I'm assuming it just breaks it?

I'm putting this in 1.7.0 for now as we are looking at reworking our C library API (so we don't just support newlib), and we should think about how to "inject" mutliple FILE back ends (though arguably once you have a VFS, perhaps it is only the stdin/stdout/stderr that you care about, so the "replacement" APIs could just call back for those

this looks great, thank you. one initial question from a quick skip read; how does overriding `_write` play with STDIN/STDOUT/STDERR handles already handled by SDK override of the same? I'm assuming it just breaks it? I'm putting this in 1.7.0 for now as we are looking at reworking our C library API (so we don't just support newlib), and we should think about how to "inject" mutliple FILE back ends (though arguably once you have a VFS, perhaps it is only the stdin/stdout/stderr that you care about, so the "replacement" APIs could just call back for those
oyama commented 2024-05-27 11:22:53 -05:00 (Migrated from github.com)

Thank you for your comment,

pico_filesystem VFS linked write(fileno(stdout), ...); returns a 9: Bad file number error or write to a user opened file. the process is passed to the VFS of the pico_filesystem, which is certainly an unintended behaviour of the user. However, calling printf("foo"); does not seem to have any negative effect, as the process is sent to the UART and USB without being _write.
A bad point of the current pico_filesystem VFS is that file descriptors are issued from 0 when a file is opened. This can conflict with STDIN_FILENO/STDOUT_FILENO/STDERR_FINENO of the standard OS and should probably be avoided.

I think pico_stdio_semihosting is more susceptible. However, this also needs to be verified.

Thank you for your comment, pico_filesystem VFS linked `write(fileno(stdout), ...);` returns a `9: Bad file number` error or write to a user opened file. the process is passed to the VFS of the pico_filesystem, which is certainly an unintended behaviour of the user. However, calling `printf("foo");` does not seem to have any negative effect, as the process is sent to the UART and USB without being `_write`. A bad point of the current pico_filesystem VFS is that file descriptors are issued from `0` when a file is opened. This can conflict with `STDIN_FILENO`/`STDOUT_FILENO`/`STDERR_FINENO` of the standard OS and should probably be avoided. I think `pico_stdio_semihosting` is more susceptible. However, this also needs to be verified.
oyama commented 2024-05-27 23:50:46 -05:00 (Migrated from github.com)

A bad point of the current pico_filesystem VFS is that file descriptors are issued from 0 when a file is opened. This can conflict with STDIN_FILENO/STDOUT_FILENO/STDERR_FINENO of the standard OS and should probably be avoided.

This problem has been fixed and the VFS file descriptor for pico_filesystem is now issued from 3. If the user calls write(fileno(stdout), ...) or read(fileno(stdin), ...) , returns -1 and errno == 9.

> A bad point of the current pico_filesystem VFS is that file descriptors are issued from `0` when a file is opened. This can conflict with `STDIN_FILENO`/`STDOUT_FILENO`/`STDERR_FINENO` of the standard OS and should probably be avoided. This problem has been fixed and the VFS file descriptor for pico_filesystem is now issued from `3`. If the user calls `write(fileno(stdout), ...)` or `read(fileno(stdin), ...) `, returns `-1` and `errno == 9`.
oyama commented 2024-05-28 09:52:29 -05:00 (Migrated from github.com)

To avoid inhibiting pico_stdio, file descriptors 0 to 2 are passed to pico_stdio instead of being processed by pico_filesystem.

To avoid inhibiting pico_stdio, file descriptors `0` to `2` are passed to pico_stdio instead of being processed by pico_filesystem.
kilograham commented 2024-08-25 13:29:27 -05:00 (Migrated from github.com)

I'm putting this in 1.7.0 for now as we are looking at reworking our C library API (so we don't just support newlib)

This is the pico_clib_interface stuff in SDK 2.0.0 ... we should now have a good place to inject FILE support

> I'm putting this in 1.7.0 for now as we are looking at reworking our C library API (so we don't just support newlib) This is the `pico_clib_interface` stuff in SDK 2.0.0 ... we should now have a good place to inject FILE support
oyama commented 2024-08-26 11:08:01 -05:00 (Migrated from github.com)

Thanks for the 2.0.0 release! This is a good hook point. I'll modify it to fit this API and test it.

Thanks for the 2.0.0 release! This is a good hook point. I'll modify it to fit this API and test it.
oyama commented 2024-09-05 01:11:25 -05:00 (Migrated from github.com)

I finally got my hands on Pico 2 and have made the following fixes and verifications

  • Changed PR to conform to pico-sdk 2.0.0
  • verified that pico-vfs, the core of the PR implementation, works with Pico/Pico 2 Arm/Pico 2 RISC-V cores

Onboard Flash and SPI SD card operation is also no longer a problem.

I finally got my hands on Pico 2 and have made the following fixes and verifications - Changed PR to conform to pico-sdk 2.0.0 - verified that `pico-vfs`, the core of the PR implementation, works with Pico/Pico 2 Arm/Pico 2 RISC-V cores Onboard Flash and SPI SD card operation is also no longer a problem.
Slion commented 2024-10-26 04:22:11 -05:00 (Migrated from github.com)

This awesome stuff and I'm looking forward using it. Thanks for sharing.

How do you define the range of flash being used by the file system? The end of the flash is notably used by btstack to store paired devices' keys, we don't want those to be overwritten. Do you check and assert if the firmware overlaps with the flash file system specified? Can I check how much space is currently used and how much is available on the file system?

This awesome stuff and I'm looking forward using it. Thanks for sharing. How do you define the range of flash being used by the file system? The end of the flash is notably used by btstack to store paired devices' keys, we don't want those to be overwritten. Do you check and assert if the firmware overlaps with the flash file system specified? Can I check how much space is currently used and how much is available on the file system?
Slion commented 2024-10-26 15:46:17 -05:00 (Migrated from github.com)

So it looks like the default configuration is not taking care of leaving the Bluetooth storage alone.
https://github.com/oyama/pico-vfs/blob/main/src/filesystem/fs_init.c#L14
It also uses a large chunk of flash and does not try to check if the firmware is small enough.

So it looks like the default configuration is not taking care of leaving the Bluetooth storage alone. https://github.com/oyama/pico-vfs/blob/main/src/filesystem/fs_init.c#L14 It also uses a large chunk of flash and does not try to check if the firmware is small enough.
Memotech-Bill commented 2024-10-27 01:34:39 -05:00 (Migrated from github.com)

How do you define the range of flash being used by the file system? The end of the flash is notably used by btstack to store paired devices' keys, we don't want those to be overwritten.

It is possible to change where bluetooth stores its keys by defining PICO_FLASH_BANK_STORAGE_OFFSET and PICO_FLASH_BANK_TOTAL_SIZE.

> How do you define the range of flash being used by the file system? The end of the flash is notably used by btstack to store paired devices' keys, we don't want those to be overwritten. It is possible to change where bluetooth stores its keys by defining **PICO_FLASH_BANK_STORAGE_OFFSET** and **PICO_FLASH_BANK_TOTAL_SIZE**.
Slion commented 2024-10-27 02:21:11 -05:00 (Migrated from github.com)

Does each flash file use at least a 4K sector or is it somehow managed at the flash page level?

Answer:
This is more a question about littlefs really and from the testing I did it looks like littlefs can pack multiple files per sector thankfully.
See: https://github.com/oyama/pico-vfs/issues/60#issuecomment-2440125408

Does each flash file use at least a 4K sector or is it somehow managed at the flash page level? **Answer:** This is more a question about littlefs really and from the testing I did it looks like littlefs can pack multiple files per sector thankfully. See: https://github.com/oyama/pico-vfs/issues/60#issuecomment-2440125408
Slion commented 2024-10-27 02:24:09 -05:00 (Migrated from github.com)

It is possible to change where bluetooth stores its keys by defining PICO_FLASH_BANK_STORAGE_OFFSET and PICO_FLASH_BANK_TOTAL_SIZE.

I'm planning to do it the other way around. I'll define a FS configuration that leaves enough space for my firmware and BT storage.

> It is possible to change where bluetooth stores its keys by defining PICO_FLASH_BANK_STORAGE_OFFSET and PICO_FLASH_BANK_TOTAL_SIZE. I'm planning to do it the other way around. I'll define a FS configuration that leaves enough space for my firmware and BT storage.
Slion commented 2024-10-27 06:57:45 -05:00 (Migrated from github.com)

I got it working fine it seems. For those interested on how it looks for real take a look at that: https://github.com/oyama/pico-vfs/issues/59

Basically the above samples forgot to mention you need to format your file system before it can be mounted and used.

I got it working fine it seems. For those interested on how it looks for real take a look at that: https://github.com/oyama/pico-vfs/issues/59 Basically the above samples forgot to mention you need to format your file system before it can be mounted and used.
Slion commented 2024-10-27 12:07:36 -05:00 (Migrated from github.com)

Why do we need the littlefs submodule in pico-sdk when it is already present in pico-vfs?

Why do we need the littlefs submodule in pico-sdk when it is already present in pico-vfs?
Slion commented 2024-10-28 13:58:58 -05:00 (Migrated from github.com)

Worth noting that this also works through C++ fstream.

Worth noting that this also works through C++ fstream.
oyama commented 2024-11-08 00:28:21 -06:00 (Migrated from github.com)

Thank you for your comment. My response is delayed as I was on a long business trip.

The reason for registering the littlefs submodule in both pico-vfs and pico-sdk is to avoid modifying the pico-sdk installation instructions. As you pointed out, a more natural layout would be to register littlefs as a submodule only once under pico-vfs. However, since git does not clone submodules recursively by default, we would need to adjust the pico-sdk installation process. To avoid this, I added littlefs directly as a submodule in pico-sdk as well, creating a flat submodule structure for the pull request.

Thank you for your comment. My response is delayed as I was on a long business trip. The reason for registering the littlefs submodule in both pico-vfs and pico-sdk is to avoid modifying the pico-sdk installation instructions. As you pointed out, a more natural layout would be to register littlefs as a submodule only once under pico-vfs. However, since git does not clone submodules recursively by default, we would need to adjust the pico-sdk installation process. To avoid this, I added littlefs directly as a submodule in pico-sdk as well, creating a flat submodule structure for the pull request.
gneverov commented 2025-02-13 17:38:59 -06:00 (Migrated from github.com)

This library is great, but I'd like to mention another library that does the same plus more. In addition to providing a VFS, it also supports concurrency (threads via FreeRTOS, a thread-safe VFS, select/poll functions), tries to emulate as much of the POSIX API as possible (not just filesystem-related stuff), integrates with lwIP for the socket API, provides a driver model for TTY devices (so you can also virtualize where stdin/stdout go), even does dynamic linking of firmware binaries. I'm not sure what the goals are for Pico SDK, but you might want to consider this library. (P.S. I am the author of the library.)

This library is great, but I'd like to mention another [library](https://github.com/gneverov/morelibc) that does the same plus more. In addition to providing a VFS, it also supports concurrency (threads via FreeRTOS, a thread-safe VFS, `select`/`poll` functions), tries to emulate as much of the POSIX API as possible (not just filesystem-related stuff), integrates with lwIP for the socket API, provides a driver model for TTY devices (so you can also virtualize where stdin/stdout go), even does dynamic linking of firmware binaries. I'm not sure what the goals are for Pico SDK, but you might want to consider this library. (P.S. I am the author of the library.)
This pull request can be merged automatically.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin 'oyama/filesystem_support:oyama/filesystem_support'
git switch 'oyama/filesystem_support'

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch 'develop'
git merge --no-ff 'oyama/filesystem_support'
git switch 'oyama/filesystem_support'
git rebase 'develop'
git switch 'develop'
git merge --ff-only 'oyama/filesystem_support'
git switch 'oyama/filesystem_support'
git rebase 'develop'
git switch 'develop'
git merge --no-ff 'oyama/filesystem_support'
git switch 'develop'
git merge --squash 'oyama/filesystem_support'
git switch 'develop'
git merge --ff-only 'oyama/filesystem_support'
git switch 'develop'
git merge 'oyama/filesystem_support'
git push origin 'develop'
Sign in to join this conversation.
No reviewers
No milestone
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
gh-raspberrypi/pico-sdk!1715
No description provided.