gh-raspberrypi/pico-sdk

Watch

Fork 0

Allow flash programming of partial pages (<256 bytes) #1169

New issue

Open

opened 2023-01-09 10:27:35 -06:00 by hattesen · 8 comments

hattesen commented

2023-01-09 10:27:35 -06:00

(Migrated from github.com)

The current hardware_flash/flash.c implementation validates parameters to flash_range_program(uint32_t flash_offs, const uint8_t *data, size_t count) ensuring that 'flash_offs' is on a page boundary (% 256 == 0) as well as byte countis a multiple of a full page (256), which is also documented in the API documentation:

// pico-sdk/src/rp2_common/hardware_flash/include/hardware/flash.h
/*! \brief  Program flash
 *  \ingroup hardware_flash
 *
 * \param flash_offs Flash address of the first byte to be programmed. Must be aligned to a 256-byte flash page.
 * \param data Pointer to the data to program into flash
 * \param count Number of bytes to program. Must be a multiple of 256 bytes (one page).
 */
void flash_range_program(uint32_t flash_offs, const uint8_t *data, size_t count);

Implementation: /src/rp2_common/hardware_flash/flash.c:

// pico-sdk/src/rp2_common/hardware_flash/flash.c
void __no_inline_not_in_flash_func(flash_range_program)(uint32_t flash_offs, const uint8_t *data, size_t count) {
    ...
    invalid_params_if(FLASH, flash_offs & (FLASH_PAGE_SIZE - 1));
    invalid_params_if(FLASH, count & (FLASH_PAGE_SIZE - 1));
    ...

When implementing EEPROM emulation or a File System backed by the QSPI (NOR) Flash (above the XIP address range), it severely restricts the potential speed and increases SRAM requirements, as every writes to Flash will require a read-modify-write cycle:

Full page (256 bytes) read
Modify page
Full page (256 bytes) program using flash_range_program()

The SPI connected Serial Flash IC used with the RP2040, does not have ANY restrictions regarding starting address or number of bytes written, when using the Page Program (02h) instruction (see Winbond W25Q16JV datasheet, section 9.2.13 for details):

The Page Program instruction allows from one byte to 256 bytes (a page) of data to be programmed at
previously erased (FFh) memory locations

Would it be possible to add a facility (library function) allowing partial page programming at arbitrary addresses, using an API similar to the flash_range_program(uint32_t flash_offs, const uint8_t *data, size_t count) function?
This function may likely restrict all data to be written to be located in ONE 256 byte page, and not allow wrapping (to the beginning of the page, which is the standard behavior of the underlying Serial Flash.

The current `hardware_flash/flash.c` implementation validates parameters to `flash_range_program(uint32_t flash_offs, const uint8_t *data, size_t count)` ensuring that 'flash_offs' is on a page boundary (% 256 == 0) as well as byte `count`is a multiple of a full page (256), which is also documented in the API documentation: ```C // pico-sdk/src/rp2_common/hardware_flash/include/hardware/flash.h /*! \brief Program flash * \ingroup hardware_flash * * \param flash_offs Flash address of the first byte to be programmed. Must be aligned to a 256-byte flash page. * \param data Pointer to the data to program into flash * \param count Number of bytes to program. Must be a multiple of 256 bytes (one page). */ void flash_range_program(uint32_t flash_offs, const uint8_t *data, size_t count); ``` [Implementation: /src/rp2_common/hardware_flash/flash.c:](https://github.com/raspberrypi/pico-sdk/blob/2e6142b15b8a75c1227dd3edbe839193b2bf9041/src/rp2_common/hardware_flash/flash.c#L86) ```C // pico-sdk/src/rp2_common/hardware_flash/flash.c void __no_inline_not_in_flash_func(flash_range_program)(uint32_t flash_offs, const uint8_t *data, size_t count) { ... invalid_params_if(FLASH, flash_offs & (FLASH_PAGE_SIZE - 1)); invalid_params_if(FLASH, count & (FLASH_PAGE_SIZE - 1)); ... ``` When implementing EEPROM emulation or a File System backed by the QSPI (NOR) Flash (above the XIP address range), it severely restricts the potential speed and increases SRAM requirements, as every writes to Flash will require a read-modify-write cycle: 1. Full page (256 bytes) read 2. Modify page 3. Full page (256 bytes) program using `flash_range_program()` The SPI connected Serial Flash IC used with the RP2040, does not have ANY restrictions regarding starting address or number of bytes written, when using the `Page Program (02h)` instruction (see [Winbond W25Q16JV datasheet](https://www.winbond.com/hq/support/documentation/levelOne.jsp?__locale=en&DocNo=DA00-W25Q16JV.1), section 9.2.13 for details): > The Page Program instruction allows from one byte to 256 bytes (a page) of data to be programmed at previously erased (FFh) memory locations Would it be possible to add a facility (library function) allowing partial page programming at arbitrary addresses, using an API similar to the `flash_range_program(uint32_t flash_offs, const uint8_t *data, size_t count)` function? This function may likely restrict all data to be written to be located in ONE 256 byte page, and not allow wrapping (to the beginning of the page, which is the standard behavior of the underlying Serial Flash.

taraskornuta commented

2023-02-06 03:09:21 -06:00

(Migrated from github.com)

Also want to support this meaning. Such an approach will also reduce the lifetime of the flash itself.

hattesen commented

2023-02-14 19:14:08 -06:00

(Migrated from github.com)

@taraskornuta

Also want to support this meaning. Such an approach will also reduce the lifetime of the flash itself.

Allowing partial page writes will not, in any way, reduce the lifetime of the flash. It is page erasures that wear out flash, not page writes (which should obviously not be overlapping). For an effective EEPROM simulation in Flash, allowing partial page writes will reduce the number of flash erase operations required, and thus reduce the wear of the flash, increasing the lifetime of it.

@taraskornuta > Also want to support this meaning. Such an approach will also reduce the lifetime of the flash itself. Allowing partial page writes will not, in any way, reduce the lifetime of the flash. It is page erasures that wear out flash, not page writes (which should obviously not be overlapping). For an effective EEPROM simulation in Flash, allowing partial page writes will reduce the number of flash erase operations required, and thus reduce the wear of the flash, increasing the lifetime of it.

lurch commented

2023-02-15 03:27:39 -06:00

(Migrated from github.com)

I know it's probably not the easiest solution, but instead of trying to write partial-pages <256 bytes, how hard would it be to modify your application to buffer write-data in RAM until you have a full page-worth to be written to flash?

I guess adding such an API / function would introduce the possibility of trying to overwrite some already-written data (i.e. not freshly-erased space), and thus introduces the possibility that what you _thought_ you'd written into flash, won't be the same as what you actually read back from flash? :thinking: I know it's probably not the easiest solution, but instead of trying to write partial-pages <256 bytes, how hard would it be to modify your application to buffer write-data in RAM until you have a full page-worth to be written to flash?

hattesen commented

2023-02-15 04:00:18 -06:00

(Migrated from github.com)

@lurch it wouldn't be at all difficult to buffer 256 bytes of data in RAM before writing to Flash, but it isn't always appropriate to do so:

It would make the system vulnerable to data loss in case of reset/power-loss, with data held in RAM and not committed to Flash.
The RP2040 (or the Pi Pico) has no (non-volatile) EEPROM to store runtime data or frequently changing configuration values, llike many other MCU's have, including the very popular ATMEL ATMEGA/ATTINY series . Being forced by the SDK to only write whole pages means that you have to use an entire page every time you want to store a 16 bit count in non-volatile memory, resulting in frequent 4K Sector Erase calls, which is SLOW and causes excessive/unneccesary flash wear.
The datasheet for the Serial Nor flash used in the Raspberry Pi Pico (W25Q16JV) specifically allows partial pages to be programmed:

"Page Program instruction allows from one byte to 256 bytes (a page) of data to be programmed"
What reason would the Pico SDK have for restricting writes to full pages (256 bytes) only?

There are many EEPROM emulation (in Flash) library designs with wear leveling and garbage collection, using partial page writes (and rewrites zeroing flag bits), and I would appreciate if the Pico SDK would not prevent me from implementing such a library.

I fail to see any good reason for this arbitrary restriction in the Pico SDK, but would be happy to gain some insight into the reason – and protecting programmers from their own stupidity isn't a good reason. A partial page program method would likely only allow writing to bytes within ONE page, with no wrapping.

@lurch it wouldn't be at all difficult to buffer 256 bytes of data in RAM before writing to Flash, but it isn't always appropriate to do so: 1. It would make the system vulnerable to data loss in case of reset/power-loss, with data held in RAM and not committed to Flash. 2. The RP2040 (or the Pi Pico) has no (non-volatile) EEPROM to store runtime data or frequently changing configuration values, llike many other MCU's have, including the very popular ATMEL ATMEGA/ATTINY series . Being forced by the SDK to only write whole pages means that you have to use an entire page every time you want to store a 16 bit count in non-volatile memory, resulting in frequent 4K Sector Erase calls, which is SLOW and causes excessive/unneccesary flash wear. 3. The datasheet for the Serial Nor flash used in the Raspberry Pi Pico (W25Q16JV) specifically allows partial pages to be programmed: > "Page Program instruction allows from one byte to 256 bytes (a page) of data to be programmed" What reason would the Pico SDK have for restricting writes to full pages (256 bytes) only? There are many EEPROM emulation (in Flash) library designs with wear leveling and garbage collection, using partial page writes (and rewrites zeroing flag bits), and I would appreciate if the Pico SDK would not prevent me from implementing such a library. - [EEPROM EMULATION WITH WEAR-LEVELING FOR 8-BIT FLASH MCUS (SILABS)](https://www.silabs.com/documents/public/application-notes/AN798.pdf) - [EEPROM Emulation Using Internal Flash (ATMEL)](http://ww1.microchip.com/downloads/en/Appnotes/Atmel-42218-EEPROM-Emulation-Using-Internal-Flash-SAM4_AT4066_AP-Note.pdf) - [EEPROM emulation techniques and software (ST)](https://www.st.com/resource/en/application_note/an4894-eeprom-emulation-techniques-and-software-for-stm32-microcontrollers-stmicroelectronics.pdf) I fail to see any good reason for this arbitrary restriction in the Pico SDK, but would be happy to gain some insight into the reason – and protecting programmers from their own stupidity isn't a good reason. A partial page program method would likely only allow writing to bytes within ONE page, with no wrapping.

👍 1

peterharperuk commented

2023-02-15 04:38:26 -06:00

(Migrated from github.com)

Sounds reasonable to me to add an API to do this. It could be made optional returning an error if the flash didn't support partial page writes (is that even a thing?).

hattesen commented

2023-02-15 04:57:00 -06:00

(Migrated from github.com)

The existing flash_range_program function could easily be extended to also support flash_offs outside page boundaries if count is less than 256, raising an error if count bytes will not fit within the page pointed to by flash_offs.

github.com/raspberrypi/pico-sdk@2062372d20/src/rp2_common/hardware_flash/flash.c (L86-L106)

... but since the implementation in flash.c calls on similarly restricted flash_page_program() methods in /bootrom/program_flash_generic.c, the only way forward is to reimplement a refactored version of the flash_page_program methods from allowing count < 256 bytes at arbitrary (byte aligned) ´flash_offs' addresses will be to refactor flash_page_program methods from /bootrom/program_flash_generic.c and place them in RAM using __no_inline_not_in_flash_func' like their pico-sdk:/src/rp2_common/hardware_flash/flash.c` brethren.

I may give it a try, implementing an updated version supporting partial page programming with a few of code samples making byte-sized updates, or even bit-sized updates (flipping flash bits individually from 1 to zero), resulting in as many as 256*8 calls to flash_range_program(uint32_t flash_offs, const uint8_t *data, size_t count) for each page in a 4KB sector (a grand total of 32,768 calls to flash_range_program on that sector), before erasing the sector and starting over flipping individual 1 bits to zero bits. Doing that 100 times will reveal whether the Flash sector has wear equalling 100 erasures, or if the 3,276,800 flash_range_program has caused additional wear to the sector, rendering it unusable.

@peterharperuk i have looked at a LOT of serial Nor flash datasheets, including those used by Adafruit, Pimoroni, seed, sparkfun and waveshare (many of which uses slow generic SPI access at half speed, compared to the "original" Winbond W25QxxJV), and they all seem to support partial page writes. They may differ in "wrap-at-end-of-page" behavior, but it would be reasonable for a partial page programming function to restrict writes to a single page without exceeding the end, typically causing wrap to beginning of (256 byte) page anyway. The existing `flash_range_program` function could easily be extended to also support `flash_offs` outside page boundaries if `count` is less than 256, raising an error if `count` bytes will not fit within the page pointed to by `flash_offs`. https://github.com/raspberrypi/pico-sdk/blob/2062372d203b372849d573f252cf7c6dc2800c0a/src/rp2_common/hardware_flash/flash.c#L86-L106 ... but since the implementation in flash.c calls on similarly restricted `flash_page_program()` methods in [/bootrom/program_flash_generic.c](https://github.com/raspberrypi/pico-bootrom/blob/ef22cd8ede5bc007f81d7f2416b48db90f313434/bootrom/program_flash_generic.c#L265-L286), the only way forward is to reimplement a refactored version of the `flash_page_program` methods from allowing `count < 256 bytes` at arbitrary (byte aligned) ´flash_offs' addresses will be to refactor flash_page_program methods from [/bootrom/program_flash_generic.c](https://github.com/raspberrypi/pico-bootrom/blob/ef22cd8ede5bc007f81d7f2416b48db90f313434/bootrom/program_flash_generic.c#L265-L286) and place them in RAM using `__no_inline_not_in_flash_func' like their `pico-sdk:/src/rp2_common/hardware_flash/flash.c` brethren. I may give it a try, implementing an updated version supporting partial page programming with a few of code samples making byte-sized updates, or even bit-sized updates (flipping flash bits individually from 1 to zero), resulting in as many as 256*8 calls to `flash_range_program(uint32_t flash_offs, const uint8_t *data, size_t count)` for each page in a 4KB sector (a grand total of 32,768 calls to `flash_range_program` on that sector), before erasing the sector and starting over flipping individual 1 bits to zero bits. Doing that 100 times will reveal whether the Flash sector has wear equalling 100 erasures, or if the 3,276,800 `flash_range_program` has caused additional wear to the sector, rendering it unusable.

hanan619 commented

2025-05-06 01:07:17 -05:00

(Migrated from github.com)

Any update on this issue?

peltsippi commented

2026-04-17 16:01:01 -05:00

(Migrated from github.com)

Just in case someone wants to know. I was able to get around this issue and write the exact size I want to flash by writing a custom function that prepares the data for 256 byte write.

I am writing structs to flash by mapping the data manually from structs to flash addresses (I'd rather control the mapping of the struct to flash myself instead of relying on compiler).

Things go roughly like this (sorry, my project is not even near the point of making it public but this part seems to work when I tested ~15 minutes rewriting values etc):

Find free address (where all data 0xFF) that is big enough
create buff[FLASH_PAGE_SIZE]
2.1 write 0xFF everywhere to the buff. Just to ensure there are no unnecessary bit flips by mistake
2.2 if the values being written next are not located at the beginning of the page, read existing values from the flash and copy to the start of the buff
2.3 if I am updating something that has old entries, flip single bit from the previous valid struct from 1 -> 0 to make it outdated (and ok to be deleted later on)
2.4 write the new values next after existing data to the buff
(2.5 rest of the buff stays 0xFF)
write buff to flash
(verify)

Seems to work pretty well and I am practically able to flip any single bit from the array. In my code I am grouping structs with identical id always within same page to ensure I can flip all old value bits to 0 during the same write and without chasing around the whole region I have reserved for those values.

I've also planned the structs accordingly. The ones being written most are the smallest by design.

Hope you are able to understand what I am trying to explain here.

Just in case someone wants to know. I was able to get around this issue and write the exact size I want to flash by writing a custom function that prepares the data for 256 byte write. I am writing structs to flash by mapping the data manually from structs to flash addresses (I'd rather control the mapping of the struct to flash myself instead of relying on compiler). Things go roughly like this (sorry, my project is not even near the point of making it public but this part seems to work when I tested ~15 minutes rewriting values etc): 1. Find free address (where all data 0xFF) that is big enough 2. create buff[FLASH_PAGE_SIZE] 2.1 write 0xFF everywhere to the buff. Just to ensure there are no unnecessary bit flips by mistake 2.2 if the values being written next are not located at the beginning of the page, read existing values from the flash and copy to the start of the buff 2.3 if I am updating something that has old entries, flip single bit from the previous valid struct from 1 -> 0 to make it outdated (and ok to be deleted later on) 2.4 write the new values next after existing data to the buff (2.5 rest of the buff stays 0xFF) 3. write buff to flash 4. (verify) Seems to work pretty well and I am practically able to flip any single bit from the array. In my code I am grouping structs with identical id always within same page to ensure I can flip all old value bits to 0 during the same write and without chasing around the whole region I have reserved for those values. I've also planned the structs accordingly. The ones being written most are the smallest by design. Hope you are able to understand what I am trying to explain here.