Change file reading from byte arrays to streams #11

Open

Nielk1 wants to merge 2 commits from Nielk1/enhancement-read_file_as_stream into main

Nielk1 commented

2026-07-06 08:26:33 -05:00

(Migrated from github.com)

This change is a big one.

The current logic loads an entire file into memory before processing it. On it's own this is fine for a simple tool, but the problem comes with files larger than Int32.MaxValue, due to how C# handles arrays. I ran into files with this problem.

The new logic uses streams whenever possible. It makes some pretty large changes to the data block traversal so this will need review.

Tests have also been altered to use the new stream based read after the old but renamed byte array based read. The tests use a new construct, a ValidationWriteStream, which is a crude fake stream formed around a byte array that will Assert if the bytes received do not match the underlying array, overflow the length of the underlying array, or if the AssertComplete function is called do not visit all array indexes.

These changes were made for my own purposes where I needed to extract large files from a UFS1 image but I hope they help here too.

Note that Dokany logic was not tested, so be sure to review that. It was adjusted to use streams and offsets so requests to read parts of files wouldn't need to load the entire file into memory first. Writing is still restricted to the old ways so this only fixes (but untested) reading large files.

This change is a big one. The current logic loads an entire file into memory before processing it. On it's own this is fine for a simple tool, but the problem comes with files larger than Int32.MaxValue, due to how C# handles arrays. I ran into files with this problem. The new logic uses streams whenever possible. It makes some pretty large changes to the data block traversal so this will need review. Tests have also been altered to use the new stream based read after the old but renamed byte array based read. The tests use a new construct, a ValidationWriteStream, which is a crude fake stream formed around a byte array that will Assert if the bytes received do not match the underlying array, overflow the length of the underlying array, or if the `AssertComplete` function is called do not visit all array indexes. These changes were made for my own purposes where I needed to extract large files from a UFS1 image but I hope they help here too. Note that Dokany logic was not tested, so be sure to review that. It was adjusted to use streams and offsets so requests to read parts of files wouldn't need to load the entire file into memory first. Writing is still restricted to the old ways so this only fixes (but untested) reading large files.

This pull request is broken due to missing fork information.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin 'Nielk1/enhancement-read_file_as_stream:Nielk1/enhancement-read_file_as_stream'

git switch 'Nielk1/enhancement-read_file_as_stream'

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch 'main'

git merge --no-ff 'Nielk1/enhancement-read_file_as_stream'

git switch 'Nielk1/enhancement-read_file_as_stream'

git rebase 'main'

git switch 'main'

git merge --ff-only 'Nielk1/enhancement-read_file_as_stream'

git switch 'Nielk1/enhancement-read_file_as_stream'

git rebase 'main'

git switch 'main'

git merge --no-ff 'Nielk1/enhancement-read_file_as_stream'

git switch 'main'

git merge --squash 'Nielk1/enhancement-read_file_as_stream'