gh-icsharpcode/AvalonEdit

Watch

Fork 0

New implementation of HeightTree #371

Open

dgrunwald wants to merge 1 commit from height-tree into master

dgrunwald commented

2022-09-04 12:19:34 -05:00

(Migrated from github.com)

Where the old implementation was based on a red-black tree, the new implementation is inspired by B+ trees.
We no longer need a separate memory allocation for every line in the document. This should dramatically reduce the memory consumption: a comment in the old code indicated that each line took 56 bytes of memory.
But that was in 32-bit days; in a 64-bit process the old implementation actually needed 88 bytes per line.

The new implementation should need 168 bytes per leaf node which holds 8-16 lines, i.e. we need 10.5-21 bytes per line for the leaf nodes.
It additionally needs 456 bytes per inner node (for the inner node and its children array). The level just above the leafs holds 64-256 lines per inner node, i.e. we need 1.8-7.2 bytes per line for the second level.
Thus the total memory consumption of the new tree should in the worst case still be below 29.2 bytes per line (168/8 + sum(456/(8**lvl) for lvl in range(2,100))).
However after freshly loading a large document from disk we should be close to the best case which in total is just 12.4 bytes per line (168/16 + sum(456/(16**lvl) for lvl in range(2,100))).

Apart from the reduced memory usage, the behavior of the new tree should be identical to the old behavior.
In particular, all operations (line insertion, line deletion, collapsing, uncollapsing, line<->Y queries) maintain their externally-visible behavior and their O(lg N) run-time.
set_DefaultLineHeight runs in O(N) now (previously could take O(N lg N)).

The HeightTree is the data structure that maps between document line numbers and visual Y positions. It handles both the varying heights of lines (e.g. due to word-wrapping) and the collapsing of lines (e.g. due to folding). Where the old implementation was based on a red-black tree, the new implementation is inspired by B+ trees. We no longer need a separate memory allocation for every line in the document. This should dramatically reduce the memory consumption: a comment in the old code indicated that each line took 56 bytes of memory. But that was in 32-bit days; in a 64-bit process the old implementation actually needed 88 bytes per line. The new implementation should need 168 bytes per leaf node which holds 8-16 lines, i.e. we need 10.5-21 bytes per line for the leaf nodes. It additionally needs 456 bytes per inner node (for the inner node and its children array). The level just above the leafs holds 64-256 lines per inner node, i.e. we need 1.8-7.2 bytes per line for the second level. Thus the total memory consumption of the new tree should in the worst case still be below 29.2 bytes per line (`168/8 + sum(456/(8**lvl) for lvl in range(2,100))`). However after freshly loading a large document from disk we should be close to the best case which in total is just 12.4 bytes per line (`168/16 + sum(456/(16**lvl) for lvl in range(2,100))`). Apart from the reduced memory usage, the behavior of the new tree should be identical to the old behavior. In particular, all operations (line insertion, line deletion, collapsing, uncollapsing, line<->Y queries) maintain their externally-visible behavior and their O(lg N) run-time. set_DefaultLineHeight runs in O(N) now (previously could take O(N lg N)).

sopgenorth commented

2023-01-02 19:45:14 -06:00

(Migrated from github.com)

I saw the comment in mainline about changing the implementation, and was curious what an improvement might look like, particularly from my other interests in trying to add some sort of lazy file loading mechanism (just a dream right now, see: #382 )

On a 750MB test file this change reduces RAM gives a healthy reduction from ~7.5GB to ~5GB.

Nice work!

This pull request has changes conflicting with the target branch.

ICSharpCode.AvalonEdit.Tests/Document/HeightTests.cs
ICSharpCode.AvalonEdit/ICSharpCode.AvalonEdit.csproj
ICSharpCode.AvalonEdit/Rendering/HeightTree.cs
ICSharpCode.AvalonEdit/Rendering/TextView.cs

View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Merge commit title

Merge pull request 'New implementation of HeightTree' (#371) from height-tree into master

Merge commit body

Reviewed-on: https://git.richey.dev/gh-icsharpcode/AvalonEdit/pulls/371

Merge pull request 'New implementation of HeightTree' (#371) from height-tree into master

Reviewed-on: https://git.richey.dev/gh-icsharpcode/AvalonEdit/pulls/371

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin 'height-tree:height-tree'

git switch 'height-tree'

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch 'master'

git merge --no-ff 'height-tree'

git switch 'height-tree'

git rebase 'master'

git switch 'master'

git merge --ff-only 'height-tree'

git switch 'height-tree'

git rebase 'master'

git switch 'master'

git merge --no-ff 'height-tree'

git switch 'master'

git merge --squash 'height-tree'

git switch 'master'

git merge --ff-only 'height-tree'

git switch 'master'

git merge 'height-tree'

git push origin 'master'

No reviewers

No milestone

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

gh-icsharpcode/AvalonEdit!371

No description provided.

Rows
Columns