General equality of first 8 bytes of XXH128 and XXH3 #1016

New issue

Closed

opened 2025-02-27 17:04:40 -06:00 by tandr · 2 comments

tandr commented

2025-02-27 17:04:40 -06:00

(Migrated from github.com)

Worked out it is not so for empty inputs! XXH3 produces a different value.


$ echo -n "" | xxhsum -H3 --tag | awk '{print $NF, substr($NF,length($NF)-15) }'
2d06800538d394c2 2d06800538d394c2

$ echo -n "" | xxhsum -H128 --tag | awk '{print $NF, substr($NF,length($NF)-15) }'
99aa06d3014798d86001c324468d497f 6001c324468d497f

$ echo -n "a" | xxhsum -H3 --tag | awk '{print $NF, substr($NF,length($NF)-15) }'
e6c632b61e964e1f e6c632b61e964e1f

$ echo -n "a" | xxhsum -H128 --tag | awk '{print $NF, substr($NF,length($NF)-15) }'
a96faf705af16834e6c632b61e964e1f e6c632b61e964e1f

Since "this is what it is, cannot change it by now" state of affairs (maybe mention it in man page and documentation?), I have a naive ELI5 question

How does it work that different starting value starts to produce exactly the same value starting first byte, always and forever?

Thank you!

Hello. I am heavily utilizing XXH3 (because it is the fastest), and at some point wanted to see if there are any collisions for what I am doing (spoiler - no collisions). So I calculated XXH128 on pretty much everything, generally remembered that first 8 bytes should be equivalent to XXH3 and I could just take if from XXH128 when I need it. Worked out it is not so for empty inputs! XXH3 produces a different value. ``` $ echo -n "" | xxhsum -H3 --tag | awk '{print $NF, substr($NF,length($NF)-15) }' 2d06800538d394c2 2d06800538d394c2 $ echo -n "" | xxhsum -H128 --tag | awk '{print $NF, substr($NF,length($NF)-15) }' 99aa06d3014798d86001c324468d497f 6001c324468d497f $ echo -n "a" | xxhsum -H3 --tag | awk '{print $NF, substr($NF,length($NF)-15) }' e6c632b61e964e1f e6c632b61e964e1f $ echo -n "a" | xxhsum -H128 --tag | awk '{print $NF, substr($NF,length($NF)-15) }' a96faf705af16834e6c632b61e964e1f e6c632b61e964e1f ``` Since "this is what it is, cannot change it by now" state of affairs (maybe mention it in man page and documentation?), I have a naive ELI5 question How does it work that different starting value starts to produce exactly the same value starting first byte, always and forever? Thank you!

👍 2

Cyan4973 commented

2025-03-03 15:27:47 -06:00

(Migrated from github.com)

Guaranteeing the equality of the first 64-bit of xxh128 with xxh3 wasn't part of the initial objectives.

It ended up happening as a consequence of sharing much of the same code, and partly because I started to notice that some users had this expectation near the end of the development. But unfortunately, it was too close to the finish line to refactor the entire project around this new property. So it ended up being a "best effort".

From what I remember, the equality happens for "large" data only.
Note that, in the context of xxhsum, "large" means > 240 bytes (XXH3_MIDSIZE_MAX).
So indeed, when checksumming files, it is likely that most files will belong to this category.

For smaller data sets, the hash values of xxh128 and xxh3 typically differ, with a notable exception for segments of length 1-3. This inequality extends beyond empty input; for example:

echo -n "abcd" | xxhsum -H3 --tag | awk '{print $NF, substr($NF,length($NF)-15) }'
# Output: 6497a96f53a89890 6497a96f53a89890

echo -n "abcd" | xxhsum -H128 --tag | awk '{print $NF, substr($NF,length($NF)-15) }'
# Output: 8d6b60383dfa90c21be79eecd1b1353d 1be79eecd1b1353d

Your note makes me appreciate that, for users manipulating mostly large data, seeing that this property is "generally" respected ends up implying that it's likely always true. Which it is not, but then maybe it should be better documented.

Guaranteeing the equality of the first 64-bit of `xxh128` with `xxh3` wasn't part of the initial objectives. It ended up happening as a consequence of sharing much of the same code, and partly because I started to notice that some users had this expectation near the end of the development. But unfortunately, it was too close to the finish line to refactor the entire project around this new property. So it ended up being a "best effort". From what I remember, the equality happens for "large" data only. Note that, in the context of `xxhsum`, "large" means > 240 bytes (`XXH3_MIDSIZE_MAX`). So indeed, when checksumming files, it is likely that most files will belong to this category. For smaller data sets, the hash values of `xxh128` and `xxh3` typically differ, with a notable exception for segments of length 1-3. This inequality extends beyond empty input; for example: ```bash echo -n "abcd" | xxhsum -H3 --tag | awk '{print $NF, substr($NF,length($NF)-15) }' # Output: 6497a96f53a89890 6497a96f53a89890 echo -n "abcd" | xxhsum -H128 --tag | awk '{print $NF, substr($NF,length($NF)-15) }' # Output: 8d6b60383dfa90c21be79eecd1b1353d 1be79eecd1b1353d ``` Your note makes me appreciate that, for users manipulating mostly large data, seeing that this property is "generally" respected ends up implying that it's likely always true. Which it is not, but then maybe it should be better documented.

👍 2

tandr commented

2025-03-03 17:34:41 -06:00

(Migrated from github.com)

Thank you kindly for the explanation. I guess I have to redo all the files now with sizes less then half a kilobyte, and use pure xxh3 next time.

No milestone

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

gh-Cyan4973/xxHash#1016

No description provided.

Rows
Columns