WIP: add a 128bit function XXH64X2 based on XXH64

gzm55 commented

2021-03-24 23:41:59 -05:00

(Migrated from github.com)

XXH64X2 is modified directly from XXH64, and the low 64bits are the same as XXH64. The purpose is that when porting to an environment without SIMD vectoring, XXH64 is is still faster than XXH3 and XXH128. The new function can cover the 128b hash usage on these environments.

The new hash has passed all the smhasher tests and the collision ratio tests as 128b hash. The performance is about 85% of the XXH64. While porting to the java, the performance is 95% of the XXH64.

`XXH64X2` is modified directly from `XXH64`, and the low 64bits are the same as `XXH64`. The purpose is that when porting to an environment without SIMD vectoring, `XXH64` is is still faster than `XXH3` and `XXH128`. The new function can cover the 128b hash usage on these environments. The new hash has passed all the smhasher tests and the collision ratio tests as 128b hash. The performance is about 85% of the `XXH64`. While porting to the java, the performance is 95% of the `XXH64`.

👍 2

Cyan4973 commented

2021-03-25 17:27:12 -05:00

(Migrated from github.com)

Thanks for the suggestion @gzm55 .

A question on the design goal :
is it supposed to be a 128-bit hash,
or a 64-bit hash which outputs a 128-bit return value ?

Thanks for the suggestion @gzm55 . A question on the design goal : is it supposed to be a 128-bit hash, or a 64-bit hash which outputs a 128-bit return value ?

gzm55 commented

2021-03-25 19:06:17 -05:00

(Migrated from github.com)

it is a 128-bit hash.

gzm55 commented

2021-03-25 21:11:03 -05:00

(Migrated from github.com)

When cannot take the benefit of the SIMD instructions, a 128-bit hash is very rare with both high quality and high speed. The XXH64 use four 64-bit integers internal state for the main 32-byte strip loop, so it could be a good candidate to output 128-bit hash. Comparing to hash twice with two different XXH64 seeds (XXH64 + XXH64), the new algorithm reuse the most avalanche operations and got a better speed. The name XXH64X2 expresses two meaning: a) it is based on XXH64, 2) it generate 128 bit output (64 x 2)

When cannot take the benefit of the SIMD instructions, a 128-bit hash is very rare with both high quality and high speed. The `XXH64` use four 64-bit integers internal state for the main 32-byte strip loop, so it could be a good candidate to output 128-bit hash. Comparing to hash twice with two different XXH64 seeds (`XXH64 + XXH64`), the new algorithm reuse the most avalanche operations and got a better speed. The name `XXH64X2` expresses two meaning: a) it is based on `XXH64`, 2) it generate 128 bit output (64 x 2)

👍 2

easyaspi314 commented

2021-03-26 11:10:10 -05:00

(Migrated from github.com)

I think this is rather redundant. We already have XXH3 which is much faster.

Cyan4973 commented

2021-03-26 11:51:51 -05:00

(Migrated from github.com)

Well, the suggested XXH64x2 uses a design that has been envisioned as a possible backup strategy for XXH128. If XXH3 had not worked as intended, I probably would have considered something similar, aka an XXH64 variant.
There are a few things that made me consider it a "second" choice rather than a first, and so it's interesting to ask a few questions about that.

Performance : Not all cpus have strong 64-bit multipliers. Modern Intel (and AMD) cpus do, and even some high end ARM ones. But most others have much weaker performance, because it costs a lot of gates. Using 64-bit multiplications on such cpus leads to sub-par performance (see for a recent example : https://github.com/Cyan4973/xxHash/pull/519#issuecomment-807375099). Yet, strong cpus also tend to feature a vector unit. In other words, cpus without a vector unit tend to have a poor 64-bit multiplier, making XXH64 also a poor fit.
So, the question is :
- in which environment are you unable to take advantage of a vector unit, yet can benefit from a good 64-bit multiplier ?
The core loop seems relatively unmodified (I guess it must be, in order to ensure that the low 64-bit remain identical to regular XXH64). That means that, although internal states is 256-bit, it's still composed of independent 64-bit lanes. Therefore, the collision ratio might be closer to 64-bit rather than 128-bit.
- have you measured it ?
- have you taken some precautions to ensure 128-bit of entropy ?
smhasher was really meant to measure entropy of 32-bit hashes. It's just unable to provide accurate measurements for 64-bit hashes, let alone 128-bit hashes.
- An incomplete yet useful work-around is to cut the output into 32-bit slices, and test smhasher on each one of them.
- It seems this could be useful for the high part of xxh64x2, since the low one is equivalent to regular xxh64 which was already validated using this method.

Well, the suggested `XXH64x2` uses a design that has been envisioned as a possible backup strategy for `XXH128`. If `XXH3` had not worked as intended, I probably would have considered something similar, aka an `XXH64` variant. There are a few things that made me consider it a "second" choice rather than a first, and so it's interesting to ask a few questions about that. - Performance : Not all cpus have strong 64-bit multipliers. Modern Intel (and AMD) cpus do, and even some high end ARM ones. But most others have much weaker performance, because it costs a lot of gates. Using 64-bit multiplications on such cpus leads to sub-par performance (see for a recent example : https://github.com/Cyan4973/xxHash/pull/519#issuecomment-807375099). Yet, strong cpus also tend to feature a vector unit. In other words, cpus without a vector unit tend to have a poor 64-bit multiplier, making `XXH64` also a poor fit. So, the question is : + in which environment are you unable to take advantage of a vector unit, yet can benefit from a good 64-bit multiplier ? - The core loop seems relatively unmodified (I guess it must be, in order to ensure that the low 64-bit remain identical to regular `XXH64`). That means that, although internal states is 256-bit, it's still composed of independent 64-bit lanes. Therefore, the collision ratio might be closer to 64-bit rather than 128-bit. + have you measured it ? + have you taken some precautions to ensure 128-bit of entropy ? - `smhasher` was really meant to measure entropy of 32-bit hashes. It's just unable to provide accurate measurements for 64-bit hashes, let alone 128-bit hashes. + An incomplete yet useful work-around is to cut the output into 32-bit slices, and test `smhasher` on each one of them. + It seems this could be useful for the high part of `xxh64x2`, since the low one is equivalent to regular `xxh64` which was already validated using this method.

👍 2

easyaspi314 commented

2021-03-26 14:31:34 -05:00

(Migrated from github.com)

In other words, cpus without a vector unit tend to have a poor 64-bit multiplier, making XXH64 also a poor fit.

Indeed. That is my main concern. ARM64 and x86_64 machine both require NEON and SSE2, respectively, where the XXH3 implementation is significantly faster than XXH64.

Additionally, XXH64 has some known flaws in its core loop.

It feels like developing a new sword when lightsabers already exist.

> In other words, cpus without a vector unit tend to have a poor 64-bit multiplier, making XXH64 also a poor fit. Indeed. That is my main concern. ARM64 and x86_64 machine both require NEON and SSE2, respectively, where the XXH3 implementation is _significantly_ faster than XXH64. Additionally, XXH64 has some known flaws in its core loop. It feels like developing a new sword when lightsabers already exist.

Cyan4973 commented

2021-03-26 15:00:30 -05:00

(Migrated from github.com)

There might be a valid reason.
Let just @gzm55 answer.

There _might_ be a valid reason. Let just @gzm55 answer.

gzm55 commented

2021-03-28 07:37:00 -05:00

(Migrated from github.com)

in which environment are you unable to take advantage of a vector unit, yet can benefit from a good 64-bit multiplier ?

The JVM on x64 is this kind of environment. After trying to port the xxh3/128 to native java code, the performance is 30%~40% of xxhash64. Even jni or new vector api of jdk16 still take much overhead and cannot keep up the the xxhash64.

have you measured it ?

Yes, all the collisions tests on the xxh128 from the wiki are passed (zero collisions). And on the high 64 bit part, 294 collisions are found within 100G inputs with length of 256byte.

have you taken some precautions to ensure 128-bit of entropy ?

xxh64x2 borrow some of methods from xxh128 for generating 128 bit results from internal state. And two independent hashes with 64-bit of entropy can be combined as a 128-bit hash with full width entropy.

It seems this could be useful for the high part of xxh64x2

For high 64 bits part, the smhasher tests and collision tests are both passed.
For two 32bits parts (65-96 and 97-128) pass all the smhashers tests except LongNeighbors giving some warnings:

Among 1455 tested bases, we expected 323.171 bads. Actually there were 307. The most striking collision has surprise score 110.765 and length 72 bytes
Among 1455 tested bases, we expected 323.171 bads. Actually there were 335. The most striking collision has surprise score 100.045 and lengths 25 and 17 bytes

> in which environment are you unable to take advantage of a vector unit, yet can benefit from a good 64-bit multiplier ? The JVM on x64 is this kind of environment. After trying to port the xxh3/128 to native java code, the performance is 30%~40% of xxhash64. Even jni or new vector api of jdk16 still take much overhead and cannot keep up the the xxhash64. > have you measured it ? Yes, all the collisions tests on the xxh128 from the wiki are passed (zero collisions). And on the high 64 bit part, 294 collisions are found within 100G inputs with length of 256byte. > have you taken some precautions to ensure 128-bit of entropy ? `xxh64x2` borrow some of methods from `xxh128` for generating 128 bit results from internal state. And two independent hashes with 64-bit of entropy can be combined as a 128-bit hash with full width entropy. > It seems this could be useful for the high part of xxh64x2 For high 64 bits part, the `smhasher` tests and collision tests are both passed. For two 32bits parts (65-96 and 97-128) pass all the smhashers tests except LongNeighbors giving some warnings: - Among 1455 tested bases, we expected 323.171 bads. Actually there were 307. The most striking collision has surprise score 110.765 and length 72 bytes - Among 1455 tested bases, we expected 323.171 bads. Actually there were 335. The most striking collision has surprise score 100.045 and lengths 25 and 17 bytes

easyaspi314 commented

2021-03-28 13:34:49 -05:00

(Migrated from github.com)

The JVM on x64 is this kind of environment.

Good point, I wasn't even considering virtual machines.

After trying to port the xxh3/128 to native java code, the performance is 30%~40% of xxhash64.

It is true that scalar XXH3 is slower than XXH64. However, that seems to be a bit much.

Even jni or new vector api of jdk16 still take much overhead and cannot keep up the the xxhash64.

What about a hybrid JNI/Java version? The overhead of JNI is pretty nasty and the short keys are all available in Java intrinsics, so it is only worth it on long keys.

Looking at the new vector API, I doubt XXH3 will work well with it, as it seems rather half-baked. It lacks shifting [wtf?!] and long multiplication.

I would like to see those implementations, though. Perhaps they can be improved.

> The JVM on x64 is this kind of environment. Good point, I wasn't even considering virtual machines. > After trying to port the xxh3/128 to native java code, the performance is 30%~40% of xxhash64. It is true that *scalar* XXH3 is slower than XXH64. However, that seems to be a bit much. > Even jni or new vector api of jdk16 still take much overhead and cannot keep up the the xxhash64. What about a hybrid JNI/Java version? The overhead of JNI is pretty nasty and the short keys are all available in Java intrinsics, so it is only worth it on long keys. Looking at the new vector API, I doubt XXH3 will work well with it, as it seems rather half-baked. It lacks shifting [wtf?!] and long multiplication. I would like to see those implementations, though. Perhaps they can be improved.

gzm55 commented

2021-03-28 21:28:11 -05:00

(Migrated from github.com)

What about a hybrid JNI/Java version? The overhead of JNI is pretty nasty and the short keys are all available in Java intrinsics, so it is only worth it on long keys.

yes, this is a major candidate method, and the split length becomes another tuning point.

Looking at the new vector API, I doubt XXH3 will work well with it, as it seems rather half-baked. It lacks shifting [wtf?!] and long multiplication.

It is still under incubator, and cannot 1-to-1 map the api to instructions of X64 arch. This set of api would not graduates in LTS JDK17, so even the api can boost the jdk to control all instructions, the industry (big data, spring, etc.) may need at least a decade to accept and deploy the new LTS JDK with vector api.

NB: OpenHFT/Zero-Allocation-Hashing#58 is the draft of pure java porting of xxh3.

Anyway, imo, XXH64X2, as a fast scalar version of 128-bit hash, could have its position, although it depends on 64bit multiplications.

> What about a hybrid JNI/Java version? The overhead of JNI is pretty nasty and the short keys are all available in Java intrinsics, so it is only worth it on long keys. yes, this is a major candidate method, and the split length becomes another tuning point. > Looking at the new vector API, I doubt XXH3 will work well with it, as it seems rather half-baked. It lacks shifting [wtf?!] and long multiplication. It is still under incubator, and cannot 1-to-1 map the api to instructions of X64 arch. This set of api would not graduates in LTS JDK17, so even the api can boost the jdk to control all instructions, the industry (big data, spring, etc.) may need at least a decade to accept and deploy the new LTS JDK with vector api. NB: OpenHFT/Zero-Allocation-Hashing#58 is the draft of pure java porting of xxh3. Anyway, imo, `XXH64X2`, as a fast scalar version of 128-bit hash, could have its position, although it depends on 64bit multiplications.

👍 1

Cyan4973 commented

2021-06-27 02:23:58 -05:00

(Migrated from github.com)

I spent some time testing the proposed variant xxh64x2.
In a test generating 100 Billion hashes, I got 1 collision.
This may not sound like much, but such an occurrence is supposed to have an unbelievably small probably, at a 1 / 59029581036420320 rate, so it doesn't feel like "just bad luck".

While it's fairly difficult to pinpoint exactly the limit,
it seems to me that this 128-bit hash doesn't provide 128-bit of entropy.
It's still larger than 64-bit, but rather around ~80-bit,
Unfortunately, I can't access a platform powerful enough to more accurately estimate the entropy.

Depending on the target application, it may still be good enough.
But better be aware of it.

I spent some time testing the proposed variant `xxh64x2`. In a test generating 100 Billion hashes, I got 1 collision. This may not sound like much, but such an occurrence is supposed to have an unbelievably small probably, at a 1 / 59029581036420320 rate, so it doesn't feel like "just bad luck". While it's fairly difficult to pinpoint exactly the limit, it seems to me that this 128-bit hash doesn't provide 128-bit of entropy. It's still larger than 64-bit, but rather around ~80-bit, Unfortunately, I can't access a platform powerful enough to more accurately estimate the entropy. Depending on the target application, it may still be good enough. But better be aware of it.

👍 1

gzm55 commented

2021-07-08 09:50:01 -05:00

(Migrated from github.com)

I spent some time testing the proposed variant xxh64x2.
In a test generating 100 Billion hashes, I got 1 collision.
This may not sound like much, but such an occurrence is supposed to have an unbelievably small probably, at a 1 / 59029581036420320 rate, so it doesn't sound like "bad luck".

While it's fairly difficult to pinpoint exactly the limit,
it seems to me that this 128-bit hash doesn't provide 128-bit of entropy.
It's still larger than 64-bit, but rather around ~80-bit,
Unfortunately, I can't access a platform powerful enough to more accurately estimate the entropy.

Depending on the target application, it may still be good enough.
But better be aware of it.

@Cyan4973 does the xxh128 pass with similar test and got 0 collision? and I also wonder for larger hash algos (256+ bits), how to prove the entropy bit.

> I spent some time testing the proposed variant `xxh64x2`. > In a test generating 100 Billion hashes, I got 1 collision. > This may not sound like much, but such an occurrence is supposed to have an unbelievably small probably, at a 1 / 59029581036420320 rate, so it doesn't sound like "bad luck". > > While it's fairly difficult to pinpoint exactly the limit, > it seems to me that this 128-bit hash doesn't provide 128-bit of entropy. > It's still larger than 64-bit, but rather around ~80-bit, > Unfortunately, I can't access a platform powerful enough to more accurately estimate the entropy. > > Depending on the target application, it may still be good enough. > But better be aware of it. @Cyan4973 does the `xxh128` pass with similar test and got 0 collision? and I also wonder for larger hash algos (256+ bits), how to prove the entropy bit.

Cyan4973 commented

2021-07-08 10:13:17 -05:00

(Migrated from github.com)

does the xxh128 pass with similar test and got 0 collision?

Yes it does.

Arguably, it's not a proof that xxh128 has achieved real 128-bit entropy. It's more a "minimum" to achieve.

In the future, systems with more RAM would make it possible to increase the measurement accuracy. I doubt we'll ever have enough to reach 128-bit, but given that the hash doesn't target cryptographic quality, that's not necessarily a "big problem". Even ~100-bit entropy is plenty-enough for most applications out there.

> does the xxh128 pass with similar test and got 0 collision? Yes it does. Arguably, it's not a proof that `xxh128` has achieved real 128-bit entropy. It's more a "minimum" to achieve. In the future, systems with more RAM would make it possible to increase the measurement accuracy. I doubt we'll ever have enough to reach 128-bit, but given that the hash doesn't target cryptographic quality, that's not necessarily a "big problem". Even ~100-bit entropy is plenty-enough for most applications out there.

gzm55 commented

2021-07-14 12:21:40 -05:00

(Migrated from github.com)

@Cyan4973 the collision test hash samples with a fixed length. It could be better to sample strings over a range of length, which is more like the real use cases and avoid the overfit of segment hash algorithm.

@Cyan4973 [the collision test](https://github.com/Cyan4973/xxHash/tree/dev/tests/collisions) hash samples with a fixed length. It could be better to sample strings over a range of length, which is more like the real use cases and avoid the overfit of segment hash algorithm.

easyaspi314 commented

2021-07-14 17:14:25 -05:00

(Migrated from github.com)

I am guessing that this collision is related to #54.

👍 1

Cyan4973 commented

2021-08-09 16:23:52 -05:00

(Migrated from github.com)

Closing. I applaud the effort, but it's difficult to introduce this variant within xxHash library without also introducing confusion for library users. From an ecosystem perspective, I would prefer to spend the effort ensuring that XXH128 is fast enough for your needs, though I understand that specific setups, such as JVM on x64, may play against its natural strengths.

Closing. I applaud the effort, but it's difficult to introduce this variant within `xxHash` library without also introducing confusion for library users. From an ecosystem perspective, I would prefer to spend the effort ensuring that `XXH128` is fast enough for your needs, though I understand that specific setups, such as JVM on x64, may play against its natural strengths.

Pull request closed

This pull request cannot be reopened because the branch was deleted.

Rows
Columns

WIP: add a 128bit function XXH64X2 based on XXH64 #518

Pull request closed