xxHash: xxh3? 64 or 128 bit?

[ThomasM]

Hi all,

I just stumbled over the GitHub-page from Cyan4973, developer of xxHash which is implemented in FFAStrans. I am wondering if there are any issues using xxHash in Version "xxh3" as described on https://github.com/Cyan4973/xxHash/releases/tag/v0.8.0 .

it says:

From now on, output values produced by XXH3 for a given input and parameter set will remain identical across systems and across future versions. It makes it possible to store these values for later comparison, or to exchange them across network connections

What does this mean? are the Hashes generated so far not compatible or comparable over Time and systems? As of now I only to xxHash from single-Machines which are comparing copied files from one machine to a NAS. this works, but will a different machine using xxHash in the future come to the same result? The xxHashes are written to a .JSON-log and so they are usable for future reference.

Furthermore, is xxh3 compatible to the FFAStrans-builtIn-xxHash?

Thanks in advance for clearing things up,
regards
tom

[emcodem]

Hey Thomas,

i fear we will need to ask Cyan4973 as i was not able to quickly get out what he is referring to. My guess would be that he means that after updating the xxxhash version that we are using, it might occur that "old, stored values, e.g. in an archiving system" are not compareable anymore to the ones that the new version will produce. (But as i said, this is only guessing).
Do you have a Github account to ask her/him for clarification or shall i do it?

[ThomasM]

Hey emcodem,

thanks for your answer! yes - it is quite unclear. In the CMD-run of xxhsum.exe -h I found:

-H# algorithm selection: 0,1,2 or 32,64,128 (default: 1)

I let it run through the different algorithms:

c:\Tom\FFAStrans>xxhsum -H0 Webinterface.zip
aa39ea3b Webinterface.zip

c:\Tom\FFAStrans>xxhsum -H1 Webinterface.zip
ec7e035a6cbfdec8 Webinterface.zip

c:\Tom\FFAStrans>xxhsum -H2 Webinterface.zip
86b18e359256ed6b417fd11edcfdf460 Webinterface.zip

That was to expect. I do not know exactly what is now going on with xxh3.

As I am no coder I have no access to github - so maybe.... you.... ahem...

Thanks a lot,
cheers

tom

[emcodem]

Hey Thomas,
well, -H0, 1 and 2 of course generate a different output for the same file, why do you wonder about this?
The thing that he is referring to is that different release versions of his tool will/can generate a different output using the SAME algorithm, also it looks like they could output different when running the same build version on linux and windows.

Does that maybe already solve your question?

[ThomasM]

Hey emcodem,

Yes, I was sure about the switches, but was wondering about xxh3. Well - I posted another question and maybe now it comes a little closer to the point:

Is the external newer Version xxhsum.exe maybe different from the FFAStrans built in? I tested both, the Built-in and the xxhsum.exe 0.8.x with the same result. Is it possible that this changes sometimes?

I just was a little unsure, because it does not make much sense to hash something and it is not comparable in the future - right...?

But IF it is fact what you are explaining, the xxHash is not good for reference and I should go for ssh or md5 - right?

regards,
tom

[emcodem]

Ah, now i see whats your issue. regarding to the release notes, xxh3 is not yet useable through the cli tool but through API only. Now that i see that, i also recognize what he writes in the release notes only regards to the xxh3 algorithm, not to the existing 32bit/64bit algorithm. So basically he says xxh3 is now production ready. I believe you already understood that part much earlier than me, sorry for the confusion.

Generally i believe you would ease your life a lot using md5 tools but i am really not sure how fast the can get. MD5 is kind of de-facto standard for checking file integrity isnt it?

[ThomasM]

MD5 is kind of de-facto standard for checking file integrity isnt it?

yes, that´s true. I was lucky to see that FFAStrans has built-in Checksum with the very fast xxHash. Maybe there is a chance to get SSH or MD5 integrated?

[emcodem]

Hm lets just summon @admin at this point and ask for his thougts about md5 vs xxhsum:-)

[admin]

Hi ThomasM,

The xxHash was introduced because it seemed to be the fastest around for large media files. Sure, any kind of hash is possible but currently there is no plan to introduce any new, unless there is a great call for it. OR, someone might want to create a custom node to do it

-steinar

[ThomasM]

Hey Steinar,

thank you for this info. As emcodem pointed out, it was a bit confusing with the xxh3 algo - but it seems that it is not already finished. So no need to change...
I twiddled a bit with xxHash on the command-line and my conclusion so far is, that if you stay in one algorithm, it will be the same. My testing was only with the standard 32Bit, 64Bit and 128Bit algos. FFAStrans uses 32Bit. Everything is fine. Works great. And just for really fast FileChecking we do not need slow MD5 or SSH with crypto necesserily.

regards
tom