xxHash: xxh3? 64 or 128 bit?

Questions and answers on how to get the most out of FFAStrans
emcodem
Posts: 1811
Joined: Wed Sep 19, 2018 8:11 am

Re: xxHash: xxh3? 64 or 128 bit?

Post by emcodem »

Hm, what i dont really understand is how the "speed of the algorithm" could be of interest here when it is actually the speed of the file reading which is the bottleneck (at least up to a approx. 2-3 Gbit/s). I guess the problem is more that most developers dont care about the art of reading large files ;-)
Of course a faster algorithm will use less CPU with that "slow" file reading but as long as the file reading was not optimized, the final tools is not guaranteed to be faster than a tool with a slower algorithm but optimized file reading.

Here is an MD5 tool i wrote quickly in python that should be a little faster than the xxh stuff at least at read speeds below ~3 Gbit/s:

https://github.com/emcodem/fast_md5/blo ... st_md5.exe

The difference compared to most other known to me checksum tools is that the file reading and the checksum is done in separate threads, also the file is being read in 8MB chunks instead of 65kB as the xxhsum tool does. So in difference to the xxhsum tool, this md5 tool should run exactly at the same speed than a file copy using windows explorer can be done.

Note that when you benchmark and you check the same file twice in a row, the second time you do it, there will be some caches helping the reading, so you can typically not compare the very first try with the second one (with the same input file from same location). Also the xxh tool will recognize the file was already checked and does not read the file a second time at all.
emcodem, wrapping since 2009 you got the rhyme?
ThomasM
Site Admin
Posts: 231
Joined: Wed Feb 22, 2017 6:36 am

Re: xxHash: xxh3? 64 or 128 bit?

Post by ThomasM »

Ooops..:

googledrive says:
Sie können auf diesen Inhalt nicht zugreifen, da er gegen unsere Nutzungsbedingungen verstößt.
emcodem
Posts: 1811
Joined: Wed Sep 19, 2018 8:11 am

Re: xxHash: xxh3? 64 or 128 bit?

Post by emcodem »

Hmmm interesting, since when cannot we share exe file anymore on google drive...

Well, lets move this to github:

https://github.com/emcodem/fast_md5/blo ... st_md5.exe
emcodem, wrapping since 2009 you got the rhyme?
ThomasM
Site Admin
Posts: 231
Joined: Wed Feb 22, 2017 6:36 am

Re: xxHash: xxh3? 64 or 128 bit?

Post by ThomasM »

...thanks a lot, @emcodem!

cheers
tom
Post Reply