Hi, I made some performance tests and ses that rsync can be a bottleneck while it is calculating hashes of basis files (not a surprise). For example, while I can read a big test file directly from the storage at 300 MB/s, rsync checksums it at about 200 MB/s. Thus impact of calculating hashes for big files can be quite important (some minutes for files of several hundreds of GB). Then, what about MD5 implementation based on SSE ? Do you think we would have a performance improvement ? Thank you ! Ben
First info / first patch there : https://lists.samba.org/archive/rsync/2020-May/032175.html
No, that patch has nothing to do with MD5. It is optimizing the rolling checksum.
Yes, and as discussed there also, sounds like xxhash could be a better solution than SSE / hardware backed MD5. Feel free to close this then if not so relevant.
Jorrit Jongma has supplied a nice patch that provides some assembler optimizations for MD5 on x86_64 which I have just merged (along with his optimizations for the rolling checksum algorithm). Also, my xxhash changes have finally landed, and it includes a way for rsync to negotiate the best checksum algorithm shared between the client & server. This will make it easier to add new checksum algorithms in the future.
Really nice additions, it looks promising, thank you very much Jorrit & Wayne !
Samba 4.11 moved to GnuTLS for our MD5 and other hash operations, and so uses any hardware optimisation available there.