I made some performance tests and ses that rsync can be a bottleneck while it is calculating hashes of basis files (not a surprise).
For example, while I can read a big test file directly from the storage at 300 MB/s, rsync checksums it at about 200 MB/s.
Thus impact of calculating hashes for big files can be quite important (some minutes for files of several hundreds of GB).
Then, what about MD5 implementation based on SSE ?
Do you think we would have a performance improvement ?
Thank you !
First info / first patch there :
No, that patch has nothing to do with MD5. It is optimizing the rolling checksum.
Yes, and as discussed there also, sounds like xxhash could be a better solution than SSE / hardware backed MD5.
Feel free to close this then if not so relevant.
Jorrit Jongma has supplied a nice patch that provides some assembler optimizations for MD5 on x86_64 which I have just merged (along with his optimizations for the rolling checksum algorithm). Also, my xxhash changes have finally landed, and it includes a way for rsync to negotiate the best checksum algorithm shared between the client & server. This will make it easier to add new checksum algorithms in the future.
Really nice additions, it looks promising, thank you very much Jorrit & Wayne !