Created attachment 15898 [details] zstd support zstd compression was announced as "good compression with high throughput" so I gave it a try. With zlib, on high speed links the CPU is usually the bottle neck. With zstd I'm able to fill a 200Mbit link :) zstd detection happens automatically via pkg-config. No zstd header means no error about missing zstd. So that should be okay. However, pkg-config is now kind of required… I duplicated the zlib code and replaced it with zstd hooks once I understood what was going on. I made a few local tests with and without tokens and it seems to work. The compression can be selected with `-Z' option. By default `0' is used as the compression level which is a special default (it currently maps to 3). The compression level can be specified by the same option as for zlib. The compressor feeds data into zstd and starts sending data once the outgoing buffer is full or when a flush is requested. That flush will close the current compression block and create a new one for the following transfer (saving the internal compression / history state). The decompressor allocates space for two blocks. Should one block contain more data, then it will loop more often.
Thanks for the patch! I have transformed it into an official "zstd.diff" file in the rsync-patches git repo using the new negotiated compression code. I imagine it will be included in master soon.
I'll note that I had to install an upgraded zstd lib to get this to work. I first tried 1.3.8 (since it was mentioned in the configure check) but it didn't have the right exported functions. Then then tried 1.4.5, which worked fine.
Actually, I decided it would be better to just go ahead and add this and the lz4 code into master and put them lower in the negotiation list, which will make them easier to test (and they're easy to disable via configure, as needed).
Thanks for the merge! Sorry for the version. According zstd's git v1.3.8 contains ZSTD_compressStream2(). Debian Buster ships v1.3.8 and it does not find the function. Buh. Looking closer it is marked as experimental and only avaible for static linking. As per header file history, since v1.4.0 it is available for linking. I'm going to send a pull for move the function for ZSTD_compressStream2() which hopefully should cover it. Regarding the compression preferences. If I understand it correctly: - "zlibx" is the external zlib. - "zlib" is the internal zlib which feeds blocks into the the zlib history/dictonary without sending it over the wire. Could those two be swapped? That zlibX sounds like zlib-eXtended. Or does it stand for zlib-eXternal? Can we move zlib* at the end of the supported algorihms in terms of preferences? My motivation was the lower for zstd was to use decent compression with _low_ CPU usage. The rsync.yo file says that this "--compress-choice=zlibx" can be used since version 3.1.1. Is this working for negotiation and would support say "zstd,lz4,zlib,zlibx" with zlibx with 3.1.1 and zstd on master or did misunderstand it? Sebastian
Wondering about git 1.3.8, which is many years old. (Git platform maintainer here)
The name "zlibx" is both for external and excluding of unsent data. It's listed first because it should be the safer choice with the weirdness in external zlib going on. The new compress items are listed afterwards because they're untested. As for what you would like the order to be, please refer to the manpage, especially the RSYNC_COMPRESS_LIST environment variable. Feel free to set it to whatever order you like.