Bug 6916 - Avoid bundling a modified zlib
Summary: Avoid bundling a modified zlib
Status: RESOLVED FIXED
Alias: None
Product: rsync
Classification: Unclassified
Component: core (show other bugs)
Version: 3.1.0
Hardware: All All
: P3 enhancement (vote)
Target Milestone: ---
Assignee: Wayne Davison
QA Contact: Rsync QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-11-21 01:45 UTC by Matt McCutchen
Modified: 2013-01-15 08:07 UTC (History)
4 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Matt McCutchen 2009-11-21 01:45:52 UTC
Fedora wants us not to bundle a modified version of zlib because it makes it harder to ensure that security updates are applied everywhere they are needed.  See:

https://bugzilla.redhat.com/show_bug.cgi?id=495310

Wayne, I don't know what (if anything) you want to do about this, but I didn't want the issue to be forgotten.
Comment 1 Wayne Davison 2009-11-25 19:54:23 UTC
The only thing for rsync to potentially do is to have a build option for using a compatible, shared zlib.  The redhat bug has links to rsync's needed changes.  If fedora patched their version of zlib with the Z_INSERT_ONLY support, it should (presumably) be usable by normal zlib apps, and rsync as well.
Comment 2 Matt McCutchen 2009-11-27 23:57:49 UTC
(In reply to comment #1)
> The only thing for rsync to potentially do is to have a build option for using
> a compatible, shared zlib.

What happened to, "Tridge has mentioned to me that he thinks the slight
savings in compression wasn't really worth all the hassle, so some
future version of rsync may get rid of this optimization and start
to use the system's zlib"?

http://lists.samba.org/archive/rsync/2005-December/014180.html

I'm not saying that is definitely the right decision, just that it deserves fair consideration.

> If fedora patched their version of zlib with the Z_INSERT_ONLY support, it
> should (presumably) be usable by normal zlib apps, and rsync as well.

The Fedora folks don't wish to be maintain the patched zlib themselves:

https://bugzilla.redhat.com/show_bug.cgi?id=495310#c7
Comment 3 Jan Zeleny 2009-12-10 05:49:35 UTC
Hi, I'm currently the maintainer of rsync in Fedora. As I was informed, zlib upstream is willing to make some changes, so rsync doesn't need its bundled version. But some discussion about this should take place first. To quote Mark Adler, zlib's developer: "I would change how to decide when to throw away the history so that it would not happen so often as the rsync patch would do it, but would retain the desired rsyncability of the compressed streams.  If I can constrain the impact to the compression efficiency, I might even make it the default behavior."
Comment 4 Matt McCutchen 2009-12-10 10:19:11 UTC
The gzip enhancement bundled with rsync is Z_INSERT_ONLY, which lets rsync enter matched blocks in the gzip history without transmitting them in the compressed stream.  The quote in comment #3 seems to be about something completely different: the --rsyncable option that makes compressed files more amenable to delta-transfer.
Comment 5 Jan Zeleny 2009-12-17 02:23:31 UTC
Well, it's possible. I'm not rsync maintainer for long, so I don't have that much knowledge about the situation. But my point was that zlib's upstream is willing to consider some changes in order to reach compromise. Could you please contact them and try to discuss the situation, so it isn't necessary to bundle modified zlib? Thanks
Comment 6 Matt McCutchen 2010-02-07 19:32:18 UTC
Some technical details: Z_INSERT_ONLY is used on the sender side to enter matched blocks in the gzip history without transmitting them in the compressed stream.  The receiver-side counterpart is the code in see_deflate_token in token.c which inserts into the input stream a synthesized zlib-format block containing the matched data.

The receiver-side behavior was possible to implement based on knowledge of the zlib format without actually modifying zlib.  Implementing the sender-side behavior in the same way would require excising the matched data from zlib's output and flushing in order to make the excision possible.  I guess that was determined to be too hard, though it's not clear to me what is so hard about it.

Wayne said previously that the receiver-side implementation is a hack and should be moved into zlib if Z_INSERT_ONLY is upstreamed:

http://lists.samba.org/archive/rsync/2005-December/014180.html

I agree.  But note that after doing that, we would still have a dependency on the zlib format in send_deflated_token, which fiddles with the "0, 0, ff, ff" bytes generated by flushing.
Comment 7 Wayne Davison 2011-12-24 21:00:51 UTC
3.1.0 has a configure option --with-included-popt=no that can be used to get rsync to use the system's popt library.  I changed the rsync code to be able to support this, and in my limited testing it appears to work fine.

There is going to be some new support for adding dictionary data during the transfer in the official zlib code, but I haven't yet tested to see if using this new support is identical to the rsync kluges or not.  If it is, the rsync code will eventually be updated to use the official support.

As for the "0, 0, ff, ff" bytes, Mark Adler mentions that these are documented, so it should be fine to tweak things based on that.
Comment 8 Matt McCutchen 2011-12-26 01:43:02 UTC
(In reply to comment #7)
> 3.1.0 has a configure option --with-included-popt=no that can be used to get
> rsync to use the system's popt library.

That should say "zlib", not "popt".