Bug 9864 - Allow permanent compression of destination files
Summary: Allow permanent compression of destination files
Alias: None
Product: rsync
Classification: Unclassified
Component: core (show other bugs)
Version: 3.1.0
Hardware: All All
: P5 enhancement (vote)
Target Milestone: ---
Assignee: Wayne Davison
QA Contact: Rsync QA Contact
Depends on:
Reported: 2013-05-07 08:12 UTC by Haravikk
Modified: 2014-01-20 20:05 UTC (History)
0 users

See Also:


Note You need to log in before you can comment on or make changes to this bug.
Description Haravikk 2013-05-07 08:12:50 UTC
Rsync already supports compression for improving transfer speed, which is great, but it would also be nice to see the option for permanent compression to reduce space on the target file-system.

Here is an example of what options might look like:

--dest-compress - enables compression of files on the destination.
--dest-compress-level - sets the compression level for files on the destination. Any value from 1-9 implies --dest-compress, while 0 disables it. If this parameter is not supplied then --compress-level will be used if set, otherwise default compression is used.
--dest-compress-ext - an optional extension to append to compressed files, e.g - 'gz'. This extension is checked when checking for files that already exist on the destination, so that compressed files are not deleted.
--source-compress-ext - specifies an extension to be checked for on the source, targeting files that need to be decompressed before sending. This is useful in cases where a destination structure created with compression needs to be synced to another target in a decompressed form. Where possible decompression should be skipped if transfer compression or destination compression are also enabled (i.e - a delta update is not being performed).

In keeping with ticket #8915:
--dest-compress-minsize - the minimum size of uncompressed file that compression will be attempted upon, e.g - if a file is under 4mb then compression may be skipped.
--dest-compress-maxsize - the maximum size of uncompressed file that compression will be attempted upon, e.g - skip compression on files larger than 1gb to save processing.

The implication of this feature of course is that a sync/backup destination can be a lot smaller than the source destination, allowing you to get more out of the space; especially handy on common file-systems such as ext4 that do not natively support compression.

However, this saving is at the cost of processing, as a file on the destination may need to be decompressed on-the-fly in order to compare checksums, and changes cannot be applied via --inplace or --append. If a file is new to the destination however then if --compress-level and --dest-compress-level are the same then it should be possible to simply write the file straight to disk in the compressed form; however if any of the compression details conflict then a gzip -d optionally followed by a gzip may be required to remove and (optionally) re-apply compression.

Of course the savings won't be as good as a single compressed archive would be, but it retains the freedom of a synced structure but with some of the space requirement reduced.
Comment 2 Wayne Davison 2014-01-19 23:27:18 UTC
I think this is better solved by a compressing filesystem, such a btrfs or a fuse overlay.
Comment 3 Haravikk 2014-01-20 12:51:56 UTC
(In reply to comment #2)
> I think this is better solved by a compressing filesystem, such a btrfs or a
> fuse overlay.

Normally I'd agree with you, but the availability is still surprisingly poor; for example most of my rsync use is onto a Synology NAS, but all it seems to support is ext4 with no real ability to customise it (trying to change mount options almost always causes it to assume the disk(s) have gone bad =/). Granted that's poor design on Synology's part, but I've worked with plenty of other limited environments where compression would have been nice.

Since rsync already has compression built-in when transferring data, it seems like a good idea to take fuller advantage of it.
Comment 4 Wayne Davison 2014-01-20 20:05:53 UTC
This is not something I'm going to support.