Bug 4383 - Idea: Store files in compressed format and visa versa
Idea: Store files in compressed format and visa versa
Status: NEW
Product: rsync
Classification: Unclassified
Component: core
3.0.0
Other All
: P3 enhancement
: ---
Assigned To: Wayne Davison
Rsync QA Contact
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2007-02-09 00:36 UTC by Kilburn Abrahams
Modified: 2009-08-07 05:24 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Kilburn Abrahams 2007-02-09 00:36:13 UTC
The -z compresses files or deltas in transist, but on the receiver side (assuming push) files are stored in uncompressed form. What would be useful is to store the file in compressed format much like gzip or bzip2 does. 

A switch to store compressed or uncompressed could be used in push or pull.

This might not be so easy because sync uses "context-compression" by "priming" the compressor with hits and discarding the compressed output, but I put it up as an idea because we are storing >3TB of data.
Comment 1 Wayne Davison 2007-02-10 23:43:42 UTC
I think that it would probably be better to implement this outside of rsync, such as creating a fuse filesystem that compresses the files that it receives.  I know that at least one project exists to implement a compressing fuse filesystem.

I'll leave this open for now to see if anyone wants to discuss this.
Comment 2 roland 2009-08-07 05:24:50 UTC
maybe the patches/source-filter_dest-filter.diff is interesting for you !?

i did not test it, but it looks related to what you want.

+dit(bf(--source-filter=COMMAND)) This option allows the user to specify a
+filter program that will be applied to the contents of all transferred
+regular files before the data is sent to destination.  COMMAND will receive
+the data on its standard input and it should write the filtered data to
+standard output.  COMMAND should exit non-zero if it cannot process the
+data or if it encounters an error when writing the data to stdout.
+
+Example: --source-filter="gzip -9" will cause remote files to be
+compressed.
+Use of --source-filter automatically enables --whole-file.
+If your filter does not output the same number of bytes that it received on
+input, you should use --times-only to disable size and content checks on
+subsequent rsync runs.
+
+dit(bf(--dest-filter=COMMAND)) This option allows you to specify a filter
+program that will be applied to the contents of all transferred regular
+files before the data is written to disk.  COMMAND will receive the data on
+its standard input and it should write the filtered data to standard
+output.  COMMAND should exit non-zero if it cannot process the data or if
+it encounters an error when writing the data to stdout.
+
+Example: --dest-filter="gzip -9" will cause remote files to be compressed.
+Use of --dest-filter automatically enables --whole-file.
+If your filter does not output the same number of bytes that it
+received on input, you should use --times-only to disable size and
+content checks on subsequent rsync runs.