Bug 8354 - --skip-compress does not appear to be working
--skip-compress does not appear to be working
Product: rsync
Classification: Unclassified
Component: core
All All
: P5 normal
: ---
Assigned To: Wayne Davison
Rsync QA Contact
Depends on:
  Show dependency treegraph
Reported: 2011-08-05 13:29 UTC by Justin Watt
Modified: 2011-09-05 06:36 UTC (History)
0 users

See Also:


Note You need to log in before you can comment on or make changes to this bug.
Description Justin Watt 2011-08-05 13:29:33 UTC
I use rsync to backup my Ubuntu home directory to an external USB hard drive. I've always used the flags -avz without thinking, and I've always wondered why my CPU shoots to 100% and my laptop gets hot enough to fry an egg. Finally it occurred to me. The vast majority of files rsync is transferring at JPEGs, which can't be compressed. So my laptop was spending all its time compressing and decompressing large files to no benefit. 

After removing the "z" flag, I noticed a tremendous speed up, not to mention a cooler, happier CPU. I dug deeper into the man page and discovered the --skip-compress option, which according to Bug 4162 should have its default file extension list enabled whenever the "z" flag is set. However I've found that not to be the case. I thought maybe it had to do with the fact that my JPEG files have an uppercase .JPG extension (Bug 7512) but even when I directly specify an uppercase extension with --skip-compress, I see no speed up.

Here are the results of my tests running rsync to backup my home directory after repeatedly deleting a destination directory of 788,709,822 bytes of photos and a movie.

without -z:

sent 791193666 bytes  received 8629 bytes  23617978.96 bytes/sec
total size is 205231570237  speedup is 259.39

with -z:

sent 788709822 bytes  received 8629 bytes  9677527.01 bytes/sec
total size is 205231570237  speedup is 260.21

with -z and --skip-compress=jpg/JPG/avi/AVI:

sent 788709822 bytes  received 8629 bytes  9560223.65 bytes/sec
total size is 205231570237  speedup is 260.21

Notice the order of magnitude difference in the bytes/sec rate.
Comment 1 Wayne Davison 2011-09-03 22:25:58 UTC
This fix was noted in 3.0.8:

- Fixed a bug in the comparing of upper-case letters in file suffixes for --skip-compress.

The issue was that the skip-compress values are always stored in lower case, but if the file had upper-case letters in the suffix, they wouldn't match the stored suffixes.
Comment 2 Justin Watt 2011-09-04 03:50:40 UTC
Wayne, but what about when I explicitly set an uppercase file extension with --skip-compress? Did it also treat that as lowercase?
Comment 3 Wayne Davison 2011-09-05 06:36:52 UTC
The way it's written, --skip-compress always ignores case.