Bug 1463 - poor performance with large block size
poor performance with large block size
Product: rsync
Classification: Unclassified
Component: core
x86 other
: P3 normal
: ---
Assigned To: Wayne Davison
Rsync QA Contact
Depends on:
  Show dependency treegraph
Reported: 2004-06-16 06:21 UTC by wallace matthews
Modified: 2005-03-16 16:48 UTC (History)
0 users

See Also:

Suggested patch from Craig Barratt (477 bytes, patch)
2004-06-18 14:45 UTC, Wayne Davison
no flags Details
Improved patch (1.51 KB, patch)
2004-07-16 10:57 UTC, Wayne Davison
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description wallace matthews 2004-06-16 06:21:13 UTC
I have a 29Gig file that is the previous version of a file and a 1.3 Gig 
incremental backup of the file. I did the transfer with no block size option 
and it takes about 6 minutes (GigEthernet). I used --block-size = 90k and it 
took about 6 minutes. I used --block-size=182000 (close to the square root of 
29 Gig) and it only completed ~50 Meg of transfer of the 1.3 Gig in a couple of 

Chris Shoemaker suggests this is a problem with the sliding window being a 
fixed size of 256k and that the block size will not allow multiple blocks in 
the window size. 

Operating system is Redhat 9.1 for both systems. Both systems have the 2.6.2 
with only 1 patch and it is the one Wayne provided for --bwlimit being bi-modal
Comment 1 Wayne Davison 2004-06-18 14:45:38 UTC
Created attachment 543 [details]
Suggested patch from Craig Barratt

Wallace Matthews confirmed that this alleviates the poor performance.

Just need to confirm that the window isn't getting too large with this (I will
check into this later unless someone else beats me to it).
Comment 2 Wayne Davison 2004-07-16 10:57:11 UTC
Created attachment 573 [details]
Improved patch

I created a patch similar to Craig's that instead modifies the MAX_MAP_SIZE
value (using a new "max_map_size" variable) which is based on the current
file's block size.  This means that we get more consistent sizing (since "len"
can vary) and that the sequence of window_size checks works out as originally
intended (maxing out at either the end of the file or at offset + len,
whichever is greater).
Comment 3 Wayne Davison 2004-07-16 11:13:19 UTC
I checked-in my improved patch, which should resolve this bug.

We may still wish to review the map_ptr() code to see if we can optimize it even
further, but this should be good enough for now.