Bug 3116 - large tar files: 1 gig size: retransmitted: rsync_rsh
large tar files: 1 gig size: retransmitted: rsync_rsh
Status: CLOSED FIXED
Product: rsync
Classification: Unclassified
Component: core
2.6.6
x86 FreeBSD
: P3 normal
: ---
Assigned To: Wayne Davison
Rsync QA Contact
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2005-09-24 07:08 UTC by Matthew
Modified: 2006-03-12 02:56 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Matthew 2005-09-24 07:08:03 UTC
Once a week, I'm performing an offsite backup. These files are each 1 gig in
size, average speedup is 1.18.

Rsync does the initial run, and then send some of the tar files again. I
initially thought the archives were too large, so I reduced them to 512M, but
had the same symtoms. My next step, is to install an rsync daemon on the remote end.

When I do a second batch of mostly static content, I don't have any issues.

Below are my options:
#!/bin/sh
TERM="VT100"
RSYNC_RSH="ssh -c blowfish -2 -i /usr/username/.ssh/id_dsa"
export RSYNC_RSH
echo ' '
/bin/date
/bin/rsync -lrtv --stats --timeout=300 --delete-after \
        /home/backup/psa username@hostname.net:/home/username/backup

source ssh version: OpenSSH_3.9p1
destination ssh version: OpenSSH_3.6.1p2

Output of rsync log:

Sat Sep 24 03:01:01 EDT 2005
building file list ... done
psa/
psa/psa.dump.full.monthly.aa
psa/psa.dump.full.monthly.ab
psa/psa.dump.full.monthly.ac
psa/psa.dump.full.monthly.ad
psa/psa.dump.full.monthly.ae
psa/psa.dump.full.monthly.af
psa/psa.dump.full.monthly.ag
psa/psa.dump.full.monthly.ah
psa/psa.dump.full.monthly.ai
psa/psa.dump.full.monthly.aj
psa/psa.dump.full.monthly.ak
psa/psa.dump.full.monthly.al
psa/psa.dump.full.monthly.am
psa/psa.dump.full.monthly.an
psa/psa.dump.full.monthly.ao
psa/psa.dump.full.monthly.aa
psa/psa.dump.full.monthly.ad
psa/psa.dump.full.monthly.ae
psa/psa.dump.full.monthly.ag
psa/psa.dump.full.monthly.ah
psa/psa.dump.full.monthly.ai
psa/psa.dump.full.monthly.aj
psa/psa.dump.full.monthly.ak
psa/psa.dump.full.monthly.al
psa/psa.dump.full.monthly.am

Number of files: 16
Number of files transferred: 25
Total file size: 15400460674 bytes
Total transferred file size: 26137878914 bytes
Literal data: 13011870082 bytes
Matched data: 13126008832 bytes
File list size: 230
File list generation time: 0.306 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 13016666532
Total bytes received: 18628384

sent 13016666532 bytes  received 18628384 bytes  892125.72 bytes/sec
total size is 15400460674  speedup is 1.18

Source Files:
-rwx------  1 root    wheel           1G Sep 22 02:19 psa.dump.full.monthly.aa
-rwx------  1 root    wheel           1G Sep 22 02:33 psa.dump.full.monthly.ab
-rwx------  1 root    wheel           1G Sep 22 02:47 psa.dump.full.monthly.ac
-rwx------  1 root    wheel           1G Sep 22 03:00 psa.dump.full.monthly.ad
-rwx------  1 root    wheel           1G Sep 22 03:16 psa.dump.full.monthly.ae
-rwx------  1 root    wheel           1G Sep 22 03:29 psa.dump.full.monthly.af
-rwx------  1 root    wheel           1G Sep 22 03:44 psa.dump.full.monthly.ag
-rwx------  1 root    wheel           1G Sep 22 03:54 psa.dump.full.monthly.ah
-rwx------  1 root    wheel           1G Sep 22 04:04 psa.dump.full.monthly.ai
-rwx------  1 root    wheel           1G Sep 22 04:15 psa.dump.full.monthly.aj
-rwx------  1 root    wheel           1G Sep 22 04:24 psa.dump.full.monthly.ak
-rwx------  1 root    wheel           1G Sep 22 04:35 psa.dump.full.monthly.al
-rwx------  1 root    wheel           1G Sep 22 04:45 psa.dump.full.monthly.am
-rwx------  1 root    wheel           1G Sep 22 05:02 psa.dump.full.monthly.an
-rwx------  1 root    wheel         351M Sep 22 05:11 psa.dump.full.monthly.ao

Second Batch Output:
-rwx------  1 backup  backup       849K Sep 23 01:01 host1-backup-etc.zip
-rwx------  1 backup  backup       581M Sep 23 01:06 host1-backup-home-a2e.zip
-rwx------  1 backup  backup       155M Sep 23 01:07 host1-backup-home-f2j.zip
-rwx------  1 backup  backup       423M Sep 23 01:09 host1-backup-home-k2o.zip
-rwx------  1 backup  backup       341M Sep 23 01:10 host1-backup-home-p2t.zip
-rwx------  1 backup  backup       374M Sep 23 01:12 host1-backup-home-u2z.zip
-rwx------  1 backup  backup        13M Sep 23 01:01 host1-backup-mysql.zip
-rwx------  1 backup  backup       264M Sep 23 01:02 host1-backup-staff.zip
-rwx------  1 backup  backup        80M Sep 23 01:01 host1-backup-var.zip
-rwx------  1 backup  backup        29B Sep 23 01:12 host1-zip-end.txt
-rwx------  1 backup  backup        29B Sep 23 01:01 host1-zip-start.txt
-rwx------  1 backup  backup       836K Sep 23 01:01 host2-backup-etc.zip
-rwx------  1 backup  backup       513M Sep 23 01:04 host2-backup-home-a2e.zip
-rwx------  1 backup  backup       393M Sep 23 01:06 host2-backup-home-f2j.zip
-rwx------  1 backup  backup       406M Sep 23 01:08 host2-backup-home-k2o.zip
-rwx------  1 backup  backup       375M Sep 23 01:10 host2-backup-home-p2t.zip
-rwx------  1 backup  backup       130M Sep 23 01:10 host2-backup-home-u2z.zip
-rwx------  1 backup  backup        11M Sep 23 01:01 host2-backup-mysql.zip
-rwx------  1 backup  backup       133M Sep 23 01:01 host2-backup-staff.zip
-rwx------  1 backup  backup        43M Sep 23 01:01 host2-backup-var.zip
-rwx------  1 backup  backup        29B Sep 23 01:10 host2-zip-end.txt
-rwx------  1 backup  backup        29B Sep 23 01:01 host2-zip-start.txt

Sat Sep 24 07:06:26 EDT 2005
building file list ... done
archive/
archive/host1-backup-etc.zip
archive/host1-backup-home-a2e.zip
archive/host1-backup-home-f2j.zip
archive/host1-backup-home-k2o.zip
archive/host1-backup-home-p2t.zip
archive/host1-backup-home-u2z.zip
archive/host1-backup-mysql.zip
archive/host1-backup-staff.zip
archive/host1-backup-var.zip
archive/host1-zip-end.txt
archive/host1-zip-start.txt
archive/host2-backup-etc.zip
archive/host2-backup-home-a2e.zip
archive/host2-backup-home-f2j.zip
archive/host2-backup-home-k2o.zip
archive/host2-backup-home-p2t.zip
archive/host2-backup-home-u2z.zip
archive/host2-backup-mysql.zip
archive/host2-backup-staff.zip
archive/host2-backup-var.zip
archive/host2-zip-end.txt
archive/host2-zip-start.txt

Number of files: 23
Number of files transferred: 22
Total file size: 4449287013 bytes
Total transferred file size: 4449287013 bytes
Literal data: 390371025 bytes
Matched data: 4058915988 bytes
File list size: 528
File list generation time: 0.016 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 391530333
Total bytes received: 1792478

sent 391530333 bytes  received 1792478 bytes  456555.79 bytes/sec
total size is 4449287013  speedup is 11.31
Comment 1 Matthew 2005-10-01 04:34:11 UTC
After upgrading both system to OpenSSH_4.2.

This issue appears to be resolved.

Number of files: 16
Number of files transferred: 15
Total file size: 15584500220 bytes
Total transferred file size: 15584500220 bytes
Literal data: 13325205804 bytes
Matched data: 2259294416 bytes
File list size: 230
File list generation time: 0.192 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 13327111326
Total bytes received: 3345929

sent 13327111326 bytes  received 3345929 bytes  1240215.59 bytes/sec
total size is 15584500220  speedup is 1.17