Bug 7120 - Variable bandwidth limit .. bwlimit
Summary: Variable bandwidth limit .. bwlimit
Alias: None
Product: rsync
Classification: Unclassified
Component: core (show other bugs)
Version: 3.1.0
Hardware: Other Linux
: P3 enhancement (vote)
Target Milestone: ---
Assignee: Wayne Davison
QA Contact: Rsync QA Contact
Depends on:
Reported: 2010-02-08 23:36 UTC by Pierre Dumuid
Modified: 2017-03-01 17:38 UTC (History)
1 user (show)

See Also:


Note You need to log in before you can comment on or make changes to this bug.
Description Pierre Dumuid 2010-02-08 23:36:28 UTC
I used rsync with the --bwlimit option quite a bit so that I can rsync without slowing down the network for others.

Often, I'd use rsync without any options to see the maximum bandwidth, then kill it,and restart with a percentage of the speed reported.

Given this situation, I was wondering if it would be possible (and you would be willing to) add an option:

or allow for:

that periodically (say 1 seconds every 30 seconds) doesn't throttle the transfer to obtain the maximum speed, then throttles down to a percentage of that measured.

Comment 1 Erik Logtenberg 2010-11-23 03:54:00 UTC
Speaking about variable bandwidth, it would also be sweet if rsync could somehow be signalled to use a different bwlimit than the one it was started with.

The use case is: on huge rsync jobs (think multiple weeks) one would like to throttle the bandwidth rather stringently during daytime (when the internet connection and/or server performance is needed for work), and speed up the rsync job during night time.
Currently this would mean killing and restarting rsync every time you'd want to change the throttle. With a huge job (think at least a couple of hours to send/receive file lists) this results in a highly inefficient usage of resources.

It would be awesome to be able to signal rsync in some way to use a different bandwidth throttle without otherwise disturbing the ongoing transfer.
Comment 2 It is me 2013-03-05 15:15:51 UTC
just a note, due to the fact I am also looking forward for this feature.

I see two options regarding usability and scritablility. :)

1# Piping (not used up to now)

mkfifo /tmp/bla
echo "1000" > /tmp/bla

cat /tmp/bla | rsync ... --bwlimit-stdin ...

changing by just write to the fifo
echo "50" > /tmp/bla

2# file and signal

mktemp > /tmp/bla2
echo "1000" > /tmp/bla2
rsync ... -bwlimit-from=/tmp/bla2 .... &
echo "50" > /tmp/bla2
kill -USR1 ${RSYNCPID}

hope the thinking helps.
Comment 3 roland 2013-03-09 12:06:15 UTC
you can do that with pipe-viewer or similar pipe transfer visualization tools with throttling feature. 


first we need to create a little wrapper to put pv into the transfer chain:

linux:/tmp # cat /tmp/pv-wrapper
pv -L 1000 | "$@"


linux:/tmp # rsync --rsh='/tmp/pv-wrapper ssh' -a /proc root@localhost:/tmp/test
Password: 4 B 0:00:01 [3.68 B/s] [ <=> ]
file has vanished: "/proc/10/exe"
file has vanished: "/proc/10/task/10/exe"
file has vanished: "/proc/11/exe"
file has vanished: "/proc/11/task/11/exe"
file has vanished: "/proc/12/exe"=> ]
4.88kiB 0:00:05 [1002 B/s] [ <=> 

You can even adjust the transfer rate at runtime, as pv can communicate with a running instance of itself - you will just need the appropriate PID. This would even make cron based tuning of transfer rates possible.

Comment 4 Pol Hallen 2014-05-05 19:15:42 UTC
Hi all and thanks for replies, any "stable" way to do rsync with variable bandwidht?


Comment 5 roland 2014-05-06 18:11:03 UTC
what`s "unstable" in the way i described it?

why build every feature in every tool when the power of unix is to concatenate tools in a way to handle requirements?
Comment 6 Wayne Davison 2014-07-21 15:59:09 UTC
I like the use of the pv tool for dynamic control.  Nice suggestion!
Comment 7 roland 2017-03-01 17:38:15 UTC
please mind the note from this posting when using pv: http://superuser.com/questions/778066/using-pv1-to-limit-rsync-speed

It seems that pv is waiting for data from rsync, and rsync is waiting for data too (stuck in select()) and not closing the input to pv. So it's a deadlock. Same happens when you substitute pv with something else (like dd). It seems that those commands just don't behave like rsync expects them to.

Haven't found a workaround short of killing everything:

export RSYNC_RSH="sh -c 'pv -qL10k | ssh \"\$@\" | (pv -qL11k; kill \$\$)' ssh"

kill is not a solution I'd be happy with. But I haven't found another.

Of course using $$ was the lazy way out, should kill pv instead but how to obtain the pid in a oneliner? pidof just gives any pv pids. Querying their respective parent pids seemed a bit odd for a oneliner.

Looking forward to a real solution... ;)