Bug 7349 - Add --limit-time=<seconds> and --fit-time=[seconds] options
Add --limit-time=<seconds> and --fit-time=[seconds] options
Status: NEW
Product: rsync
Classification: Unclassified
Component: core
3.1.0
All All
: P3 enhancement
: ---
Assigned To: Wayne Davison
Rsync QA Contact
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2010-04-11 04:58 UTC by Simon Arlott
Modified: 2013-03-09 12:08 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Simon Arlott 2010-04-11 04:58:53 UTC
More (cheaper) bandwidth is available between 2am and 6am, so it is the ideal time to run rsync for backup purposes. However, in order to allow the maximum possible time for rsync to complete it must be started at 2am even if it would only take a shorter period of time (e.g. 2 hours). If there are a lot of users (of an ISP) doing this then there will be a large undesireable spike at 2am.

Ideally they would all schedule times so they don't overlap but this is impractical. Instead, rsync could automatically reduce the bandwidth limit so that the transfer takes the full time. This would require determining the size of all files to be transferred in advance and then adjusting the bandwidth limit based on remaining time.

--limit-time would limit the maximum duration of the process
--fit-time would lower the --bwlimit (which may not be set; this will dynamically change if it's not going fast enough) so that the transfer takes the intended time

The default for --fit-time should be 95% of the --limit-time, allowing some time for the inexact nature of adjusting bandwidth dynamically so that the transfer can complete successfully. Transfers should never exceed --bwlimit even if the time would not be met. Compression is likely to complicate the situation so that transfers get progressively slower, but the assumption is that these files are already compressed so this will not be used.

Any checksum-based options would need to be disabled unless they could avoid reading files twice (once at the start and once at the transfer stage). Partial transfers and appending are probably an implicit requirement of limiting the transfer time. Currently I have a cron job kill rsync at 6am.
Comment 1 Simon Arlott 2010-04-11 05:32:43 UTC
An additional option, which requires --limit-time (and optionally --fit-time) as well as --bwlimit to be set accurately to the expected minimum available bandwidth:

--delay-start=<seconds> which would delay starting of the actual transfer by up to the specified amount of time if it is expected to then complete before --limit-time

If --fit-time is specified, the transfer should also be slowed down if it will complete within the given time.

Example:
Bandwidth between 18:00 and 09:00 is "off peak", and between 02:00 and 06:00 is "night". Ideally transfers should complete during "night" but may use all of the "off peak" period. The actual size of the transfer varies from day to day.

Run rsync with --limit-time=54000 --fit-time=14400 --bwlimit=200 --delay-start=28800. The limit is 15 hours (the "off peak" period), but the transfer may be delayed by up to 8 hours (bringing it to 02:00) if it would still complete by 09:00. Based on the bandwidth limit set, if the transfer could be completed by 06:00 at a slower rate, the bandwidth would be dynamically reduced as in the original proposal.

If the transfer would not finish by 09:00 when starting at 02:00 it would adjust the start delay backwards, e.g. to 01:00 or 18:00 if necessary or there is too much data.
Comment 2 roland 2013-03-09 12:08:37 UTC
have a look at https://bugzilla.samba.org/show_bug.cgi?id=7120 , as it describes a method of externally adjusting bandwidth at runtime without changing rsync at all