Bug 2423 - Feature Request: Ability to transfer files newer than date or file. (--newer switch).
Summary: Feature Request: Ability to transfer files newer than date or file. (--newer ...
Status: ASSIGNED
Alias: None
Product: rsync
Classification: Unclassified
Component: core (show other bugs)
Version: 2.6.3
Hardware: All Linux
: P3 enhancement (vote)
Target Milestone: ---
Assignee: Wayne Davison
QA Contact: Rsync QA Contact
URL:
Keywords:
: 3210 (view as bug list)
Depends on:
Blocks:
 
Reported: 2005-03-07 05:43 UTC by Roman Gaufman
Modified: 2014-06-04 06:57 UTC (History)
4 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Roman Gaufman 2005-03-07 05:43:35 UTC
--newer [file|date]    transfer only files newer than file or date 
                       (e.g. rsync --newer "15/03/05 21:22:00" -a dir1 dir2)

That would really make my day when transferring daily cumulative sql changelogs
and not having to keep older ones.

E.G:
changelog-200502-09.sql 100Mb
changelog-200503-01.sql 100Mb
changelog-200503-02.sql 100Mb
changelog-200503-03.sql 56Mb (and growing)

The first 3 files were transferred, applied to the backup database and deleted.
The third file is then partially transferred until it reaches 100Mb. There's no
way however to just rsync files newer than changelog-03.sql (when they are
added), I also dont have the disk space to keep the tens of files.

The only solution I can find is to use --exclude, but pretty soon the list will
grow to hundreds of files and will be hard to maintain.

I would really appreciate a --newer switch.
Comment 1 Wayne Davison 2005-03-12 16:35:25 UTC
I'd suggest using a combination of "find" and the --files-from option to rsync:

find . -newer changelog-200503-02.sql | rsync -av --files-from=- . host:/dest/

or

find /src -mtime -1 | rsync -av --files-from=- / host:/dest/

Unfortunately, that only works (easily) with a push, which is rather annoying. 
Even so, I'm currently doubtful that I'll include this, but I'll leave it open
for now as a future feature request.
Comment 2 Roman Gaufman 2005-03-12 17:31:18 UTC
Thanks for that. Havent tested yet, but if it works I suppose that will do for
now. Would still like to see this in rsync, but I guess its not a priority.

Thanks again!
Comment 3 Roman Gaufman 2005-03-12 17:37:41 UTC
Actually, that cant work. I dont have the filenames I need to rsync. I'm
rsyncing from client, not from server.

The destination is '.' not 'host:/dest/'
Comment 4 Roman Gaufman 2005-03-12 17:42:20 UTC
Oh, and especially since the sql file is commulative until it reaches 100Mb, I
then have to rsync it from client to server, then sync back to client. Thats no
good.
Comment 5 James R Grinter (mail bounces back) 2005-04-28 03:27:05 UTC
I'd find this kind of feature really useful too.

I'm pushing files, but I'm using --include and --exclude patterns to build the 
list of files I want to transfer, in some cases avoiding directory recursion. --
files-from and include/exclude don't mix.

It's also really hard to avoid recursion with find, and it would be nicer to be 
able to do it all within rsync for efficiency reasons (it's already stat()ing 
files, for one.)

Perhaps something matching find's syntax for these things:

--newer file
  newer than a reference file. (perhaps also -older, as you wouldn't have 
the '! -newer' syntax of find)
--mtime n
  (if the file was modified n days ago, also +n, -n)
--ctime n
  (ditto, for inode change time)
--atime n
  (ditto, for access time)
Comment 6 Peter Eckel 2007-01-04 01:34:32 UTC
I can only second this suggestion.

I'm currently working on a project where I have to import tons (about 150 GB) of data from a remote site and process them, with subsequent updates to the source files where only very few files are changed or added and the majority of the processed files lie around motionless, possibly for years. No use at all in keeping it around, except that rsync (without the suggested --newer switch) insists on having them about.
Comment 7 Matt McCutchen 2007-01-04 18:53:02 UTC
(In reply to comment #3)
> Actually, that cant work. I dont have the filenames I need to rsync. I'm
> rsyncing from client, not from server.
> 
> The destination is '.' not 'host:/dest/'

You could write a script on the server that pipes the results of the find to rsync and then specify that script as the --rsync-path.

Roman and Peter, another way to accomplish what you want is to use --update and have the receiver truncate files it is finished with to zero length instead of deleting them.  James, I don't know whether this technique accomplishes what you want.

An alternative to adding --newer is to add a new kind of filter rule that makes rsync pass file names to a user-specified program, which decides whether they "match" the rule.  The specifics could be as follows.  A "|" modifier means to interpret the text of the rule as a command line instead of a filename pattern.  Rsync runs the command and gives it one filename per line on standard input.  The command must print a "0" or "1" character for each file to indicate whether the custom rule matches the file.  For example, the following option would have the same effect as `--newer=TIME':

--filter='-!| newer-filter TIME'

where newer-filter is the following script:

#!/bin/bash
# We convert times to seconds since the Epoch so
# we can compare them with [ -gt ].
cutoff_time=$(date +%s -d "$1") 
while IFS='' read fname; do
    # Compare file's mtime to cutoff time
    if [ $(stat --format=%Y $fname) -gt $cutoff_time ]; then
        echo -n 1
    else
        echo -n 0
    fi
done

The "|" modifier would support all non-name-based filters at once and would have the additional advantage that they could be prioritized anywhere in the filter list.  One disadvantage is that many rsync daemons would want to refuse the modifier for security.
Comment 8 Wayne Davison 2007-08-23 19:19:03 UTC
*** Bug 3210 has been marked as a duplicate of this bug. ***
Comment 9 Roman Gaufman 2007-08-29 06:25:02 UTC
any update on this?
Comment 10 Rainer Glaschick 2014-06-04 06:56:27 UTC
I vote for the original proposal.
Using --files-from does not work with --delete, which is absolutely correct,
but --newer could delete all files on the target that are not in the newer list.

I use rsync for backup to large external harddisk monthly, that is disconnected and moved to a save place otherwise.
In between these full mirror backups, I use a small device (actually a raspberry pi in a separate room) receiving an incremental backup of all the files that are newer.
While in this case cleaning the incremental backup after the full backup works, the --newer option would be better, as I am alyways reluctant erasing files on backup devices manually.