The Samba-Bugzilla – Bug 2423
Feature Request: Ability to transfer files newer than date or file. (--newer switch).
Last modified: 2014-06-04 06:57:13 UTC
--newer [file|date] transfer only files newer than file or date
(e.g. rsync --newer "15/03/05 21:22:00" -a dir1 dir2)
That would really make my day when transferring daily cumulative sql changelogs
and not having to keep older ones.
changelog-200503-03.sql 56Mb (and growing)
The first 3 files were transferred, applied to the backup database and deleted.
The third file is then partially transferred until it reaches 100Mb. There's no
way however to just rsync files newer than changelog-03.sql (when they are
added), I also dont have the disk space to keep the tens of files.
The only solution I can find is to use --exclude, but pretty soon the list will
grow to hundreds of files and will be hard to maintain.
I would really appreciate a --newer switch.
I'd suggest using a combination of "find" and the --files-from option to rsync:
find . -newer changelog-200503-02.sql | rsync -av --files-from=- . host:/dest/
find /src -mtime -1 | rsync -av --files-from=- / host:/dest/
Unfortunately, that only works (easily) with a push, which is rather annoying.
Even so, I'm currently doubtful that I'll include this, but I'll leave it open
for now as a future feature request.
Thanks for that. Havent tested yet, but if it works I suppose that will do for
now. Would still like to see this in rsync, but I guess its not a priority.
Actually, that cant work. I dont have the filenames I need to rsync. I'm
rsyncing from client, not from server.
The destination is '.' not 'host:/dest/'
Oh, and especially since the sql file is commulative until it reaches 100Mb, I
then have to rsync it from client to server, then sync back to client. Thats no
I'd find this kind of feature really useful too.
I'm pushing files, but I'm using --include and --exclude patterns to build the
list of files I want to transfer, in some cases avoiding directory recursion. --
files-from and include/exclude don't mix.
It's also really hard to avoid recursion with find, and it would be nicer to be
able to do it all within rsync for efficiency reasons (it's already stat()ing
files, for one.)
Perhaps something matching find's syntax for these things:
newer than a reference file. (perhaps also -older, as you wouldn't have
the '! -newer' syntax of find)
(if the file was modified n days ago, also +n, -n)
(ditto, for inode change time)
(ditto, for access time)
I can only second this suggestion.
I'm currently working on a project where I have to import tons (about 150 GB) of data from a remote site and process them, with subsequent updates to the source files where only very few files are changed or added and the majority of the processed files lie around motionless, possibly for years. No use at all in keeping it around, except that rsync (without the suggested --newer switch) insists on having them about.
(In reply to comment #3)
> Actually, that cant work. I dont have the filenames I need to rsync. I'm
> rsyncing from client, not from server.
> The destination is '.' not 'host:/dest/'
You could write a script on the server that pipes the results of the find to rsync and then specify that script as the --rsync-path.
Roman and Peter, another way to accomplish what you want is to use --update and have the receiver truncate files it is finished with to zero length instead of deleting them. James, I don't know whether this technique accomplishes what you want.
An alternative to adding --newer is to add a new kind of filter rule that makes rsync pass file names to a user-specified program, which decides whether they "match" the rule. The specifics could be as follows. A "|" modifier means to interpret the text of the rule as a command line instead of a filename pattern. Rsync runs the command and gives it one filename per line on standard input. The command must print a "0" or "1" character for each file to indicate whether the custom rule matches the file. For example, the following option would have the same effect as `--newer=TIME':
--filter='-!| newer-filter TIME'
where newer-filter is the following script:
# We convert times to seconds since the Epoch so
# we can compare them with [ -gt ].
cutoff_time=$(date +%s -d "$1")
while IFS='' read fname; do
# Compare file's mtime to cutoff time
if [ $(stat --format=%Y $fname) -gt $cutoff_time ]; then
echo -n 1
echo -n 0
The "|" modifier would support all non-name-based filters at once and would have the additional advantage that they could be prioritized anywhere in the filter list. One disadvantage is that many rsync daemons would want to refuse the modifier for security.
*** Bug 3210 has been marked as a duplicate of this bug. ***
any update on this?
I vote for the original proposal.
Using --files-from does not work with --delete, which is absolutely correct,
but --newer could delete all files on the target that are not in the newer list.
I use rsync for backup to large external harddisk monthly, that is disconnected and moved to a save place otherwise.
In between these full mirror backups, I use a small device (actually a raspberry pi in a separate room) receiving an incremental backup of all the files that are newer.
While in this case cleaning the incremental backup after the full backup works, the --newer option would be better, as I am alyways reluctant erasing files on backup devices manually.