Bug 8367 - Add a feature to --rename-existing files
Summary: Add a feature to --rename-existing files
Status: NEW
Alias: None
Product: rsync
Classification: Unclassified
Component: core (show other bugs)
Version: 3.1.2
Hardware: All All
: P3 enhancement (vote)
Target Milestone: ---
Assignee: Wayne Davison
QA Contact: Rsync QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-08-09 18:45 UTC by Devin Nate
Modified: 2021-01-15 14:21 UTC (History)
1 user (show)

See Also:


Attachments
Patch file to add --move-existing (4.87 KB, text/plain)
2011-08-09 18:45 UTC, Devin Nate
no flags Details
second version of patch (5.05 KB, patch)
2011-08-24 15:06 UTC, Devin Nate
no flags Details
third version (5.10 KB, patch)
2011-09-22 22:55 UTC, Devin Nate
no flags Details
rsync-3.1.2-rename-existingV6.patch (7.99 KB, patch)
2016-05-15 21:29 UTC, Devin Nate
no flags Details
rsync-3.1.2-rename-existingV7.patch (7.75 KB, patch)
2016-05-16 16:24 UTC, Devin Nate
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Devin Nate 2011-08-09 18:45:31 UTC
Created attachment 6766 [details]
Patch file to add --move-existing

The use case is one where rsync is being used to replicate files from a source to a destination, with the --remove-source-files flag being used. The files on the source are created sequentially, based on the file existing or not.

Essentially, software creates a files named 1.tif, 2.tif, 3.tif, and so on, based on the existance of the files. If rsync moves the files from the source to the destination, then the files 1.tif, 2.tif, 3.tif, etc., no longer exist on the source, so the software on the source re-starts it's numbering. Then, when rsync is run again either: (a) it'll sync the files up, which essentially means all prior data was lost, since they are totally different files, or (b) if --ignore-existing is used, the files back up on the source (good in that there's no data loss, bad that the files don't move). 

The attached patch is a first attempt at adding a --move-existing functionality to rsync. It does not honour --dry-run and probably a ton of other use cases. It also uses strftime and some other functions which I see are in IFDEF blocks in other areas. The new filename generator should probably be in util.c, and it's logging is ugly right now. It's not well documented (aka no man page entry).

It will rename a file in this format:
SOURCE: filename.ext
DEST POST RENAME: filename_2011-08-09-13-44-22-1.ext

It's somewhat contrary to rsync individual file sync, since it suggests we specifically don't want to sync a particular file. That said, it is in line with the concept of wanting to sync a (poorly) created series of files in a safe manner.

Thanks,
Devin
Comment 1 Devin Nate 2011-08-24 15:06:11 UTC
Created attachment 6803 [details]
second version of patch

This is an enhancement of the original patch.

1. Adds better logging.
1a. errors are logged at all verbosity levels
1b. info messages at verbosity level > 2 (aka 3 -v's)

2. Confirmed it behaves properly with --dry-run, it does

3. CHANGED the flag from --move-existing to be --rename-existing

4. CHANGED the format of the rename logic. No longer does a long string based on date. Now does epoch time in hex.

5. Moved to using snprintf for additional safety.

Thanks,
Devin
Comment 2 Devin Nate 2011-09-22 22:55:15 UTC
Created attachment 6931 [details]
third version

- removes memory leak if an error occurs
Comment 3 Devin Nate 2016-05-15 21:29:01 UTC
Created attachment 12104 [details]
rsync-3.1.2-rename-existingV6.patch
Comment 4 Devin Nate 2016-05-15 21:30:57 UTC
Comment on attachment 12104 [details]
rsync-3.1.2-rename-existingV6.patch

Please find attached an updated version of this patch. It has been reworked for additional safety (checking MAXPATHLEN and NAME_MAX better). This applies to rsync 3.1.2, including the new debug and info message handling.

It's a pretty specific use case to need this patch, but we find it useful. Posting here in case others have use for this also.

Thanks,
Devin
Comment 5 Devin Nate 2016-05-15 21:33:05 UTC
Comment on attachment 12104 [details]
rsync-3.1.2-rename-existingV6.patch

Please find attached an updated version of this patch. It has been reworked for additional safety (checking MAXPATHLEN and NAME_MAX better), and a different renaming scheme (it simply appends a number starting from 1 to 9999 to the filename, making sure to respect any file extension). There were a few other changes and bug fixes made too. This applies to rsync 3.1.2, including the new debug and info message handling.

It's a pretty specific use case to need this patch, but we find it useful. Posting here in case others have use for this also.

Thanks,
Devin
Comment 6 Devin Nate 2016-05-16 16:24:02 UTC
Created attachment 12106 [details]
rsync-3.1.2-rename-existingV7.patch

minor cleanups
Comment 7 Claudius Ellsel 2021-01-15 14:21:20 UTC
I didn't have a very close look. Just for understanding, what is the difference to `-b` option? That the files remain in the folder and are not moved to the backup directory?