Bug 13388 - Feature request: When deleting files only delete files that are over a certain age.
Summary: Feature request: When deleting files only delete files that are over a certai...
Status: NEW
Alias: None
Product: rsync
Classification: Unclassified
Component: core (show other bugs)
Version: 3.1.3
Hardware: All All
: P5 enhancement (vote)
Target Milestone: ---
Assignee: Wayne Davison
QA Contact: Rsync QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-04-17 06:38 UTC by Arthur Gouros
Modified: 2018-04-17 06:38 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Arthur Gouros 2018-04-17 06:38:50 UTC
Hi,

Feature request: New option: --min-age-delete=SECONDS    Only delete files that are over a certain age.

This situation came about when I have source servers in multiple data centres that very frequently update the same target via rsync using the --update option. 

Being separate data centres they eventually generate the same file manifests but sometimes there is a small period of time where the manifests of the source servers differ between each data centre.

Using the --update option solves the problem of only updating the target with the latest file, however if the source server generates files that incorporate timestamps in their names then a condition can arise at the target server where one source server's rsync deletes the timestamped files of another source server's rsync. 

Consider this example. 

source-server-1
    example.html
        image_201801011400.gif

source-server-2
    example.html
        image_201801011401.gif

Suppose the resulting gif file is generated by some local process on the source server (e.g. an ingestion process).

Using the --update option, whichever source server has the latest example.html file then that will end up at the target destination. 
Unfortunately the same cannot be said for the gif file so what ends up at the target destination is dependent on which source server ran the last rsync (remember they run rsync very frequently).
This means there is a potential that the file example.html contains a reference to a file that doesn't exist on the target rsync destination.

By having the following option available to rsync --min-age-delete=60 then any files that qualify for deletion will need to be greater than 60 seconds of age.

Used in conjunction with the --update option this will allow me to use multiple source servers to update the same target destination via rsync.

The default value for the proposed option --min-age-delete can be 0 which preserves backwards compatability.

The proposed enhancement can be implemented around the Posix unlink(3c) function call which will also preserve compatability with the existing --delete-* options.

At the momement, to overcome this issue I need to run a two step process.
1. Produce a list of files that haven't changed in the last X seconds.
2. Add that file list to an rsync exclude option.
This is not very elegant and I would prefer to have this capability as a built-in feature of rsync.

Thank you,
Arthur Gouros.