Bug 7249 - Add an option to use O_NOATIME
Summary: Add an option to use O_NOATIME
Status: RESOLVED FIXED
Alias: None
Product: rsync
Classification: Unclassified
Component: core (show other bugs)
Version: 3.1.0
Hardware: Other Linux
: P3 enhancement (vote)
Target Milestone: ---
Assignee: Wayne Davison
QA Contact: Rsync QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-03-14 17:08 UTC by Nicolas George
Modified: 2020-04-23 21:34 UTC (History)
5 users (show)

See Also:


Attachments
adds a --noatime option (3.58 KB, patch)
2010-03-14 17:09 UTC, Nicolas George
no flags Details
--noatime option for 3.1.0 (3.86 KB, patch)
2013-10-13 09:21 UTC, Martin von Gagern
Martin.vGagern: review?
Details
Adds --noatime option for 3.1.3 (5.13 KB, patch)
2019-10-10 00:05 UTC, Samuel Henrique
no flags Details
Adds --noatime option for 3.1.3 (5.13 KB, patch)
2019-10-10 00:09 UTC, Samuel Henrique
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Nicolas George 2010-03-14 17:08:52 UTC
Some systems (at least Linux since 2.6.8) have the O_NOATIME open flag that avoids altering the atime of files opened for reading. An option to enable this flag is supported would be useful for backup systems based on rsync, as it would avoid interfering with tools that relies on atime (such as new mail detection, whatever may be thought about the correctness of the method) and reduce disc activity on the source filesystem.

We use a patched rsync that always sets O_ATIME with rsnapshot since quite some time and we have not noticed any drawback.
Comment 1 Nicolas George 2010-03-14 17:09:41 UTC
Created attachment 5497 [details]
adds a --noatime option
Comment 2 roland 2010-08-22 06:15:03 UTC
this looks very useful . i vote for it.

seems there is already an atimes.diff in [rsync-patches.git] which has a similar purpose !?
Comment 3 Nicolas George 2010-08-22 07:49:16 UTC
They have a similar purpose, but they do not work the same way at all.

My patch (comment #2) politely asks the kernel not to update the atime of opened files. It only works if the O_NOATIME flags exists on that particular platform but has no other drawbacks.

AFAIK, the patch on rsync-patches.git explicitly puts back the atime to what it was. It has the drawback of updating the ctime instead.
Comment 4 Martin von Gagern 2011-12-10 19:53:32 UTC
+1 vote.

Should someone set the review flag to '?' to get Nicolas' patch reviewed? Who?

rsync3.txt in the source tree has this to say about atimes:

"Desirable features:
 [...]
 - Propagate atimes and do not modify them.  This is very ugly on
   Unix.  It might be better to try to add O_NOATIME to kernels, and
   call that."

So apparently someone with commit privileges (Martin Pool to be precise) already has decided that such a feature would be desirable (back in 2001). Does this add weight to this request and patch?
Comment 5 Martin von Gagern 2013-10-13 09:21:06 UTC
Created attachment 9278 [details]
--noatime option for 3.1.0

I rebased Nicholas' patch to 3.1.0, with modifications as described below.

One thing I'm not sure is whether the --noatime flag should automatically be forwarded to the server. In a general backup scenario, the client would wish to preserve atimes, but on the backup server I see no reason to do so. Therefore I made the forwarding part a comment. Users can still explicitely include the --noatime in the list of server arguments if they whish this behaviour on both ends.

I've been using this form on 3.0.9 for quite some time, and would like to see this included officially one day soon. I hope setting it to “review=?” the way I did is the way to go for that.
Comment 6 don311 2015-02-26 06:54:08 UTC
+1 from here as well.

I understand the arguments about the limited usefulness of the atime field in general, and I agree with much of it, but to me there are more use cases than what's mentioned in what I've read so far - such as providing helpful clues for certain kinds of "forensic" debugging.  atime has been around for a very long time (every *NIX variant I've worked on, going back to the 1980's), so even though its utility may be a bit "specialized", there are clearly people who are surprised to see it ignored by a tool like rsync.  IMO back-up/replication/sync tools should strive to record or replicate the "originals" as faithfully as possible[*], with minimal impact on the contents or meta-data of files being backed up or replicated.

That is to say, one should ideally be able to use rsync to do things like moving a directory tree to a new place (file system, machine), or copying/sync'ing it to a back-up location, and later restoring the originals, without the user of those files being able to discern any differences in them.

I built rsync with Nicolas George's O_NOATIME patch on the latest Git, and IMO this change brings rsync one step closer to that ideal (on Linux at least).  It not only avoids clobbering the atime values for the original files as they're being read during a first copy operation, but also when they're being checksummed (on both sides) if one uses the "-c" option in a later re-sync.

Martin von Gagern: That's the main reason that I think "--noatime" should be sent to the remote side.  I agree that in some/many cases, rsync users would not care about atimes being preserved in the copy.  For backups, supposing atimes had been copied to the backup machine[*] (perhaps not part of the scenario you had in mind), they would need to be preserved during re-syncs in order to be able to available when sync'd back to the original place during a restore.

One problem with unconditionally sending a new option to the remote side is that if it isn't supported on the remote side, it fails, whereas I would still see benefit (for some use cases) in using O_NOATIME on the local side only in that case.  Not sure how easy it would be to handle that situation.

Aside from that, I would really like to see this enhancement be integrated as soon as possible.  Since it's an option, and doesn't affect the protocol etc, it doesn't seem as though it would have any negative impact when it isn't being used.  Seems like a win-win. :)  Could it be integrated before the next release?


[*] Which brings us to the --atime patch -- but that's another topic.
Comment 7 cyril42e 2016-11-15 22:31:53 UTC
+1
I'm currently using a locally patched version, because I'm using rsync to mirror my whole data daily, plus using a deduplicating backup tool weekly that stores atime (so if rsync sets atime, the metadata are never deduplicated).
It is logical for a backup tool to avoid setting atime, and logical that it is an option for rsync that may or may not be used as a backup tool.
Comment 8 Michal Ruprich 2018-01-12 10:59:18 UTC
Is there any update on this feature? Is there a plan to merge the patch into master branch of rsync?
Comment 9 Michal Ruprich 2018-03-01 15:41:08 UTC
Did any of you who uses rsync with noatime patch had problems with running 'make check' command? It seems to me that some of the source files might be compiling in different order than with a simple make since the build ends up with undefined reference to `noatime' in the syscall.c file.
Comment 10 Michal Ruprich 2019-09-25 10:42:23 UTC
Hi,

I think that ideally you should decide what to do with this. This almost 10 years old bug is still here and it seems no one from the rsync developers paid any attention to this. If you are not planning to do anything, maybe you should just close it.

Best regards,
Michal
Comment 11 Samuel Henrique 2019-09-29 23:56:34 UTC
This patch has been being applied to the Debian packaging of rsync since 2014 (5 years now).

As Michal Ruprich said, this patch requires the build to be made in a certain way as it can lead to undefined references to noatime in syscall.c.

I stumbled upon this when converting the Debian packaging to use debhelper, we start having this problem when using it.

It would be nice to have this properly upstreamed.
Comment 12 Samuel Henrique 2019-10-08 22:24:01 UTC
The patch is currently introducing a regression on the target "test" (make test).

/usr/bin/ld: syscall.o: in function `do_open':
./syscall.c:206: undefined reference to `noatime'

It happens when the target "tls$(EXEEXT)" gets run.

I'm trying to debug it so we can have an updated noatime patch AND run make test when building rsync. Any help is appreciated.

It looks like currently neither Debian nor Fedora nor Redhat is making use of make test, I know this is the blocker on the Debian case.
Comment 13 Samuel Henrique 2019-10-10 00:05:08 UTC
Created attachment 15524 [details]
Adds --noatime option for 3.1.3

Updated the patch to fix "make test" with the help of
Paul Slootman.

I'm happy to say that we are now using this patch and running the tests
both at build time and in our pipelines (CI) on Debian.
Comment 14 Samuel Henrique 2019-10-10 00:09:39 UTC
Created attachment 15525 [details]
Adds --noatime option for 3.1.3

There were two typos in the previous patch, not a big deal
but I updated again to be 100%.
Comment 15 Michal Ruprich 2019-10-10 08:16:04 UTC
Thanks Samuel,

now I can enable tests during build in Fedora and keep the noatime option as well. Still though, it would be good to have upstream approval on this :/
Comment 16 Wayne Davison 2020-04-23 21:02:11 UTC
The --set-noatime option was just added to the main git (along with the --atimes option for copying access times).
Comment 17 Wayne Davison 2020-04-23 21:34:33 UTC
I just tweaked the option name to --open-noatime, which I think is clearer.