Bug 10074 - rsync reorders --from-files alphabetically
rsync reorders --from-files alphabetically
Product: rsync
Classification: Unclassified
Component: core
All All
: P5 enhancement
: ---
Assigned To: Wayne Davison
Rsync QA Contact
Depends on:
  Show dependency treegraph
Reported: 2013-08-08 19:57 UTC by Nicholas Andre
Modified: 2014-11-05 19:10 UTC (History)
1 user (show)

See Also:

Add the --unsorted option (1.44 KB, patch)
2013-08-12 06:00 UTC, Wayne Davison
no flags Details
Add the --unsorted option (1.81 KB, patch)
2013-08-13 16:27 UTC, Wayne Davison
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Nicholas Andre 2013-08-08 19:57:58 UTC
This is request to add a flag to request that rsync does not automatically sort --from-files alphabetically, due to its use case with LTFS media.

LTFS is a linear media with exceedingly slow seek times when compared with raw read/write speed. To do efficient reads, each file on the tape must be read off in the order it was written. Each file has an extended attribute (ltfs.startblock) which returns its exact location (startblock) on the tape. In this manner, you can easily obtain a list of files in the optimal order for reads (sorted, ascending, in order of their startblock).

rsync's behavior of copying all files on and off in alphabetical order works fine if all you ever do is one write, as the order is consistent. If you do anything else to that volume, however, (using, for example, rsync -u to copy additional files), it will copy them on in alphabetical order but multiple times. You will end up with n number of a-z writes distributed across 6 minutes of seek time.

Trying to restore alphabetically after doing more than one copy will result in one of the least optimal restore paths imaginable, with average read rates dropping by 50% in my case when compared with copying them one at a time using rsync. I, however, was copying fairly large files. Any number of smaller sizes or wider distribution of files, or both will decimate the performance. You could easily create a file structure for which rsync would blindly shoot back and fourth across the entire tape for every single file.

Basically: batch file copying with rsync, if it always restores alphabetically, is useless for LTFS volumes.

While I can write a python script to execute rsync on individual files, it would be nice to be able to pass an argument to rsync (--maintain-file-order) which would prevent rsync from taking my optimally ordered list and butchering the LTFS restore with an alphabetical sort.

--Nicholas Andre

(Reposted from http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=640492)
Comment 1 Wayne Davison 2013-08-12 05:57:59 UTC
It doesn't seem very hard to get something going that may meet your needs, but I don't think that I will add this as an official feature.  I'll attach a patch for 3.1.0 pre1.  To use it, you'll need to use --no-i-r (to avoid incremental-recursion reordering) and possibly --no-i-d (if you need to avoid implied directories).
Comment 2 Wayne Davison 2013-08-12 06:00:04 UTC
Created attachment 9124 [details]
Add the --unsorted option

This patch adds the --unsorted option, slightly extending the need_unsorted_flist flag variable that is used by the iconv code.
Comment 3 Wayne Davison 2013-08-13 16:27:05 UTC
Created attachment 9130 [details]
Add the --unsorted option

This version of the patch also insures that a remote rsync gets told about the --unsorted option.
Comment 4 Nicholas Andre 2013-08-14 20:59:17 UTC
(In reply to comment #3)
> Created attachment 9130 [details]
> Add the --unsorted option
> This version of the patch also insures that a remote rsync gets told about the
> --unsorted option.

Tested, works great! Thanks!
Comment 5 Christian Campbell 2013-09-17 04:56:39 UTC
Thank you, Wayne -- works great for me, too! I was delighted to find that a solution to my problem had so recently come into existence.  =-)

I'm curious why you don't think --unsorted should be an official feature. It's obviously critical for the OP's slow linear media problem, but it seems it would eliminate a great amount of time lost to needlessly wide-angle random seek distances even when reading from HDDs. (That's my use case, though I'm not profiling to prove the hypothesis.)
Comment 6 Dave Rice 2014-11-05 19:10:44 UTC
I agree with Christian Campbell this patch would be helpful as an official part of rsync and encourage that this be considered as there's more and more rsync users working with LTFS.
Dave Rice