The Samba-Bugzilla – Bug 10074
rsync reorders --from-files alphabetically
Last modified: 2014-11-05 19:10:44 UTC
This is request to add a flag to request that rsync does not automatically sort --from-files alphabetically, due to its use case with LTFS media.
LTFS is a linear media with exceedingly slow seek times when compared with raw read/write speed. To do efficient reads, each file on the tape must be read off in the order it was written. Each file has an extended attribute (ltfs.startblock) which returns its exact location (startblock) on the tape. In this manner, you can easily obtain a list of files in the optimal order for reads (sorted, ascending, in order of their startblock).
rsync's behavior of copying all files on and off in alphabetical order works fine if all you ever do is one write, as the order is consistent. If you do anything else to that volume, however, (using, for example, rsync -u to copy additional files), it will copy them on in alphabetical order but multiple times. You will end up with n number of a-z writes distributed across 6 minutes of seek time.
Trying to restore alphabetically after doing more than one copy will result in one of the least optimal restore paths imaginable, with average read rates dropping by 50% in my case when compared with copying them one at a time using rsync. I, however, was copying fairly large files. Any number of smaller sizes or wider distribution of files, or both will decimate the performance. You could easily create a file structure for which rsync would blindly shoot back and fourth across the entire tape for every single file.
Basically: batch file copying with rsync, if it always restores alphabetically, is useless for LTFS volumes.
While I can write a python script to execute rsync on individual files, it would be nice to be able to pass an argument to rsync (--maintain-file-order) which would prevent rsync from taking my optimally ordered list and butchering the LTFS restore with an alphabetical sort.
(Reposted from http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=640492)
It doesn't seem very hard to get something going that may meet your needs, but I don't think that I will add this as an official feature. I'll attach a patch for 3.1.0 pre1. To use it, you'll need to use --no-i-r (to avoid incremental-recursion reordering) and possibly --no-i-d (if you need to avoid implied directories).
Created attachment 9124 [details]
Add the --unsorted option
This patch adds the --unsorted option, slightly extending the need_unsorted_flist flag variable that is used by the iconv code.
Created attachment 9130 [details]
Add the --unsorted option
This version of the patch also insures that a remote rsync gets told about the --unsorted option.
(In reply to comment #3)
> Created attachment 9130 [details]
> Add the --unsorted option
> This version of the patch also insures that a remote rsync gets told about the
> --unsorted option.
Tested, works great! Thanks!
Thank you, Wayne -- works great for me, too! I was delighted to find that a solution to my problem had so recently come into existence. =-)
I'm curious why you don't think --unsorted should be an official feature. It's obviously critical for the OP's slow linear media problem, but it seems it would eliminate a great amount of time lost to needlessly wide-angle random seek distances even when reading from HDDs. (That's my use case, though I'm not profiling to prove the hypothesis.)
I agree with Christian Campbell this patch would be helpful as an official part of rsync and encourage that this be considered as there's more and more rsync users working with LTFS.