Bug 2947 - stdout with [-v] -H --link-dest and slink/sock/fifo/regf
stdout with [-v] -H --link-dest and slink/sock/fifo/regf
Status: CLOSED FIXED
Product: rsync
Classification: Unclassified
Component: core
2.6.5
All FreeBSD
: P3 normal
: ---
Assigned To: Wayne Davison
Rsync QA Contact
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2005-08-02 01:00 UTC by hoffa
Modified: 2006-03-12 02:56 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description hoffa 2005-08-02 01:00:28 UTC
on the first time the destdir is populated by reference to the --link-dest...
 regfiles are listed upon create/mtime but not delete in ./0/.
  should be all three.
 slinks/sockets/fifos are always listed even if no change in ./0/.
  should be only upon create/mtime/delete.

to repeat...
use the familiar srcdir full of slink/sock/fifo/regf types
 in both hardlinked/non-hardlinked versions.
sudo rsync -Haxv --delete --link-dst=`pwd`/1/ ./0/ ./2/
play with create/mtime/delete of slink/sock/fifo/regf in ./0/
increment link-dest and dest dir indexes and rerun
examine rsync output
repeat with next index

a duplicate rerun with the same indexes and no play is clean as it should be.
a duplicate rerun with the same indexes and play is properly listed.
use without -v is similarly weird in respect to the above tests.
not sure the output needs to differ based on type anymore but rather
be consistent instead. hope this helps.
Comment 1 Wayne Davison 2005-08-03 16:45:50 UTC
Keep in mind that --link-dest only hard-links regular files, and nothing else
(and I want to keep it that way, since not all systems can hard-link things such
as symlinks, and there's not that much of a savings in hard-linking devices and
symlinks anyway).  Also, while rsync's output treats files that are already
up-to-date in a link-dest dir just like they were up-to-date in the destination
dir (i.e. it doesn't output their names as transferred files without an extra
-v), rsync does not currently consider the contents of the link-dest dirs when
it is deleting (though that might be interesting to do so, it could get very
complicated when using multiple --link-dest options).

So, the only thing that I see that I'd consider improving is the output of a
hard-linked file that had an up-to-date counterpart in the link-dest dir and a
linked-"buddy" in the current destination hierarchy -- in that case rsync
outputs that it is hard-linking the latter files into cluster to the first file
of the cluster, when it should arguably be silent about that file.
Comment 2 hoffa 2005-08-03 23:36:46 UTC
>link-dest only hard-links regular files...savings in hard-linking [non-regular
files]
 yeah, they all take the same directory space, any savings would be in inode
count.
>multiple --link-dest options
 never used this, can someone using this post a useful context?
 isn't this just an OR of reference hiers that result in a final same/notsame
 decision to print/do? if it's hard to code maybe caveat support to just one
 link-dest option?

hmm, maybe put another way... it would be nice to be able to run
 rsync -nHaxv --delete --link-dest=`pwd`/1/ ./0/ ./2/
and see only what the changes would be. then run without -n and have it print
and do exactly the same. right now it says [by printing output] that, hey, i'm
going to make a bunch of new slink/socket/fifo/device and hardlinks of same/regf
in ./2/ [as if they were newly created in ./0/ post when ./1/ was created
earlier,
even though that is false]...and by the way, i might tell you about that
new/modified regfile that showed up in ./0/ [post ./1/], but i'm not going to
warn/tell you that file disappeared. that behaviour seems wrong...er, point is
to
accurately say what will be and is done between src and dst, with or without
the link-dest reference. create/mtime_change/delete should be what's printed
with one -v regardless of S_IFMT type. as is done today with
 rsync -Haxv --delete ./src/ ./dst/
where the src contains only regular files and dirs with no hardlinks. the
nothing
but errors without -v is great idea.
is this more helpful to anyone?

example:

ls -liRT
240876 drwxr-xr-x  2 tooba  wheel  512 Aug  3 22:04:30 2005 0
./0:
240865 -rw-r--r--  1 tooba  wheel  0 Aug  3 22:04:23 2005 a
240855 -rw-r--r--  2 tooba  wheel  0 Aug  3 21:51:56 2005 h1
240855 -rw-r--r--  2 tooba  wheel  0 Aug  3 21:51:56 2005 h2

rsync -Haxv --delete  ./0/ ./1/         <-- make first copy, lev0
created directory ./1
./
a               <-- ok
h2              <-- ok
h1 => h2        <-- ok

240876 drwxr-xr-x  2 tooba  wheel  512 Aug  3 22:04:30 2005 0
240850 drwxr-xr-x  2 tooba  wheel  512 Aug  3 22:04:30 2005 1
./0:
240865 -rw-r--r--  1 tooba  wheel  0 Aug  3 22:04:23 2005 a
240855 -rw-r--r--  2 tooba  wheel  0 Aug  3 21:51:56 2005 h1
240855 -rw-r--r--  2 tooba  wheel  0 Aug  3 21:51:56 2005 h2
./1:
240856 -rw-r--r--  1 tooba  wheel  0 Aug  3 22:04:23 2005 a
240857 -rw-r--r--  2 tooba  wheel  0 Aug  3 21:51:56 2005 h1
240857 -rw-r--r--  2 tooba  wheel  0 Aug  3 21:51:56 2005 h2

rsync -Haxv --delete --link-dest=`pwd`/1/ ./0/ ./2/  <-- first referential copy
lev1
created directory ./2
./              <-- ignoring these at the moment, not as important
h2 => h1        <-- no need to know, this is not a difference between 0 and 2

rm 0/a          <-- now nuke this
rsync -Haxv --delete --link-dest=`pwd`/2/ ./0/ ./3/
created directory ./3
./
h2 => h1        <-- no need to know, this is not a difference between 0 and 2
                <-- but hey, where'd my informative 0/a deletion printf go...

touch 0/b       <-- now create this
rsync -Haxv --delete --link-dest=`pwd`/3/ ./0/ ./4/
created directory ./4
./
b               <-- ... because i correctly tell you of create/update_mtime here
h2 => h1        <-- no need to know, this is not a difference between 0 and 2

rm 0/h1         <-- now nuke this
rsync -Haxv --delete --link-dest=`pwd`/4/ ./0/ ./5/
created directory ./5
./
                <-- but hey, where'd my informative 0/h1 deletion printf go...

ln 0/h2 0/h3    <-- now create this
rsync -Haxv --delete --link-dest=`pwd`/5/ ./0/ ./6/
created directory ./6
./
h3 => h2        <-- ok, this is a true difference


if using the usual full test srcdir as ./0/, i get...
240903 lrw-rw----  1 tooba  wheel         1 Aug  2 00:47:11 2005 a -> b
240912 -rw-rw----  1 tooba  wheel         0 Jul 31 22:39:05 2005 b
240904 prw-rw----  1 tooba  wheel         0 Aug  4 02:03:22 2005 fifo0
240905 prw-rw----  2 tooba  wheel         0 Aug  2 00:47:06 2005 fifohard1
240905 prw-rw----  2 tooba  wheel         0 Aug  2 00:47:06 2005 fifohard2
240878 -rw-rw----  1 tooba  wheel         0 Aug  4 02:03:22 2005 file0
240914 -rw-rw----  2 tooba  wheel         0 Jul 31 22:58:45 2005 filehard1
240914 -rw-rw----  2 tooba  wheel         0 Jul 31 22:58:45 2005 filehard2
240906 brw-rw----  1 tooba  wheel    2,   2 Aug  4 02:03:22 2005 nullb0
240907 brw-rw----  2 tooba  wheel    2,   2 Aug  4 01:48:48 2005 nullbhard1
240907 brw-rw----  2 tooba  wheel    2,   2 Aug  4 01:48:48 2005 nullbhard2
240908 crw-rw----  1 tooba  wheel    2,   2 Aug  4 02:03:22 2005 nullc0
240909 crw-rw----  2 tooba  wheel    2,   2 Aug  4 01:50:01 2005 nullchard1
240909 crw-rw----  2 tooba  wheel    2,   2 Aug  4 01:50:01 2005 nullchard2
240910 srw-rw----  1 tooba  wheel         0 Aug  4 02:03:22 2005 socket0
240911 srw-rw----  2 tooba  wheel         0 Aug  4 02:05:00 2005 sockethard1
240911 srw-rw----  2 tooba  wheel         0 Aug  4 02:05:00 2005 sockethard2

rsync -Hax --delete ./0/ ./1/
rsync -Haxvi --delete --link-dest=`pwd`/1/ ./0/ ./2/
created directory ./2
.d..t.o.. ./
cL+++++++ a -> b
cD+++++++ fifo0
cD+++++++ fifohard2
hD+++++++ fifohard1 => fifohard2
hf+++++++ filehard2 => filehard1        <-- note reversal of 1 => 2 number
cD+++++++ nullb0                             order compared to first copy wayyy
cD+++++++ nullbhard2                         up ^^^^ there ;-]
hD+++++++ nullbhard1 => nullbhard2
cD+++++++ nullc0
cD+++++++ nullchard2
hD+++++++ nullchard1 => nullchard2
cD+++++++ socket0
cD+++++++ sockethard2
hD+++++++ sockethard1 => sockethard2

none of which have actually changed.

for the reader...
-H used here as a standard addition to -a, for more identical mirroring.
 with that, df -i also happens to match up between mirrors as an added quick
check.
the -H preserves the link relationships that exist between names in ./0/
 through to the mirror in ./2/. link-dest just saves space between ./1/
 and ./2/. with millions of files, each consuming more than their directory
space,
 link-dest is just plain cool;-]
sockets/[bc]devices/fifos consume zero bytes, slinks/dirs some bytes. and
regulars
 use a lot in comparison.
being careful discussing 'devices' is important as block and char devices are
 different from each other and from sockets, symlinks and fifos which are all
 independant types and NOT devices proper. see stat(2).
imo, rsync has done an excellent job of becoming aware of the seven common file
 types in the past year or two with just a little more to go. ie: log-format,
 a type section in the man page, this series of hopefully useful reports.
 hats off to the coders.

Comment 3 Wayne Davison 2005-12-15 21:26:42 UTC
I have checked in fixes for several of the items that you mentioned:

* Rsync now keeps quiet about an item in a hard-link cluster if it was properly hard-linked in the --link-dest dir.

* Rsync now treats fifos/devices/symlinks the same as files when it is running with --link-dest/--compare-dest/--copy-dest (I changed my mind on this point).  This means that if those items are already up-to-date in a --FOO-dest dir, that rsync won't mention it being in need of an update (and will hard-link it to the prior version in the case of --link-dest and not create a new version at all in the case of --compare-dest).

As mentioned in my first response, the delete issues you cited are not bugs: rsync only mentions deletions from the actual destination directory, not extra items that might be found in one or more of the adjunct --FOO-dest dirs.  Yes, this does mean that using --delete along with --link-dest is usually useless (because --link-dest works best into an empty hierarchy, and there's nothing to delete in an empty hierarchy).