Bug 10495 - "skipping directory foo" (does not transfer directories by default)
Summary: "skipping directory foo" (does not transfer directories by default)
Status: RESOLVED WONTFIX
Alias: None
Product: rsync
Classification: Unclassified
Component: core (show other bugs)
Version: 3.1.0
Hardware: All All
: P5 major (vote)
Target Milestone: ---
Assignee: Wayne Davison
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-03-11 16:24 UTC by Filipus Klutiero
Modified: 2014-03-25 19:41 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Filipus Klutiero 2014-03-11 16:24:10 UTC
rsync's manual does not specify its usage. However, examples are provided in the USAGE section. Unfortunately, there is a huge error in the very first example:

rsync -t *.c foo:src/ 


 This would transfer all files matching the pattern *.c from the current directory to the directory src on the machine foo. If any of the files already exist on the remote system then the rsync remote-update protocol is used to update the file by sending only the differences. See the tech report for details. 

In fact, only the files matching the pattern and with the right type are copied. I do not know which types are copied by default, but I know that directories are not (unless this would be a Debian-specific behavior). If --dirs is not specified, trying the above will throw a "skipping directory x" for all matching directories.

By the way, it is far from obvious which "tech report" this description refers to.
Comment 1 Wayne Davison 2014-03-14 00:03:46 UTC
The example is copying *.c files, not directories.  The manpage clearly states "all files matching the pattern *.c".
Comment 2 Kevin Korb 2014-03-14 00:12:53 UTC
BTW, there is also a recursive example in the include/exclude pattern rules:
>  The combination  of  "+ */", "+ *.c", and "- *" would include all directories and C source files but nothing else (see also the --prune-empty-dirs option)

In other words:
rsync -rt --prune-empty-dirs --include="*/" --include="*.c" --exclude="*" /path/to/source/ /path/to/target/
will copy all *.c files and the directories leading to them in an entire tree.
Comment 3 Filipus Klutiero 2014-03-14 03:05:10 UTC
Files and directories are not exclusive. Directories are files, as the base specifications explain.
Comment 4 Kevin Korb 2014-03-14 03:16:04 UTC
Directories are only followed if you use -r aka --recursive.  Note that in my example I did so.  However, if you added -r to your example you would get any directory in the current directory named *.c recursively.  IOW it would recursively copy a directory named foo.c but would not copy foo/bar.c

Simply put, *.c as the source is expanded by the shell globbing.  Rsync is actually given a list of every file/directory that matches *.c.  Then without --recursive it ignores the ones that are directories.  Without --specials it ignores those too.
Comment 5 Filipus Klutiero 2014-03-14 03:36:27 UTC
I understand that --recursive will consider directories. The problem is that the example doesn't contain --recursive.

I managed to get my backup working, I was just pointing out the problem in the documentation (as your answers show, the problem is not an unexpected rsync behavior I was getting, it's just the documentation).
Comment 6 Kevin Korb 2014-03-14 03:41:18 UTC
Because the user probably doesn't want *.c to match files and directories.  If they want *.c recursively they probably need the include/exclude example that is listed elsewhere.  This is exactly how cp behaves btw.
Comment 7 Filipus Klutiero 2014-03-14 04:39:59 UTC
Sorry if I was unclear, but I didn't mean to say that rsync's behavior should be changed. I do believe that users would expect rsync to transfer directories by default, but this doesn't necessarily mean that changing the default is desirable at this point. Indeed, cp has the same issue as rsync, so changing one without the other would reduce consistency.

What I meant to say was that the documentation is misleading. If directories are not copied by default, the manpage should probably have a fat warning, but at least, it should not mislead readers.

The example description should probably say that only regular files will be transferred, or that all files other than directories will be transferred. Or something more complicated (I do not know rsync's default behavior). For sure, it would be more accurate to say "all files other than directories".
Comment 8 Wayne Davison 2014-03-14 06:38:48 UTC
That example you cited shows how to copy some c files to a remote src dir.  There are no dirs involved, thus -r is not needed.
Comment 9 Filipus Klutiero 2014-03-14 15:23:38 UTC
Ths example is clearly intended to copy C files, but that's not the description given:
 This would transfer all files matching the pattern *.c from the current directory to the directory src on the machine foo.
Comment 10 Wayne Davison 2014-03-14 23:20:01 UTC
... which is exactly what it does.
Comment 11 Filipus Klutiero 2014-03-16 17:15:28 UTC
No, [it] would transfer *some* files matching the pattern *.c from the current
directory to the directory src on the machine foo.

At least directories will not be transferred.
Comment 12 Paul Slootman 2014-03-19 06:57:03 UTC
(In reply to comment #11)
> No, [it] would transfer *some* files matching the pattern *.c from the current
> directory to the directory src on the machine foo.
> 
> At least directories will not be transferred.

I think you're confusing "current directory" with "current directory and all subdirectories".

The example is clear.
Comment 13 Filipus Klutiero 2014-03-22 05:18:35 UTC
That's not what I meant. Here is what this example will do on a simple directory having 2 files matching *.c:

$ ls -lR
.:
total 4
drwxr-xr-x 2 test test 4096 mar 22 01:04 bar.c
-rw-r--r-- 1 test test    0 mar 22 01:04 foo.c

./bar.c:
total 0
test@vinci:~/testsrc$ rsync -t *.c ~/testdst/ 
skipping directory bar.c
test@vinci:~/testsrc$ ls -lR ~/testdst/
/home/test/testdst/:
total 0
-rw-r--r-- 1 test test 0 mar 22 01:04 foo.c
test@vinci:~/testsrc$

All files in testsrc/ are directory entries of testsrc/. Yet, after the transfer, even though all of these files match *.c, only some have been transferred. bar.c was not transferred, since it is a directory and rsync does not transfer directories unless some options which weren't given are given.
Comment 14 Wayne Davison 2014-03-22 06:56:32 UTC
People don't name their directories with .c suffixes.
Comment 15 Filipus Klutiero 2014-03-22 20:54:31 UTC
People name directories in many ways. Almost all UNIX systems will have directories matching *.d. The very first system I connected to has hundreds of matches for locate *.c/. Is that enough to convince you to leave this ticket alone until the bug has been fixed? If you'd rather spend more time in a way which doesn't help rsync, you'll have to do that without me.
Comment 16 Wayne Davison 2014-03-25 05:35:36 UTC
Examples are not the correct place to get into minutia details.  The example as given is correct for what it is trying to convey.
Comment 17 Paul Slootman 2014-03-25 10:55:29 UTC
(In reply to comment #13)
> That's not what I meant. Here is what this example will do on a simple
> directory having 2 files matching *.c:
> 
> $ ls -lR
> .:
> total 4
> drwxr-xr-x 2 test test 4096 mar 22 01:04 bar.c
> -rw-r--r-- 1 test test    0 mar 22 01:04 foo.c

Your example is wrong, there are NOT 2 files matching *.c;
there is 1 file and 1 directory.

Get over it.
Comment 18 Filipus Klutiero 2014-03-25 19:41:12 UTC
Paul, a directory is a file. The directory has 2 files matching *.c and 1 directory matching *.c.


Wayne, the problem isn't that the command given is invalid. The problem is that the description is incorrect and leads readers to craft commands which won't do what the reader expects. Every single example would be a waste of time if it wasn't for the fact that examples inspire readers to adapt rsync to their needs. Unfortunately, the first example is a misleading inspiration.