Bug 13320 - file contents cause rsync to fail (with certains args and dir structure)
file contents cause rsync to fail (with certains args and dir structure)
Status: NEW
Product: rsync
Classification: Unclassified
Component: core
3.1.3
All All
: P5 normal
: ---
Assigned To: Wayne Davison
Rsync QA Contact
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2018-03-05 17:19 UTC by Dave Gordon
Modified: 2018-08-20 06:16 UTC (History)
2 users (show)

See Also:


Attachments
Simple although probably suboptimal fix (1.45 KB, patch)
2018-03-05 17:36 UTC, Dave Gordon
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Dave Gordon 2018-03-05 17:19:50 UTC
From xftroxgpx on mailing list, see

https://lists.samba.org/archive/rsync/2018-March/031507.html

script to reproduce:

#!/bin/bash

#tested to fail as below: ArchLinux's rsync-3.1.3-1-x86_64.pkg.tar.xz
#tested to fail as below: ArchLinux's rsync-3.1.3pre1-1-x86_64.pkg.tar.xz
#tested to work ok      : ArchLinux's rsync-3.1.2-8-x86_64.pkg.tar.xz

if test "$1" == "clean"; then
  rm -vrf destdir sourcedir sourcedir2 sourcedir3
  exit 0
fi

echo '!! test 1:'

mkdir -p destdir
mkdir -p sourcedir/a
#one \0 followed by a non-\0 (so, using a space) required:
echo -ne '\0 ' > sourcedir/a/b
#non zero size file required:
echo -ne 'c' > sourcedir/c
echo 'sourcedir' >/tmp/filesfrom.lst.tmp
rsync --recursive --perms --checksum --delay-updates --numeric-ids --preallocate --sparse --files-from=/tmp/filesfrom.lst.tmp -- ./ ./destdir/
#rsync: write failed on "/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/destdir/sourcedir/a/b": No such file or directory (2)
#rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3]

# ^ this happens first and any subsequent times!

echo '!! test 2:'

mkdir -p sourcedir2/a
#one \0 followed by a non-\0 (so, using an M) required:
echo -ne '\0M' > sourcedir2/a/b
#one \0 followed by a non-\0 (so, using an M) required:
echo -ne '\0M' > sourcedir2/c

#in order to see this error: (for file 'c')

#non-zero file size required and it must be prefixed by '.' aka dot
#echo -ne '1' > sourcedir2/.d
#XXX: ^ (un)comment, don't forget to ./go clean afterwards, then ./go

#otherwise, the error is for file "a/b"

echo 'sourcedir2' >/tmp/filesfrom.lst.tmp
rsync --recursive --perms --checksum --delay-updates --numeric-ids --preallocate --sparse --files-from=/tmp/filesfrom.lst.tmp -- ./ ./destdir/
#rsync: write failed on "/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/destdir/sourcedir2/a/b": No such file or directory (2)
#rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3pre1]

echo '!! test 3:' #same as 2 but an extra file '.d' exists!

mkdir -p sourcedir3/a
#one \0 followed by a non-\0 (so, using an M) required:
echo -ne '\0M' > sourcedir3/a/b
#one \0 followed by a non-\0 (so, using an M) required:
echo -ne '\0M' > sourcedir3/c

#non-zero file size required and it must be prefixed by '.' aka dot
echo -ne '1' > sourcedir3/.d
echo 'sourcedir3' >/tmp/filesfrom.lst.tmp
rsync --recursive --perms --checksum --delay-updates --numeric-ids --preallocate --sparse --files-from=/tmp/filesfrom.lst.tmp -- ./ ./destdir/
#rsync: write failed on "/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/destdir/sourcedir2/c": No such file or directory (2)
#rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3]

A pristine copy of this script can be found here: https://github.com/xftroxgpx/a3/blob/37ebff3e0fe9d294aeec899a082dc2c51c486eb4/system/Z575/OSes/3archlinux/on_baremetal/filesystem_now/archlinux/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/go

Just run ./go to start all 3 tests.

All 3 tests should fail with:
ArchLinux's rsync-3.1.3-1-x86_64.pkg.tar.xz
ArchLinux's rsync-3.1.3pre1-1-x86_64.pkg.tar.xz

All 3 tests will succeed with ArchLinux's rsync-3.1.2-8-x86_64.pkg.tar.xz
(aka rsync  version 3.1.2  protocol version 31)
Comment 1 Dave Gordon 2018-03-05 17:30:46 UTC
Bug will be triggered if:

options include both --sparse and --preallocate, AND
the second (or subsequent) file to be copied begins with one or more zero bytes.

.Dave.
Comment 2 Dave Gordon 2018-03-05 17:36:20 UTC
Created attachment 14018 [details]
Simple although probably suboptimal fix
Comment 3 Dave Gordon 2018-03-05 17:37:54 UTC
Problem introduced by
    commit f3873b3d88b61167b106e7b9227a20147f8f6197
    Author: Wayne Davison <wayned@samba.org>
    Date:   Mon Oct 10 11:49:50 2016 -0700

    Support --sparse combined with --preallocate or --inplace.

    The new code tries to punch holes in the destination file using newer
    Linux fallocate features. It also supports a --whole-file + --sparse +
    --inplace copy on any filesystem by truncating the destination file.

Root cause is that the new variable 'sparse_past_write' is not reset
when switching files, thus leading to miscalculation of the current
seek position in do_punch_hole().

Quick fix is to reset this variable to the current seek position on
each call to write_sparse() - probably not the most efficient fix,
but it seems to work.

.Dave.
Comment 4 Marcus Linsner 2018-08-20 05:02:20 UTC
Thanks for this fix Dave, worksforme!