Created attachment 11733 [details]
reenable tarmode verbose, and skip inaccessible files

Besides this problem, there is a second glitch in the new
TAR stuff.  In 3.6.1 tar had the problem that it it could
no access a file, (busy etc.) it skipped the remainder of the directory.
(Bug report 10605.)

In 4.3.1 if it can't access a file, instead of just skipping the
file, it writes a tar element of all zeros, making it look like
is did back up something.

I'm attaching a "diff -c" with my changes to renable
   tarmode verbose
and properly skip inaccessible files.

Unlike my proposed patches for 10605, this time there is no
   tarmode errorskip

Created attachment 11734 [details]
Modified clitar.c for tarmode verbose

this is just the whole clitar.c that I'm now using.

Comment on attachment 11733 [details]
reenable tarmode verbose, and skip inaccessible files

Please submit the patch as a git commit, as that will use the accepted 'diff -u' format, using 

git commit -s -m 'A useful commit message'

git format-patch -s -1

https://wiki.samba.org/index.php/Using_Git_for_Samba_Development has some further information. 

We also need a test, as otherwise we will just regress again in the future.

Well I tried installing GIT and doing the clone and it fails
with a network timewout.

I already attached my modified clitar.c so it would seem much easier
if you just used that to do the GIT stuff since clearly GIT works for you.

As for a test.
1) create a Windows directory with at least 3 files
   mkdir junk
   echo >junk/x1 x1
   echo >junk/x2 x2
   echo >junk/x3 x3
2) remove all perms from and intermediate file
   chmod 000 junk/x2
3) on the linux system create a tar file of the test directory
   e.g. 
   smbclient //oliver/c\$ -A pw -D /home/atbowler/junk -Tc xxx.tar

The output should look something like this
atbowler@dory:~$ smbclient //oliver/c\$ -A pw -D /home/atbowler/junk -Tc xxx.tar
Domain=[THINKAGE] OS=[Windows 8.1 Enterprise 9600] Server=[Windows 8.1 Enterprise 6.3]
           3 (    2.9 kb/s) \home\atbowler\junk\x1
NT_STATUS_ACCESS_DENIED opening remote file \home\atbowler\junk\x2
           3 (    2.9 kb/s) \home\atbowler\junk\x3
tar: dumped 2 files and 0 directories
Total bytes written: 9

and a tar listing should look like
atbowler@dory:~$ tar tvf xxx.tar
-rw-r--r-- 0/0               3 2015-12-21 16:13 ./home/atbowler/junk/x1
-rw-r--r-- 0/0               3 2015-12-21 16:13 ./home/atbowler/junk/x3

Note that "x2" file was skipped and "x3" did get included

the lines like 
   3 (    2.9 kb/s) \home\atbowler\junk\x1
   3 (    2.9 kb/s) \home\atbowler\junk\x3
and summary lines
  tar: dumped 2 files and 0 directories
  Total bytes written: 9

Are what is important for BackUpPC.  Clearly the exact transfer
speed need not mach from test to test

To be clear, by a test I mean a patch to our 'make test'.  

Specifically source3/script/tests/test_smbclient_tarmode.sh

Please persist in getting the git patches, it really does make things easier for us.

You might try cloning https://git.samba.org/samba.git if your firewall blocks git://, or else see our github page at https://github.com/samba-team/samba

Thanks!

never mind, I have the change for the verbose more ready here, just testing some and getting it out later on.

Alan: can you tell me in which way BackupPC insists on the format of the verbose output of smbclient here? I'd like actually to make the output fit more to the output of a "v" or "vv" tar command.

A more standard tar conforming output would in the long term probably also help any other application using the tar mode. Worst case BackupPC might have to be modified if we change the output of the verbose mode.

I've been out of town over the Christmas period and had
other stuff to do first when I got back.

I should point out that I'm not really a expert on Linux or Windows.
I primarily do system and programmer tools for GCOS.
(GCOS is an old mainframe system and one of the ancestors of Unix).
I have not done serious Unix programming since Unix V6 (around 1976),
and sort of inherited our backup setup when someone else left the company.

BackupPC is written in Perl, and this is the first time I've ever
looked at any Perl.

As far as I can tell BackupPC uses the follwing code to
recognize when it has received a file and count up to
number of files, and file space, and get a file name.
(Although the name might just be used in the log).

if ( /^\s*(-?\d+) \(\s*\d+[.,]\d kb\/s\) (.*)$/ ) {
    my $sambaFileSize = $1;
    my $pcFileName    = $2;
    (my $fileName = $pcFileName) =~ s/\\/\//g;
    $sambaFileSize += 1024 * 1024 * 4096 if ( $sambaFileSize < 0 );
    $fileName =~ s/^\/*//;
    $t->{byteCnt} += $sambaFileSize;
    $t->{fileCnt}++;
    $t->{XferLOG}->write(\"$_\n") if ( $t->{logLevel} >= 2 u);

note that in parsing a line like
     3 (    2.9 kb/s) \home\atbowler\junk\x1
it is discarding the "(    2.9 kb/s)".
If at the end of the Tar operation the "fileCnt" is non-zero
it concludes the operation whas failed.

It also checks for  a line like
    tar: dumped 2 files and 0 directories
with the code

} elsif ( /^\s*tar: dumped \d+ files/ ) {
    $t->{xferOK} = 1;
    $t->{XferLOG}->write(\"$_\n") if ( $t->{logLevel} >= 0 );

I really don't know how I could set up an automated test in
  source3/script/tests/test_smbclient_tarmode.sh

For smbclient to handle a denial on a TAR, and continue
processing the rest of the directory.  Our Linux system is not
configured as an SMB server, so I can't run a test where I
point smbclient back at a directory on the same machine
to do a test like I described in Comment 4

okay, I think it is a bit unrealistic if BackupPC expects that the syntax never change. Even a move from kb/s to kiB/s in the output would break that parsing.

I think now that the verbose output got broken anyway, it is a good moment to fix it and to make the output a bit more tar standard like as in SUSv2.

a /etc/X11/xim 4515 bytes, 9 tape blocks
a /etc/X11/xkb/ directory

For adding files to archive a leading "a" followed by the filename, followed by any free formed aditional information. Programs would then also be able to use standard tar output parsers, I guess also BackupPC already contains parsing rules for that.

Actually, the comments in BackupPC do acknowledge that the
patterns it looks for will likely need to be updated to match
new versions of smbclient.  I think I could modify the copy we
run to cope with the verbose output like you suggest.  What is
important, is that file and directory names as well as the
file sizes be output, and that a summary line with the info
like
    tar: dumped 4 files and 1 directories
be available.

The TAR "vv" output might not be a good idea.
Windows group names may or may not have spaces in them
making it harder to pick out the fields.

Your
   
a /etc/X11/xim 4515 bytes, 9 tape blocks
a /etc/X11/xkb/ directory

style could be easily handled.

Created attachment 12263 [details]
updted context diff for the 4.4.4 version of clitar.c

The changes do the same as the previous context diff,
but are based on Samba 4.4.4.
tarmode verbose is default (as per the smbclient man page)
tarmode verbose does generate outout (as opposed to nothing),
  and the verbose output is compatible with what backuppc expects.

fixed in master and in the upcoming 4.12 with d512b27563b32f7bcb35da21781e87473830c906