Bug 11642 - tarmode verbose is ignored in libarchive version of smbclient
tarmode verbose is ignored in libarchive version of smbclient
Status: NEW
Product: Samba 4.1 and newer
Classification: Unclassified
Component: Other
4.3.1
All Linux
: P5 regression
: ---
Assigned To: Björn Jacke
Samba QA Contact
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-11 21:53 UTC by Alan Bowler
Modified: 2016-07-06 20:13 UTC (History)
3 users (show)

See Also:


Attachments
reenable tarmode verbose, and skip inaccessible files (4.50 KB, patch)
2015-12-18 19:06 UTC, Alan Bowler
no flags Details
Modified clitar.c for tarmode verbose (41.20 KB, text/plain)
2015-12-18 19:09 UTC, Alan Bowler
no flags Details
updted context diff for the 4.4.4 version of clitar.c (6.16 KB, patch)
2016-07-06 20:13 UTC, Alan Bowler
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Alan Bowler 2015-12-11 21:53:56 UTC
In 3.6.23 (before libarchive) the tar command in verbose mode (default)
would output a line for each file and directory that was written to the
archive.  

BackUpPC looks for these messages.
The session below shows this effect even if it is not exactly
the command BackUpPC is issuing on our system.   (Debian)

atbowler@dory:~$
atbowler@dory:~$ smbclient --version
Version 3.6.23
atbowler@dory:~$ smbclient //oliver/c\$ -D /home/atbowler/backtest -A pw -c "tarmode verbose" -Tc xxx.tar
Unknown parameter encountered: "realm"
Ignoring unknown parameter "realm"
WARNING: Ignoring invalid value 'ADS' for parameter 'security'
Domain=[THINKAGE] OS=[Windows 8.1 Enterprise 9600] Server=[Windows 8.1 Enterprise 6.3]
tarmode is now full, system, hidden, noreset, verbose, noerrorskip
                directory \home\atbowler\backtest\dir\
           7 (    3.4 kb/s) \home\atbowler\backtest\dir\x4.txt
           3 (    1.5 kb/s) \home\atbowler\backtest\x1
          30 (   14.6 kb/s) \home\atbowler\backtest\x2.txt
          30 (   14.6 kb/s) \home\atbowler\backtest\x3.txt
tar: dumped 5 files and directories
Total bytes written: 2048
atbowler@dory:~$ smbclient //oliver/c\$ -D /home/atbowler/backtest -A pw -c "tarmode quiet" -Tc xxx.tar
Unknown parameter encountered: "realm"
Ignoring unknown parameter "realm"
WARNING: Ignoring invalid value 'ADS' for parameter 'security'
Domain=[THINKAGE] OS=[Windows 8.1 Enterprise 9600] Server=[Windows 8.1 Enterprise 6.3]
tarmode is now full, system, hidden, noreset, quiet, noerrorskip
tar: dumped 5 files and directories
Total bytes written: 2048

(note the "errorskip" tarmode is a result of small local modifications
 Ref report 10605)

smbclient in 4.3.1  ignores the verbose mode 
E.G.

atbowler@dory:~$ work-samba/bin/smbclient --version
Version 4.3.1
atbowler@dory:~$ work-samba/bin/smbclient //oliver/c\$ -D /home/atbowler/backtest -A pw -c "tarmode verbose" -Tc xxx.tar
WARNING: The "syslog" option is deprecated
Domain=[THINKAGE] OS=[Windows 8.1 Enterprise 9600] Server=[Windows 8.1 Enterprise 6.3]
tarmode is now full, system, hidden, noreset, verbose
tar: dumped 4 files and 1 directories
Total bytes written: 70
atbowler@dory:~$ work-samba/bin/smbclient //oliver/c\$ -D /home/atbowler/backtest -A pw -c "tarmode quiet" -Tc xxx.tar
WARNING: The "syslog" option is deprecated
Domain=[THINKAGE] OS=[Windows 8.1 Enterprise 9600] Server=[Windows 8.1 Enterprise 6.3]
tarmode is now full, system, hidden, noreset, quiet
tar: dumped 4 files and 1 directories
Total bytes written: 70
atbowler@dory:~$

Note that there is no list of files in the verbose setting.
Comment 1 Alan Bowler 2015-12-18 19:06:11 UTC
Created attachment 11733 [details]
reenable tarmode verbose, and skip inaccessible files

Besides this problem, there is a second glitch in the new
TAR stuff.  In 3.6.1 tar had the problem that it it could
no access a file, (busy etc.) it skipped the remainder of the directory.
(Bug report 10605.)

In 4.3.1 if it can't access a file, instead of just skipping the
file, it writes a tar element of all zeros, making it look like
is did back up something.

I'm attaching a "diff -c" with my changes to renable
   tarmode verbose
and properly skip inaccessible files.

Unlike my proposed patches for 10605, this time there is no
   tarmode errorskip
Comment 2 Alan Bowler 2015-12-18 19:09:29 UTC
Created attachment 11734 [details]
Modified clitar.c for tarmode verbose

this is just the whole clitar.c that I'm now using.
Comment 3 Andrew Bartlett 2015-12-18 20:12:48 UTC
Comment on attachment 11733 [details]
reenable tarmode verbose, and skip inaccessible files

Please submit the patch as a git commit, as that will use the accepted 'diff -u' format, using 

git commit -s -m 'A useful commit message'

git format-patch -s -1

https://wiki.samba.org/index.php/Using_Git_for_Samba_Development has some further information. 

We also need a test, as otherwise we will just regress again in the future.
Comment 4 Alan Bowler 2015-12-21 21:27:41 UTC
Well I tried installing GIT and doing the clone and it fails
with a network timewout.

I already attached my modified clitar.c so it would seem much easier
if you just used that to do the GIT stuff since clearly GIT works for you.

As for a test.
1) create a Windows directory with at least 3 files
   mkdir junk
   echo >junk/x1 x1
   echo >junk/x2 x2
   echo >junk/x3 x3
2) remove all perms from and intermediate file
   chmod 000 junk/x2
3) on the linux system create a tar file of the test directory
   e.g. 
   smbclient //oliver/c\$ -A pw -D /home/atbowler/junk -Tc xxx.tar

The output should look something like this
atbowler@dory:~$ smbclient //oliver/c\$ -A pw -D /home/atbowler/junk -Tc xxx.tar
Domain=[THINKAGE] OS=[Windows 8.1 Enterprise 9600] Server=[Windows 8.1 Enterprise 6.3]
           3 (    2.9 kb/s) \home\atbowler\junk\x1
NT_STATUS_ACCESS_DENIED opening remote file \home\atbowler\junk\x2
           3 (    2.9 kb/s) \home\atbowler\junk\x3
tar: dumped 2 files and 0 directories
Total bytes written: 9

and a tar listing should look like
atbowler@dory:~$ tar tvf xxx.tar
-rw-r--r-- 0/0               3 2015-12-21 16:13 ./home/atbowler/junk/x1
-rw-r--r-- 0/0               3 2015-12-21 16:13 ./home/atbowler/junk/x3

Note that "x2" file was skipped and "x3" did get included

the lines like 
   3 (    2.9 kb/s) \home\atbowler\junk\x1
   3 (    2.9 kb/s) \home\atbowler\junk\x3
and summary lines
  tar: dumped 2 files and 0 directories
  Total bytes written: 9

Are what is important for BackUpPC.  Clearly the exact transfer
speed need not mach from test to test
Comment 5 Andrew Bartlett 2015-12-21 21:34:39 UTC
To be clear, by a test I mean a patch to our 'make test'.  

Specifically source3/script/tests/test_smbclient_tarmode.sh

Please persist in getting the git patches, it really does make things easier for us.

You might try cloning https://git.samba.org/samba.git if your firewall blocks git://, or else see our github page at https://github.com/samba-team/samba

Thanks!
Comment 6 Björn Jacke 2015-12-23 09:09:11 UTC
never mind, I have the change for the verbose more ready here, just testing some and getting it out later on.
Comment 7 Björn Jacke 2015-12-23 10:04:43 UTC
Alan: can you tell me in which way BackupPC insists on the format of the verbose output of smbclient here? I'd like actually to make the output fit more to the output of a "v" or "vv" tar command.

A more standard tar conforming output would in the long term probably also help any other application using the tar mode. Worst case BackupPC might have to be modified if we change the output of the verbose mode.
Comment 8 Alan Bowler 2016-01-05 23:14:35 UTC
I've been out of town over the Christmas period and had
other stuff to do first when I got back.

I should point out that I'm not really a expert on Linux or Windows.
I primarily do system and programmer tools for GCOS.
(GCOS is an old mainframe system and one of the ancestors of Unix).
I have not done serious Unix programming since Unix V6 (around 1976),
and sort of inherited our backup setup when someone else left the company.

BackupPC is written in Perl, and this is the first time I've ever
looked at any Perl.

As far as I can tell BackupPC uses the follwing code to
recognize when it has received a file and count up to
number of files, and file space, and get a file name.
(Although the name might just be used in the log).

if ( /^\s*(-?\d+) \(\s*\d+[.,]\d kb\/s\) (.*)$/ ) {
    my $sambaFileSize = $1;
    my $pcFileName    = $2;
    (my $fileName = $pcFileName) =~ s/\\/\//g;
    $sambaFileSize += 1024 * 1024 * 4096 if ( $sambaFileSize < 0 );
    $fileName =~ s/^\/*//;
    $t->{byteCnt} += $sambaFileSize;
    $t->{fileCnt}++;
    $t->{XferLOG}->write(\"$_\n") if ( $t->{logLevel} >= 2 u);

note that in parsing a line like
     3 (    2.9 kb/s) \home\atbowler\junk\x1
it is discarding the "(    2.9 kb/s)".
If at the end of the Tar operation the "fileCnt" is non-zero
it concludes the operation whas failed.

It also checks for  a line like
    tar: dumped 2 files and 0 directories
with the code

} elsif ( /^\s*tar: dumped \d+ files/ ) {
    $t->{xferOK} = 1;
    $t->{XferLOG}->write(\"$_\n") if ( $t->{logLevel} >= 0 );

I really don't know how I could set up an automated test in
  source3/script/tests/test_smbclient_tarmode.sh

For smbclient to handle a denial on a TAR, and continue
processing the rest of the directory.  Our Linux system is not
configured as an SMB server, so I can't run a test where I
point smbclient back at a directory on the same machine
to do a test like I described in Comment 4
Comment 9 Björn Jacke 2016-01-06 16:32:19 UTC
okay, I think it is a bit unrealistic if BackupPC expects that the syntax never change. Even a move from kb/s to kiB/s in the output would break that parsing.

I think now that the verbose output got broken anyway, it is a good moment to fix it and to make the output a bit more tar standard like as in SUSv2.

a /etc/X11/xim 4515 bytes, 9 tape blocks
a /etc/X11/xkb/ directory

For adding files to archive a leading "a" followed by the filename, followed by any free formed aditional information. Programs would then also be able to use standard tar output parsers, I guess also BackupPC already contains parsing rules for that.
Comment 10 Alan Bowler 2016-01-06 18:18:35 UTC
Actually, the comments in BackupPC do acknowledge that the
patterns it looks for will likely need to be updated to match
new versions of smbclient.  I think I could modify the copy we
run to cope with the verbose output like you suggest.  What is
important, is that file and directory names as well as the
file sizes be output, and that a summary line with the info
like
    tar: dumped 4 files and 1 directories
be available.
Comment 11 Alan Bowler 2016-01-07 00:19:26 UTC
The TAR "vv" output might not be a good idea.
Windows group names may or may not have spaces in them
making it harder to pick out the fields.

Your
   
a /etc/X11/xim 4515 bytes, 9 tape blocks
a /etc/X11/xkb/ directory

style could be easily handled.
Comment 12 Alan Bowler 2016-07-06 20:13:10 UTC
Created attachment 12263 [details]
updted context diff for the 4.4.4 version of clitar.c

The changes do the same as the previous context diff,
but are based on Samba 4.4.4.
tarmode verbose is default (as per the smbclient man page)
tarmode verbose does generate outout (as opposed to nothing),
  and the verbose output is compatible with what backuppc expects.