In 3.6.23 (before libarchive) the tar command in verbose mode (default) would output a line for each file and directory that was written to the archive. BackUpPC looks for these messages. The session below shows this effect even if it is not exactly the command BackUpPC is issuing on our system. (Debian) atbowler@dory:~$ atbowler@dory:~$ smbclient --version Version 3.6.23 atbowler@dory:~$ smbclient //oliver/c\$ -D /home/atbowler/backtest -A pw -c "tarmode verbose" -Tc xxx.tar Unknown parameter encountered: "realm" Ignoring unknown parameter "realm" WARNING: Ignoring invalid value 'ADS' for parameter 'security' Domain=[THINKAGE] OS=[Windows 8.1 Enterprise 9600] Server=[Windows 8.1 Enterprise 6.3] tarmode is now full, system, hidden, noreset, verbose, noerrorskip directory \home\atbowler\backtest\dir\ 7 ( 3.4 kb/s) \home\atbowler\backtest\dir\x4.txt 3 ( 1.5 kb/s) \home\atbowler\backtest\x1 30 ( 14.6 kb/s) \home\atbowler\backtest\x2.txt 30 ( 14.6 kb/s) \home\atbowler\backtest\x3.txt tar: dumped 5 files and directories Total bytes written: 2048 atbowler@dory:~$ smbclient //oliver/c\$ -D /home/atbowler/backtest -A pw -c "tarmode quiet" -Tc xxx.tar Unknown parameter encountered: "realm" Ignoring unknown parameter "realm" WARNING: Ignoring invalid value 'ADS' for parameter 'security' Domain=[THINKAGE] OS=[Windows 8.1 Enterprise 9600] Server=[Windows 8.1 Enterprise 6.3] tarmode is now full, system, hidden, noreset, quiet, noerrorskip tar: dumped 5 files and directories Total bytes written: 2048 (note the "errorskip" tarmode is a result of small local modifications Ref report 10605) smbclient in 4.3.1 ignores the verbose mode E.G. atbowler@dory:~$ work-samba/bin/smbclient --version Version 4.3.1 atbowler@dory:~$ work-samba/bin/smbclient //oliver/c\$ -D /home/atbowler/backtest -A pw -c "tarmode verbose" -Tc xxx.tar WARNING: The "syslog" option is deprecated Domain=[THINKAGE] OS=[Windows 8.1 Enterprise 9600] Server=[Windows 8.1 Enterprise 6.3] tarmode is now full, system, hidden, noreset, verbose tar: dumped 4 files and 1 directories Total bytes written: 70 atbowler@dory:~$ work-samba/bin/smbclient //oliver/c\$ -D /home/atbowler/backtest -A pw -c "tarmode quiet" -Tc xxx.tar WARNING: The "syslog" option is deprecated Domain=[THINKAGE] OS=[Windows 8.1 Enterprise 9600] Server=[Windows 8.1 Enterprise 6.3] tarmode is now full, system, hidden, noreset, quiet tar: dumped 4 files and 1 directories Total bytes written: 70 atbowler@dory:~$ Note that there is no list of files in the verbose setting.
Created attachment 11733 [details] reenable tarmode verbose, and skip inaccessible files Besides this problem, there is a second glitch in the new TAR stuff. In 3.6.1 tar had the problem that it it could no access a file, (busy etc.) it skipped the remainder of the directory. (Bug report 10605.) In 4.3.1 if it can't access a file, instead of just skipping the file, it writes a tar element of all zeros, making it look like is did back up something. I'm attaching a "diff -c" with my changes to renable tarmode verbose and properly skip inaccessible files. Unlike my proposed patches for 10605, this time there is no tarmode errorskip
Created attachment 11734 [details] Modified clitar.c for tarmode verbose this is just the whole clitar.c that I'm now using.
Comment on attachment 11733 [details] reenable tarmode verbose, and skip inaccessible files Please submit the patch as a git commit, as that will use the accepted 'diff -u' format, using git commit -s -m 'A useful commit message' git format-patch -s -1 https://wiki.samba.org/index.php/Using_Git_for_Samba_Development has some further information. We also need a test, as otherwise we will just regress again in the future.
Well I tried installing GIT and doing the clone and it fails with a network timewout. I already attached my modified clitar.c so it would seem much easier if you just used that to do the GIT stuff since clearly GIT works for you. As for a test. 1) create a Windows directory with at least 3 files mkdir junk echo >junk/x1 x1 echo >junk/x2 x2 echo >junk/x3 x3 2) remove all perms from and intermediate file chmod 000 junk/x2 3) on the linux system create a tar file of the test directory e.g. smbclient //oliver/c\$ -A pw -D /home/atbowler/junk -Tc xxx.tar The output should look something like this atbowler@dory:~$ smbclient //oliver/c\$ -A pw -D /home/atbowler/junk -Tc xxx.tar Domain=[THINKAGE] OS=[Windows 8.1 Enterprise 9600] Server=[Windows 8.1 Enterprise 6.3] 3 ( 2.9 kb/s) \home\atbowler\junk\x1 NT_STATUS_ACCESS_DENIED opening remote file \home\atbowler\junk\x2 3 ( 2.9 kb/s) \home\atbowler\junk\x3 tar: dumped 2 files and 0 directories Total bytes written: 9 and a tar listing should look like atbowler@dory:~$ tar tvf xxx.tar -rw-r--r-- 0/0 3 2015-12-21 16:13 ./home/atbowler/junk/x1 -rw-r--r-- 0/0 3 2015-12-21 16:13 ./home/atbowler/junk/x3 Note that "x2" file was skipped and "x3" did get included the lines like 3 ( 2.9 kb/s) \home\atbowler\junk\x1 3 ( 2.9 kb/s) \home\atbowler\junk\x3 and summary lines tar: dumped 2 files and 0 directories Total bytes written: 9 Are what is important for BackUpPC. Clearly the exact transfer speed need not mach from test to test
To be clear, by a test I mean a patch to our 'make test'. Specifically source3/script/tests/test_smbclient_tarmode.sh Please persist in getting the git patches, it really does make things easier for us. You might try cloning https://git.samba.org/samba.git if your firewall blocks git://, or else see our github page at https://github.com/samba-team/samba Thanks!
never mind, I have the change for the verbose more ready here, just testing some and getting it out later on.
Alan: can you tell me in which way BackupPC insists on the format of the verbose output of smbclient here? I'd like actually to make the output fit more to the output of a "v" or "vv" tar command. A more standard tar conforming output would in the long term probably also help any other application using the tar mode. Worst case BackupPC might have to be modified if we change the output of the verbose mode.
I've been out of town over the Christmas period and had other stuff to do first when I got back. I should point out that I'm not really a expert on Linux or Windows. I primarily do system and programmer tools for GCOS. (GCOS is an old mainframe system and one of the ancestors of Unix). I have not done serious Unix programming since Unix V6 (around 1976), and sort of inherited our backup setup when someone else left the company. BackupPC is written in Perl, and this is the first time I've ever looked at any Perl. As far as I can tell BackupPC uses the follwing code to recognize when it has received a file and count up to number of files, and file space, and get a file name. (Although the name might just be used in the log). if ( /^\s*(-?\d+) \(\s*\d+[.,]\d kb\/s\) (.*)$/ ) { my $sambaFileSize = $1; my $pcFileName = $2; (my $fileName = $pcFileName) =~ s/\\/\//g; $sambaFileSize += 1024 * 1024 * 4096 if ( $sambaFileSize < 0 ); $fileName =~ s/^\/*//; $t->{byteCnt} += $sambaFileSize; $t->{fileCnt}++; $t->{XferLOG}->write(\"$_\n") if ( $t->{logLevel} >= 2 u); note that in parsing a line like 3 ( 2.9 kb/s) \home\atbowler\junk\x1 it is discarding the "( 2.9 kb/s)". If at the end of the Tar operation the "fileCnt" is non-zero it concludes the operation whas failed. It also checks for a line like tar: dumped 2 files and 0 directories with the code } elsif ( /^\s*tar: dumped \d+ files/ ) { $t->{xferOK} = 1; $t->{XferLOG}->write(\"$_\n") if ( $t->{logLevel} >= 0 ); I really don't know how I could set up an automated test in source3/script/tests/test_smbclient_tarmode.sh For smbclient to handle a denial on a TAR, and continue processing the rest of the directory. Our Linux system is not configured as an SMB server, so I can't run a test where I point smbclient back at a directory on the same machine to do a test like I described in Comment 4
okay, I think it is a bit unrealistic if BackupPC expects that the syntax never change. Even a move from kb/s to kiB/s in the output would break that parsing. I think now that the verbose output got broken anyway, it is a good moment to fix it and to make the output a bit more tar standard like as in SUSv2. a /etc/X11/xim 4515 bytes, 9 tape blocks a /etc/X11/xkb/ directory For adding files to archive a leading "a" followed by the filename, followed by any free formed aditional information. Programs would then also be able to use standard tar output parsers, I guess also BackupPC already contains parsing rules for that.
Actually, the comments in BackupPC do acknowledge that the patterns it looks for will likely need to be updated to match new versions of smbclient. I think I could modify the copy we run to cope with the verbose output like you suggest. What is important, is that file and directory names as well as the file sizes be output, and that a summary line with the info like tar: dumped 4 files and 1 directories be available.
The TAR "vv" output might not be a good idea. Windows group names may or may not have spaces in them making it harder to pick out the fields. Your a /etc/X11/xim 4515 bytes, 9 tape blocks a /etc/X11/xkb/ directory style could be easily handled.
Created attachment 12263 [details] updted context diff for the 4.4.4 version of clitar.c The changes do the same as the previous context diff, but are based on Samba 4.4.4. tarmode verbose is default (as per the smbclient man page) tarmode verbose does generate outout (as opposed to nothing), and the verbose output is compatible with what backuppc expects.
fixed in master and in the upcoming 4.12 with d512b27563b32f7bcb35da21781e87473830c906