Bug 563 - smbclient tar 8GB files fail on tar extract, and on big endian tar create
smbclient tar 8GB files fail on tar extract, and on big endian tar create
Status: RESOLVED FIXED
Product: Samba 3.0
Classification: Unclassified
Component: smbclient
3.0.0
All Linux
: P3 normal
: none
Assigned To: Karolin Seeger
:
Depends on:
Blocks: 8595
  Show dependency treegraph
 
Reported: 2003-10-04 14:05 UTC by Craig Barratt
Modified: 2012-01-04 20:22 UTC (History)
0 users

See Also:


Attachments
proposed patch (906 bytes, patch)
2003-10-11 23:19 UTC, Craig Barratt
no flags Details
Patch from combining comment 1 and comment 4 proposals (2.00 KB, patch)
2011-11-02 10:28 UTC, David Disseldorp
asn: review+
Details
git-am fix for 3.5.next (2.02 KB, patch)
2012-01-04 01:27 UTC, Jeremy Allison
ddiss: review+
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Craig Barratt 2003-10-04 14:05:41 UTC
This code in client/clitar.c::writetarheader() only works on
little-endian machines:

        if (size > (SMB_BIG_UINT)077777777777LL) {

                /* This is a non-POSIX compatible extention to store files
                        greater than 8GB. */

                memset(hb.dbuf.size, 0, 4);
                hb.dbuf.size[0]=128;
                for (i = 8, jp=(char*)&size; i; i--)
                        hb.dbuf.size[i+3] = *(jp++);
        }

This causes smbclient to fail on big endian machines when creating tar
files with >=8GB files.

Secondly, client/clitar.c::readtarheader() fails to handle the
case of size >= 8GB (ie: hd->dbuf.size[0] == 0x80).  It just does
this:

    finfo->size = unoct(hb->dbuf.size, sizeof(hb->dbuf.size));

I will submit a patch for both bugs after I complete some testing
(it's slow with such large files).
Comment 1 Craig Barratt 2003-10-11 23:19:30 UTC
Created attachment 195 [details]
proposed patch

Submitted proposed patch.  Tested with >= 8GB smbclient tar create.
However, smbclient tar extract still only writes a maximum 2GB file;
not sure why.
Comment 2 Gerald (Jerry) Carter 2005-11-14 09:29:24 UTC
database cleanup
Comment 3 Brad Ellis (mail address dead) 2005-12-04 05:07:34 UTC
Have there been any recent updates on this ?

We're seeing a failure with a 4 GB tarball. 
2GB (exactly) is being written to the target, 
and the following errors are reported:

command:
--------
smbclient //<target ip>/<share> "<password>" -U <user> -Tx <tarball>

errors:
-------
Error writing remote file
Abandoning restore

Sounds a bit like a 32bit counter somewhere.

This was tried with the following versions:
Version 3.0.3-5
Version 3.0.20b

Using smbclient and a "put" of the 4gb tarball works fine.
Comment 4 Brad Ellis (mail address dead) 2005-12-13 03:04:28 UTC
client/clitar.c seems to be the cause - should the pos,dsize
(and possibly bpos) variables be off_t's and not ints ?

----
static int get_file(file_info2 finfo)
{
/*      int fnum = -1, pos = 0, dsize = 0, bpos = 0; existing variable declarations */
/* new declarations */
        int fnum = -1;
        off_t pos = 0, dsize = 0, bpos = 0;

        SMB_BIG_UINT rsize = 0;

        DEBUG(5, ("get_file: file: %s, size %.0f\n", finfo.name, (double)finfo.size));

        if (ensurepath(finfo.name) &&
                        (fnum=cli_open(cli, finfo.name, O_RDWR|O_CREAT|O_TRUNC, DENY_NONE)) == -1) {
                DEBUG(0, ("abandoning restore\n"));
                return(False);
        }

        /* read the blocks from the tar file and write to the remote file */

        rsize = finfo.size;  /* This is how much to write */

        while (rsize > 0) {

                /* We can only write up to the end of the buffer */
                dsize = MIN(tbufsiz - (buffer_p - tarbuf) - bpos, 65520); /* Calculate the size to write */
                dsize = MIN(dsize, rsize);  /* Should be only what is left */
                DEBUG(5, ("writing %i bytes, bpos = %i ...\n", dsize, bpos));

                if (cli_write(cli, fnum, 0, buffer_p + bpos, pos, dsize) != dsize) {
                        DEBUG(0, ("Error writing remote file\n"));


-----
from include/proto.h
size_t cli_write(struct cli_state *cli,
                 int fnum, uint16 write_mode,
                 const char *buf, off_t offset, size_t size);

offset (or pos in clitar.c) should be a off_t and not a int as declared.
I'd suspect this is the cause of the 2Gb limit.

However my changes result in some 9728 bytes extra being written to the file. These extra bytes seem to be zeros.
Comment 5 SPAMMER 2007-04-28 02:40:47 UTC
Created attachment 2468


test
Comment 6 SPAMMER 2007-04-28 03:23:03 UTC
Created attachment 2469
Comment 7 SPAMMER 2007-04-28 03:23:57 UTC
Created attachment 2470
Comment 8 SPAMMER 2007-04-28 03:24:29 UTC
Created attachment 2471
Comment 9 SPAMMER 2007-04-28 03:24:57 UTC
Created attachment 2472
Comment 10 SPAMMER 2007-04-28 03:25:44 UTC
Created attachment 2473
Comment 11 SPAMMER 2007-04-28 03:26:08 UTC
Created attachment 2474
Comment 12 SPAMMER 2007-04-28 03:26:45 UTC
Created attachment 2475
Comment 13 SPAMMER 2007-04-28 03:27:13 UTC
Created attachment 2476
Comment 14 SPAMMER 2007-04-28 03:27:38 UTC
Created attachment 2477
Comment 15 SPAMMER 2007-04-28 03:28:08 UTC
Created attachment 2478
Comment 16 SPAMMER 2007-04-28 03:28:40 UTC
Created attachment 2479
Comment 17 SPAMMER 2007-04-28 03:29:07 UTC
Created attachment 2480
Comment 18 SPAMMER 2007-04-28 03:29:36 UTC
Created attachment 2481
Comment 19 SPAMMER 2007-04-28 03:29:59 UTC
Created attachment 2482
Comment 20 SPAMMER 2007-04-28 03:30:44 UTC
Created attachment 2483
Comment 21 SPAMMER 2007-04-28 03:31:06 UTC
Created attachment 2484
Comment 22 SPAMMER 2007-04-28 03:31:34 UTC
Created attachment 2485
Comment 23 SPAMMER 2007-04-28 03:31:56 UTC
Created attachment 2486
Comment 24 SPAMMER 2007-04-28 03:32:22 UTC
Created attachment 2487
Comment 25 SPAMMER 2007-04-28 03:32:48 UTC
Created attachment 2488
Comment 26 SPAMMER 2007-04-28 03:33:12 UTC
Created attachment 2489
Comment 27 SPAMMER 2007-04-28 03:33:39 UTC
Created attachment 2490
Comment 28 SPAMMER 2007-04-28 03:34:03 UTC
Created attachment 2491
Comment 29 SPAMMER 2007-04-28 03:34:27 UTC
Created attachment 2492
Comment 30 Gerald (Jerry) Carter 2007-04-28 07:58:16 UTC
Comment on attachment 2477
Comment 31 Stefan Metzmacher 2010-04-26 03:22:22 UTC
If it's still broken in 3.5, please reopen.
3.0 isn't supported anymore.
Comment 32 David Disseldorp 2011-11-02 09:39:25 UTC
This bug is still present in 3.6.
Comment 33 David Disseldorp 2011-11-02 10:28:37 UTC
Created attachment 7049 [details]
Patch from combining comment 1 and comment 4 proposals

Provided by Masafumi Nakayama.
Comment 34 Andreas Schneider 2012-01-03 13:56:23 UTC
Comment on attachment 7049 [details]
Patch from combining comment 1 and comment 4 proposals

Looks good.
Comment 35 Jeremy Allison 2012-01-04 01:27:16 UTC
Created attachment 7224 [details]
git-am fix for 3.5.next

Back port of the 3.6.x patch above to 3.5.x.

David, please confirm then I'll re-assign to Karolin for inclusion in 3.5.next and 3.6.next.

Jeremy.
Comment 36 David Disseldorp 2012-01-04 09:43:13 UTC
Back port looks good, thanks Jeremy.

Karolin, could you please apply the attached patch "git-am fix for 3.5.next" to 3.5.next and cherry pick the master commit (909056a2daacd961b40158b86bc117650a897054) for 3.6.next?
Comment 37 Karolin Seeger 2012-01-04 20:22:34 UTC
(In reply to comment #36)
> Back port looks good, thanks Jeremy.
> 
> Karolin, could you please apply the attached patch "git-am fix for 3.5.next" to
> 3.5.next and cherry pick the master commit
> (909056a2daacd961b40158b86bc117650a897054) for 3.6.next?

Done.

Closing out bug report.

Thanks!