Bug 3124 - xcopy /d with samba shares works not as aspected
Summary: xcopy /d with samba shares works not as aspected
Status: RESOLVED FIXED
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: File services (show other bugs)
Version: 4.1.7
Hardware: x86 Windows XP
: P1 regression (vote)
Target Milestone: ---
Assignee: Karolin Seeger
QA Contact: Samba QA Contact
URL:
Keywords:
Depends on:
Blocks: 2279
  Show dependency treegraph
 
Reported: 2005-09-29 14:52 UTC by Thomas Bork
Modified: 2014-06-10 09:00 UTC (History)
5 users (show)

See Also:


Attachments
debug log level 10 with only one file (test.txt) (243.40 KB, text/plain)
2005-10-30 08:32 UTC, Thomas Bork
no flags Details
git-am fix for master. (16.75 KB, patch)
2014-05-21 19:05 UTC, Jeremy Allison
ddiss: review+
Details
git-am fix for 4.0.x and 4.1.x. (16.71 KB, patch)
2014-05-21 19:07 UTC, Jeremy Allison
ddiss: review+
Details
git-am fix for 3.6.x. (6.75 KB, patch)
2014-05-21 19:43 UTC, Jeremy Allison
no flags Details
tcpdump dump of xcopy writing a unchanged file (26.54 KB, application/octet-stream)
2014-05-23 19:36 UTC, Thomas Bork
no flags Details
log.smbd with debug level 10 with only one file (test.txt) (1.21 MB, application/octet-stream)
2014-05-23 19:39 UTC, Thomas Bork
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Thomas Bork 2005-09-29 14:52:52 UTC
Copying files and folders to mapped samba shares with xcopy works not as to
windows servers, if using the xcopy switch /d (only copy newer or changed files).

If copying to an other directory on the XP client or to a windows share, the
xcopy option /d works as designed - only copying the first time, don't copy
anything the second time, because files and folders are not changed.
If copying to the mapped samba share, xcopy is transfering the files _every_
time but files and folders are not changed in the meantime.

This is the source:

F:\Eigene Dateien\tb>cd testordner

F:\Eigene Dateien\tb\Testordner>dir
 Datenträger in Laufwerk F: ist DATEN
 Volumeseriennummer: F8E7-9E4E

 Verzeichnis von F:\Eigene Dateien\tb\Testordner

25.09.2005  18:43    <DIR>          .
25.09.2005  18:43    <DIR>          ..
15.01.2001  13:38    <DIR>          Ordner1
15.01.2001  13:57    <DIR>          Ordner2
15.01.2001  13:38    <DIR>          Ordner3
01.09.2005  14:49             2.088 test.txt
               1 Datei(en)          2.088 Bytes
               5 Verzeichnis(se), 80.333.746.176 Bytes frei

Z: is the mapped samba share testshare:

[testshare]
   comment = testshare on %h
   browseable = yes
   writeable = yes
   path = /testshare
   read list =
   write list =
   create mode = 0777
   force create mode = 0777
   directory mode = 0777
   force directory mode = 0777
   oplocks = no
   level2 oplocks = no
   blocking locks = no
   hide files = /desktop.ini/Thumbs.db/
   dos filemode = yes

Copying on the XP client:
(first time)
F:\Eigene Dateien\tb>xcopy Testordner Testordner1 /z /e /d /c /i /r /h /y
Testordner\test.txt
Testordner\Ordner3\test.txt
2 Datei(en) kopiert

(second time)
F:\Eigene Dateien\tb>xcopy Testordner Testordner1 /z /e /d /c /i /r /h /y
0 Datei(en) kopiert

OK, no files are copied because the files are not newer and not changed.


Copying to the mapped samba share:
(first time)
F:\Eigene Dateien\tb>xcopy Testordner Z: /z /e /d /c /i /r /h /y
Testordner\test.txt
Testordner\Ordner3\test.txt
2 Datei(en) kopiert

(second time)
F:\Eigene Dateien\tb>xcopy Testordner Z: /z /e /d /c /i /r /h /y
Testordner\test.txt
Testordner\Ordner3\test.txt
2 Datei(en) kopiert

Not ok, two files were copied which are not newer or changed.


mg2 # ls -alR /testshare
/testshare:
insgesamt 24
drwxrwxrwx   5 root root  4096 2005-09-25 18:43 .
drwxr-xr-x  44 root root  4096 2005-09-25 18:35 ..
drwxrwxrwx   2 tb   users 4096 2001-01-15 13:38 Ordner1
drwxrwxrwx   2 tb   users 4096 2001-01-15 13:57 Ordner2
drwxrwxrwx   2 tb   users 4096 2001-01-15 13:38 Ordner3
-rwxrwxrwx   1 tb   users 2088 2005-09-01 14:49 test.txt

/testshare/Ordner1:
insgesamt 8
drwxrwxrwx  2 tb   users 4096 2001-01-15 13:38 .
drwxrwxrwx  5 root root  4096 2005-09-25 18:43 ..

/testshare/Ordner2:
insgesamt 8
drwxrwxrwx  2 tb   users 4096 2001-01-15 13:57 .
drwxrwxrwx  5 root root  4096 2005-09-25 18:43 ..

/testshare/Ordner3:
insgesamt 12
drwxrwxrwx  2 tb   users 4096 2001-01-15 13:38 .
drwxrwxrwx  5 root root  4096 2005-09-25 18:43 ..
-rwxrwxrwx  1 tb   users 2088 2001-01-15 13:38 test.txt


From a debug level 10 log:
[2005/09/25 19:40:44.962308, 10, pid=1109]
smbd/trans2.c:call_trans2setfilepathinfo(3976)
  call_trans2setfilepathinfo: Set end of file info for file test.txt to 2088
[2005/09/25 19:40:44.962350, 6, pid=1109]
smbd/trans2.c:call_trans2setfilepathinfo(4406)
  actime: Thu Sep  1 14:49:27 2005
   modtime: Sun Sep 25 19:40:44 2005
   size: 2088 dosmode: 20

actime is the time called "Erstellt am" (created on) in windows

Win-Create-Time:
F:\Eigene Dateien\tb\Testordner>dir /T:C test.txt
 Datenträger in Laufwerk F: ist DATEN
 Volumeseriennummer: F8E7-9E4E

 Verzeichnis von F:\Eigene Dateien\tb\Testordner

01.09.2005  14:49             2.088 test.txt
               1 Datei(en)          2.088 Bytes
               0 Verzeichnis(se), 80.333.312.000 Bytes frei

modtime is the time called "Letzter Zugriff" (last access on) in windows and not
"Geändert am" (modified on) as it should be.

Win-Access-Time (accessed this file later but not changed it):
F:\Eigene Dateien\tb\Testordner>dir /T:A test.txt
 Datenträger in Laufwerk F: ist DATEN
 Volumeseriennummer: F8E7-9E4E

 Verzeichnis von F:\Eigene Dateien\tb\Testordner

25.09.2005  22:55             2.088 test.txt
               1 Datei(en)          2.088 Bytes
               0 Verzeichnis(se), 80.333.312.000 Bytes frei

Win-Write-Access-Time:
F:\Eigene Dateien\tb\Testordner>dir /T:W test.txt
 Datenträger in Laufwerk F: ist DATEN
 Volumeseriennummer: F8E7-9E4E

 Verzeichnis von F:\Eigene Dateien\tb\Testordner

01.09.2005  14:49             2.088 test.txt
               1 Datei(en)          2.088 Bytes
               0 Verzeichnis(se), 80.333.312.000 Bytes frei

Maybe thats why samba thinks, the file has changed...
I have a debug level 10 log from the second xcopy command.


der tom
Comment 1 Marc Kaplan 2005-09-29 15:19:52 UTC
I did some analysis of this previously that I will move from my bug tracking
system over here (note mcomp refers to an internal compare utility):

This may be due to a time granularity problem. Windows uses file time stamps
that have a much greater granularity than what it displays (microseconds?
nanoseconds?), and NTFS is capable of storing this granularity. It's probably
the case that XFS and/or Samba support less time granularity than Windows+NTFS,
and thus some files, though time stamped with the same user-visible time, have a
different low-granularity time stamp on the file system.

You can see this with mcomp on a file that has the "same" time stamp that gets
re-copied with xcopy /d when copying to the Samba server:

C:\test>mcomp.exe wirelessperf.txt x:\yermom\wirelessperf.txt /u/d
Source: wirelessperf.txt
Target: x:\yermom\wirelessperf.txt
Options specified: /u/d
Buffer size: 65536

  wirelessperf.txt

Source dwLowDateTime = 61b1e7ae, Source dwHighDateTime = 1c4525a
Target dwLowDateTime = 61872e80, Target dwHighDateTime = 1c4525a

Date/time mismatch:  source = 6/14/2004 14:56: 2  target = 6/14/2004 14:56: 2

Number of source files                    1
Number of source directories              0
Number of date/time mismatches            1
Number of file image mismatches           0
Number of missing target files/dirs       0


Errors were encountered.


As you can see, the user-visible time of the file on source and target is
"6/14/2004 14:56: 2". As you can see, the low resolution time codes match (both
1c4525a), and the high resolution time codes differ (61b1e7ae and 61b1e7ae). If
 I convert these values to decimal, I get:

61b1e7ae    1639049134
61872e80    1636249216

Put a decimal place in front of these and they should represent fractions of
seconds. So the first three digits match, but the rest don't. This could suggest
that we retain time granularity only to three decimal places (milliseconds)
while Windows stores...um, is that nanoseconds?
Comment 2 Thomas Bork 2005-10-20 03:52:05 UTC
> This may be due to a time granularity problem. Windows uses file time stamps
> that have a much greater granularity than what it displays (microseconds?
> nanoseconds?), and NTFS is capable of storing this granularity. It's probably
> the case that XFS and/or Samba support less time granularity than Windows+NTFS,
> and thus some files, though time stamped with the same user-visible time, have a
> different low-granularity time stamp on the file system.

Do you have other results, when Samba reports an other file system type than NTFS?
Comment 3 Thomas Bork 2005-10-30 08:32:07 UTC
Created attachment 1554 [details]
debug log  level 10 with only one file (test.txt)
Comment 4 Gerald (Jerry) Carter (dead mail address) 2006-04-20 08:03:35 UTC
severity should be determined by the developers and not the reporter.
Comment 5 Andrew Klosterman 2014-05-20 14:03:35 UTC
Troubleshooting a similar problem with SMB2 I noticed differences in timestamps as reported by SMB2 FIND and SMB2 CREATE.  Program was comparing timestamps and they turned out different from the directory listing vs. checking individual files.

The FIND code __rounds__ timestamps based on the discovered resolution of the underlying filesystem as stored in the connection structure.  The CREATE code does __no__ rounding.

* smbd_smb2_request_dispatch()
  * smbd_smb2_request_process_find()
    * smbd_smb2_find_send()
      * case SMB2_FIND_ID_BOTH_DIRECTORY_INFO:
        * info_level = SMB_FIND_ID_BOTH_DIRECTORY_INFO;
      * smbd_dirptr_lanman2_entry()
        * smbd_dirptr_get_entry()
        * smbd_marshall_dir_entry()
          * mdate_ts = smb_fname->st.st_ex_mtime;
          * adate_ts = smb_fname->st.st_ex_atime;
          * create_date_ts = get_create_timespec(conn, NULL, smb_fname);
          * cdate_ts = get_change_timespec(conn, NULL, smb_fname);
          * if (lp_dos_filetime_resolution(SNUM(conn))) {
            * dos_filetime_timespec(&create_date_ts);
            * dos_filetime_timespec(&mdate_ts);
            * dos_filetime_timespec(&adate_ts);
            * dos_filetime_timespec(&cdate_ts);
          * create_date = convert_timespec_to_time_t(create_date_ts);
          * mdate = convert_timespec_to_time_t(mdate_ts);
          * adate = convert_timespec_to_time_t(adate_ts);
          * c_date = convert_timespec_to_time_t(cdate_ts);
          * case SMB_FIND_ID_BOTH_DIRECTORY_INFO:
            * put_long_date_timespec(conn->ts_res,p,create_date_ts); p += 8;       <<<<<<< timestamp resolution from connection structure!!
              * round_timespec(res, &ts);
                * {{{
void round_timespec(enum timestamp_set_resolution res, struct timespec *ts)
{
        switch (res) {
                case TIMESTAMP_SET_SECONDS:
                        round_timespec_to_sec(ts);
                        break;
                case TIMESTAMP_SET_MSEC:
                        round_timespec_to_usec(ts);
                        break;
                case TIMESTAMP_SET_NT_OR_BETTER:
                        /* No rounding needed. */
                        break;
        }
}
}}}
                  * {{{
void round_timespec_to_sec(struct timespec *ts)
{
        ts->tv_sec = convert_timespec_to_time_t(*ts);
        ts->tv_nsec = 0;                                                          <<<<<<< drop the sub-second resolution from the timestamp
}
}}}
              * unix_timespec_to_nt_time(&nt, ts);
            * put_long_date_timespec(conn->ts_res,p,adate_ts); p += 8;
            * put_long_date_timespec(conn->ts_res,p,mdate_ts); p += 8;
            * put_long_date_timespec(conn->ts_res,p,cdate_ts); p += 8;


{{{
typedef struct connection_struct {
...
        enum timestamp_set_resolution ts_res;
...
} connection_struct;
}}}

* smbd_smb2_request_dispatch()
  * case SMB2_OP_CREATE:
  * return_value = smbd_smb2_request_process_create(req);
    * tsubreq = smbd_smb2_create_send(smb2req,
      * status = SMB_VFS_CREATE_FILE(smb1req->conn,
      * unix_timespec_to_nt_time(&last_write_time, result->fsp_name->st.st_ex_mtime);
        * unix_timespec_to_nt_time() converts to a 64-bit integer that counts 100ns units since 1601 AD, Gregorian Calendar https://ftp.samba.org/pub/samba/specs/cifs6.txt
      * get_file_infos(result->file_id, 0, NULL, &write_time_ts);
      * {{{
        unix_timespec_to_nt_time(&state->out_creation_time,
                        get_create_timespec(smb1req->conn, result,
                                        result->fsp_name));
        unix_timespec_to_nt_time(&state->out_last_access_time,
                        result->fsp_name->st.st_ex_atime);
        unix_timespec_to_nt_time(&state->out_last_write_time,
                        result->fsp_name->st.st_ex_mtime);
        unix_timespec_to_nt_time(&state->out_change_time,
                        get_change_timespec(smb1req->conn, result,
                                        result->fsp_name));
}}}

There may be other places where time resolution rounding is not applied.
Comment 6 Jeremy Allison 2014-05-21 17:19:44 UTC
Great analysis - I'll look into a fix for this asap !

Thanks

Jeremy.
Comment 7 Jeremy Allison 2014-05-21 19:05:24 UTC
Created attachment 9961 [details]
git-am fix for master.
Comment 8 Jeremy Allison 2014-05-21 19:07:55 UTC
Created attachment 9962 [details]
git-am fix for 4.0.x and 4.1.x.
Comment 9 Jeremy Allison 2014-05-21 19:09:51 UTC
Andrew, can you test the fix for 4.0.x (or 4.1.x, whichever you're running) and let me know if it fixes your bug.

It should, as it now causes all code paths in the SMB2 server that write file times onto the wire to use the common put_long_date_timespec() function that correctly rounds the times based on what the underlying filesystem can report.

Plase let me know asap - this is a very important bug !

Cheers,

Jeremy.
Comment 10 Andrew Klosterman 2014-05-21 19:25:37 UTC
(In reply to comment #9)
> Andrew, can you test the fix for 4.0.x (or 4.1.x, whichever you're running) and
> let me know if it fixes your bug.
> ...
> Plase let me know asap - this is a very important bug !

Alas, I found this issue in Samba 3.6.3 ... and that's what I'm running ... nothing approaching 4.x!  :-/
{{{
# smbd --version
Version 3.6.3
}}}

repro possibilties:
1. use a filesystem that reports timestamp resolution of 1.0seconds .. or change code in a VFS "fs_capabilities" call to return 1.0s resolution.
1. start a pcap
1. do a SMB2 CREATE to open a file ... watch the timestamp attributes in the pcap ... they're high precision
1. do a SMB2 directory listing / SMB2_FIND_ID_BOTH_DIRECTORY_INFO and see file attributes rounded to 1s resolution
1. stop the pcap
1. discard your modified VFS "fs_capabilities" change
Comment 11 Jeremy Allison 2014-05-21 19:27:39 UTC
Give me a few minutes and I'll create a 3.6.x back-port for you to test :-).
Comment 12 Jeremy Allison 2014-05-21 19:43:07 UTC
Created attachment 9963 [details]
git-am fix for 3.6.x.

OK Andrew, if you can try this on 3.6.x I'd appreciate it !

Thanks,

Jeremy.
Comment 13 Andrew Klosterman 2014-05-21 20:12:21 UTC
Comment on attachment 9963 [details]
git-am fix for 3.6.x.

> source3/smbd/smb2_create.c | 85 +++++++++++++++++++++++-----------------------
> 1 file changed, 43 insertions(+), 42 deletions(-)
>
>diff --git a/source3/smbd/smb2_create.c b/source3/smbd/smb2_create.c
>index 0862990..8a589e8 100644
>--- a/source3/smbd/smb2_create.c
>+++ b/source3/smbd/smb2_create.c
> ...
>@@ -263,10 +263,11 @@ static void smbd_smb2_request_create_done(struct tevent_req *tsubreq)
> 	DATA_BLOB outdyn;
> 	uint8_t out_oplock_level = 0;
> 	uint32_t out_create_action = 0;
>-	NTTIME out_creation_time = 0;
>-	NTTIME out_last_access_time = 0;
>-	NTTIME out_last_write_time = 0;
>-	NTTIME out_change_time = 0;
>+	connection_struct *conn = smb2req->tcon->compat_conn;
> ...

This last line, for "connection_struct", looks to be superfluous.
(At least it appears so as I read the master branch in git that I pulled a couple days ago.)
Comment 14 Jeremy Allison 2014-05-21 20:21:05 UTC
(In reply to comment #13)

> This last line, for "connection_struct", looks to be superfluous.
> (At least it appears so as I read the master branch in git that I pulled a
> couple days ago.)

No it's not for 3.6.x, as later on in that function new code does:

        put_long_date_timespec(conn->ts_res,
              (char *)outbody.data + 0x08,
              out_creation_ts);                 /* creation time */
        put_long_date_timespec(conn->ts_res,
              (char *)outbody.data + 0x10,
              out_last_access_ts);              /* last access time */
        put_long_date_timespec(conn->ts_res,
              (char *)outbody.data + 0x18,
              out_last_write_ts);               /* last write time */
        put_long_date_timespec(conn->ts_res,
              (char *)outbody.data + 0x20,
              out_change_ts);                   /* change time */

instead of the direct BVALs that were there before.
Comment 15 Andrew Klosterman 2014-05-21 20:24:10 UTC
(In reply to comment #14)
> (In reply to comment #13)
> 
> > This last line, for "connection_struct", looks to be superfluous.
> > (At least it appears so as I read the master branch in git that I pulled a
> > couple days ago.)
> 
> No it's not for 3.6.x, as later on in that function new code does:
>...
> instead of the direct BVALs that were there before.

Aha!  Now I see what you did there!  :-)

I needed to look for the new "connection_struct" variable in the diff ... rather than the pre-existing code!
Comment 16 Richard Sharpe 2014-05-21 21:45:30 UTC
Hmmm, we found this issue with Samba 3.6.x as well, and fixed it on FreeBSD 8.0 by back-porting utimensat et al.

Since Windows uses 100nS time resolution, and the times differed from those that Windows has, we figured we needed to store the correct resolution time stamps. Since ZFS has at least a 32-bit field for that, it was relatively simple.

It's not clear to me that rounding will prevent robocopy from recopying and it already has a flag to use DOS time resolution for that.
Comment 17 Jeremy Allison 2014-05-21 21:51:39 UTC
(In reply to comment #16)
> Hmmm, we found this issue with Samba 3.6.x as well, and fixed it on FreeBSD 8.0
> by back-porting utimensat et al.

Thanks for logging the bug... :-).

> Since Windows uses 100nS time resolution, and the times differed from those
> that Windows has, we figured we needed to store the correct resolution time
> stamps. Since ZFS has at least a 32-bit field for that, it was relatively
> simple.
> 
> It's not clear to me that rounding will prevent robocopy from recopying and it
> already has a flag to use DOS time resolution for that.

Yes it should. What is clear in the codepaths is that currently, all of them except for SMB2_CREATE round the returned timestamps, so SMB2_CREATE can return different values than SMB2_GETINFO for the same file.

So this is an obviously necessary fix.

Jeremy.
Comment 18 Jeremy Allison 2014-05-21 21:54:25 UTC
(In reply to comment #16)

> It's not clear to me that rounding will prevent robocopy from recopying and it
> already has a flag to use DOS time resolution for that.

Just one more point - the DOS filetime resolution setting is completely different - it arbitrarily rounds any timestamp to a 2 second boundary (as that's what DOS supports) ignoring what the underlying filesystem can do. So they really are different things.
Comment 19 Richard Sharpe 2014-05-21 22:18:48 UTC
(In reply to comment #17)
> > Since Windows uses 100nS time resolution, and the times differed from those
> > that Windows has, we figured we needed to store the correct resolution time
> > stamps. Since ZFS has at least a 32-bit field for that, it was relatively
> > simple.
> > 
> > It's not clear to me that rounding will prevent robocopy from recopying and it
> > already has a flag to use DOS time resolution for that.
> 
> Yes it should. What is clear in the codepaths is that currently, all of them
> except for SMB2_CREATE round the returned timestamps, so SMB2_CREATE can return
> different values than SMB2_GETINFO for the same file.
> 
> So this is an obviously necessary fix.

Sure, but the original bug was with XP, so it really seems to be related to not storing the full time stamp resolution.

If you are limited to microsecond resolution, then nine times out of ten the stored time will be different than what Windows has, and it will recopy. (Nine times out of ten because when the Windows timestamp--last change time, I think--has a zero in the hundred nanoseconds area, your stored time will match.

If you round, then by definition you are returning a different value than Windows has and I think robocopy will simply recopy.

At least, when we started storing nanosecond time resolutions, robocopy started behaving correctly.
Comment 20 Jeremy Allison 2014-05-21 22:37:44 UTC
Yeah I guess it depends if it's comparing with what is stored locally, or if it's doing a comparison between what is returned from SMB2_GETINFO and what came back from SMB2_CREATE. Even if it doesn't completely fix the robocopy issue this is a nasty bug that we must fix (IMHO). Can't have inconsistent return times between CREATE and GETINFO on the same file :-(.
Comment 21 Thomas Bork 2014-05-21 22:55:42 UTC
Nice to see that only 9 years after I reported this bug someone tries to 
fix it :)

> Sure, but the original bug was with XP, so it really seems to be related to not
> storing the full time stamp resolution.

Yes, this was XP and samba 3.x. No relation to SMB2.

der tom
Comment 22 Andrew Klosterman 2014-05-22 19:03:52 UTC
(In reply to comment #12)
> Created attachment 9963 [details]
> git-am fix for 3.6.x.
> 
> OK Andrew, if you can try this on 3.6.x I'd appreciate it !

I'm sorry, but my "day job" is preventing me from moving forward with trying the patch.  Maybe I can revisit it later on...  :-/

I sure hope that there are no more "oops, we skipped the rounding!" situations and this is really the last one remaining to be found.

A 9 year old bug ... I'm happy to help squash that one!  ;-)
Comment 23 Jeremy Allison 2014-05-22 19:06:31 UTC
Well I'm pretty confident in the fix, but it would really help to have confirmation from you, so if you could get around to it soon... (else why did you report it in the first place ? :-).

Now I've forced all date/time sets to go via the standard put_long_date_timespec() function even in the SMB2 server we shouldn't have any more problems like this.

Jeremy.
Comment 24 Thomas Bork 2014-05-22 20:11:16 UTC
> Well I'm pretty confident in the fix, but it would really help to have
> confirmation from you, so if you could get around to it soon... (else why did
> you report it in the first place ? :-).

A user of us had this problem in 2005.
I will try your patch with 4.1.7. First I confirmed, that the bug still exist (samba 4.1.7, Win7, SMB2):

D:\TOM\Documents>xcopy Testordner Z: /z /e /d /c /i /r /h /y
Testordner\test.txt
Testordner\Ordner3\test.txt
2 Datei(en) kopiert

D:\TOM\Documents>xcopy Testordner Z: /z /e /d /c /i /r /h /y
Testordner\test.txt
Testordner\Ordner3\test.txt
2 Datei(en) kopiert

Next step is to patch 4.1.7 and see, if your patch helps in case of SMB2 and SMB1 (forced via smb.conf).

> Now I've forced all date/time sets to go via the standard
> put_long_date_timespec() function even in the SMB2 server we shouldn't have any
> more problems like this.

Shure this works now with SMB1 too?
Comment 25 Jeremy Allison 2014-05-22 20:26:20 UTC
None of these patches change the SMB1 code, that code always correctly goes through the put_long_date_timespec() function.
Comment 26 Thomas Bork 2014-05-22 21:05:40 UTC
The bug I reported was with XP and samba 3.x - so not SMB2 related (see comment 21).

And sorry, this bug is still the same with your patch for SMB2 and Win7 and samba 4.1.7. Also with Win7 and samba 4.1.7 with a forced SMB1 connection (max protocol = NT1):

D:\TOM\Documents>xcopy Testordner Z: /z /e /d /c /i /r /h /y
Testordner\test.txt
Testordner\Ordner3\test.txt
2 Datei(en) kopiert

D:\TOM\Documents>xcopy Testordner Z: /z /e /d /c /i /r /h /y
Testordner\test.txt
Testordner\Ordner3\test.txt
2 Datei(en) kopiert
Comment 27 Jeremy Allison 2014-05-22 21:48:05 UTC
(In reply to comment #26)
> The bug I reported was with XP and samba 3.x - so not SMB2 related (see comment
> 21).
> 
> And sorry, this bug is still the same with your patch for SMB2 and Win7 and
> samba 4.1.7. Also with Win7 and samba 4.1.7 with a forced SMB1 connection (max
> protocol = NT1):
> 
> D:\TOM\Documents>xcopy Testordner Z: /z /e /d /c /i /r /h /y
> Testordner\test.txt
> Testordner\Ordner3\test.txt
> 2 Datei(en) kopiert
> 
> D:\TOM\Documents>xcopy Testordner Z: /z /e /d /c /i /r /h /y
> Testordner\test.txt
> Testordner\Ordner3\test.txt
> 2 Datei(en) kopiert

So I think you're looking at a different issue (or this bug has been hijacked by a different issue, depending on how you look at it :-).

The fix certainly addresses the timestamp rounding resolution reported by Andrew Klosterman, so I'm guessing your problem is a different issue which I didn't fix (sorry).
Comment 28 Jeremy Allison 2014-05-22 21:50:10 UTC
Thomas, what filesystem are you backing the Samba share with ? Does it actually store the full 100ns timestamp resolution ? If not then as Richard Sharpe pointed out it's a problem that can't be fixed for your filesystem.
Comment 29 Richard Sharpe 2014-05-23 17:25:33 UTC
(In reply to comment #28)
> Thomas, what filesystem are you backing the Samba share with ? Does it actually
> store the full 100ns timestamp resolution ? If not then as Richard Sharpe
> pointed out it's a problem that can't be fixed for your filesystem.

It might also be a failure to detect the availability of utimensat etc.
Comment 30 Thomas Bork 2014-05-23 18:34:30 UTC
> Thomas, what filesystem are you backing the Samba share with ? Does it actually
> store the full 100ns timestamp resolution ? If not then as Richard Sharpe
> pointed out it's a problem that can't be fixed for your filesystem.

This is ext4. How can I check the time resolution? Searching brings the 
information, that ext4 has a time resolution of nanoseconds.

Output from stat for a copied file:
samba 2.2.8 # stat /testshare/test.txt
   File: `/testshare/test.txt'
   Size: 6               Blocks: 8          IO Block: 4096   regular file
Device: 803h/2051d      Inode: 179175      Links: 1
Access: (0777/-rwxrwxrwx)  Uid: ( 2001/      tb)   Gid: (  100/   users)
Access: 2014-05-22 23:03:18.024555000 +0200
Modify: 2014-05-22 21:58:34.042671300 +0200
Change: 2014-05-23 00:02:26.965831055 +0200
  Birth: -
Comment 31 Jeremy Allison 2014-05-23 18:37:02 UTC
In that case, with the patch applied it certainly should be returning the correct timestamps that match the ones stored locally on NTFS on the client.

Is there any debug option from robocopy to let us know why it decided is needs to update a file ?

A debug level 10 log + capture trace with Robocopy making that decision for one file only in one directory would be helpful.

Cheers,

Jeremy.
Comment 32 Thomas Bork 2014-05-23 19:34:20 UTC
(In reply to comment #31)

> Is there any debug option from robocopy to let us know why it decided is needs
> to update a file ?

This is _xcopy_ - not robocopy. I cannot find such a debug switch in xcopy.

> A debug level 10 log + capture trace with Robocopy making that decision for one
> file only in one directory would be helpful.

Ok.
Comment 33 Thomas Bork 2014-05-23 19:36:41 UTC
Created attachment 9972 [details]
tcpdump dump of xcopy writing a unchanged file
Comment 34 Thomas Bork 2014-05-23 19:39:10 UTC
Created attachment 9973 [details]
log.smbd with debug level 10 with only one file (test.txt)
Comment 35 Jeremy Allison 2014-05-23 21:56:58 UTC
(In reply to comment #34)
> Created attachment 9973 [details]
> log.smbd with debug level 10 with only one file (test.txt)

There's something horribly wrong with the timestamp resolution returned on the files on that share. Everything is being rounded to 1 second (no sub-second resolution is being returned).

Can you tell me the contents of the following defines in your config.h
file:

HAVE_STAT_HIRES_TIMESTAMPS
HAVE_STRUCT_STAT_ST_MTIM_TV_NSEC
HAVE_STRUCT_STAT_ST_MTIMENSEC
HAVE_STRUCT_STAT_ST_MTIME_N
HAVE_STRUCT_STAT_ST_UMTIME
HAVE_STRUCT_STAT_ST_MTIMESPEC_TV_NSEC

Here's what I get here in my bog-standard Ubuntu 12.10 box:

#define HAVE_STRUCT_STAT_ST_MTIM_TV_NSEC 1
/* #undef HAVE_STRUCT_STAT_ST_MTIMENSEC */
/* #undef HAVE_STRUCT_STAT_ST_MTIMESPEC_TV_NSEC */
/* #undef HAVE_STRUCT_STAT_ST_MTIME_N */
/* #undef HAVE_STRUCT_STAT_ST_UMTIME */
#define HAVE_STAT_HIRES_TIMESTAMPS 1
/* #undef HAVE_STRUCT_STAT_ST_BIRTHTIME */
/* #undef HAVE_STRUCT_STAT_ST_BIRTHTIMESPEC_TV_NSEC */
/* #undef HAVE_STRUCT_STAT_ST_BIRTHTIMENSEC */
Comment 36 Thomas Bork 2014-05-23 22:51:25 UTC
> Can you tell me the contents of the following defines in your config.h
> file:
> HAVE_STAT_HIRES_TIMESTAMPS

#define HAVE_STAT_HIRES_TIMESTAMPS 1

> HAVE_STRUCT_STAT_ST_MTIM_TV_NSEC

#define HAVE_STRUCT_STAT_ST_MTIM_TV_NSEC 1

> HAVE_STRUCT_STAT_ST_MTIMENSEC

/* #undef HAVE_STRUCT_STAT_ST_MTIMENSEC */

> HAVE_STRUCT_STAT_ST_MTIME_N

/* #undef HAVE_STRUCT_STAT_ST_MTIME_N */

> HAVE_STRUCT_STAT_ST_UMTIME

/* #undef HAVE_STRUCT_STAT_ST_UMTIME */

> HAVE_STRUCT_STAT_ST_MTIMESPEC_TV_NSEC

/* #undef HAVE_STRUCT_STAT_ST_MTIMESPEC_TV_NSEC */

> Here's what I get here in my bog-standard Ubuntu 12.10 box:
> #define HAVE_STRUCT_STAT_ST_MTIM_TV_NSEC 1
> /* #undef HAVE_STRUCT_STAT_ST_MTIMENSEC */
> /* #undef HAVE_STRUCT_STAT_ST_MTIMESPEC_TV_NSEC */
> /* #undef HAVE_STRUCT_STAT_ST_MTIME_N */
> /* #undef HAVE_STRUCT_STAT_ST_UMTIME */
> #define HAVE_STAT_HIRES_TIMESTAMPS 1
> /* #undef HAVE_STRUCT_STAT_ST_BIRTHTIME */
> /* #undef HAVE_STRUCT_STAT_ST_BIRTHTIMESPEC_TV_NSEC */
> /* #undef HAVE_STRUCT_STAT_ST_BIRTHTIMENSEC */

I have the same:
#define HAVE_STRUCT_STAT_ST_MTIM_TV_NSEC 1
/* #undef HAVE_STRUCT_STAT_ST_MTIMENSEC */
/* #undef HAVE_STRUCT_STAT_ST_MTIMESPEC_TV_NSEC */
/* #undef HAVE_STRUCT_STAT_ST_MTIME_N */
/* #undef HAVE_STRUCT_STAT_ST_UMTIME */
#define HAVE_STAT_HIRES_TIMESTAMPS 1
/* #undef HAVE_STRUCT_STAT_ST_BIRTHTIME */
/* #undef HAVE_STRUCT_STAT_ST_BIRTHTIMESPEC_TV_NSEC */
/* #undef HAVE_STRUCT_STAT_ST_BIRTHTIMENSEC */

Related to VMWare? Compiled in virtual machine and executed in virtual 
machine.

deveisneu # /lib/libc.so.6
GNU C Library stable release version 2.11.3, by Roland McGrath et al.
Copyright (C) 2009 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.5.4.
Compiled on a Linux 2.6.32 system on 2013-05-29.
Available extensions:
         crypt add-on version 2.1 by Michael Glad and others
         GNU Libidn by Simon Josefsson
         NoVersion patch for broken glibc 2.0 binaries
         Native POSIX Threads Library by Ulrich Drepper et al
         BIND-8.2.3-T5B
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.
deveisneu # gcc --version
gcc (GCC) 4.5.4
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

deveisneu #

On the executing machine:

samba # smbd -b | grep TIME
    HAVE_SYS_TIMEB_H
    HAVE_SYS_TIMES_H
    HAVE_SYS_TIME_H
    HAVE_TIME_H
    HAVE_UTIME_H
    HAVE_UT_UT_TIME
    HAVE_CLOCK_GETTIME
    HAVE_CLOCK_PROCESS_CPUTIME_ID
    HAVE_CLOCK_REALTIME
    HAVE_DECL_STRPTIME
    HAVE_FUTIMENS
    HAVE_FUTIMES
    HAVE_GETTIMEOFDAY_TZ
    HAVE_KRB5_CC_GET_LIFETIME
    HAVE_KRB5_SET_REAL_TIME
    HAVE_LUTIMES
    HAVE_MKTIME
    HAVE_SETITIMER
    HAVE_STAT_HIRES_TIMESTAMPS
    HAVE_STRFTIME
    HAVE_STRPTIME
    HAVE_STRUCT_TIMESPEC
    HAVE_TIMEGM
    HAVE_UTIME
    HAVE_UTIMENSAT
    HAVE_UTIMES
    HAVE_WORKING_STRPTIME
    SIZEOF_TIME_T
    TIME_WITH_SYS_TIME
Comment 37 Thomas Bork 2014-05-23 23:02:29 UTC
> Related to VMWare? Compiled in virtual machine and executed in virtual
> machine.

Or related to

dos filetime resolution = yes

in smb.conf?
Comment 38 Jeremy Allison 2014-05-23 23:05:07 UTC
Oh god, I didn't know you had that set :-).

Yeah, remove that and your problems with xcopy will go away :-).

dos filetime resolution = yes

rounds internal time to a *2 second* resolution (as that's all DOS will support). We used to need it for some strange VisualC++ build errors people had off DOS filesystems..
Comment 39 Thomas Bork 2014-05-23 23:22:17 UTC
> Yeah, remove that and your problems with xcopy will go away :-).

Mmpf, this compatibility option is set since years (samba 2.x?) in my 
package. Sorry for the noise...
Comment 40 Jeremy Allison 2014-05-23 23:28:11 UTC
Oh that's cool - that means that the git-am fixes *are* actually correct for the 4.1.x and 4.0.x code for this bug. Also means that the change David got me to make to remove the bug ref. in the master fix was incorrect. Never mind :-).

I'll ask him to review the 4.1.x and 4.0.x patch now it's gone into master :-).

Jeremy.
Comment 41 Jeremy Allison 2014-05-23 23:29:14 UTC
Comment on attachment 9962 [details]
git-am fix for 4.0.x and 4.1.x.

Adding David now it's gone into master, and these patches are actually correct for this bug :-).

Jeremy.
Comment 42 David Disseldorp 2014-05-24 19:35:28 UTC
(In reply to comment #41)
> Comment on attachment 9962 [details]
> git-am fix for 4.0.x and 4.1.x.
> 
> Adding David now it's gone into master, and these patches are actually correct
> for this bug :-).

Oops, sorry to ask for the tag removal... I take it the SMB1 path was fixed long ago then too.
Comment 43 Jeremy Allison 2014-05-27 19:02:46 UTC
Re-assigning to Karolin for inclusion in 4.1.next, 4.0.next.

Thanks !

Jeremy.
Comment 44 Karolin Seeger 2014-05-27 19:08:49 UTC
(In reply to comment #43)
> Re-assigning to Karolin for inclusion in 4.1.next, 4.0.next.
> 
> Thanks !
> 
> Jeremy.

Pushed to autobuild-v4-[0|1]-test.
Comment 45 Karolin Seeger 2014-06-03 06:55:47 UTC
(In reply to comment #44)
> (In reply to comment #43)
> > Re-assigning to Karolin for inclusion in 4.1.next, 4.0.next.
> > 
> > Thanks !
> > 
> > Jeremy.
> 
> Pushed to autobuild-v4-[0|1]-test.

Pushed to v4-1-test.
autobuild-v4-0-test failed, re-trying.
Comment 46 Karolin Seeger 2014-06-10 09:00:56 UTC
(In reply to comment #45)
> (In reply to comment #44)
> > (In reply to comment #43)
> > > Re-assigning to Karolin for inclusion in 4.1.next, 4.0.next.
> > > 
> > > Thanks !
> > > 
> > > Jeremy.
> > 
> > Pushed to autobuild-v4-[0|1]-test.
> 
> Pushed to v4-1-test.
> autobuild-v4-0-test failed, re-trying.

Pushed to both branches.
Closing out bug report.

Thanks!