Bug 6926 - Server (and Linux client) doesn't honor file lock
Summary: Server (and Linux client) doesn't honor file lock
Status: RESOLVED FIXED
Alias: None
Product: CifsVFS
Classification: Unclassified
Component: kernel fs (show other bugs)
Version: 2.4
Hardware: Other Linux
: P3 normal
Target Milestone: ---
Assignee: Jeff Layton
QA Contact: Samba QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-11-24 13:59 UTC by Jörg Sommer (mail address dead)
Modified: 2010-02-05 10:09 UTC (History)
2 users (show)

See Also:


Attachments
strace of smbd of the writing client while reproducing this bug (172.56 KB, text/plain)
2009-11-24 14:01 UTC, Jörg Sommer (mail address dead)
no flags Details
strace of smbd of the locking (reading) client while reproducing this bug (226.16 KB, text/plain)
2009-11-24 14:01 UTC, Jörg Sommer (mail address dead)
no flags Details
Network traffic between server and both clients while reproducing this bug (132.08 KB, application/octet-stream)
2009-11-24 14:04 UTC, Jörg Sommer (mail address dead)
no flags Details
strace of the writing process on (wireshark) client W while reproducing this bug (159.74 KB, text/plain)
2009-11-24 14:12 UTC, Jörg Sommer (mail address dead)
no flags Details
Network traffic with client R == Linux && client W == Windows (2.83 KB, application/octet-stream)
2009-11-25 10:33 UTC, Jörg Sommer (mail address dead)
no flags Details
test patch 1 -- allow cifs to invalidate cached data for open files (1.26 KB, patch)
2009-12-07 09:54 UTC, Jeff Layton
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jörg Sommer (mail address dead) 2009-11-24 13:59:43 UTC
While a client owns a file lock for reading another client can modify this file. Then the client never detects the file change.

Client R holds a read lock of a file:

# smbstatus

Samba version 3.4.3
PID     Username      Group         Machine                        
-------------------------------------------------------------------
21485     joerg         adm           __ffff_10.10.10.26 (::ffff:10.10.10.26)
20634     krause        entwickler    __ffff_10.10.10.28 (::ffff:10.10.10.28)

Service      pid     machine       Connected at
-------------------------------------------------------
projects     20634   __ffff_10.10.10.28  Tue Nov 24 08:19:58 2009
projects     21485   __ffff_10.10.10.26  Tue Nov 24 15:32:31 2009
all-homes    21485   __ffff_10.10.10.26  Tue Nov 24 15:32:31 2009
all-homes    20634   __ffff_10.10.10.28  Tue Nov 24 08:19:58 2009

Locked files:
Pid          Uid        DenyMode   Access      R/W        Oplock           SharePath   Name   Time
--------------------------------------------------------------------------------------------------
21485        1000       DENY_NONE  0x81        RDONLY     NONE             /home   Öffentlich/joerg/x-smb-network.dump   Tue Nov 24 20:25:36 2009

But the client W can write to this file. After the write the file is modified on the server filesystem, but client R doesn't detect this change:

server% md5sum x-smb-network.dump
bdc3db9ef626db402de0de594bbca90d  x-smb-network.dump
client R% md5sum x-smb-network.dump
bdc3db9ef626db402de0de594bbca90d  x-smb-network.dump
client W% md5sum x-smb-network.dump
bdc3db9ef626db402de0de594bbca90d  x-smb-network.dump

== now client W modifies the file ==
server% md5sum x-smb-network.dump
2931b76f94312bd864a92297d29a9131  /home/Öffentlich/joerg/x-smb-network.dump
client W% md5sum x-smb-network.dump
2931b76f94312bd864a92297d29a9131  /home/Öffentlich/joerg/x-smb-network.dump
client R% md5sum x-smb-network.dump
bdc3db9ef626db402de0de594bbca90d  x-smb-network.dump

Client R runs a Linux 2.6.31-rc5, client W runs Linux 2.6.28-16 and the server runs Samba 3.4.3.

I think the server should not allow a write access to a file that is locked by a client or at least the Linux client must not assume file locks work.
Comment 1 Jörg Sommer (mail address dead) 2009-11-24 14:01:02 UTC
Created attachment 4998 [details]
strace of smbd of the writing client while reproducing this bug
Comment 2 Jörg Sommer (mail address dead) 2009-11-24 14:01:42 UTC
Created attachment 4999 [details]
strace of smbd of the locking (reading) client while reproducing this bug
Comment 3 Jörg Sommer (mail address dead) 2009-11-24 14:04:35 UTC
Created attachment 5000 [details]
Network traffic between server and both clients while reproducing this bug
Comment 4 Jörg Sommer (mail address dead) 2009-11-24 14:12:53 UTC
Created attachment 5001 [details]
strace of the writing process on (wireshark) client W while reproducing this bug

In the jitter, you can find this line:

12076 20:43:55.104443 open("/mnt/homes/\303\226ffentlich/joerg/x-smb-network.dump", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 8

i.e. the client write access to the locked file.
Comment 5 Jeremy Allison 2009-11-24 14:17:23 UTC
I don't see any fcntl lock requests in your strace. You do know that POSIX locks are *advisory* don't you. I.e. Samba won't prevent a POSIX client writing to a file - they client has to coordinate the locks themselves with fcntl calls.

Jeremy.
Comment 6 Jörg Sommer (mail address dead) 2009-11-24 15:46:38 UTC
(In reply to comment #5)
> I don't see any fcntl lock requests in your strace. You do know that POSIX
> locks are *advisory* don't you. I.e. Samba won't prevent a POSIX client writing
> to a file - they client has to coordinate the locks themselves with fcntl
> calls.

I don't know where this Lock in smbstatus comes from, but to me it doesn't look like the application installs this lock. Start less on a client and then run smbstatus. At me, I get a file lock, while less does nothing else than open, read and keeping the file open. It seems this is enough to get an entry in the file lock section of smbstatus. So these locks aren't handled on application level.

And even if this is a POSIX lock and the application (wireshark) misbehaves, the client R should get access to the new file or get a hint that the file has changed.
Comment 7 Jeremy Allison 2009-11-24 19:11:54 UTC
I don't see any locks in your trace. Don't be confused by the "Locked files" comment in smbstatus. This should really say "Open files". In fact I think I'll change it....
Jeremy.
Comment 8 Jörg Sommer (mail address dead) 2009-11-25 06:36:35 UTC
But trying to change a "locked/opened" file from a Windows client results in access denied. And when a Windows client has opened a file I can't write to this file on the linux client.

And even if all this is fine, why does the linux client R not get the file update after client W has changed the file? (See the md5sums in the bug report) Is this a bug in the linux client?
Comment 9 Jörg Sommer (mail address dead) 2009-11-25 10:33:48 UTC
Created attachment 5010 [details]
Network traffic with client R == Linux && client W == Windows

When opening a file with less on a Linux client (== client R) and try to save something to this file on a Windows XP client (== client W) then the Windows client refuses to save the file.

# smbstatus

Samba version 3.4.3
PID     Username      Group         Machine                        
-------------------------------------------------------------------
9138      joerg         adm           __ffff_10.10.10.26 (::ffff:10.10.10.26)
13067     joerg         adm           muffi        (::ffff:10.10.10.21)

Service      pid     machine       Connected at
-------------------------------------------------------
all-homes    13067   muffi         Wed Nov 25 17:06:30 2009
projects     9138   __ffff_10.10.10.26  Wed Nov 25 12:29:35 2009
all-homes    9138   __ffff_10.10.10.26  Wed Nov 25 12:29:35 2009

Locked files:
Pid          Uid        DenyMode   Access      R/W        Oplock           SharePath   Name   Time
--------------------------------------------------------------------------------------------------
9138         1000       DENY_NONE  0x81        RDONLY     NONE             /home   joerg/file   Wed Nov 25 17:20:41 2009
13067        1000       DENY_NONE  0x100081    RDONLY     NONE             /home   joerg   Wed Nov 25 17:17:19 2009
13067        1000       DENY_NONE  0x100081    RDONLY     NONE             /home   .   Wed Nov 25 17:17:16 2009

This differs from the behaviour explained in the bug report. A Linux client can change the file, while a Windows client can't.
Comment 10 Volker Lendecke 2009-12-06 14:52:26 UTC
The Windows behaviour is understandable. Windows tries to open the file with DENY_ALL (or rather FILE_SHARE set to 0). As the file is already open, Samba denies this second open. Linux clients don't do that.

The fact that a second client does not see the changes made by the first one is something I would like Jeff Layton to look at. Re-assigning the bug to him.

Volker
Comment 11 Jeff Layton 2009-12-07 07:52:36 UTC
Thanks for the bug report. Sounds like there's a problem with the revalidation code -- i.e. we're getting new attributes for the inode but not invalidating the cache. Then when the client goes to read the file again, it trusts that the cache is OK because it doesn't notice the timestamp changes.

Can you outline exactly how you're reproducing that problem?
Comment 12 Jörg Sommer (mail address dead) 2009-12-07 09:01:17 UTC
I can reproduce this bug this way:

shell-1% echo 1 > file
shell-1% less 1

shell-2% ssh remote-host echo 2\> file\; cat file
2
shell-2% cat file
1
Comment 13 Jeff Layton 2009-12-07 09:54:38 UTC
Created attachment 5062 [details]
test patch 1 -- allow cifs to invalidate cached data for open files

Ahh ok. I think I see the problem...

CIFS currently doesn't allow the kernel to invalidate the pagecache for files that it has open. It's not clear to me why this is the case.

This patch changes that behavior, but note that it's completely untested. The patch is also racy. It's easily possible for it to end up invalidating pages that were dirtied after the writeback. Fixing this the right way will mean making sure that can't happen.

Still though, it would be good to have you test this patch to make sure we're on the right track here. If you have a non-critical place to do so, please test it and let me know if it fixes the issue.

In the example above, patching just the "reader" client should be sufficient.
Comment 14 Jeff Layton 2009-12-08 05:30:56 UTC
Adding Steve and Shirish...

I think this problem is due to the fact that CIFS doesn't invalidate the cache on open files. The patch I attached is just for testing. A real patch will probably need to convert this code to use invalidate_inode_page2 and check the return value.

We'll probably also need to change this code not to do the invalidation in d_revalidate but instead just flag the inode for invalidation when an operation needs it -- on reads in particular, but probably a few other calls too.

Comment 15 Jörg Sommer (mail address dead) 2009-12-08 07:33:12 UTC
(In reply to comment #13)
> Created an attachment (id=5062) [details]
> test patch 1 -- allow cifs to invalidate cached data for open files
> 
> Ahh ok. I think I see the problem...

But his patch fixes the problem only partly. With this patch I get the new content after closing less. But while less keeps the file open, a cat in a second terminal shows the old content.

This works:
shell-1% echo 1 > file
shell-1% less file

shell-2% ssh remote-host echo 2\> file\; cat file
2
shell-1% <close less>
shell-1% cat file
1

This doesn't work:
shell-1% echo 1 > file
shell-1% less file

shell-2% ssh remote-host echo 2\> file\; cat file
2
shell-2% cat file # less is still open in shell-1
1
Comment 16 shirishpargaonkar@gmail.com 2009-12-08 09:48:44 UTC
I can't recreate this problem with fairly recent cifs client and 
[root@linux-nfs2 smb_t]# /usr/sbin/smbd -V
Version 3.0.28-0.el5.8
what am I missing?

-------------------------------------------------------------------------------
cifstest6:/mnt/smb_b # tty
/dev/pts/0

cifstest6:/mnt/smb_b # mount

//linux-nfs2/smbt on /mnt/smb_b type cifs (rw,mand)

cifstest6:/mnt/smb_b # pwd
/mnt/smb_b

cifstest6:/mnt/smb_b # echo 1 > file3
cifstest6:/mnt/smb_b # less file3

1
file3 lines 1-1/1 (END)

------------------------------------------------------------------------------

cifstest6:/mnt/smb_b # tty
/dev/pts/2

cifstest6:/mnt/smb_b # pwd
/mnt/smb_b

cifstest6:/mnt/smb_b # ssh linux-nfs2.  echo 2 > file3; cat file3
root@linux-nfs2's password:
2
-------------------------------------------------------------------------------

cifstest6:/mnt/smb_b # cat file3
2
Comment 17 shirishpargaonkar@gmail.com 2009-12-08 10:03:00 UTC
Forgot to run this command,  cat file3 in pts2 with or without less keeping
the file open still displays correct contents, i.e. 2 and not 1.
Comment 18 Jörg Sommer (mail address dead) 2009-12-08 11:21:52 UTC
(In reply to comment #16)
> I can't recreate this problem with fairly recent cifs client and 
> [root@linux-nfs2 smb_t]# /usr/sbin/smbd -V
> Version 3.0.28-0.el5.8
> what am I missing?

> cifstest6:/mnt/smb_b # ssh linux-nfs2.  echo 2 > file3; cat file3

Did you really run this command? Then the only thing you did on the remote side is 'echo 2.' The redirect (i.e. the write) and the cat happend on the client that runs less. But you have to modify the file on the remote side.

Do you access the filesystem on the remove side via cifs? What I did was accessing the file from two different clients. I've never accessed the filesystem exported by Samba.

# smbd --version
Version 3.4.3
Comment 19 shirishpargaonkar@gmail.com 2009-12-08 11:33:14 UTC
(In reply to comment #18)
> > cifstest6:/mnt/smb_b # ssh linux-nfs2  echo 2 > file3; cat file3
> Did you really run this command? Then the only thing you did on the remote side

Yes, I have cd'ed to the mounted directory and doing all the writing.
I did check the contents of the file on the server as well and it is 
the second echo i.e. 2.
Comment 20 Jörg Sommer (mail address dead) 2009-12-08 12:58:22 UTC
Me too. I can't reproduce it with 3.0, but with 3.4. A second problem is
that I can't write to files on the 3.0 server.

% cat /mnt/other/test
1
% echo 2 > /mnt/other/test
zsh: no such file or directory: /mnt/other/test
% ls -l /mnt/other/test
-rwxrwSrwx 1 root root 2 Dec  8 19:10 /mnt/other/test

Hence, I have to do the access with smbclient.

% grep /mnt/home /proc/mounts 
//cifs-homes.net.de/all-homes /mnt/home cifs rw,mand,nosuid,nodev,relatime,unc=\\cifs-homes.net.de\all-homes,username=joerg,domain=ATeO,addr=10.10.10.6,posixpaths,serverino,acl,rsize=16384,wsize=57344 0 0
% echo 1 > /mnt/home/joerg/test
% less /mnt/home/joerg/test
---------------------------------------------------
% echo 2 > /tmp/test
% cd /tmp; PAGER=cat smbclient -U joerg //cifs-homes.net.de/all-homes -D joerg -c 'more test; put test; more test'
Enter joerg's password: 
Domain=[ATEO] OS=[Unix] Server=[Samba 3.4.3]
getting file \joerg\test of size 2 as /tmp/smbmore.Zkfs9Y (0,2 KiloBytes/sec) (average 0,2 KiloBytes/sec)
1
putting file test as \joerg\test (0,1 kb/s) (average 0,1 kb/s)
getting file \joerg\test of size 2 as /tmp/smbmore.wvHMmt (0,2 KiloBytes/sec) (average 0,2 KiloBytes/sec)
2
% cat /mnt/home/joerg/test
1

Now, a different server:

% grep /mnt/other /proc/mounts
//saturn93b/www /mnt/other cifs rw,mand,nosuid,nodev,relatime,unc=\\saturn93b\www,username=poolteam,addr=10.10.10.7,posixpaths,serverino,acl,rsize=16384,wsize=57344 0 0
% echo 1 > /mnt/other/test
% less /mnt/other/test
---------------------------------------------------
% echo 2 > /tmp/test
% cd /tmp; PAGER=cat smbclient -U poolteam //saturn93b/www -c 'more test; put test; more test'
Enter poolteam's password: 
Domain=[INFONET] OS=[Unix] Server=[Samba 3.0.23d-6-1083-SUSE-SL10.2]
getting file \test of size 2 as /tmp/smbmore.Bpehly (0,5 KiloBytes/sec) (average 0,5 KiloBytes/sec)
1
putting file test as \test (0,5 kb/s) (average 0,5 kb/s)
getting file \test of size 2 as /tmp/smbmore.t76xwe (1,0 KiloBytes/sec) (average 0,7 KiloBytes/sec)
2
% cat /mnt/other/test
2
Comment 21 Jeff Layton 2010-01-11 10:01:10 UTC
I've tried to reproduce this in several ways and have had no luck. The server is running:

samba-3.4.2-47.fc12.x86_64

...I've tested this with the following kernels:

kernel-2.6.31.9-174.fc12.x86_64
kernel-2.6.32.2-18.fc13.x86_64

...I'm afraid I don't have anything older handy. I noticed in the original description that one of the hosts is running a 2.6.28 kernel. Can you reproduce this when that client is running a newer one?

Also, it might be helpful to see if you can reproduce this on a non -rc kernel. Perhaps this was a bug fixed in the -rc phase?
Comment 22 Jörg Sommer (mail address dead) 2010-02-05 10:06:25 UTC
(In reply to comment #21)
> I've tried to reproduce this in several ways and have had no luck. The server
> is running:
> 
> samba-3.4.2-47.fc12.x86_64
> 
> ...I've tested this with the following kernels:
> 
> kernel-2.6.31.9-174.fc12.x86_64
> kernel-2.6.32.2-18.fc13.x86_64

me too. My server is running Samba 3.4.3 and my client Linux 2.6.32-trunk-686 (Debian-Kernel).

> ...I'm afraid I don't have anything older handy. I noticed in the original
> description that one of the hosts is running a 2.6.28 kernel.

In the meantime, I've updated all client systems. I don't have a system with 2.6.28.

I think this bus is resolved.
Comment 23 Jeff Layton 2010-02-05 10:09:37 UTC
Ok then...

Closing bug with a resolution of FIXED. Please reopen if it returns.