==Summary== This problem appears to be in the CIFS client, and only happens for self-mounts. I confirmed this by: - Mounting the share on Windows 7 and using winexe to read the file remotely. - Mounting the share on another Linux machine and using ssh to read the file remotely. In a nutshell, the problem is this: if a machine has a CIFS mount to the same machine, the samba client can intermittently return cached data from deleted files, if the file being read happens to have the same inode as the delete file. I repro'd this problem on CentOS 6.3 with both of these versions: samba-client.x86_64 3.5.10-125.el6 samba-client.x86_64 3.6.9-151.el6 ==Steps to reproduce== 1. mkdir /tmp/public; chmod 0777 /tmp/public 2. Create a public share in smb.conf, using this: [global] <snip other stuff> guest account = nobody map to guest = bad user [public] comment = Public Test Share browseable = yes path = /tmp/public public = yes writeable = no guest ok = yes 3. /etc/init.d/smb restart 4. Save the script below to samba_test.pl. The script creates a temporary file with 6 random numbers. It then reads the file directly from the local disk as well as from the mounted share and compares the two. Eventually you'll see that it fails because the two text strings differ. You'll notice that content of the previous (now deleted) file is being returned as the content from the mount. 5. perl samba_test.pl > samba_test.log ==Expected Output== Script runs without stopping. Please stop it after 60 seconds with CTRL-c. ==Actual Output== After less than 60 seconds (usually 10 seconds or so) the script stops. Running "tail samba_test.log" shows something like this: ---- Iteration 570, file tmp.m7sAXnesuN (inode #5474)... Same! /tmp/public/tmp.m7sAXnesuN : 11015 5368 22743 3105 20133 22460 /tmp/mount/tmp.m7sAXnesuN: 11015 5368 22743 3105 20133 22460 ---- Iteration 571, file tmp.1oSZnnqLLi (inode #5474)... Different! /tmp/public/tmp.1oSZnnqLLi : 9274 23766 26247 21653 6126 18578 /tmp/mount/tmp.1oSZnnqLLi: 11015 5368 22743 3105 20133 22460 This shows that the samba client is providing the content of /tmp/mount/tmp.1oSZnnqLLi as the content from iteration 570, which is a different file that was deleted at the end of iteration 570! If you run it multiple times, you'll also see that the data read from the file occasionally gets truncated: ---- Iteration 330, file tmp.KnYmJFv9kc (inode #5788)... Same! /tmp/public/tmp.KnYmJFv9kc : 14341 26110 16704 28754 15253 21681 /tmp/mount/tmp.KnYmJFv9kc: 14341 26110 16704 28754 15253 21681 ---- Iteration 331, file tmp.o0KFtMXSOK (inode #5788)... Different! /tmp/public/tmp.o0KFtMXSOK : 4659 27100 7872 24673 15969 22105 /tmp/mount/tmp.o0KFtMXSOK: 14341 26110 16704 28754 15253 2168<no newline> In this case the "1\n" at the end of the previously (deleted) file is missing. One can confirm both of these cases after the script runs by simply using "cat": # cat /tmp/public/tmp.o0KFtMXSOK 4659 27100 7872 24673 15969 22105 # cat /tmp/mount/tmp.o0KFtMXSOK 14341 26110 16704 28754 15253 2168# Let me know if you need any more information. ==Script== #!/usr/bin/perl use strict; # Unbuffered output $| = 1; my $share = 'public'; my $local_shared_dir = '/tmp/public'; my $mount_point = '/tmp/mount'; mkdir $mount_point unless -d $mount_point; # Make sure it's not already mounted system "umount $mount_point 2>/dev/null"; system("mount -t cifs -oguest //localhost/$share $mount_point") == 0 or die "Couldn't mount!"; my $iteration = 0; while (1) { $iteration++; # Create a temp file with random numbers my $temp_file = `mktemp --tmpdir=$local_shared_dir`; chomp $temp_file; system "echo \$RANDOM \$RANDOM \$RANDOM \$RANDOM \$RANDOM \$RANDOM > $temp_file"; chmod 0666, $temp_file; my $temp_filename = `basename $temp_file`; chomp $temp_filename; my $inode = (stat "$local_shared_dir/$temp_filename")[1]; print "----\nIteration $iteration, file $temp_filename (inode #$inode)... "; # Read the line from the newly created file from the local file system open IN, "$local_shared_dir/$temp_filename" or die "Couldn't open local: $!"; my $original_text = <IN>; close IN; # Now read the line from the newly created file from the mounted file system open IN, "$mount_point/$temp_filename" or die "Couldn't open mounted: $!"; my $mount_text = <IN>; close IN; if ($original_text eq $mount_text) { print "Same!\n"; print "$local_shared_dir/$temp_filename : $original_text"; print "$mount_point/$temp_filename: $mount_text"; } else { print "Different!\n"; print "$local_shared_dir/$temp_filename : $original_text"; print "$mount_point/$temp_filename: $mount_text"; exit; } unlink "$local_shared_dir/$temp_filename"; }
We faced this bug after migration from centos 5 to centos 6. During customer's file integration our application server was often reading incorrect file, which lead to integration fail. It took us a while, after a lot of additional logging and rollbacks, to understand that the problem might be with cifs cache (btw, thanks a lot to David for this report and test case - took us just half a day to reproduce and fix it after we found this bug) After we disabled cifs cache the problem had disappeared. Please consider to increase priority of this bug. It's really critical in multi-customer environment as it can lead to situation when process of one customer reads file that belongs to different customer.