Bug 10125 - cifs client returns cached data of deleted files
Summary: cifs client returns cached data of deleted files
Status: NEW
Alias: None
Product: CifsVFS
Classification: Unclassified
Component: kernel fs (show other bugs)
Version: 3.x
Hardware: x64 Linux
: P5 normal
Target Milestone: ---
Assignee: Steve French
QA Contact: cifs QA contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-09-02 15:42 UTC by David Coppit
Modified: 2020-12-14 07:37 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description David Coppit 2013-09-02 15:42:48 UTC
==Summary==

This problem appears to be in the CIFS client, and only happens for self-mounts. I confirmed this by:
- Mounting the share on Windows 7 and using winexe to read the file remotely.
- Mounting the share on another Linux machine and using ssh to read the file remotely.

In a nutshell, the problem is this: if a machine has a CIFS mount to the same machine, the samba client can intermittently return cached data from deleted files, if the file being read happens to have the same inode as the delete file.

I repro'd this problem on CentOS 6.3 with both of these versions:

samba-client.x86_64 3.5.10-125.el6
samba-client.x86_64 3.6.9-151.el6

==Steps to reproduce==

1. mkdir /tmp/public; chmod 0777 /tmp/public

2. Create a public share in smb.conf, using this:

[global]
    <snip other stuff>
    guest account = nobody
    map to guest = bad user

[public]
    comment = Public Test Share
    browseable = yes
    path = /tmp/public
    public = yes
    writeable = no
    guest ok = yes

3. /etc/init.d/smb restart

4. Save the script below to samba_test.pl.

The script creates a temporary file with 6 random numbers. It then reads the file directly from the local disk as well as from the mounted share and compares the two. Eventually you'll see that it fails because the two text strings differ. You'll notice that content of the previous (now deleted) file is being returned as the content from the mount.

5. perl samba_test.pl > samba_test.log

==Expected Output==

Script runs without stopping. Please stop it after 60 seconds with CTRL-c.

==Actual Output==

After less than 60 seconds (usually 10 seconds or so) the script stops. Running "tail samba_test.log" shows something like this:

----
Iteration 570, file tmp.m7sAXnesuN (inode #5474)... Same!
/tmp/public/tmp.m7sAXnesuN     : 11015 5368 22743 3105 20133 22460
/tmp/mount/tmp.m7sAXnesuN: 11015 5368 22743 3105 20133 22460
----
Iteration 571, file tmp.1oSZnnqLLi (inode #5474)... Different!
/tmp/public/tmp.1oSZnnqLLi     : 9274 23766 26247 21653 6126 18578
/tmp/mount/tmp.1oSZnnqLLi: 11015 5368 22743 3105 20133 22460

This shows that the samba client is providing the content of /tmp/mount/tmp.1oSZnnqLLi as the content from iteration 570, which is a different file that was deleted at the end of iteration 570!

If you run it multiple times, you'll also see that the data read from the file occasionally gets truncated:

----
Iteration 330, file tmp.KnYmJFv9kc (inode #5788)... Same!
/tmp/public/tmp.KnYmJFv9kc     : 14341 26110 16704 28754 15253 21681
/tmp/mount/tmp.KnYmJFv9kc: 14341 26110 16704 28754 15253 21681
----
Iteration 331, file tmp.o0KFtMXSOK (inode #5788)... Different!
/tmp/public/tmp.o0KFtMXSOK     : 4659 27100 7872 24673 15969 22105
/tmp/mount/tmp.o0KFtMXSOK: 14341 26110 16704 28754 15253 2168<no newline>

In this case the "1\n" at the end of the previously (deleted) file is missing.

One can confirm both of these cases after the script runs by simply using "cat":

# cat /tmp/public/tmp.o0KFtMXSOK
4659 27100 7872 24673 15969 22105
# cat /tmp/mount/tmp.o0KFtMXSOK
14341 26110 16704 28754 15253 2168# 

Let me know if you need any more information.

==Script==

#!/usr/bin/perl

use strict;

# Unbuffered output
$| = 1;

my $share = 'public';
my $local_shared_dir = '/tmp/public';
my $mount_point = '/tmp/mount';

mkdir $mount_point unless -d $mount_point;
# Make sure it's not already mounted
system "umount $mount_point 2>/dev/null";

system("mount -t cifs -oguest //localhost/$share $mount_point") == 0
   or die "Couldn't mount!";

my $iteration = 0;

while (1) {
  $iteration++;

  # Create a temp file with random numbers
  my $temp_file = `mktemp --tmpdir=$local_shared_dir`;
  chomp $temp_file;
  system "echo \$RANDOM \$RANDOM \$RANDOM \$RANDOM \$RANDOM \$RANDOM > $temp_file";
  chmod 0666, $temp_file;

  my $temp_filename = `basename $temp_file`;
  chomp $temp_filename;

  my $inode = (stat "$local_shared_dir/$temp_filename")[1];
  print "----\nIteration $iteration, file $temp_filename (inode #$inode)... ";

  # Read the line from the newly created file from the local file system
  open IN, "$local_shared_dir/$temp_filename" or die "Couldn't open local: $!";
  my $original_text = <IN>;
  close IN;

  # Now read the line from the newly created file from the mounted file system
  open IN, "$mount_point/$temp_filename" or die "Couldn't open mounted: $!";
  my $mount_text = <IN>;
  close IN;

  if ($original_text eq $mount_text) {
    print "Same!\n";
    print "$local_shared_dir/$temp_filename     : $original_text";
    print "$mount_point/$temp_filename: $mount_text";
  } else {
    print "Different!\n";
    print "$local_shared_dir/$temp_filename     : $original_text";
    print "$mount_point/$temp_filename: $mount_text";
    exit;
  }

  unlink "$local_shared_dir/$temp_filename";
}
Comment 1 Vitaly Gorbunov 2017-03-06 19:54:52 UTC
We faced this bug after migration from centos 5 to centos 6.
During customer's file integration our application server was often reading incorrect file, which lead to integration fail.

It took us a while, after a lot of additional logging and rollbacks, to understand that the problem might be with cifs cache (btw, thanks a lot to David for this report and test case - took us just half a day to reproduce and fix it after we found this bug)

After we disabled cifs cache the problem had disappeared.

Please consider to increase priority of this bug. It's really critical in multi-customer environment as it can lead to situation when process of one customer reads file that belongs to different customer.