This bug may be related to #2117 (which is closed but with no apparent solution posted). The setup is as follows: Setup 1: ======== - Server: P4 3.2 Ghz w/ 2xSATA II HDD's in RAID 1, partition is JFS, Intel Gigabit Ethernet, running Slackware (current) with latest Samba compiled and installed by hand. Linux Kernel is 2.6.16 (32bit) - Client(s): Various, all running Windows XP SP2 The problem: - Via FTP, Download (Server->Client) works at around 30MBytes/sec, and Upload (Client->Server) works at around 20Mbytes/sec. This means the network/hardware are fine. - Via SAMBA, Download works at around 20MBytes/sec., but the problem appears when Uploading: for a ~700MByte file, the first ~128MBytes are uploaded at 15MBytes/sec. After that point (presumably when some sort of cache is filled), the speed drops to 300KBytes/sec(!). That is about 50x slower! - The same thing happens with various clients and different files. It is not noticeable for files smaller than 128MB. - The default (i.e. minimal) smb.conf is used in this case. A workaround is to disable oplocks: ---------- oplocks = no level2 oplocks = no ---------- - In this case (oplocks = no) the Download speed is unaffected, however the Upload speed is constant at 8-9MBytes/sec from start fo finish. - Downloaded and installed latest (3.0.22) version and the problem persists. Setup 2: ======== - Server: AMB Athlon64 3500+ w/ 1x SATA HDD, partition is also JFS, running Slamd64 (current) with packaged Samba (version 3.0.20). Linux Kernel is 2.6.15.4 (64bit) - Client(s): Various, all running Windows XP SP2 The problem: - In this setup, the problem is similar yet different: via FTP everything works as in the other example, but with SAMBA it is like this: - Downloading is fast, at around 15Mbytes/sec. - Uploading does not slow after the first 128MB. Instead, throughout the whole Upload (no matter how large/small the file) the speed is instable, with spykes and periods of pause, looking as if it writes a bit, then stops, then writes another bit and stops, etc. - The overall Upload speed in this case is around 3-4MBytes/sec (fluctuating). - The weird part happens when disabling oplocks. With "oplocks = no" in smb.conf, Downloading is not affected, but Upploading slows even further, fluctuating between 500KBytes/sec and 2.5MBytes/sec. Could this have something to do with the fact that the partition in both cases was JFS? Maybe there are some performance issues regarding samba+oplocks+jfs? Good luck fixing this... I can provide more details and tests if needed (however, only from Setup #2, as the first one has been shipped to customers and is now in production with the "oplocks = no" workaround set).
Update: just tested this with Setup #2 and the latest version of Samba (3.0.22). The behaviour is now identical to the other Setup: - with "oplocks=yes", the Upload performance is terrible after the first 128MB or so. A bandwidth monitor shows fluctuations between 0 and 3-4Mbytes/sec, with spikes up to 11Mbytes/sec, but the average speed is around 500KBytes/sec; - with "oplocks=no" (the only change), Upload speed varies between 9-11Mbytes/sec for the whole file.
I think this was fixed in 3.0.23 by Jeremy O_SYNC chanegs.
I tested with the latest stable version of samba and, unfortunately, it does not solve the issue. While the situation is improved, something still is very wrong. These are the results of testing with the latest version and the exact same "Setup 2" as above: ===== 1. oplocks = yes Download: 13 MBytes/sec Upload: varies wildly, with the average ending up around 4 MBytes/sec. 2. oplocks = no Download: 14 MBytes/sec Upload: varies wildly with even lower speeds than case #1. The average is around 2 MBytes/sec. ===== What can be derived from this is that the latest samba version reverted to the behaviour of version 3.0.21 (at least on the setup I have available). I also tested with the same hardware/network but booting in Linux and mounting with smbmount, and the performance was somewhat worse on download and the same with upload. Various test files were used, around 700MBytes each. Needless to say, the same files uploaded to the same machine but via FTP instead of samba achieve stable speeds above 25 MBytes/sec. So this bug still crawls :(
submitter wrote : > - Via SAMBA, Download works at around 20MBytes/sec., but the problem appears > when Uploading: for a ~700MByte file, the first ~128MBytes are uploaded at > 15MBytes/sec. After that point (presumably when some sort of cache is filled), > the speed drops to 300KBytes/sec(!). That is about 50x slower! Looks like you ran out of kernel cache. Have you tried tuning the kernel tunables for vm useage ? They are documented in /proc/sys/vm. This doesn't look like a Samba bug to me at all. Try using a different filesystem to JFS and see if the behaviour changes (I'd recommend xfs - it's what I use for large filesystems). Jeremy.
(In reply to comment #4) > submitter wrote : > > > - Via SAMBA, Download works at around 20MBytes/sec., but the problem appears > > when Uploading: for a ~700MByte file, the first ~128MBytes are uploaded at > > 15MBytes/sec. After that point (presumably when some sort of cache is filled), > > the speed drops to 300KBytes/sec(!). That is about 50x slower! > > Looks like you ran out of kernel cache. Have you tried tuning the kernel > tunables for vm useage ? They are documented in /proc/sys/vm. No, no kernel tuning was performed, seeing how everything except samba worked perfectly... Is there anything particular in the way samba interracts with the kernel cache? > This doesn't look like a Samba bug to me at all. Have you read the whole message? It clearly states that the above situation happened with samba 3.0.21/3.0.22 and _only_ when oplocks were set to "yes". Disabling oplocks in samba prevented that from happening. Also, in all of the cases presented here, other means of file transfer like FTP worked perfectly with the exact same setup... So how can this not be a samba bug? > Try using a different filesystem to JFS and see if the behaviour changes (I'd > recommend xfs - it's what I use for large filesystems). JFS is better in some cases, XFS in others. I agree that there may be a certain incompatibility between samba and JFS, both of them working fine sepparately but turning up with issues when combined. However, switching filesystems would only be a workaround, and while I'd like to test XFS on the same system to determine that for sure, it is not an option at the moment. The fact remains that if samba and JFS can't work together, this _is_ a bug and it would need to be researched and documented in a README somewhere, if the fault is with JFS. In any case, I'm awaiting suggestions on what to try (apart from changing filesystems or hardware) on the affected systems in order to help with debugging this. Best regards, Mihnea
Have you looked at the network trace at the slowdown point ? The key to this (you've jogged my memory) is the slowdown only occurs with oplocks on. This may then be a Windows client issue. With oplocks on the client caches as much of the file in memory as possible, with oplocks off it doesn't cache at all - so goes in a loop doing : read off disk write to network. With oplocks on the client pattern is : read/read/read/read/read (up to cache limit) off disk. write/write/write -> network. Have you tried using Windows registry settings to tune the Windows client cache ? You may be able to move the change point. Adding more memory to a client will also move it (the cache size on the client is auto tuning by default). Still doesn't look like a Samba bug to me.
(In reply to comment #6) > Have you looked at the network trace at the slowdown point ? The key to this > (you've jogged my memory) is the slowdown only occurs with oplocks on. This > may > then be a Windows client issue. With oplocks on the client caches as much of > the file in memory as possible, with oplocks off it doesn't cache at all Ok, let's recap: 1. The tests where oplocks being on and off had an effect were done with samba versions prior to 3.0.23rc3. They indeed showed a type of behaviour that indicated some sort of cache problems. However: - the same windows clients worked properly with another windows server - ftp and other transfer means worked perfectly, only samba showed this issue - changing the samba version influenced the behaviour - see below 2. With samba 3.0.23rc3, the setting of oplocks (either on or off) has a much lesser impact. It actually seems to work as it should - that is, with oplocks "off" the performance is measurably lower than with them on. However, even with this version, uploads to the samba server are unacceptably slow, at 1/6th the speed of FTP with the same setup! Also, the transfer is not steady, it is constantly fluctuating betweem low and high speeds (a graph would show a lot of smaller and a few larger spikes instead of a smooth line, like the one created by FTP). > Have you tried using Windows registry settings to tune the Windows client cache? What are those settings? I will try them and report the behaviour observed for various values... > Still doesn't look like a Samba bug to me. Well, the behaviour certainly changed when the _only_ thing modified in the setup was the version of samba. It also happens on different computers and network environments, and it _does_not_ affect other means of file transfer (or the windows driver for that matter). Sure sounds like a samba problem to me...
I am also seeing similar strange upload behavior from WinXP clients to IRIX servers (6.5.29f) running samba 3.0.23. In my case I work with lots of small files (20Kb - 1Mb) and see hangs a) traversing folder hierarchies [with Windows Explorer and also with application File dialogs], b) uploading a file or series of files, c) within an application (e.g. Adobe Acrobat, save file dialog) trying to save a document to a file. In each of these cases I see hangs of 2-3 minutes and then the operation usually completes and I am able to continue [copying, saving, etc]. I definitely did not see this behavior with samba 2.2.8. I am not sure about 3.0.22. I am in the process of doing more testing to see if 3.0.22 works better (in this regard) than 3.0.23. I am using a fairly vanilla smb.conf.
It seems this bug is related to the one at https://bugs.launchpad.net/ubuntu/+source/samba/+bug/84782 and discussed at http://ubuntuforums.org/showthread.php?t=277807 and http://ubuntuforums.org/showthread.php?t=424063 and http://ubuntuforums.org/showthread.php?t=143982. Running Ubuntu Server 6.06 LTS and Windows XP Prof client - both connected via Gigabit NICs and switch, I observed the following interesting fact: Upload of a single file is extremely slow. Concurrent upload of two files is much faster (not only more throughput but even much faster for each file) Download of a single or two files faster than single upload but slower than double upload. Number of concurrent connections don't matter. Have a look at the attachment, which nicely shows what I've just tried to explain.
Created attachment 3088 [details] Perfmon showing upload/download performance
i don't think if this problem was valid that it's still there in 4.1. please retest with samba 4.1