Bug 9070 - File writes are slow from Windows 7 to Samba 3.6.6
File writes are slow from Windows 7 to Samba 3.6.6
Status: NEW
Product: Samba 3.6
Classification: Unclassified
Component: File services
3.6.6
All All
: P5 normal
: ---
Assigned To: Volker Lendecke
Samba QA Contact
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2012-07-28 22:42 UTC by Florin Iucha
Modified: 2012-10-29 15:58 UTC (History)
1 user (show)

See Also:


Attachments
1st trace iSCSI (2.66 KB, application/octet-stream)
2012-07-28 22:42 UTC, Florin Iucha
no flags Details
2nd trace iSCSI (27.34 KB, application/octet-stream)
2012-07-28 22:43 UTC, Florin Iucha
no flags Details
1st trace Samba (6.25 KB, application/octet-stream)
2012-07-28 22:43 UTC, Florin Iucha
no flags Details
2nd trace Samba (6.03 KB, application/octet-stream)
2012-07-28 22:43 UTC, Florin Iucha
no flags Details
SSD to RAM-disk, using max protocol = smb2 (6.39 KB, application/octet-stream)
2012-07-29 19:15 UTC, Florin Iucha
no flags Details
Full smb.conf (12.88 KB, application/octet-stream)
2012-07-29 20:42 UTC, Florin Iucha
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Florin Iucha 2012-07-28 22:42:14 UTC
Context:  I have a workstation W (running Windows 7 Professional 64 bit)
and a server S running Debian GNU/Linux 64 bit).  The workstation is a
6-core Xeon, the server is a 4-core i7, both with hyperthreading.  The
workstation has 12GB of RAM, the server has 16.  The workstation has
on-board dual Intel gigabit Ethernet controllers (with Jumbo frames
enabled, IP checksum offload, TCP checksum offload, ...). The server
has dual Marvell SysKonnect gigabit Ethernet controllers (with Jumbo
frames enabled, IP checksum offload, TCP checksum offload, ...). They
are connected via a Cisco SG300-10 managed gigabit switch with Jumbo
frames enabled.

On the server, I have installed a fresh copy of Debian wheezy (kernel
3.2.0, samba 3.6.6).  On the system hard drive (Samsung 1TB, 7200 RPM)
I have created two logical volumes, one formatted with XFS and exported
via samba, and the other exported through iSCSI.

iperf with 128KB window on the Linux side and 1024KB window and 1024KB
length of buffer to send shows the network speed to be ~800mbit/second.

Importing the iSCSI partition on Windows, formatting it with NTFS,
then copying 11GB (basically tarring up C:\Program Files using 7-zip,
no compression) and monitoring the performance with 'dstat 5' on Linux
produces a steady stream of network receives and disk writes at 75-79
MBytes/second (attached).

Connecting the mounted filesystem that is exported via Samba and writing
the same test file produces network receive rates that fluctuate between
21 MBytes/s and 51 MBytes/s (but clustered around 35-43 MBytes/s)
The disk writes are also interesting, as they go between 2.5KBytes/s
to 136MBytes/s.

At the recommendation of a fellow LUG member, I have created a 13GB tmpfs
filesystem and exported it via Samba, then run my workload.  The 'dstat 5'
output is attached as well.

Other than the test, the boxes are completely idle.
Comment 1 Florin Iucha 2012-07-28 22:42:44 UTC
Created attachment 7721 [details]
1st trace iSCSI
Comment 2 Florin Iucha 2012-07-28 22:43:03 UTC
Created attachment 7722 [details]
2nd trace iSCSI
Comment 3 Florin Iucha 2012-07-28 22:43:17 UTC
Created attachment 7723 [details]
1st trace Samba
Comment 4 Florin Iucha 2012-07-28 22:43:34 UTC
Created attachment 7724 [details]
2nd trace Samba
Comment 5 Volker Lendecke 2012-07-29 09:33:31 UTC
dtrace is not really informative here. A tunables you might try:

max protocol = smb2
aio write size = 1
aio read size = 1
vfs objects = aio_fork
Comment 6 Florin Iucha 2012-07-29 14:50:51 UTC
aio_fork seems to not be available in Debian:

[2012/07/29 09:46:28.453375,  0] smbd/vfs.c:173(vfs_init_custom)
  error probing vfs module 'aio_fork': NT_STATUS_UNSUCCESSFUL
[2012/07/29 09:46:28.453407,  0] smbd/vfs.c:315(smbd_vfs_init)
  smbd_vfs_init: vfs_init_custom failed for aio_fork
[2012/07/29 09:46:28.453427,  0] smbd/service.c:902(make_connection_snum)
  vfs_init failed for service space
[2012/07/29 09:46:28.454402,  0] smbd/vfs.c:173(vfs_init_custom)
  error probing vfs module 'aio_fork': NT_STATUS_UNSUCCESSFUL
[2012/07/29 09:46:28.454434,  0] smbd/vfs.c:315(smbd_vfs_init)
  smbd_vfs_init: vfs_init_custom failed for aio_fork
[2012/07/29 09:46:28.454454,  0] smbd/service.c:902(make_connection_snum)
  vfs_init failed for service space
[2012/07/29 09:46:28.455098,  0] smbd/vfs.c:173(vfs_init_custom)
  error probing vfs module 'aio_fork': NT_STATUS_UNSUCCESSFUL
[2012/07/29 09:46:28.455129,  0] smbd/vfs.c:315(smbd_vfs_init)
  smbd_vfs_init: vfs_init_custom failed for aio_fork
[2012/07/29 09:46:28.455148,  0] smbd/service.c:902(make_connection_snum)
  vfs_init failed for service space
[2012/07/29 09:46:28.457339,  0] smbd/vfs.c:173(vfs_init_custom)
  error probing vfs module 'aio_fork': NT_STATUS_UNSUCCESSFUL
[2012/07/29 09:46:28.457407,  0] smbd/vfs.c:315(smbd_vfs_init)
  smbd_vfs_init: vfs_init_custom failed for aio_fork
[2012/07/29 09:46:28.457427,  0] smbd/service.c:902(make_connection_snum)
  vfs_init failed for service space
[2012/07/29 09:46:28.458609,  0] smbd/vfs.c:173(vfs_init_custom)
  error probing vfs module 'aio_fork': NT_STATUS_UNSUCCESSFUL
[2012/07/29 09:46:28.458636,  0] smbd/vfs.c:315(smbd_vfs_init)
  smbd_vfs_init: vfs_init_custom failed for aio_fork
[2012/07/29 09:46:28.458655,  0] smbd/service.c:902(make_connection_snum)
  vfs_init failed for service space
Comment 7 Volker Lendecke 2012-07-29 18:42:04 UTC
Well, then no sensible aio is available. This leaves you at "max protocol = smb2"
Comment 8 Florin Iucha 2012-07-29 18:58:56 UTC
(In reply to comment #7)
> Well, then no sensible aio is available. This leaves you at "max protocol =
> smb2"

As I have used an SSD on the client and a RAM-disk on the server, I believe I was able to take the disk-io out of the picture (and it is only the disk-io on the server might be helped by aio).

My main concern right now is the lower network throughput that we see in the dstat log with Samba when compared with the dstat log with iSCSI.  With iSCSI we are pushing 75-78MB/s while Samba only manages 26-55MB/s .
Comment 9 Florin Iucha 2012-07-29 19:15:51 UTC
Created attachment 7725 [details]
SSD to RAM-disk, using max protocol = smb2

The range in network throughput is very interesting: from 2MB/s to 62MB/s.
Comment 10 Volker Lendecke 2012-07-29 19:34:07 UTC
Well, as I said: dstat output is not really relevant here. We need network traces and your full smb.conf file.
Comment 11 Florin Iucha 2012-07-29 20:42:55 UTC
Created attachment 7726 [details]
Full smb.conf
Comment 12 Florin Iucha 2012-07-29 20:44:00 UTC
(In reply to comment #10)
> Well, as I said: dstat output is not really relevant here. We need network
> traces and your full smb.conf file.

I have attached the smb.conf that I'm using.

I do have a managed switch and I can mirror a port - what options should I give to tcpdump so I can get a capture that is useful for you?  I suppose I start my test, leave it running for 30 seconds, then start a capture for 30 seconds or so?
Comment 13 Volker Lendecke 2012-07-30 05:51:14 UTC
See http://wiki.samba.org/index.php/Capture_Packets

Please make sure the TCP connection setup is part of the trace. We need to see the options negotiated when the client connects