Bug 10111 - We need to enable TCP_CORK option so 'use sendfile=YES' can have better performance
We need to enable TCP_CORK option so 'use sendfile=YES' can have better perfo...
Status: ASSIGNED
Product: Samba 4.1 and newer
Classification: Unclassified
Component: Other
unspecified
All Linux
: P5 enhancement
: ---
Assigned To: Jeremy Allison
Samba QA Contact
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2013-08-24 01:51 UTC by nanericwang
Modified: 2013-09-12 08:17 UTC (History)
0 users

See Also:


Attachments
proposed patch (951 bytes, patch)
2013-08-26 08:49 UTC, Björn Jacke
jra: review-
Details

Note You need to log in before you can comment on or make changes to this bug.
Description nanericwang 2013-08-24 01:51:21 UTC
In 'static const smb_socket_option socket_options[]'@/lib/util/util_net.c, we need to add:

"
{"TCP_CORK", IPPROTO_TCP, TCP_CORK, 0, OPT_BOOL},
"


Imagine that the application using sendfile() transfers bulk data. Application protocols usually require sending some information that helps interpret the data first, known as a header. Typically, the header is small, and the TCP_NODELAY is set on the socket. The packet with the header will be transmitted immediately and, in some cases (depending on internal packet counters), it could even cause a request of acknowledgement that this packet was successfully received by the other side. Thus, the transfer of bulk data will be delayed and unnecessary network traffic exchanged. **In particular, to minimize the number of network operations, the sendfile() syscall should be used together with the TCP/IP option called TCP_CORK.** We recommend setting the TCP_CORK option when you're sure that you will be sending multiple data sets together (such as header and a body of HTTP response), with no delays between them. This can greatly benefit the performance of WWW, FTP, and file servers, as well as simplifying your life.
Comment 1 Jeremy Allison 2013-08-24 03:22:50 UTC
This is of course very Linux-specific. I'm willing to consider this but it'll have to be wrapped in code that makes it not impact other systems.

If you have a spacific patch, that would be very welcome :-).

Jeremy.
Comment 2 nanericwang 2013-08-24 08:29:02 UTC
(In reply to comment #1)
> This is of course very Linux-specific. I'm willing to consider this but it'll
> have to be wrapped in code that makes it not impact other systems.
> 
> If you have a spacific patch, that would be very welcome :-).
> 
> Jeremy.

I'm not a Linux programmer, but I'm will to study about patch/git/Linux debugging. :-)

Another (easier) way is to offer users an option (in smb.conf) to disable socket_options validation/limitation.
Comment 3 nanericwang 2013-08-24 08:30:04 UTC
(In reply to comment #2)
> (In reply to comment #1)
> > This is of course very Linux-specific. I'm willing to consider this but it'll
> > have to be wrapped in code that makes it not impact other systems.
> > 
> > If you have a spacific patch, that would be very welcome :-).
> > 
> > Jeremy.
> 
> I'm not a Linux programmer, but I'm will to study about patch/git/Linux
> debugging. :-)
> 
> Another (easier) way is to offer users an option (in smb.conf) to disable
> socket_options validation/limitation.

'will' -> 'willing'

sorry i'm not a native English speaker. :(
Comment 4 Björn Jacke 2013-08-26 08:32:42 UTC
I'm going to make a patch for this if you don't mind. I'll also add the FreeBSD equivalent TCP_NOPUSH.

Do you have real life performance comparisons with TCP_CORK dis/enabled ?
Comment 5 Björn Jacke 2013-08-26 08:49:25 UTC
Created attachment 9161 [details]
proposed patch
Comment 6 nanericwang 2013-08-26 10:01:26 UTC
(In reply to comment #4)
> I'm going to make a patch for this if you don't mind. I'll also add the FreeBSD
> equivalent TCP_NOPUSH.
> 
Wow!!! Thanks a lot!


> Do you have real life performance comparisons with TCP_CORK dis/enabled ?

Nope, but it seems like Mozilla has been using sendfile()+TCP_CORK since 2002:
https://bugzilla.mozilla.org/show_bug.cgi?id=132208
Comment 7 Jeremy Allison 2013-08-26 16:48:19 UTC
Comment on attachment 9161 [details]
proposed patch

Unfortunately TCP_CORK (or TCP_NOPUSH) can't be a generic option that users set inside smb.conf, as TCP_CORK has to be removed once the header+sendfile data is sent. The code needs to be more complex than this. The patch should be setting TCP_CORK once we know we're doing a sendfile send (before we send the header) and then removing it once the sendfile send has been done.

See here:

http://baus.net/on-tcp_cork/

for details.

Jeremy.
Comment 8 Jeremy Allison 2013-08-26 16:58:03 UTC
Look inside source3/lib/sendfile.c for the line:

        /*
         * Send the header first.
         * Use MSG_MORE to cork the TCP output until sendfile is called.
         */

in the Linux-specific version of the code. Hmmm. Looking into the man page for send() we have:

MSG_MORE (Since Linux 2.4.4)
The caller has more data to send. This flag is used with TCP sockets to obtain the same effect as the TCP_CORK socket option (see tcp(7)), with the difference that this flag can be set on a per-call basis.
Since Linux 2.6, this flag is also supported for UDP sockets, and informs the kernel to package all of the data sent in calls with this flag set into a single datagram which is only transmitted when a call is performed that does not specify this flag. (See also the UDP_CORK socket option described in udp(7).)

So it looks like we're already doing the equivalent of TCP_CORK with sendfile() (on Linux anyway).

Jeremy.
Comment 9 Volker Lendecke 2013-08-27 06:35:42 UTC
> So it looks like we're already doing the equivalent of TCP_CORK with sendfile()
> (on Linux anyway).

The check is easy: Look at a network trace, possibly by adding a little sleep between sending the header and the sendfile call. 1 packets or two? :-)
Comment 10 nanericwang 2013-09-11 03:27:45 UTC
(In reply to comment #9)
> > So it looks like we're already doing the equivalent of TCP_CORK with sendfile()
> > (on Linux anyway).
> 
> The check is easy: Look at a network trace, possibly by adding a little sleep
> between sending the header and the sendfile call. 1 packets or two? :-)

MSG_MORE doesn't improve too much. Can we try TCP_CORK?
Comment 11 Jeremy Allison 2013-09-12 00:20:26 UTC
Your message doesn't answer Volker's question. To reiterate:

"The check is easy: Look at a network trace, possibly by adding a little sleep
between sending the header and the sendfile call. 1 packets or two? :-)"

Please do that work (add the sleep and get a wireshark trace). One packet or two ?

Jeremy.
Comment 12 Björn Jacke 2013-09-12 08:17:44 UTC
(In reply to comment #10)
> MSG_MORE doesn't improve too much. Can we try TCP_CORK?

didn't you say that you have to real live performance comparison? I even saw setups where the the network throughput with sendfile enabled was going down. What did you test actually and what did you measure?