In 'static const smb_socket_option socket_options[]'@/lib/util/util_net.c, we need to add: " {"TCP_CORK", IPPROTO_TCP, TCP_CORK, 0, OPT_BOOL}, " Imagine that the application using sendfile() transfers bulk data. Application protocols usually require sending some information that helps interpret the data first, known as a header. Typically, the header is small, and the TCP_NODELAY is set on the socket. The packet with the header will be transmitted immediately and, in some cases (depending on internal packet counters), it could even cause a request of acknowledgement that this packet was successfully received by the other side. Thus, the transfer of bulk data will be delayed and unnecessary network traffic exchanged. **In particular, to minimize the number of network operations, the sendfile() syscall should be used together with the TCP/IP option called TCP_CORK.** We recommend setting the TCP_CORK option when you're sure that you will be sending multiple data sets together (such as header and a body of HTTP response), with no delays between them. This can greatly benefit the performance of WWW, FTP, and file servers, as well as simplifying your life.
This is of course very Linux-specific. I'm willing to consider this but it'll have to be wrapped in code that makes it not impact other systems. If you have a spacific patch, that would be very welcome :-). Jeremy.
(In reply to comment #1) > This is of course very Linux-specific. I'm willing to consider this but it'll > have to be wrapped in code that makes it not impact other systems. > > If you have a spacific patch, that would be very welcome :-). > > Jeremy. I'm not a Linux programmer, but I'm will to study about patch/git/Linux debugging. :-) Another (easier) way is to offer users an option (in smb.conf) to disable socket_options validation/limitation.
(In reply to comment #2) > (In reply to comment #1) > > This is of course very Linux-specific. I'm willing to consider this but it'll > > have to be wrapped in code that makes it not impact other systems. > > > > If you have a spacific patch, that would be very welcome :-). > > > > Jeremy. > > I'm not a Linux programmer, but I'm will to study about patch/git/Linux > debugging. :-) > > Another (easier) way is to offer users an option (in smb.conf) to disable > socket_options validation/limitation. 'will' -> 'willing' sorry i'm not a native English speaker. :(
I'm going to make a patch for this if you don't mind. I'll also add the FreeBSD equivalent TCP_NOPUSH. Do you have real life performance comparisons with TCP_CORK dis/enabled ?
Created attachment 9161 [details] proposed patch
(In reply to comment #4) > I'm going to make a patch for this if you don't mind. I'll also add the FreeBSD > equivalent TCP_NOPUSH. > Wow!!! Thanks a lot! > Do you have real life performance comparisons with TCP_CORK dis/enabled ? Nope, but it seems like Mozilla has been using sendfile()+TCP_CORK since 2002: https://bugzilla.mozilla.org/show_bug.cgi?id=132208
Comment on attachment 9161 [details] proposed patch Unfortunately TCP_CORK (or TCP_NOPUSH) can't be a generic option that users set inside smb.conf, as TCP_CORK has to be removed once the header+sendfile data is sent. The code needs to be more complex than this. The patch should be setting TCP_CORK once we know we're doing a sendfile send (before we send the header) and then removing it once the sendfile send has been done. See here: http://baus.net/on-tcp_cork/ for details. Jeremy.
Look inside source3/lib/sendfile.c for the line: /* * Send the header first. * Use MSG_MORE to cork the TCP output until sendfile is called. */ in the Linux-specific version of the code. Hmmm. Looking into the man page for send() we have: MSG_MORE (Since Linux 2.4.4) The caller has more data to send. This flag is used with TCP sockets to obtain the same effect as the TCP_CORK socket option (see tcp(7)), with the difference that this flag can be set on a per-call basis. Since Linux 2.6, this flag is also supported for UDP sockets, and informs the kernel to package all of the data sent in calls with this flag set into a single datagram which is only transmitted when a call is performed that does not specify this flag. (See also the UDP_CORK socket option described in udp(7).) So it looks like we're already doing the equivalent of TCP_CORK with sendfile() (on Linux anyway). Jeremy.
> So it looks like we're already doing the equivalent of TCP_CORK with sendfile() > (on Linux anyway). The check is easy: Look at a network trace, possibly by adding a little sleep between sending the header and the sendfile call. 1 packets or two? :-)
(In reply to comment #9) > > So it looks like we're already doing the equivalent of TCP_CORK with sendfile() > > (on Linux anyway). > > The check is easy: Look at a network trace, possibly by adding a little sleep > between sending the header and the sendfile call. 1 packets or two? :-) MSG_MORE doesn't improve too much. Can we try TCP_CORK?
Your message doesn't answer Volker's question. To reiterate: "The check is easy: Look at a network trace, possibly by adding a little sleep between sending the header and the sendfile call. 1 packets or two? :-)" Please do that work (add the sleep and get a wireshark trace). One packet or two ? Jeremy.
(In reply to comment #10) > MSG_MORE doesn't improve too much. Can we try TCP_CORK? didn't you say that you have to real live performance comparison? I even saw setups where the the network throughput with sendfile enabled was going down. What did you test actually and what did you measure?