The brokenness of the sendfile() system call

The sendfile() system call has a bad reputation. In theory it is pretty nice: the system call allows you to send a file over a socket, without having to read out the file and writing to the socket block-by-block. Instead, the kernel can do the copying for you. This saves CPU time and context switches.

However, pretty much every production-grade web server -- such as Apache and Nginx -- have a configuration option to disable using the sendfile() system call, and have had those options long before I started writing networking programs. It is said that sendfile() is broken on various operating systems, but I have never been able to find concrete information about which operating systems it is broken on and what exactly is broken.

I have always wondered whether those the ability to turn off using sendfile() is still necessary. The system call has been around forever, so surely kernel developers must have long fixed any bugs. In 2015, can't we just use sendfile() all the time? Having the configuration option around just causes confusion in my opinion.

Sadly, I have received a wake up call today. The answer to that question is a big "no". While debugging a mod_xsendfile-related issue for a Passenger Enterprise customer, I discovered that sendfile() on OS X is broken.

My situation is as follows. Apache is installed with mod_xsendfile and tries to send a file beyond a specific size. In my case, the file was 5369 bytes. Armed with dtruss, OS X's system call profiling tool, I observed that the sendfile() system call returns successfully, but doesn't actually cause the client to receive the data until some kind of Apache timeout kicks in and calls shutdown() on the socket. This causes response to have a delay of ~5 seconds.

If I make the file four times as large, then the client receives the file data partially, blocks until Apache calls shutdown() a few seconds later, then receives the rest of the data.

Setting the Apache UseSendfile off configuration option solved the problem.

Apparently this bug is not new. I found a report on the Internet in which someone else suffered from the same problem, outside of the context of Passenger.

And this is the sad state of the sendfile system call in 2015. Hopefully there will come one day when we can use these kinds of system calls without nasty surprises. In my opinion, software should be optimal without requiring tweaking from the user, who probably doesn't know any better than the programmer anyway.