Phusion white papers Phusion overview

Phusion Blog

Baseimage-docker, fat containers and “treating containers as VMs”

By Hongli Lai on January 20th, 2015

boat

Baseimage-docker is a minimal Ubuntu base image that is modified for Docker-friendliness. People can pull Baseimage-docker from the Docker Registry and use it as a base image for their own images.

We were early adopters of Docker, using Docker for continuous integration and for building development environments way before Docker hit 1.0. We developed Baseimage-docker in order to solve some problems with the way Docker works, most notably the PID 1 zombie reaping problem.

We figured that:

  1. The problems that we solved are applicable to a lot of people.
  2. Most people are not even aware of these problems, so things can break in unexpected ways (Murphey’s law).
  3. It’s inefficient if everybody has to solve these problems over and over.

So in our spare time we extracted our solution into a reusable base image that everyone can use: Baseimage-docker. We didn’t want to see the community reinventing the wheel over and over. Our solution seems to be well-received: we are the most popular third party image on the Docker Registry, only ranking below the official Ubuntu and CentOS images.

Fat containers, “treating containers as VMs”

Over time, many people got the impression that Baseimage-docker advocates “fat containers”, or “treating containers as VMs”. The Docker developers strongly advocate small, lightweight containers where each container has a single responsibility. The fact that Baseimage-docker advocates the use of multiple processes seems to go against this philosophy.

However, what the Docker developers advocate is running a single logical service per container. Or more generally, a single responsibility per container. Baseimage-docker does not dispute this. Consider that a single logical service can consist of multiple OS processes. Baseimage-docker does not advocate fat containers or treating containers as VMs at all.

Does Baseimage-docker advocate running multiple logical services in a single container? Not necessarily, but we do not prohibit it either. Although the Docker philosophy advocates slim containers, we believe that sometimes it makes sense to run multiple services in a single container, and sometimes it doesn’t.

Why multiple processes?

The most important reason why Baseimage-docker advocates multiple OS processes is because it’s necessary to solve the PID 1 zombie reaping problem. If you’re not familiar with it, you should have a look.

Zombies

The second reason is that splitting your logical service into multiple OS processes also makes sense from a security standpoint. By running different parts of your service as different processes with different users, you can limit the impact of security vulnerabilities. Baseimage-docker provides tools to encourage running processes as different users, e.g. the setuser tool.

The third reason is to automatically restart processes that have crashed. We saw that a lot of people use Supervisord for this purpose, but Baseimage-docker advocates Runit instead because we think it’s easier to use, more efficient and less resource-hungry. Before Docker 1.2, if your main process crashes then the container is down. With the advent of Docker 1.2 — which introduced automatic restarts of containers — this has reason has become less relevant. However, Runit is still useful for the purpose of running different parts of your service as different users, for security reasons. And sometimes it may make sense to restart only a part of the container instead of the container as a whole.

Baseimage-docker is about freedom

Although following the Docker philosophy is a good thing, we believe that ultimately you should decide what makes sense. We see Docker more as a general-purpose tool, comparable to FreeBSD jails and Solaris zones. Our primary use cases for Docker include:

  • Continuous integration.
  • Building portable development environments (e.g. replacing Vagrant for this purpose).
  • Building controlled environments for compiling software (e.g. Traveling Ruby and passenger_rpm_automation).

For these reasons, Baseimage-docker was developed to accept the Docker philosophy where possible, but not to enforce it.

How does Baseimage-docker play well with the Docker philosophy?

So when we say that Baseimage-docker modifies Ubuntu for “Docker friendliness” and that it “accepts the Docker philosophy”, what do we mean? Here are a few examples.

Environment variables

Using environment variables to pass parameters to Docker containers is very much in line with “the Docker way”. However, if you use multiple processes inside a container then the original environment variables can quickly get lost. For example, if you use sudo then sudo will nuke all environment variables for security reasons. Other software, like Nginx, nuke environment variables for security reasons too.

Baseimage-docker provides a mechanism for accessing the original environment variables, but it is sufficiently secured so that only processes that you explicitly allow can access them.

“docker logs” integration to become better

Baseimage-docker tries its best to integrate with docker logs where possible. Daemons have the tendency to log to log files or to syslog, but logging to stdout/stderr (which docker logs exposes) is much more in line with the Docker way.

In the next version of Baseimage-docker, we will adhere better to the Docker philosophy by redirecting all syslog output to docker logs.

SSH to be replaced by “docker exec”

SSH

Baseimage-docker provides a mechanism to easily login to the container using SSH. This also contributes to why people believe that Baseimage-docker advocates fat containers.

However, fat containers have never been the reason why we include SSH. The rationale was that there should be some way to login to the container for the purpose of debugging, inspection or maintenance. Before Docker 1.4 — which introduced docker exec — there was no mechanism built into Docker for logging into a container or running a command inside a container, so we had to introduce our own.

There are people who advocate that containers should be treated as black boxes. They say that if you have to login to the container, then you’re designing your containers wrong. Baseimage-docker does not dispute this either. SSH is not included because we encourage people to login. SSH is included mainly to handle contingencies. No matter how well you design your containers, if it’s used seriously in production then there will come one day when you have to look inside it in order to debug a problem. Baseimage-docker prepares for that day.

Despite this, the SSH mechanism has been widely criticized. Before Docker 1.4, most critics advocated the use of lxc-attach and nsenter. But lxc-attach soon became obsolete because Docker 0.7 moved away from LXC as backend. Nsenter was a better alternative, but suffered from its own problems, such as the fact that it was not included in most distributions which were widely used back then, as well as the fact that using nsenter requires root access on the Docker (which, depending on your requirements, may or may not be acceptable). Of course, SSH also had its own problems. We knew that there is no one-size-fits-all solution. So instead of replacing SSH with lxc-attach/nsenter, we chose to support both SSH and nsenter, and we clearly documented the pros and cons of each approach.

Docker 1.4 finally introduced the docker exec command. This command is like nsenter; indeed, it appears to be a wrapper around a slightly modified nsenter binary that is included by default with Docker. This is great: it means that for a large number of use cases, neither SSH nor nsenter are necessary. However, some of the issues that are inherent with nsenter are still applicable. For example, running docker exec requires access to the Docker daemon, but users who have access to the Docker daemon effectively have root access.

However, we definitely acknowledge “docker exec” as more in line with “the Docker way”. So in the next version of Baseimage-docker, we will adopt “docker exec” as the default mechanism for logging into a container. But because of the issues in “docker exec”, we will continue to support SSH as an alternative, although it will be disabled by default. And we will continue to clearly document the pros and cons of each approach, so that users can make informed decisions instead of blindly jumping on bandwagons.

Conclusion

Baseimage-docker is not about fat containers or about treating containers as VMs, and the fact that it encourages multiple processes does not go against the Docker philosophy. Furthermore, the Docker philosophy is not binary, but a continuum. So we are even actively developing Baseimage-docker to become increasingly in line with the Docker philosophy.

Is Baseimage-docker the only possible right solution?

Of course not. What Baseimage-docker aims to do is:

  1. To make people aware of several important caveats and pitfalls of Docker containers.
  2. To provide pre-created solutions that others can use, so that people do not have to reinvent solutions for these issues.

This means that multiple solutions are possible, as long as they solve the issues that we describe. You are free to reimplement solutions in C, Go, Ruby or whatever. But why should you when we already have a perfectly fine solution?

Maybe you do not want to use Ubuntu as base image. Maybe you use CentOS. But that does not stop Baseimage-docker from being useful to you. For example, our passenger_rpm_automation project uses CentOS containers. We simply extracted Baseimage-docker’s my_init and imported it there.

So even if you do not use, or do not want to use Baseimage-docker, take a good look at the issues we describe, and think about what you can do to solve them.

Happy Dockering.

Discuss this on Hacker News

If you liked this article then maybe you would be interested in our newsletter. It’s low volume, but we regularly post interested updates there. Just enter your email address and name. No spam, we promise.



Docker and the PID 1 zombie reaping problem

By Hongli Lai on January 20th, 2015

boat

When building Docker containers, you should be aware of the PID 1 zombie reaping problem. That problem can cause unexpected and obscure-looking issues when you least expect it. This article explains the PID 1 problem, explains how you can solve it, and presents a pre-built solution that you can use: Baseimage-docker.

When done, you may want to read part 2: Baseimage-docker, fat containers and “treating containers as VMs”.

Introduction

About a year ago — back in the Docker 0.6 days — we first introduced Baseimage-docker. This is a minimal Ubuntu base image that is modified for Docker-friendliness. Other people can pull Baseimage-docker from the Docker Registry and use it as a base image for their own images.

We were early adopters of Docker, using Docker for continuous integration and for building development environments way before Docker hit 1.0. We developed Baseimage-docker in order to solve some problems with the way Docker works. For example, Docker does not run processes under a special init process that properly reaps child processes, so that it is possible for the container to end up with zombie processes that cause all sorts of trouble. Docker also does not do anything with syslog so that it’s possible for important messages to get silently swallowed, etcetera.

However, we’ve found that a lot of people have problems understanding the problems that we’re solving. Granted, these are low-level Unix operating system-level mechanisms that few people know about or understand. So in this blog article we will describe the most important problem that we’re solving — the PID 1 problem zombie reaping problem — in detail.

Zombies

We figured that:

  1. The problems that we solved are applicable to a lot of people.
  2. Most people are not even aware of these problems, so things can break in unexpected ways (Murphy’s law).
  3. It’s inefficient if everybody has to solve these problems over and over.

So in our spare time we extracted our solution into a reusable base image that everyone can use: Baseimage-docker. This image also adds a bunch of useful tools that we believe most Docker image developers would need. We use Baseimage-docker as a base image for all our Docker images.

The community seemed to like what we did: we are the most popular third party image on the Docker Registry, only ranking below the official Ubuntu and CentOS images.

The PID 1 problem: reaping zombies

Recall that Unix processes are ordered in a tree. Each process can spawn child processes, and each process has a parent except for the top-most process.

This top-most process is the init process. It is started by the kernel when you boot your system. This init process is responsible for starting the rest of the system, such as starting the SSH daemon, starting the Docker daemon, starting Apache/Nginx, starting your GUI desktop environment, etc. Each of them may in turn spawn further child processes.

Unix process hierarchy

Nothing special so far. But consider what happens if a process terminates. Let’s say that the bash (PID 5) process terminates. It turns into a so-called “defunct process”, also known as a “zombie process”.

Zombie process

Why does this happen? It’s because Unix is designed in such a way that parent processes must explicitly “wait” for child process termination, in order to collect its exit status. The zombie process exists until the parent process has performed this action, using the waitpid() family of system calls. I quote from the man page:

“A child that terminates, but has not been waited for becomes a “zombie”. The kernel maintains a minimal set of information about the zombie process (PID, termination status, resource usage information) in order to allow the parent to later perform a wait to obtain information about the child.”

In every day language, people consider “zombie processes” to be simply runaway processes that cause havoc. But formally speaking — from a Unix operating system point of view — zombie processes have a very specific definition. They are processes that have terminated but have not (yet) been waited for by their parent processes.

Most of the time this is not a problem. The action of calling waitpid() on a child process in order to eliminate its zombie, is called “reaping”. Many applications reap their child processes correctly. In the above example with sshd, if bash terminates then the operating system will send a SIGCHLD signal to sshd to wake it up. Sshd notices this and reaps the child process.

Zombie process reaping

But there is a special case. Suppose the parent process terminates, either intentionally (because the program logic has determined that it should exit), or caused by a user action (e.g. the user killed the process). What happens then to its children? They no longer have a parent process, so they become “orphaned” (this is the actual technical term).

And this is where the init process kicks in. The init process — PID 1 — has a special task. Its task is to “adopt” orphaned child processes (again, this is the actual technical term). This means that the init process becomes the parent of such processes, even though those processes were never created directly by the init process.

Consider Nginx as an example, which daemonizes into the background by default. This works as follows. First, Nginx creates a child process. Second, the original Nginx process exits. Third, the Nginx child process is adopted by the init process.

Orphaned process adoption

You may see where I am going. The operating system kernel automatically handles adoption, so this means that the kernel expects the init process to have a special responsibility: the operating system expects the init process to reap adopted children too.

This is a very important responsibility in Unix systems. It is such a fundamental responsibility that many many pieces of software are written to make use of this. Pretty much all daemon software expect that daemonized child processes are adopted and reaped by init.

Although I used daemons as an example, this is in no way limited to just daemons. Every time a process exits even though it has child processes, it’s expecting the init process to perform the cleanup later on. This is described in detail in two very good books: Operating System Concepts by Silberschatz et al, and Advanced Programming in the UNIX Environment by Stevens et al.

Operating System Concepts by Silberschatz et al Avanced Programming in the Unix Environment by Stevens et al

Why zombie processes are harmful

Why are zombie processes a bad thing, even though they’re terminated processes? Surely the original application memory has already been freed, right? Is it anything more than just an entry that you see in ps?

You’re right, the original application memory has been freed. But the fact that you still see it in ps means that it’s still taking up some kernel resources. I quote the Linux waitpid man page:

“As long as a zombie is not removed from the system via a wait, it will consume a slot in the kernel process table, and if this table fills, it will not be possible to create further processes.”

Relationship with Docker

So how does this relate to Docker? Well, we see that a lot of people run only one process in their container, and they think that when they run this single process, they’re done. But most likely, this process is not written to behave like a proper init process. That is, instead of properly reaping adopted processes, it’s probably expecting another init process to do that job, and rightly so.

Let’s look at a concrete example. Suppose that your container contains a web server that runs a CGI script that’s written in bash. The CGI script calls grep. Then the web server decides that the CGI script is taking too long and kills the script, but grep is not affected and keeps running. When grep finishes, it becomes a zombie and is adopted by the PID 1 (the web server). The web server doesn’t know about grep, so it doesn’t reap it, and the grep zombie stays in the system.

This problem applies to other situations too. We see that people often create Docker containers for third party applications — let’s say PostgreSQL — and run those applications as the sole process inside the container. You’re running someone elses code, so can you really be sure that those applications don’t spawn processes in such a way that they become zombies later? If you’re running your own code, and you’ve audited all your libraries and all their libraries, then fine. But in the general case you should run a proper init system to prevent problems.

But doesn’t running a full init system make the container heavyweight and like a VM?

An init system does not have to be heavyweight. You may be thinking about Upstart, Systemd, SysV init etc with all the implications that come with them. You may be thinking that full system needs to be booted inside the container. None of this is true. A “full init system” as we may call it, is neither necessary nor desirable.

The init system that I’m talking about is a small, simple program whose only responsibility is to spawn your application, and to reap adopted child processes. Using such a simple init system is completely in line with the Docker philosophy.

A simple init system

Is there already an existing piece of software that can run another application and that can reap adopted child processes at the same time?

There is almost a perfect solution that everybody has — it’s plain old bash. Bash reaps adopted child processes properly. Bash can run anything. So instead having this in your Dockerfile…

CMD ["/path-to-your-app"]

…you would be tempted to have this instead:

CMD ["/bin/bash", "-c", "set -e && /path-to-your-app"]

(The -e directive prevents bash from detecting the script as a simple command and exec()‘ing it directly.)

This would result in the following process hierarchy:

bash

But unfortunately, this approach has a key problem. It doesn’t handle signals properly! Suppose that you use kill to send a SIGTERM signal to bash. Bash terminates, but does not send SIGTERM to its child processes!

bash_signal

When bash terminates, the kernel terminates the entire container with all processes inside. These processes are terminated uncleanly through the SIGKILL signal. SIGKILL cannot be trapped, so there is no way for processes to terminate cleanly. Suppose that the app you’re running is busy writing a file; the file could get corrupted if the app is terminated uncleanly in the middle of a write. Unclean terminations are bad. It’s almost like pulling the power plug from your server.

But why should you care whether the init process is terminated by SIGTERM? That’s because docker stop sends SIGTERM to the init process. “docker stop” should stop the container cleanly so that you can start it later with “docker start”.

Bash experts would now be tempted to write an EXIT handler that simply sends signals to child processes, like this:

#!/bin/bash
function cleanup()
{
    local pids=`jobs -p`
    if [[ "$pids" != "" ]]; then
        kill $pids >/dev/null 2>/dev/null
    fi
}

trap cleanup EXIT
/path-to-your-app

Unfortunately, this does not solve the problem. Sending signals to child processes is not enough: the init process must also wait for child processes to terminate, before terminating itself. If the init process terminates prematurely then all children are terminated uncleanly by the kernel.

So clearly a more sophisticated solution is required, but a full init system like Upstart, Systemd and SysV init are overkill for lightweight Docker containers. Luckily, Baseimage-docker has a solution for this. We have written a custom, lightweight init system especially for use within Docker containers. For the lack of a better name, we call this program my_init, a 350 line Python program with minimal resource usage.

Several key features of my_init:

  • Reaps adopted child processes.
  • Executes subprocesses.
  • Waits until all subprocesses are terminated before terminating itself, but with a maximum timeout.
  • Logs activity to “docker logs”.

Will Docker solve this?

Ideally, the PID 1 problem is solved natively by Docker. It would be great if Docker supplies some builtin init system that properly reaps adopted child processes. But as of January 2015, we are not aware of any effort by the Docker team to address this. This is not a criticism — Docker is very ambitious, and I’m sure the Docker team has bigger things to worry about, such as further developing their orchestration tools. The PID 1 problem is very much solvable at the user level. So until Docker has officially solved this, we recommend people to solve this issue themselves, by using a proper init system that behaves as described above.

Is this really such a problem?

At this point, the problem might still sound hypothetical. If you’ve never seen any zombie processes in your container then you may be inclined to think that everything is all right. But the only way you can be sure that this problem never occurs, is when you have audited all your code, audited all your libraries’ code, and audited all the code of the libraries that your libraries depend on. Unless you’ve done that, there could be a piece of code somewhere that spawns processes in such a way that they become zombies later on.

You may be inclined to think, I’ve never seen it happen, so the chance is small. But Murphy’s law states that when things can go wrong, they will go wrong.

Apart from the fact that zombie processes hold kernel resources, zombie processes that don’t go away can also interfere with software that check for the existence of processes. For example, the Phusion Passenger application server manages processes. It restarts processes when they crash. Crash detection is implemented by parsing the output of ps, and by sending a 0 signal to the process ID. Zombie processes are displayed in ps and respond to the 0 signal, so Phusion Passenger thinks the process is still alive even though it has terminated.

And think about the trade off. To prevent problems with zombie processes from ever happening, all you have to do is to is to spend 5 minutes, either on using Baseimage-docker, or on importing our 350 lines my_init init system into your container. The memory and disk overhead is minimal: only a couple of MB on disk and in memory to prevent Murphy’s law.

Conclusion

So the PID 1 problem is something to be aware of. One way to solve it is by using Baseimage-docker.

Is Baseimage-docker the only possible solution? Of course not. What Baseimage-docker aims to do is:

  1. To make people aware of several important caveats and pitfalls of Docker containers.
  2. To provide pre-created solutions that others can use, so that people do not have to reinvent solutions for these issues.

This means that multiple solutions are possible, as long as they solve the issues that we describe. You are free to reimplement solutions in C, Go, Ruby or whatever. But why should you when we already have a perfectly fine solution?

Maybe you do not want to use Ubuntu as base image. Maybe you use CentOS. But that does not stop Baseimage-docker from being useful to you. For example, our passenger_rpm_automation project uses CentOS containers. We simply extracted Baseimage-docker’s my_init and imported it there.

So even if you do not use, or do not want to use Baseimage-docker, take a good look at the issues we describe, and think about what you can do to solve them.

Happy Dockering.

There is a part 2: We will discuss the phenomenon that a lot of people associate Baseimage-docker with “fat containers”. Baseimage-docker is not about fat containers at all, so what is it then? See Baseimage-docker, fat containers and “treating containers as VMs”

Discuss this on Hacker News

If you liked this article then maybe you would be interested in our newsletter. It’s low volume, but we regularly post interested updates there. Just enter your email address and name. No spam, we promise.



Interview by The Changelog about Phusion Passenger 5 and the Raptor campaign

By Hongli Lai on January 8th, 2015

The Changelog

We have recently been interviewed by the awesome guys at The Changelog, a weekly blog and podcast about open source projects. In this interview I explained a bit about Phusion’s history, what’s new in Phusion Passenger 5, how the Raptor campaign came to be, why we chose to run the campaign like that, the challenges of open source, future Passenger developments, etc.

Indeed, running an open source company has been a big challenge:

A lot of people say, “Hey, if you’re open source, and you want to make money, try selling support.” We tried doing that, but it didn’t work at all, because Passenger is too good.

– Hongli Lai – 12:10

You can listen the interview at The Changelog’s website. You can stream it online or download it as an MP3.

Researching a potential new form of HTTP caching optimization

By Hongli Lai on January 6th, 2015

PIe in the sky

It’s been a while since we released the first beta of Phusion Passenger 5 (codename “Raptor”), the application server for Ruby, Python and Node.js web apps. We have received a lot of great feedback from the community regarding its performance, stability and features.

Passenger 5 isn’t production-ready yet, but we are getting close because 5.0 beta 3 will soon be released. But in the mean time, we would like to share a major new idea with you.

While Passenger 5 introduced many performance optimizations and is much faster than Passenger 4, the impact on real-world application performance varies greatly per application. This is because in many cases the overall performance is more dependent on the application than on the app server.

It’s obvious that just making the app server itself fast is not enough to improve overall performance. So what else can the app server do? After contemplating this question for some time, we believe we have found an answer in the form of a modified HTTP caching mechanism. Its potential is huge.

Update: Over the course of the day, readers have made us aware that some of the functionality can also be achieved through Varnish and through the use of Edge Side Includes, but there are also some ideas which cannot be achieved using only Varnish. These ideas require support from the app. Please read this article until the end before drawing conclusions.

Please also note that the point of this article is not to show we can “beat” Varnish. The point is to share our ideas with the community, to have a discussion about these ideas and to explore the possibilities and feasibility.

Turbocaching

One of the main new features in Passenger 5 is turbocaching. This is an HTTP cache built directly in Passenger so that it can achieve much higher performance than external HTTP caches like Varnish (update: no, we’re not claiming to be better than Varnish). It is fast and small, specifically designed to handle large amounts of traffic to a limited number of end points. For that reason, we described it as a “CPU L1 cache for the web”.


Turbocaching is a major contributor of Passenger 5’s performance

The turbocache has the potential to improve app performance dramatically, no matter how much work the app does. This is seen in the chart above. A peculiar property is that the relative speedup is inversely proportional to the app’s native performance. That is, the slower your app is, the bigger the speedup multiplier you can get from caching. At worst, caching does not hurt. In extreme cases — if the app is really slow — you can see a hundred fold performance improvement.

The limits of caching

So far for the potential of caching, but reality is more nuanced. We have received a lot of feedback from the community about the Passenger 5 beta, including feedback about its turbocache.

As expected, the turbocache performs extremely well in applications that serve data that is publicly cacheable by everyone, i.e. they do not serve data that is login-specific. This includes blogs and other sites that consist mostly of static content. The Phusion Passenger website itself is also an example of a mostly static site. But needless to say, this still makes the turbocache’s usefulness rather limited. Most sites serve some login-specific data, even if it’s just a navigation bar displaying the username.

However, these limitations apply to all other HTTP caches too. For example, the Varnish HTTP cache has been used very successfully to speed up WordPress. But Varnish doesn’t help a lot with logged-in traffic.

Even the CloudFlare CDN — which is essentially a geographically distributed HTTP cache — does not help a lot with logged-in traffic. Although CloudFlare can reduce the bandwidth between the origin server and the cache server through its Railgun technology, it doesn’t reduce the load on the origin server, which is what we are after.

Update: some readers have pointed out that Varnish supports Edge Side Include (ESI), which is like a text postprocessor at the web server/cache level. But using ESI only solves half of the problem. Read on for more information.

A glimmer of hope

Hope is not all lost though. We have identified two classes of apps for which there is hope:

  1. Apps which have more anonymous traffic than logged in traffic. Examples of such apps include Ted.com, Wikipedia, Imgur, blogs, news sites, video sites, etc. Let’s call these mostly-anonymous apps. What if we can cache responses by user, so that anonymous users share a single cache entry?
  2. Apps which serve public data for the most part. Examples of such apps include: Twitter, Reddit, Discourse, discussion forums. Let’s call these mostly-public apps. Most of the data that they serve is the same for everyone. There are only minor variations, e.g. the a navigation bar that displays the username, and secured pages. What if we can cache the cacheable content, and skip the rest?

Class 1: caching mostly-anonymous apps

There is almost a perfect solution for making apps in the first class cacheable: the HTTP Vary header. This header allows you to send a different cached response, based on the value of some header that is sent by the client.

For example, suppose that your app…

  • …serves gzip-compressed responses to browsers that support gzip compression.
  • …serves regular responses to browsers that don’t support gzip compression.

You don’t want a cache to serve gzipped responses to browsers that don’t support gzip. Browsers tell the server whether they support gzip by sending the Accept-Encoding: gzip header. If the application sets the Vary: Accept-Encoding header in its responses, then the cache will know that that particular response should only be served to clients with the particular Accept-Encoding value that it has received now.

HTTP caching Vary header effect
The Vary response header makes HTTP caches serve different cached responses based on the headers the browsers send.

In theory, we would be able to cache responses differently based on cookies (Vary: Cookie). Each logged in user would get its own cached version. And because most traffic is anonymous, all anonymous users can share cache entries.

Unfortunately, on the modern web, cookies are not only set by the main site, but also by third-party services which the site uses. This includes Google Analytics, Youtube and Twitter share buttons. The values of their cookies can change very often and often differ on a per-user basis, probably for the purpose of user tracking. Because these widely different values are also included in the cache variation key, they make it impossible for anonymous users to share cache entries if we were to try to vary the cache by the Cookie header. The situation is so bad that Varnish has decided not to cache any requests containing cookies by default.

Even using Edge Side Include doesn’t seem to help here. The value of the cookie header can change quickly even for the same user, so when using Edge Side Include the cache may not even be able to cache the previous user-specific response.

The eureka moment: modifying Vary

While the Vary header is almost useless in practice, the idea of varying isn’t so bad. What we actually want is to vary the cache by user, not by the raw cookie value. What if the cache can parse cookies and vary the cached response by the value of a specific cookie, not the entire header?

And this is exactly what we are researching for Passenger 5 beta 3. Initial tests with a real-world application — the Discourse forum software — show promising results. Discourse is written in Ruby. We have modified Discourse to set a user_id cookie on login.

# lib/auth/default_current_user_provider.rb

TURBOCACHE_COOKIE = "user_id"

def log_on_user(user, session, cookies)
  ...
  cookies.permanent[TURBOCACHE_COOKIE] = { value: user.id, httponly: true }
  ...
end

We modified Passenger to parse cookies, and to vary turbocache responses based on the value of this user_id cookie. We invoke Passenger like this:

passenger start --vary-turbocache-by-cookie user_id

This changeset can be seen in Github commit a760649cd7.

The result is a Discourse where all anonymous users share the same cache entry. Uncached, Discourse performance is pretty constant at 97 req/sec no matter which app server you use. But with turbocaching, performance is 19 000 req/sec.

This is caching that Varnish and other “normal” HTTP caches (including CloudFlare) could not have done*. The benefit that turbocaching adds in this scenario is exactly in line with our vision of a “CPU L1 cache” for the web. You can still throw in Varnish for extra caching on top of Passenger’s turbocaching, but Passenger’s turbocaching’s provides an irreplaceable service.

* Maybe Varnish’s VCL allows it, but we have not been able to find a way so far. If we’re wrong, please let us know in the comments section.

Class 2: caching mostly-public apps

Apps that serve pages where most data is publicly cacheable, except for small fragments, appear not to be cacheable at the HTTP level at all. Currently these apps utilize caching at the application level, e.g. using Rails fragment caching or Redis. View rendering typically follows this sort of pseudo-algorithm:

send_response("<nav>Welcome " + current_user_name "!</nav>")
cached_fragment = fetch_from_cache("rest_of_page")
if cached_fragment != null
    send_response(cached_fragment)
else
    fragment = render_rest_of_page()
    store_in_cache("rest_of_page", fragment)
    send_response(fragment)

However, this still means the request has to go through the application. If there is a way to cache this at the Passenger level then we can omit the entire application, boosting the performance even further.

We’ve come to the realization that this is possible, if we change the app into a “semi single page app”:

  1. Instead of rendering pages on the server side, render them on the client side, e.g. using Ember. This way, the view templates can be simple static HTML files, which are easily HTTP cacheable.
  2. PushState is then used to manipulate the location bar, making it feel like a regular server-side web app.
  3. The templates are populated using JSON data from the server. We can categorize this JSON data in two categories:
    1. User-independent JSON data, which is HTTP-level cacheable. For example, the list of subforums.
    2. User-specific JSON data, which is not HTTP-level cacheable. For example, information about the logged in user, such as the username and profile information.

      And here lies the trick: we only load this data once, when the user loads the page. When the user clicks on any links, instead of letting the browser navigate there, the Javascript loads the user-independent JSON data (which is easily cacheable), updates the views and updates the location bar using PushState.

By using this approach, we reduce the performance impact of non-cacheable fragments tremendously. Normally, non-cacheable page fragments would make every page uncacheable. But by using the approach we described, you would only pay the uncacheability penalty once, during the initial page load. Any further requests are fully cacheable.

And because of the use of HTML PushState, each page has a well-defined URL. This means that, despite the app being a semi-single-page app, it’s indexable by crawlers as long as they support Javascript. GoogleBot supports Javascript.

Discourse is a perfect example of an app that’s already architected this way. Discourse displays the typical “navigation bar with username”, but this is only populated on the first page load. When the user clicks on any of the links, Discourse queries JSON from the server and updates the views, but does not update the navbar username.

An alternative to this semi-single page app approach is by using Edge Side Include technology, but adoption of the technology is fairly low at this point. Most developers don’t run Varnish in their development environment. In any case, ESI doesn’t solve the whole problem: just half of it. Passenger’s cookie varying turbocaching feature is still necessary.

Even when there are some protected/secured subforums, the turbocache cookie varying feature is powerful enough make even this scenario cacheable. Suppose that the Discourse content depends on the user’s access level, and that there are 3 access levels: anonymous users, regular registered members, staff. You can put the access level in a cookie, and vary the cache by that:

cookies.permanent[TURBOCACHE_COOKIE] = { value: user.access_level, httponly: true }

That way, all users with the same access level share the same cache entry.

Due to time constraints we have not yet fully researched modifying Discourse this way, but that leads us to the following point.

Call for help: please participate in our research

The concepts we proposed in this blog post are ideas. Until tested in practice, they remain theory. This is why we are looking for people willing to participate in this research. We want to test these ideas in real-world applications, and we want to look for further ways to improve the turbocache’s usefulness.

Participation means:

  • Implementing the changes necessary to make your app turbo-cache friendly.
  • Benchmarking or testing whether performance has improved, and by how much.
  • Actively working with Phusion to test ideas and to look for further room for improvements. We will happily assist active participants should they need any help.

If you are interested, please send us an email at info@phusion.nl and let’s talk.

Discuss this on Hacker News

Also, if you liked this article then maybe you would be interested in our newsletter. It’s low volume, but we regularly post interested updates there. Just enter your email address and name. No spam, we promise.



Phusion Passenger 4.0.57: better Ruby 2.2 support, Nginx 1.7.9 support

By Hongli Lai on January 5th, 2015

We’ve just released version 4.0.57 of the Phusion Passenger application server for Ruby, Python and Node.js. This release fixes two minor but important issues.

Phusion Passenger 4 is the current stable branch, in which we release bug fixes from time to time. At the same time there is also Phusion Passenger 5, which is the not-yet-ready-for-production development branch, with major changes and improvements and terms of performance application behavior visibility. Version 5.0 beta 3 will soon be released, but until the 5.x branch is considered stable, we will keep releasing bug fixes under the 4.x branch.

Phusion Passenger also has an Enterprise version which comes with a wide array of additional features. By buying Phusion Passenger Enterprise you will directly sponsor the development of the open source version.

Improved Ruby 2.2 support

Version 4.0.56 already introduced Ruby 2.2 support, but due to an issue in the way we compile the Phusion Passenger native extension, it didn’t work with all Ruby 2.2 installations. In particular, 4.0.56 worked with Ruby 2.2 installations that were compiled with a shared libruby, which is the case if you installed Ruby 2.2 with RVM or though operating system packages. But it did not work with Ruby 2.2 installations that were compiled with a static libruby, which is the case if you installed manually from source, or using rbenv and chruby, or when you are using Heroku.

At first, we suspected a bug in Ruby 2.2’s build system, but after feedback from the MRI core developers, it turned out to be an issue in our own build system. The issue is caused by a commit from 4 years ago, GH-168, which attempted to fix a different issue. It seems there is no way to fix Ruby 2.2 compatibility while at the same time fixing GH-168, so we had to make a choice. Since GH-168 is quite old and was made at a time when Ruby 1.8.6 was the latest Ruby version, we believe that the issue is no longer relevant. We reverted GH-168 in favor of Ruby 2.2 compatibility.

Many thanks to the different users who have supplied feedback. You can read the discussions at the Github issue tracker.

Nginx 1.7.9 support

Nginx 1.7.9 was released a while ago and changed its internal API yet again. 4.0.57 fixes compatibility with Ngixn 1.7.9.

Installing or upgrading to 4.0.57

OS X OS X Debian Debian Ubuntu Ubuntu
Heroku Heroku Ruby gem Ruby gem Tarball Tarball

Final

Phusion Passenger’s core is open source. Please fork or watch us on Github. :)

If you would like to stay up to date with Phusion news, please fill in your name and email address below and sign up for our newsletter. We won’t spam you, we promise.



Phusion Passenger 5 beta 2

By Hongli Lai on December 22nd, 2014

The first Passenger 5 beta was released about 3 weeks ago. This major new version brought many changes, such as performance improvements, better tools for application-level visibility and a new HTTP JSON API for accessing Passenger’s internals.

Beta 1 was not considered production ready. In the past 3 weeks we’ve been working hard to fix bugs. The result is beta 2, which fixes 8 issues.

  • Fixed handling of multiple Set-Cookie headers. Closes GH-1296.
  • passenger-config system-metrics now works properly if the agent is installed in ~/.passenger. Closes GH-1304.
  • Documentation enhancements by Igor Vuk. Closes GH-1318.
  • Fixed some crasher bugs.
  • [Standalone] User switching is now correctly disabled.
  • [Standalone] Fixed the --thread-count parameter.
  • [Apache] IPs set by mod_remoteip are now respected. Closes GH-1284.
  • [Apache] Fixed support for gzipped chunked responses. Closes GH-1309.

Installing or upgrading

Here’s a quickstart:

gem install passenger --pre -v 5.0.0.beta2
cd /path-to-your-app
passenger start

The above is a very short excerpt of how to install or upgrade Phusion Passenger. For detailed instructions (which, for example, take users and permissions into account), please refer to the “RubyGems” section of the installation manuals:

Please note:

  • 5.0.0 beta 2 is a beta release. There are still major bugs open and we do not recommend using it in production yet.
  • There are no Homebrew or Debian packages for this release, because this release is still in beta!
  • There is also a 5.0.0 beta 2 release of Phusion Passenger Enterprise available. Please refer to the Customer Area.

Phusion Passenger 4.0.56: facepalming at file descriptor leak, Node.js load balancing

By Hongli Lai on December 19th, 2014

We’ve just released version 4.0.56 of the Phusion Passenger application server for Ruby, Python and Node.js, which fixes a number of interesting and important bugs. They’re the kind of typical bugs that make me go “what the **** was I thinking?!” after I’ve analyzed them, because the fixes are very simple.

Facepalm

Leaking file descriptors

The first bug is a file descriptor leak. A file descriptor is number which represents a kernel resource, such as an open file or a socket. Every time you call File.open or TCPSocket.new in Ruby, you get an IO object that’s internally backed by a file descriptor. In Node.js you even often work with file descriptors directly: most fs functions return a file descriptor. Since Phusion Passenger is written in C++, we also work with file descriptors directly.

Schematically, it looks like this:

File descriptors

You’re probably familiar with memory leaks. A file descriptor leak is very similar. If you ever lose track of a file descriptor number, you’ve leaked it. In Ruby this is not possible because all IO objects own their file descriptor, and IO objects are garbage collected (thus closing the corresponding file descriptor). However in Node.js and in C++ this can easily happen if you’re not careful. When leaked, the kernel resource stays allocated until your process exits.

What went wrong

In Passenger, we leaked a file descriptor when creating an error report file. This file is created if your app can’t spawn for some reason (e.g. it throws an exception during startup). The code that was responsible for rendering the file looked like this, in semi C++ pseudocode:

void processAndLogNewSpawnException(SpawnException &e) {
    int fd = -1;
    FdGuard guard(fd);
    fd = createNewReportFile();
    if (fd != -1) {
        renderReportFileContents(fd, e);
    }
}

Notice the guard variable. In C++, it is a so-called RAII object: “Resource Acquisition Is Initialization”. It is a common coding pattern in C++ to ensure that things are cleaned up when exceptions are thrown, kind of like the C++ equivalent of the ensure keyword in Ruby or the finally keyword in Javascript. When this function exits for any reason, be it a normal return or an exception, the guard destructor is called, which is supposed to close the file descriptor.

The facepalm moment was when Paul “popox” B reported that the guard was on the wrong line. The guard was created before the file descriptor was assigned to fd, so the guard did nothing all this time. Every time a report file was created, a file descriptor was leaked.

The one-line fix

The solution was to move the guard object a little bit:

void processAndLogNewSpawnException(SpawnException &e) {
    int fd = -1;
    // Guard object was here
    fd = createNewReportFile();
    // It is now here
    FdGuard guard(fd); 
    if (fd != -1) {
        renderReportFileContents(fd, e);
    }
}

Thank you Paul B!

Node.js load balancing

The other issue fixed in 4.0.56 is a Node.js load balancing issue. In Passenger we load balance requests between application processes as much as possible. Traditionally, the reason for load balancing has been to minimize latency. This utilizes the concept of “application concurrency”: the maximum number of concurrent requests a single app process can handle. For Ruby apps, the concurrency is 1 (unless you configured multithreading, in which case the concurrency is equal to the number of threads). Since Ruby apps have finite I/O concurrency, Passenger load balances a request to a different process only if one process has run out of concurrency.

Node.js is different in that it’s fully asynchronous. It can effectively have an unlimited amount of concurrency.

Passenger orders processes in a priority queue by “busyness”. Load balancing is achieved by routing a new request to the process with the least busyness.

What went wrong

What went wrong with the Node.js case is the fact that we had special rules for application processes with unlimited concurrency. The busyness for such processes is calculated as follows:

if (sessions == 0) {
    return 0;
} else {
    return 1;
}

sessions indicates the number of requests that a process is currently handling. This piece of code effectively sorted Node.js processes in two categories only: idle processes and non-idle processes.

From a concurrency point of view, there is nothing wrong with this. Node.js apps have unlimited concurrency after all. However this resulted in lots of requests “sticking” to a few processes, as Charles Vallières reported:

  * PID: 24526   Sessions: 84      Processed: 1       Uptime: 9s
  * PID: 24545   Sessions: 1       Processed: 0       Uptime: 9s
  * PID: 24571   Sessions: 83      Processed: 0       Uptime: 8s
  * PID: 24596   Sessions: 1       Processed: 0       Uptime: 8s

Then it dawned to me that I forgot something. An even distribution of requests is desirable here, because now the reason for load balancing becomes different. It’s to maximize CPU core usage, because single Node.js process can only use 1 CPU core.

The fix

The fix was incredibly easy:

return sessions;

Yes, facepalm time.

Thank you Charles Vallières!

Installing or upgrading to 4.0.56

OS X OS X Debian Debian Ubuntu Ubuntu
Heroku Heroku Ruby gem Ruby gem Tarball Tarball

Final

Phusion Passenger’s core is open source. Please fork or watch us on Github. :)

If you would like to stay up to date with Phusion news, please fill in your name and email address below and sign up for our newsletter. We won’t spam you, we promise.



Phusion Passenger 4.0.55 released, supports Ruby 2.2

By Hongli Lai on December 15th, 2014


Phusion Passenger is a fast and robust web server and application server for Ruby, Python, Node.js and Meteor. Passenger takes a lot of complexity out of deploying web apps, and adds powerful enterprise-grade features that are useful in production. High-profile companies such as Apple, New York Times, AirBnB, Juniper, American Express, etc are already using it, as well as over 350.000 websites.

Phusion Passenger is under constant maintenance and development. Version 4.0.55 is a bugfix release.

Phusion Passenger also has an Enterprise version which comes with a wide array of additional features. By buying Phusion Passenger Enterprise you will directly sponsor the development of the open source version.

Recent changes

Version 4.0.54 has been skipped because it was a hotfix for Enterprise customers. The changes in 4.0.54 and 4.0.55 combined are as follows:

  • Supports Ruby 2.2. Ruby 2.2 isn’t released yet and only exists as an alpha, but we’re supporting it anyway. Closes GH-1314.
  • Fixed Linux OS name detection.
  • Contains a licensing-related hot fix for Enterprise customers.

Installing or upgrading to 4.0.55

OS X OS X Debian Debian Ubuntu Ubuntu
Heroku Heroku Ruby gem Ruby gem Tarball Tarball

Final

Phusion Passenger’s core is open source. Please fork or watch us on Github. :)

If you would like to stay up to date with Phusion news, please fill in your name and email address below and sign up for our newsletter. We won’t spam you, we promise.



Introducing Traveling Ruby

By Hongli Lai on December 8th, 2014

Ruby is one of our favorite programming languages. Most people use it for web development, but Ruby is so much more. We at Phusion have been using Ruby for years for writing sysadmin automation scripts, developer command line tools and more. Heroku’s Toolbelt and Chef have also demonstrated that Ruby is an excellent language for these sorts of things.

However, distributing Ruby apps to non-Ruby-programmer end users on Linux and OS X is problematic. If you require users to install Ruby or to use RubyGems, they can get into trouble or become frustrated.

Creating platform-specific packages for each Linux distro and each OS requires a lot of work. Because building such packages requires a fleet of VMs, building packages takes a lot of time.

Our solution to this problem is Traveling Ruby, which lets you create self-contained Ruby app packages for Linux and OS X. It is a project which supplies self-contained, “portable” Ruby binaries: Ruby binaries that can run on any Linux distribution and any OS X machine. This allows Ruby app developers to bundle these binaries with their Ruby app, so that they can distribute a single package to end users, without needing end users to first install Ruby or gems.

Learn more about the motivation behind Traveling Ruby.

Key benefits

  • Self-contained: apps packaged with Traveling Ruby are completely self-contained and don’t require the user to install any further dependencies or runtimes.
  • Simple & easy: Traveling Ruby is very simple to use and very easy to learn. No complicated tooling to learn. You can grasp the basics in just 5 minutes.
  • Fast & lightweight: produce packages for multiple OS targets, regardless of which OS you are developing on. This is achieved without the need for heavyweight tools like VMs.

Learn more

logo

You can learn more about Traveling Ruby here:

Introducing Phusion Passenger 5 beta 1, codename “Raptor”

By Hongli Lai on November 25th, 2014

Phusion Passenger Raptor

Executive summary

Phusion Passenger is an app server that supports Ruby. We have released version 5 beta 1, codename “Raptor”. This new version is much faster, helps you better identify and solve problems, and has a ton of other improvements. If you’ve followed the Raptor campaign then you may wonder why we held the campaign like this. Read on if you’re interested. If you just want to try it, scroll all the way down for the changelog and the installation and upgrade instructions.

Learn more about Phusion Passenger

Introduction

A month ago, we released a website in which we announced “Raptor”, supposedly a new Ruby app server that’s much faster than others. It has immediately received a lot of community attention. It was covered by Fabio Akita and by RubyInside’s Peter Cooper. From the beginning, “Raptor” was Phusion Passenger 5, a new major version with major internal overhauls. In the weeks that followed, we blogged about how we made “Raptor” fast.

Even though “Raptor” is Phusion Passenger, it doesn’t make the impact any less powerful. The performance improvements that we claim are real, and they are open source. Because “Raptor” is Phusion Passenger 5, it means that it automatically has a mature set of features:

  • Handle more traffic
    Phusion Passenger 5 is up to 4x faster than other Ruby app servers, allowing you to handle more traffic with the same hardware.
  • Reduce maintenance
    Automates more system tasks than other app servers. Spend less time micromanaging software, and more time building your business.
  • Identify & fix problems quickly
    Why is your app behaving the way it does? What is it doing? Phusion Passenger 5 provides tools that give you the insights you need.
  • Keep bugs & issues in check
    Limit the impact of bugs and issues, making downtime and user dissatisfaction less likely. Reduce pressure on developers while the root problem is being fixed.
  • Excellent support
    We have excellent documentation and a vibrant community discussion forum. And with our professional and enterprise support contracts, you can consult our team of experts directly.
  • Improve security
    Provides extra layers of defense, reduces attack surfaces on applications and limits the impact of security vulnerabilities.

However, the authors behind “Raptor” remained unknown — until today. The reason why we ran the campaign like this is explained in this article.

A brief history

It is perhaps hard to fathom now, but in the early days of Ruby, getting an app into a production environment was a painful task in itself. Many hours were spent by developers on tedious tasks such as manually managing ports and performing other error-prone configuration sit-ups. The status quo of deployment back then wasn’t exactly in line with what Rails advocates through its “convention over configuration” mantra. Far from it in fact.

When we first introduced Phusion Passenger back in 2008, we wanted to “fix” this. We wanted Ruby deployment to be as easy as PHP so that developers could focus on their apps and lower the barrier of entry for newcomers.

Even though we have been able to help power some of the largest sites on the Internet over the past few years through Phusion Passenger, we have always remained vigilant as to not become complacent: we have been eagerly listening to the community as to what they expect the next big thing to be.

The observations we have made over the years have eventually culminated into Phusion Passenger 5, which was codenamed Raptor for a number of reasons.

A greater focus on performance and efficiency

Whether you are deploying a small web app on a VPS or spinning up tens of thousands of instances to power your e-commerce business, we all want the most bang for our buck. Being able to reduce the number of required servers would be beneficial in reducing costs and it is for this reason that developers seek to employ the most efficient software stack currently available. When it comes to making that choice, benchmarks from third parties often seem to play an important part in the decision making process. Even though they are convenient to consult, it is easy to overlook a couple of important things that we would like to underline.

When it comes to performance benchmarks for example, it does not always become clear how the results have been obtained and how they will affect the reader. This is mostly due to the fact that benchmarks are often performed on synthetic applications in synthetic environments that don’t take into consideration real world workloads and latencies. This often leads to skewed results when compared to real time workloads.

A good example of this is the “Hello World” benchmark, where people tend to benchmark app servers against a so-called “Hello World” application: an app that basically returns “Hello World”. Needless to say, this is hardly a real world application.

Anyone who has ever deployed a real world application will know that these kinds of benchmarks don’t really say anything useful as they effectively measure how fast an app server is at “doing nothing”.

In real world Ruby applications on the other hand, processing time quickly gets overtaken by the app itself, plus network overhead, rather than the app server: the differences in performance between the app servers basically become insignificant when compared to the time spent in the hosted app itself.

Despite this, benchmarks remain a popular source to consult when trying to figure out what the “best” software solution is. The danger here lies in the possibility that a developer might be tempted to base their decision solely on what bar chart sticks out the most, without so much as looking at what the other solutions all bring to the table.

There is a reason for example why Phusion Passenger — even though hot on the heels of its competitors — has not been leading these kinds of Hello World benchmarks in the past: we do much more than the competition does when it comes to ease of use, memory efficiency, security and features. All these things are not free, and this is basically what is being measured with said benchmarks.

When benchmarked against real world applications however, the differences in performance between app servers becomes almost indistinguishable. The focus should then be put on what feature-set is most suitable for your production app. We believe that on that front, Phusion Passenger is leading the pack.

We have tried many times to explain this in the comment sections of benchmarks, but have unfortunately had to infer that such explanations often fall on deaf ears. We think that’s a shame, but rather than continue to fight it, we have decided to try to beat our competitors on performance as well. This has led to a series of internal optimizations and innovations which we have documented in the Raptor articles. Not only do we believe we are now able to win these kinds of benchmarks, we believe we have been able to do so with mechanisms that are incredibly useful in real world scenarios too (e.g. Turbocaching).

A greater focus on showcasing the technology

Software that is easy to use runs the risk of being considered “boring” to hackers, or worse, “simple”. In the latter case, the underlying technology facilitating the ease of use gets taken granted for. Over the years, we felt this was happening to Phusion Passenger to the extent that we wanted to set the record straight.

A lot of thought went into using the right algorithms and applying the right optimizations to allow Phusion Passenger to do what it does best: being able to deliver an “upload-and-go” deployment experience second to none in a secure and performant manner is by no means a trivial task. We chose to abstract these implementation details however from the end-user as we wanted them to be able to focus more on their app and business rather than the nitty gritty when it came down to how “the soup was made”. Who cares right?

Well, as it turned out, a lot of hackers do. Articles about “Unicorn being Unix” sparked a lot of interest from hackers allowing it to quickly garner a following. We thought articles such as these were great, but felt somewhat disappointed that people seemed to forget that Phusion Passenger was already doing the majority of what was written in such articles a full year earlier. It then dawned on us that we were no longer being considered to be the new shiny thing, but rather considered being part of the establishment.

In hindsight, it was perhaps also an error of judgement of us to focus our marketing efforts mostly on businesses rather than the grassroots hacker community we originated from ourselves: they are not mutually exclusive and instead of mostly underlining the business advantages, we should have underlined the technological advantages much more as well.

In an effort to set this straight, we chose to document Raptor’s underlying technology in great detail which was incredibly well received by the Hacker News community. It was in fact on the front page for most of the day with close to 200 upvotes and sparked a lot of discussions on Twitter too.

Besides the new optimizations and features found in Raptor, a lot of technology discussed in those articles was already available in Phusion Passenger 4 and its precursors. If there is a lesson to take from all this, it is that marketing is indeed the art of repetition. And that it probably helps to present it as the new kid on the block to get rid of any preconceived notions/misconceptions people might have had about Phusion Passenger.

Smoke and mirrors

Whether or not people would be just as excited if they knew that it was Phusion Passenger 5 all along is perhaps another discussion to be had: some actually found out ahead of time due to the similar writing style of our tech articles and / or through nslookups (a fedora hat-tip goes out to you folks! May your sense of scrutiny live long and prosper!).

What we do however know is that our Raptor approach over the past month has produced more subscribers to our newsletter than we have been able to accomplish over the past 6 years through the Phusion Passenger moniker. We still have a hard time comprehending this, but there is no denying the numbers: we — the community — seem to like shiny new things.

Truth be told, we didn’t really bother trying to cover up the fact that it was in fact Phusion all along that was behind Raptor. We kind of left it as an exercise to the reader to figure this out amidst all the “hype” and claims of “vaporware” to see if people still remembered Phusion Passenger’s fortes.

We were not disappointed when it came to that and felt incredibly proud that a lot of people questioned why Phusion Passenger was not included within the Raptor benchmarks and requested it to be included. Needless to say, this was something we were unable to do because it already was included all along as Raptor itself ;-) We were also happy to see that some even pointed out that some of the features look like they came straight out of Phusion Passenger.

What’s in a name?

You might be wondering why we chose to market Phusion Passenger 5 under the Raptor code name and went through so many hoops in doing so. To quote Shakespeare: “Conceal me what I am, and be my aid, for a disguise as haply shall become, the form of my intent”.

With all the new improvements and features pertaining to performance and introspection, we felt Phusion Passenger deserved new consideration from its audience in an objective manner. To circumvent any preconceived notions and/or misconceptions people may have had about Phusion Passenger over the years, we decided to market it as Raptor. We felt the codename was particularly appropriate for our renewed commitment to performance.

Just to be clear, Phusion Passenger will not be renamed to Raptor. Raptor is just a codename that has served its purpose by the time of this writing for Phusion Passenger 5. We will drop this name starting from today: from now on, “Raptor” is Phusion Passenger 5.

With a little help from our friends

The success of the Raptor campaign would not have been possible without the help and support of our friends. In particular, we would like to thank Peter Cooper and Fabio Akita for their in-depth write-ups on Phusion Passenger 5 / Raptor and its precursors. Their articles carried the necessary weight to allow us to focus on explaining and improving the technology itself rather than having to spend time on trying to debunk “vaporware” claims. In a similar manner, we would also like to thank David Heinemeier Hansson for helping us out via his tweets and feedback.

Lastly, we would like to thank the community and our customers for being so supportive. At the end of the day, it is you folks who make this all possible and we can’t wait to show you what we have in store for the future.

Final words before upgrade instructions

Phusion Passenger’s core is open source. Learn more about Phusion Passenger or fork or watch us on Github. :)

If you would like to stay up to date with Phusion news, please fill in your name and email address below and sign up for our newsletter. We won’t spam you, we promise.



Changelog

5.0.0 beta 1 is a beta release. It may still have bugs and we do not recommend using it in production yet.

Version 5.0.0 beta 1 contains major changes. It is mostly compatible with version 4, but there are a few minor breakages, which are described below. Major changes and notable breakages are:

  • Performance has been much improved. This is thanks to months of optimization work. You can learn more at www.rubyraptor.org.
  • We have published a server optimization guide for those who are interested in tuning Phusion Passenger.
  • Support for Rails 1.2 – 2.2 has been removed, for performance reasons. Rails 2.3 is still supported.
  • Phusion Passenger now supports integrated HTTP caching, which we call turbocaching. If your app sets the right HTTP headers then Phusion Passenger can tremendously accelerate your app. It is enabled by default, but you can disable it with --disable-turbocaching (Standalone), PassengerTurbocaching off (Apache), or passenger_turbocaching off (Nginx).
  • Touching restart.txt will no longer restart your app immediately. This is because, for performance reasons, the stat throttle rate now defaults to 10. You can still get back the old behavior by setting PassengerStatThrottleRate 0 (Apache) or passenger_stat_throttle_rate 0 (Nginx), but this is not encouraged. Instead, we encourage you to use the passenger-config restart-app tool to initiate restarts, which has immediate effect.
  • Websockets are now properly disconnected on application restarts.
  • The Phusion Passenger log levels have been completely revamped. If you were setting a log level before (e.g. through passenger_log_level), please read the latest documentation to learn about the new log levels.
  • If you use out-of-band garbage collection, beware that the X-Passenger-Request-OOB-Work header has now been renamed to !~Request-OOB-Work.
  • When using Rack’s full socket hijacking, you must now output an HTTP status line.
  • [Nginx] The passenger_set_cgi_param option has been removed and replaced by passenger_set_header and passenger_env_var.
  • [Nginx] passenger_show_version_in_header is now only valid in the http context.
  • [Apache] The PassengerStatThrottleRate option is now global.

Minor changes:

  • The minimum required Nginx version is now 1.6.0.
  • The instance directory is now touched every hour instead of every 6 hours. This should hopefully prevent more problems with /tmp cleaner daemons.
  • Applications are not grouped not only on the application root path, but also on the environment. For example, this allows you to run the same app in both production and staging mode, with only a single directory, without further configuration. Closes GH-664.
  • The passenger_temp_dir option (Nginx) and the PassengerTempDir option (Apache) have been replaced by two config options. On Nginx they are passenger_instance_registry_dir and passenger_data_buffer_dir. On Apache they are PassengerInstanceRegistryDir and PassengerDataBufferDir. On Apache, PassengerUploadBufferDir has been replaced by PassengerDataBufferDir.
  • Command line tools no longer respect the PASSENGER_TEMP_DIR environment variable. Use PASSENGER_INSTANCE_REGISTRY_DIR instead.
  • passenger-status --show=requests has been deprecated in favor of passenger-status --show=connections.
  • Using the SIGUSR1 signal to restart a Ruby app without dropping connections, is no longer supported. Instead, use passenger-config detach-process.
  • Introduced the passenger-config reopen-logs command, which instructs all Phusion Passenger agent processes to reopen their log files. You should call this after having rotated the web server logs.
  • [Standalone] The Phusion Passenger Standalone config template has changed. Users are encouraged to update it.
  • [Standalone] passenger-standalone.json has been renamed to Passengerfile.json.
  • [Standalone] passenger-standalone.json/Passengerfile.json no longer overrides command line options. Instead, command line options now have the highest priority.

Installing or upgrading

Here’s a quickstart:

gem install passenger --pre -v 5.0.0.beta1
cd /path-to-your-app
passenger start

The above is a very short excerpt of how to install or upgrade Phusion Passenger. For detailed instructions (which, for example, take users and permissions into account), please refer to the “RubyGems” section of the installation manuals:

Please note:

  • 5.0.0 beta 1 is a beta release. It may still have bugs and we do not recommend using it in production yet.
  • There are no Homebrew or Debian packages for this release, because this release is still in beta!
  • There is also a 5.0.0 beta 1 release of Phusion Passenger Enterprise available. Please refer to the Customer Area.