Asynchronous spawning

Previously, when application processes are being spawned, Phusion Passenger is unable to handle HTTP requests until the processes are done spawning, because Phusion Passenger is holding the lock on the application pool while this is happening. Some websites have apps that need a very long time to spawn (30+ seconds) and this would be a problem for them. This behavior would also cause a “thundering herd” problem: suppose that a traffic spike appears, then the first request will cause Phusion Passenger to spawn a process. The other requests in the spike are blocked until that’s done, and then all of a sudden they are processed. Phusion Passenger thinks it needs to spawn more, so it spawns another one and blocks the rest, and so on. This can cause a large number of processes to be spawned all of a sudden, causing a long delay.

In Phusion Passenger 3 spawning happens in the background so that no clients have to be blocked. This turns out to work so well that application process spawning is now virtually unnoticeable, except when spawning the first application process.

Ability to configure minimum number of processes

Phusion Passenger automatically shuts down processes that haven’t been accessed for a while, where “a while” is configurable through the PassengerPoolIdleTime directive. Many web applications are rarely accessed during the night, so what happens is that all application processes are shut down during the night and the first person in the morning who accesses the web application has to wait for some time while the first application process is being spawned. This problem could be solved by setting PassengerPoolIdleTime to 0 or to a large number which means that processes are kept around forever or for a long time, but this also means that application processes are not shut down during the night, which might still be desirable for resource utilization reasons.

Phusion Passenger 3 introduces a new, long-awaited configuration directive: PassengerMinInstances. As its name implies, PassengerMinInstances makes sure that at least the given number of processes will be started and kept around. This, in combination with asynchronous spawning, turns out to work so well that we’ve assigned a default value of 1 for PassengerMinInstances. With Phusion Passenger 3, spawning delays should become a thing of the past!

Smart spawning support for all Rack applications

Smart spawning is a core feature of Phusion Passenger since version 1.0. It can reduce the spawning time of Rails processes by as much as 90%, and in combination with Ruby Enterprise Edition it allows you to save 33% memory on average.

However, smart spawning was limited to Rails applications only, not for Rack applications. Starting from Rails 3, all Rails 3 applications are also Rack applications, and Phusion Passenger 2 only supports smart spawning of Rails 3 applications if you remove config.ru (thereby forcing Phusion Passenger to detect it as a Rails app, not a Rack app).

Phusion Passenger 3 now supports smart spawning for all Rack applications. This works transparently and without further configuration.

Ability to access individual application processes over HTTP

When one sends a request to the web server, Phusion Passenger routes the request to one of the application processes, but one never knows up front which one that’s going to be, nor is there a way to control it. This is fine for normal requests, but sometimes one wants to send a request to a specific application process, e.g. to debug something. Another use case for accessing individual application processes directly is broadcasting messages: e.g. telling every application process to clear some local in-memory caches.

This has always been possible with reverse proxy app servers like Mongrel and Thin because each of their processes listen on their own port. Phusion Passenger 3 now also allows one to access individual application processes directly. Each application process can now be accessed through its own TCP socket port. The port numbers are randomly allocated by the operating system and the protocol is plain HTTP, so you can use existing tools like ‘curl’.

We take security very seriously. These sockets are bound to 127.0.0.1 so it’s not possible to access them from remote computers. Furthermore, each socket is protected by its own unique randomly generated secure password. One can use ‘passenger-status’ to query the ports and passwords.

Global queuing now on by default

Many people with web apps that have long-running requests are familiar with the “slow Mongrel queue problem”. When one sets up Mongrel or Thin behind Nginx or Apache, it’s possible for new requests to be queued behind a Mongrel/Thin instance which is currently processing a long-running request. When other Mongrel/Thin instances become available, this new request is already queued behind the long-running request and cannot migrate to the other free instances. The more long-running requests one has the bigger of a problem this can become, resulting to very long response times for some users.

There are multiple ways to solve this problem, but Phusion Passenger has already solved this problem for a long time with a feature that we call global queuing. It was disabled by default because turning global queuing off would yield a little bit of extra performance in microbenchmarks. However for version 3.0 we’ve decided to turn it on by default: rather than saving those few milliseconds in benchmarks, we believe it’s much more important that all users get to have fair response times.

Ability to disable friendly error pages

One of the innovations that Phusion Passenger has brought us is the ability to show friendly error messages directly in the browser, e.g. when the web application fails to spawn because of a syntax error. This dramatically improves usability for developers, new and experienced alike, because one doesn’t have to manually dig into log files anymore. However this is not always desirable in production: although the error page is developer-friendly, it isn’t necessarily user-friendly. It might also expose information that the system administrator would rather not expose such as filenames.

Phusion Passenger 3 introduces a new configuration directive for controlling whether friendly error pages should be shown: ‘PassengerFriendlyErrorPages’. When turned off, Phusion Passenger will display the standard Apache/Nginx 500 Internal Server Error page instead, but all error messages are still logged to the web server error log file.

Nginx-specific improvements

In Phusion Passenger 3, Nginx is compiled with SSL support by default due to popular demand.

We’ve also introduced the following new configuration options:

passenger_set_cgi_param (name) (value)

This is the Phusion Passenger equivalent of proxy_module’s proxy_set_header. It allows you to pass arbitrary CGI environment parameters the web application.

passenger_buffer_response (on|off)

This is the Phusion Passenger equivalent of proxy_buffer. It was and still is off by default to allow streaming responses, but when streaming responses aren’t necessary one can turn this option on so that Nginx can gracefully handling clients that are slow at receiving responses.

passenger_ignore_client_abort (on|off)

This is the Phusion Passenger equivalent of proxy_ignore_client_abort.

In other news

In Technology Preview 3 we unveiled Phusion Passenger Lite. Based on various feedback that we’ve gotten since then, we’ve decided to rename this thing to Phusion Passenger Standalone in order to reduce confusion about what it is.

Towards the future

If there is one thing we’ve come to understand over time is that different businesses have different needs and constraints when it comes to deploying their applications. In order to provide the best experience and support to these businesses, we’re working on different versions of Phusion Passenger to accommodate them even better in their respective environments. In light of this, we want to underline that the technology previews we’re currently writing about will and have described the cool stuff that will be incorporated in the version intended for the most high demand environments. More information on this will follow. Having said that, almost everything we’ve blogged about up till this point will be included in the version that’s available for everybody.

While developing Phusion Passenger for Apache, we decided to follow a PHP-like model because ease of use was one of our main goals. No manual setups of app servers. No manual proxy configuration. Upload and go. For Phusion Passenger for Nginx, we continued to follow this model. Let’s call this the automatic model. As of 2010, Phusion Passenger appears to be the only widely-used Ruby app server that implements this model; the other widely-used Ruby app servers implement the reverse proxy model.

Reverse proxy vs automatic model: which one is better?

Ever since Phusion Passenger was first released, debates popped up about which one is superior. We believe that no model is inherently superior to the other. They are just different, meaning that both models have their own pros and cons. Which one is better for you depends a lot on your server infrastructure and your system administrators’ preferences.

Phusion Passenger’s automatic model:

• Integrated into the web server. Processes are managed along with the web server itself, and configuration happens in the web server config file.
• Easier to comprehend for most people. Appears more “standard stack” to system administrators who are not familiar with Ruby specifically.
• Can spawn and shutdown processes dynamically according to traffic patterns.
• Processes are automatically monitored: if they crash they are automatically restarted.
• Less manual control over individual processes because they can come and go at any time.

Reverse proxy model as implemented by most other Ruby app servers:

• App server is a separate entity. Processes are managed distinctly from the web server itself. Configuration happens outside the web server.
• Many people have a hard time comprehending this and they generally find setups like this cumbersome, but to experts this model can be seen as simple, elegant and sensible.
• Most app servers do not automatically restart crashed processes and one needs to monitor processes separately with things like Monit.
• One needs to specify the number of processes up front: no dynamic process count scaling according to traffic.
• Allows fine-grained manual control over individual processes.

We are not commenting on which points are supposed to be pros and which points are supposed to be cons because they are highly subjective. For us, integration into the web server is a strong plus because we host dozens of apps on our server(s) and we don’t like to spend time managing app server processes for each app, but other people are uncomfortable with having the web server manage things automatically and would prefer to keep a close eye on everything.

The automatic model can also be problematic to people who were on the reverse proxy model because they already had their web servers and infrastructures configured in a certain way. Switching to Phusion Passenger could mean changing a lot of web server configuration.

The hidden but unutilized potential

Reverse proxy model app servers can potentially have an extra advantage, but for some reason this hasn’t been implemented to its full potential so far:
Reverse proxy app servers are just easier to get started with. When you’ve just created a new Rails app, you can start it with script/server and you’re ready to go.

This works great in development but totally blows up in production. Reverse proxy model app servers must be put behind a reverse proxy e.g. Nginx or HAProxy for a variety of reasons such as security, load balancing between processes, handling of slow clients, etc. In production environments nobody exposes Mongrel or Thin directly to the Internet. Unicorn even explicitly documents that it is designed to be put behind a reverse proxy and that it doesn’t bother with slow clients at all.

In contrast, Phusion Passenger 2.x requires one to configure the web server, meaning the user must first install a web server. This is cumbersome when you’re in development and just want to get started. It is also cumbersome if you’re a newcomer and aren’t familiar with Apache or Nginx, and you just want to get your app running on your server.

Do you type script/server in development instead of creating a virtual host in the Apache or Nginx? Well you’re not the only one: we also do this until we eventually get sick of it, but there’s always a mental blockade that tells us that editing the web server configuration file is too much work to bother with.

Well, until Phusion Passenger 3 comes along.

Phusion Passenger Lite: fusion between the reverse proxy and the automatic model

In addition to Phusion Passenger for Apache and Phusion Passenger for Nginx, Phusion Passenger 3 introduces a new component to the existing lineup: Phusion Passenger Lite.

When it comes to usage, its interface is almost identical to that of Mongrel and Thin. To run your Ruby web app, just type this in the terminal and you’re ready to go:

passenger start

Closing the gap between development and production

Phusion Passenger Lite consists of an Nginx core. Nginx is known to be extremely scalable, high-performance and lightweight. You do not need to have Nginx already installed; this is automatically taken care of. You also do not need to have any Nginx experience: Nginx is hidden from the user but its power is automatically utilized.

Unlike Mongrel, Thin and Unicorn, Phusion Passenger Lite can be directly exposed to the Internet. It can serve static files at blazing speeds thanks to the Nginx core. Mongrel and Thin can serve static files but they aren’t very good at it. Unicorn doesn’t even try.

Easy migration from existing reverse proxy app servers

Because the interface is so similar, you can easily swap Mongrel, Thin or Unicorn in your existing reverse proxy setup and replace it with Phusion Passenger Lite. Unlike Mongrel and Thin, Phusion Passenger Lite only has to listen on a single socket instead of multiple, vastly simplifying your reverse proxy configurations. Phusion Passenger Lite can listen on a Unix domain socket instead of a TCP socket, just like Thin and Unicorn. In reverse proxy setups this can yield much higher performance than TCP sockets.

Advantages over existing reverse proxy app servers

Unlike Mongrel, Thin and Unicorn, Phusion Passenger Lite can dynamically spawn and shutdown processes according to traffic. However you can also configure it to use a static number of processes! In fact you can configure a minimum and a maximum and have Phusion Passenger Lite automatically figure out the number of processes to use for the current traffic.

Like Phusion Passenger for Apache/Nginx and Unicorn, worker processes that have crashed are automatically restarted.

That said, bear in mind that this advantage can be a disadvantage to some people. At its heart, Phusion Passenger Lite still manages processes for you, so you don’t have as much fine-grained control over the processes as you do with other reverse proxy app servers.

Advantage over Phusion Passenger for Apache and Phusion Passenger for Nginx

Another unintended advantage of Phusion Passenger Lite is that it runs as the same user as the shell and respects environment variables that are defined for your shell, e.g. things like PATH, LD_LIBRARY_PATH, RUBYOPT, GEM_PATH and GEM_HOME.

• Some people find that their app cannot load a certain library when the app is started in Phusion passenger, but can when the app is started with e.g. Mongrel or Thin. This is almost always caused by some environment variable that’s set in the shell but not in the web server: everything you set in /etc/bashrc and friends don’t have effect on processes started from the web server.
• Some people say that their application does not start on Phusion Passenger, but does under Mongrel/Thin. Very often this turns out to be just a permission issue or some web server configuration issue.

With Phusion Passenger Lite, even confusion like this will be a thing of the past.

Rethinking the word “easy”: automatic mass deployment

Phusion Passenger is considered by many people the easiest Ruby app server out there. But can it be easier? After some heavy thinking outside the box, we believe the answer is yes, it certainly can!

Imagine having a directory full of Ruby web apps, e.g. ~/Sites. To deploy an app, just drop your application’s directory into ~/Sites. To undeploy it, remove the application’s directory. The application directory’s name is used as the domain name. No manually signaling the web server for a restart.

This is exactly what we’ve done with Phusion Passenger Lite. We call this feature automatic mass deployment. Check out this demo.

So from now on, if you have a bunch of Ruby web apps in the same directory, just run this command in that directory

sudo passenger start -p 80 -u (some_unprivileged_username)

and you’ve immediately deployed every single app!

Conclusion

Phusion Passenger Lite does not replace Phusion Passenger for Apache or Phusion Passenger for Nginx. Rather, it is a complement to our existing lineup of Phusion Passenger products, optimized for different use cases. Phusion Passenger Lite closes the gap between development and production and can be used comfortably and easily in both. It can act as a drop-in replacement for your existing reverse proxy based setup. It makes Ruby web app deployment even easier than before: now you don’t even need a separate web server. On the other hand, if you need integration into the web server, then Phusion Passenger for Apache/Nginx is for you.

We hope you’ve enjoyed this Technology Preview. Please stay tuned for the next one because we have more exciting news for you!

What’s new in version 2.2.15?

[Apache] Fixed incorrect temp dir cleanup by passenger-status

On some systems, running passenger-status could print the following message:

*** Cleaning stale folder /tmp/passenger.1234

…after which Phusion Passenger breaks because that directory is necessary for it to function properly. The cause of this problem has been found and has been fixed.

[Apache] Fixed some upload handling problems

Previous versions of Phusion Passenger check whether the size of the received upload data matches the contents of the Content-Length header as sent by the client. It turns out that there could be a mismatch e.g. because of mod_deflate input compression, so we can’t trust Content-Length anyway and we were being too strict. The check has now been removed.

[Nginx] Fixed compilation issues with Nginx >= 0.7.66

Thanks to Potamianos Gregory for reporting this issue. Issue #500.

[Nginx] Default Nginx version changed to 0.7.67

The previous default version was 0.7.65.

Fixed more Bundler problems

Previous versions of Phusion Passenger would preload some popular libraries such as mysql and sqlite3 in order to utilize copy-on-write optimizations better. However this behavior conflicts with Bundler so we’ve removed it.

How do I upgrade to 2.2.15?

Via a gem

Please install it with the following command:

gem install passenger

Next, run:

passenger-install-apache2-module

Or, if you’re an Nginx user:

passenger-install-nginx-module

Please don’t forget to copy & paste the Apache/Nginx config snippet that the installer gives you.

Via a native Linux package

John Leach from Brightbox has kindly provided Ubuntu packages for Phusion Passenger. These packages are available from the Brightbox repository which you can find at:

http://apt.brightbox.net

Add the following line to the Third Party Software Sources:

deb http://apt.brightbox.net hardy main

(The simplest way to do that is to create a file in /etc/apt/sources.list.d/ containing the deb instruction, and then run ‘apt-get update’).

Once you’ve done this then you can install Phusion Passenger by running:

sudo apt-get install libapache2-mod-passenger

-or-

sudo apt-get install nginx-brightbox

(Note that John is currently packaging 2.2.15, so it might take a while before this release shows up in the apt repository.)

Final

Phusion Passenger is provided to the community for free. If you like Phusion Passenger, please consider sending us a donation. Thank you!

Although stability, robustness and availability aren’t as hot and fashionable as performance, for Phusion Passenger 3 we have not neglected these areas. In fact we’ve been working on hard on implementing additional safeguards, as well as refactoring our designs to make things more stable, robust and available.

Self-healing

In Phusion Passenger 2.2′s architecture, there are a number of processes that work together. At the very front there is the web server, which could consist of multiple processes. If you’ve ever typed passenger-memory-stats then you’ve seen the web server processes at work. Apache typically has a dozen of processes (prefork MPM) and Nginx typically has 3.

For Phusion Passenger to work, there must be some kind of global state, shared by all web server processes. In this global state information is stored such as which Ruby app processes exist, which ones are currently handling requests, etc. This allows Phusion Passenger to make decisions such as which Ruby app process to route a request to, whether there is any need to spawn another process, etc. This global state exists in a separate process which all web server processes communicate with. On Apache this is the ApplicationPoolServerExecutable, on Nginx this is the HelperServer. For simplicity’s sake, let’s call both of them HelperServer (in Passenger 3 they’ve both been renamed to PassengerHelperAgent). The HelperServer is written in C++ and is extremely fast and lightweight, consuming only about 500 KB of real memory.

As you can see the HelperServer is essentially the core of Phusion Passenger. The problem with 2.2 is that if the HelperServer goes down, Phusion Passenger goes down with it entirely. Phusion Passenger will stay down until the web server is restarted. For various architectural reasons in Apache and Nginx, it is not easily possible to restart the HelperServer upon a crash in a reliable way.

Now, why would the HelperServer ever crash?

1. Bugs. We are humans too and we can make mistakes, so it’s possible that there are crasher bugs in the HelperServer. In the past 2 years we’ve spent a lot of effort into making the HelperServer stable. For example we check all system calls for error results, and we’ve spent a lot of effort into making sure that uncaught exceptions are properly logged and handled. However one can never prove that a system is entirely bug free. We aren’t aware of any crasher bugs at this time but they might still exist.
2. Limited system resources. For example if the system is very low on memory, the kernel will invoke the Out-Of-Memory Killer (OOM Killer). Properly selecting a process to kill in low-memory conditions is actually a pretty hard problem, and more often than not the OOM Killer selects the wrong process, e.g. our HelperServer.
3. System administrator mistakes. Passing the wrong PID to the kill command and things like that.
4. System configuration problems, hardware problems (faulty RAM and stuff) and operating system bugs.

There are some people who have reported problems with the HelperServer. Their HelperServer crashes tend to happen sporadically and they usually cannot reproduce the problem reliably themselves. For many people (2) is often the cause of HelperServer crashes, and increasing the amount of swap is reported to help, but for other people the problems lie elsewhere.

The crashes aren’t always our fault (i.e. not bugs), but they are always our problem. It saddens us to say that we’ve been unable to help these people so far because we simply cannot reproduce their problems even when we mimic their system configuration.

But this is going to change.

Enter Phusion Passenger 3 with self-healing architecture

Phusion Passenger 3 now introduces a lightweight watchdog process into the architecture. It monitors both the web server and the HelperServer. If the HelperServer crashes, then the watchdog restarts the HelperServer immediately.

Of course if the watchdog is killed then it’s still game over, but we’ve taken extra care in terms of code to try to make this extremely unlikely to happen. The watchdog for starters is extremely lightweight, even more so than the HelperServer. It is written in C++ and uses about 150-200 KB of memory. Its only job is to start the HelperServer and other Phusion Passenger helper processes and to monitor them. The codebase extensively uses C++ idioms that promote code stability, such as smart pointers and RAII. By employing heavy testing as well, we’re expecting to have brought the possibility that the watchdog contains crashing bugs to a minimum. The small footprint and the fact that it does nothing most of the time minimizes the chances that it’s killed by the OOM Killer. In fact, if the watchdog is running on Linux and has root access, it will register itself as not OOM-killable.

No longer will HelperServer crashes take down Phusion Passenger, even if the crash isn’t our fault.

Restarts are fast

It only takes a few hundred miliseconds to restart the HelperServer.

Crashing signals are logged

If the HelperServer crashes then the watchdog will tell you whether it crashed because of a signal, e.g. SIGSEGV. This makes it much easier for system administrators to see why a component crashed so that they might fix the underlying cause. In Phusion Passenger 2.2 this was not possible.

Guaranteed cleanup

Upon shutting down or restarting the web server, Phusion Passenger 2.2 gracefully notifies application processes to shut down. It does not force them to. This would pose a problem for broken web applications that don’t shut down properly, e.g. web applications that are stuck in an infinite loop, stuck in a database call, etc.

Phusion Passenger 3 guarantees that all application processes are properly shut down when you shutdown/restart the web server. It gives application processes a deadline of 30 seconds to shutdown gracefully; if any of them fail to do that, they’ll be terminated with SIGKILL.

This mechanism works so well that it even extends to background processes that have been spawned off by the web application processes. All of those processes belong to the same process group. Phusion Passenger sends SIGKILL to the entire process group and terminates everything. No longer will you have to manually clean up processes; you can be confident that everything is gone if you shutdown/restart the web server.

Zero-downtime web server restart

In Phusion Passenger 2.2, whenever you restart the web server, HTTP requests that are currently in progress are dropped and the clients receive ugly “Connection reset by server” or similar error messages. This can be a major problem for large websites, because during the 1 second that Phusion Passenger is restarting hundreds of people could be getting errors. If your visitor happens to be clicking on that “Buy” button, well, tough luck.

In Phusion Passenger 3 we’ve implemented zero-downtime web server restart. Phusion Passenger and the web server are restarted in the background, and while this is happening, the old web server instance (with the old Phusion Passenger instance) will continue to process requests.

The architecture is actually a little bit more complicated than what’s shown in the diagram because behind the web server there are a bunch of Phusion Passenger processes, but you get the gist of it.

When the new web server (along with the new Phusion Passenger) has been started, it will immediately begin accepting new requests. Old requests that aren’t finished yet will continue to be processed by the old web server and Phusion Passenger instance. The old instance will shut down 5 seconds after all requests have been finished, to counter the possibility that the kernel still has leftover requests in the socket backlog that hit the old instance after its done processing everything already in its queue.

This works so well that we can restart the web server while running an ‘ab’ benchmark with 100 concurrent users without dropping a single request!

Zero-downtime application shutdown

Suppose that a Ruby application process has gone rogue and you want to shut it down. The most obvious way to do that is by sending it a SIGTERM or SIGKILL signal. However this would also abort whatever request it is currently processing.

In Phusion Passenger 2.2, you could also send SIGUSR1 to the process, causing it to shut down gracefully after it has processed all requests in its socket backlog. However this introduces two problems:

• If the website is very busy then the process’s socket backlog will never be empty, and so the process will never exit.
• Exiting after the process has detected an empty backlog can introduce a race condition. Suppose that, right after the process has determined that its backlog is empty but before it has shutdown completely, Phusion Passenger tries to send another request to the process. This request would be lost.

In Phusion Passenger 3, SIGUSR1 will now cause the application process to first unregister itself so that Phusion Passenger won’t route any new requests to it anymore. It will then proceed with exiting 5 seconds after its socket backlog has become empty. This way you can gracefully shutdown a process without losing a single request.

Conclusion

Although Phusion Passenger has been powering many high-traffic Ruby websites for a while now, some people still have some doubts about whether Phusion Passenger is fit for production. Instead of using words convince them, we would rather convince them with real results. Phusion Passenger 3 raises the bar in the areas of performance, stability, robustness and availability yet higher, but it doesn’t stop here. Please stay tuned for the next Technology Preview in which will unveil even more of Phusion Passenger 3.

• Ease of use.
• Stability.
• Robustness.

Notice that “performance” is not on the above list. We strived to make Phusion Passenger “fast enough”, e.g. not ridiculously slower than the alternatives. Lately it would appear that competitors are once again focusing on performance. We can of course not afford to stay behind. We’ve been working on Phusion Passenger 3 for a while now. Today we will begin unveiling the technology behind this new major Phusion Passenger version. This blog post is the first of the multiple technology previews to come.

The performance test

It’s not very useful to benchmark Phusion Passenger performance using a Rails application because most of the time is spent in Rails and the application itself. Therefore we’ll benchmark with a simple Rack application. Consider the following hello world Rack application:

app = proc do |env|
  [200, { "Content-Type" => "text/html" }, ["hello world\n"]]
end

run app

How fast does this run on Phusion Passenger 2.2?

• Operating system: OS X Snow Leopard
• ab -c 25 -n 10000 http://rack.test/, pool size 25
• Apache: 1628 req/sec
• Nginx: 1843 req/sec

Now let’s look at Phusion Passenger 3:

• Operating system: OS X Snow Leopard
• Apache: 2225 req/sec; 36% faster
• Nginx: 2864 req/sec; 55% faster

That’s right, the Nginx version is over 50% faster than 2.2 on OS X!

A graph is worth more than a thousand words

Suffice it to say, even though Phusion Passenger was already pretty fast, we believe we’ve created some pretty significant improvements in terms of performance and it will be interesting to see how the final version of Phusion Passenger 3 will stack up against the competition. Needless to say, we’ve performed our own benchmarks already and have concluded that “the self-proclaimed fastest deployment solution” really isn’t the fastest deployment solution compared to Phusion Passenger 3. That said, benchmarks are lies, lies, lies, damn lies of course and your mileage may definitely vary so we will encourage you to perform any kind of benchmark you’d like when we release 3. For us, the most important issue still lies in the trade off of how much time you have to spend actually maintaining your setup, but as the graphs indicate, we’ve made some pretty monstrous improvements to performance as well.

How did we do it?

When it comes to optimizing software, there’s the saying that 20% of the code is responsible for 80% of the time. Not so with Phusion Passenger: we’ve found that there were no obvious performance bottlenecks. Even profilers turned out to be totally useless because all the times are so small and so close to each other. Phusion Passenger was already pretty fast.

Instead, we optimized the hard way: with lots and lots of micro-optimizations. 2% here, 3% there, etc etc. In other words, blood, sweat, tears and lots of sleepless nights. The optimizations can be summed up as follows:

Reducing system calls

System calls are pretty expensive compared to userspace computation. They require a context switch to the kernel. For example, all I/O operations (read(), write()) are system calls. We’ve performed an extensive code inspection and removed and coalesced a lot of redundant system calls.

The beginning of a zero-copy I/O architecture

The CPU is very fast nowadays. In fact it is so fast that RAM speed cannot keep up with the CPU. This makes memory access very expensive. In case of I/O intensive applications such as web servers, one would benefit from copying I/O data as little as possible. In order to optimize memory access, we’ve implemented the beginning of a zero-copy I/O architecture. This architecture covers both the C++ and the Ruby parts of Phusion Passenger.

Less Ruby garbage production

The garbage collector in Ruby can be a significant bottleneck. We’ve heavily optimized the Ruby part of the request handler and reduced creation of Ruby objects to a minimum. This made the request handler significantly faster in our tests.

Optimizing algorithms and optimizing Ruby code in C

Some algorithms have been optimized, e.g. some O(N) algorithms have been replaced by O(log N) or O(1) algorithms. Some key Ruby code has been replaced by C code. The former didn’t give us a lot of performance because all the O(N) algorithms weren’t doing a lot of work in the first place, but the latter gave us a much more noticeable boost.

Reducing context switches

Phusion Passenger is heavily multithreaded and consists of multiple threads and service processes. However, some communication between threads and processes required round trips, which caused more context switches than necessary. We’ve optimized our internal protocols and reduced context switching to a minimum.

The future

As stated in this blog post, this is just a glimpse of what we’ve got in store for you and as you’ve come to expect from us, we want to make sure that our findings will hold up in real life scenarios as well. With close to two years of field testing with Phusion Passenger 2, witnessing some of the most high demanding environments in web hosting of our clients, we’ve been working for the last few months now on forging this experience back into Phusion Passenger 3. Through beta testing in these high-demand Rails environments, we hope to ensure that they will give you the best experience both in an enterprise environment as well as for your personal use. Performance has been touched upon in this blog post, and in the coming period leading up to the release of Phusion Passenger 3, we’ll start to unveil bit by bit what we’ve been tinkering on for the last few months. In particular, we’re looking forward how the zero-copy I/O architecture will unfold in a real life scenario as well as the optimizations we’ve performed over the last months. Even though we’re not done yet in terms of optimizing, we will likely hit a ceiling at some point where optimizations will get harder and harder and this in particular is true if you want to retain features such as ease of use that define Phusion Passenger. One thing is for sure, we want this release to be nothing less than stunning so we encourage you to submit your wish list to us as well. We’ve likely implemented a lot of them already, but we just want to make sure that we’re not missing anything.

Changes since 2.2.12

Fixed some Rails 3 compatibility issues that were recently introduced.

About a week ago the Rails team committed a change which broke our Rails loader. This has now been fixed. Rails 3 remains to be a moving target but we’ll keep moving along with it.

[Nginx] Fix a localtime() crash on FreeBSD

This was caused by insufficient stack space for threads. Issue #499.

Added support for Rubinius

Patch contributed by Evan Phoenix.

Fixed a mistake in the SIGQUIT backtrace message.

Patch contributed by Christoffer Sawicki.

Fixed a typo that causes config/setup_load_paths.rb not to be loaded correctly.

This is related to the new Bundler support.

Recent changes

Phusion Passenger is under constant maintenance and development. We are pleased to announce Phusion Passenger version 2.2.11. This is a bug fix release.

Improved Bundler support.

Previous versions might not be able to correctly load gems bundled by Bundler. We’ve also documented how our Bundler support works and how to override our support if you need special behavior. Please refer to the Phusion Passenger Users Guide, section "Bundler support".

Worked around some user account handling bugs in Ruby.

Issue #192.

[Apache] Suppress bogus mod_xsendfile-related error messages.

When mod_xsendfile is being used, Phusion Passenger might print bogus error messages like "EPIPE" or "Apache stopped forwarding the backend’s response" to the log file. These messages are normal, are harmless and can be safely ignored, but they pollute the log file. So in this release we’ve added code to suppress these messages when mod_xsendfile is being used. Issue #474.

[Nginx] Fixed “passenger_user_switching off” permission problems

If Nginx is running as root and passenger_user_switching is turned off, then Phusion Passenger would fail to initialize because of a permission problem. This has been fixed. Issue #458.

[Nginx] Nginx >= 0.8.38 is now supported.

Thanks to Sergey A. Osokin for reporting the problem.

[Nginx] passenger-install-nginx-module upgraded

It now defaults to installing Nginx 0.7.65 instead of 0.7.64.

Others

• Fixed some Ruby 1.9 tempfile.rb compatibility problems.
• Fixed some compilation problems on some ARM Linux platforms.

How do I upgrade to 2.2.12?

Via a gem

Please install it with the following command:

gem install passenger

Next, run:

passenger-install-apache2-module

Or, if you’re an Nginx user:

passenger-install-nginx-module

Please don’t forget to copy & paste the Apache/Nginx config snippet that the installer gives you.

Via a native Linux package

John Leach from Brightbox has kindly provided an Ubuntu 8.04 package for Phusion Passenger. The package is available from the Brightbox repository which you can find at:

http://apt.brightbox.net

Add the following line to the Third Party Software Sources:

deb http://apt.brightbox.net hardy main

(The simplest way to do that is to create a file in /etc/apt/sources.list.d/ containing the deb instruction, and then run ‘apt-get update’).

Once you’ve done this then you can install Phusion Passenger by running:

sudo apt-get install libapache2-mod-passenger

-or-

sudo apt-get install nginx-brightbox

(Note that John is currently packaging 2.2.12, so it might take a while before this release shows up in the apt repository.)

Final

Phusion Passenger is provided to the community for free. If you like Phusion Passenger, please consider sending us a donation. Thank you!

This problem does not affect Nginx; you only have to upgrade if you’re using Apache.

More information about the problem can be found at the following discussion thread: http://groups.google.com/group/phusion-passenger/t/d5bb2f17c8446ea0

How do I upgrade to 2.2.11?

Via a gem

Please install it with the following command:

gem install passenger

Next, run:

passenger-install-apache2-module

Or, if you’re an Nginx user:

passenger-install-nginx-module

Please don’t forget to copy & paste the Apache/Nginx config snippet that the installer gives you.

Via a native Linux package

John Leach from Brightbox has kindly provided an Ubuntu 8.04 package for Phusion Passenger. The package is available from the Brightbox repository which you can find at:

http://apt.brightbox.net

Add the following line to the Third Party Software Sources:

deb http://apt.brightbox.net hardy main

(The simplest way to do that is to create a file in /etc/apt/sources.list.d/ containing the deb instruction, and then run ‘apt-get update’).

Once you’ve done this then you can install Phusion Passenger by running:

sudo apt-get install libapache2-mod-passenger

-or-

sudo apt-get install nginx-brightbox

(Note that John is currently packaging 2.2.11, so it might take a while before this release shows up in the apt repository.)

Final

Phusion Passenger is provided to the community for free. If you like Phusion Passenger, please consider sending us a donation. Thank you!

Recent changes

Phusion Passenger is under constant maintenance and development. We are pleased to announce Phusion Passenger version 2.2.10. This is a bug fix release.

• Fixed some Bundler compatibility problems.
• Fixed some file descriptor passing problems, which previously could lead to mysterious crashes.
• Fixed some compilation problems on newer GCC versions. Issue #430.
• Support #size method in rack.input.

How do I upgrade to 2.2.10?

Via a gem

Please install it with the following command:

gem install passenger

Next, run:

passenger-install-apache2-module

Or, if you’re an Nginx user:

passenger-install-nginx-module

Please don’t forget to copy & paste the Apache/Nginx config snippet that the installer gives you.

Via a native Linux package

John Leach from Brightbox has kindly provided an Ubuntu 8.04 package for Phusion Passenger. The package is available from the Brightbox repository which you can find at:

http://apt.brightbox.net

Add the following line to the Third Party Software Sources:

deb http://apt.brightbox.net hardy main

(The simplest way to do that is to create a file in /etc/apt/sources.list.d/ containing the deb instruction, and then run ‘apt-get update’).

Once you’ve done this then you can install Phusion Passenger by running:

sudo apt-get install libapache2-mod-passenger

-or-

sudo apt-get install nginx-brightbox

(Note that John is currently packaging 2.2.10, so it might take a while before this release shows up in the apt repository.)

Final

Phusion Passenger is provided to the community for free. If you like Phusion Passenger, please consider sending us a donation. Thank you!