Phusion Corporate Blog

Ruby Enterprise Edition 1.8.7-20090928 released

By Hongli Lai September 29th, 2009

In the not so distant past we launched a Ruby Enterprise Edition sponsorship campaign. We’ve been working hard on the code since that time. And although the campaign target hasn’t been reached, we’ve decided to release it anyway because we believe that this release is of great importance to the community.

Just how important is this release? Well, Twitter has beta tested this release and they’re reporting roughly 30% improvement in throughput. Keep reading to learn more and follow us on @phusion_nl for the latest updates.

Summary of changes

It’s now based on Ruby 1.8.7-p174

Previous REE releases were based on 1.8.6. We were initially reluctant about switching to 1.8.7: many people reported various incompatibility problems introduced by 1.8.7, and it seemed that alternative Ruby implementors were boycotting 1.8.7. However, the tide has now changed. Rails 3.0 will drop support for Ruby versions earlier than 1.8.7, JRuby had recently decided to support 1.8.7 and OS X Snow Leopard as well as all the Linux distros ship 1.8.7. We’ve received a lot of requests from the community for a 1.8.7-based version.

In short, everybody is moving towards 1.8.7, and we can’t afford to stay behind.

We were initially planning to release a final 1.8.6-based version, but after carefully considering the current state of things, we decided to skip the 1.8.6 release and jump directly to 1.8.7. 1.8.7 is also more compatible with 1.9, allowing a smoother transition to future Ruby versions.

The MBARI patches have been integrated

Brent Roman’s MBARI patch set uses a variety of techniques to significantly reduce Ruby’s memory usage. It also adds a bunch of performance optimizations, useful features and bug fixes.

Ruby’s garbage collector scans the C stack conservatively. This means that any data on the C stack that might look like a pointer pointing to a valid Ruby object will be treated like a real pointer, causing the object to be retained during garbage collection. Because of the way Ruby is written and because of compiler optimizations, there may be a lot of such “pointers” on the stack, even when they’re not really pointers but just garbage data that like look pointers.

Brent has refactored and modified many parts of Ruby in order to reduce the possibility of such “garbage pointers” occurring on the stack. As a result, garbage collection will be much more effective. Many objects that would otherwise not have been freed will be freed thanks to the MBARI patches.

As a side-effect of the reduced memory usage, applications typically run faster as well. Not only because applications are less likely to be swapped out, but also because modern hardware’s performance depends greatly on the amount of memory traffic.
Ruby uses an inefficient, linear time algorithm for extending the stack space when switching context between threads and/or continuations. This is mostly noticeable when you have a lot of threads. Brent replaces this with a constant time algorithm, which is much faster.
It adds the methods __file__ and __line__ to the Method, UnboundMethod and Proc classes. Very useful for inspecting large code bases that make heavy use of metaprogramming. These methods are documented in the Ruby Enterprise Edition manual.

We’ve ported his patch set to 1.8.7-p174. Especially Twitter benefited a lot from these patches.

The zero-copy context switching patch have been integrated as an optional feature

Joe Damato and Aman Gupta have created a patch which implements zero-copy context switching in Ruby’s threading system. In Ruby 1.8, whenever Ruby switches context to another thread, it has to copy over the entire thread stack. This is very expensive, and the deeper your function calls are nested, the slower it becomes. The zero-copy context switching patch removes the need to copy any stack data; instead, it keeps all thread stacks at different areas in memory and simply changes the stack pointer. In extreme situations, this patch can increase threading performance by as much as 10 times!

Phusion Passenger is not really affected by this patch because its request handling core is not multithreaded, at least not at this moment. But Mongrel is heavily multithreaded and will benefit tremendously from this patch.

We’ve made this patch compatible with the MBARI patch set, but there are currently some issues with 64-bit platforms, so we’ve disabled it by default. You can enable it by running the REE installer with --fast-threading

Various other minor changes

When compiling REE, binaries are optimized for size (-Os), not speed. See below for an explanation.
The tcmalloc memory allocator has been upgraded to version 1.3
RubyGems upgraded to version 1.3.5.
The Debian package now installs files to /usr/local, so that one doesn’t have to mess with $PATH to get things working.

In practice

This release has been beta tested with Twitter. Evan Weaver reported significant improvements in Twitter’s throughput after having switched to this REE release.

Twitter especially benefited from the MBARI patch set. We’ve found a few unexpected things during the beta testing phase:

We’ve found that optimizing Ruby for code size (compilation with -Os) yields better results that optimizing Ruby for speed (compiling with -O2). Ruby is a very unconventional program and we suspect that optimizing for code size allows more instructions to fit in the CPU instruction cache, resulting in better performance.
Microbenchmarks and synthetic benchmarks such as the Ruby benchmark suite do not show any performance improvements at all, although Twitter was able to measure a significant performance improvement. As always, benchmark results should be taken with a grain of salt.

Other major beta testers include 37signals and Novell. They’re still in the progress of testing, and we’re eager to receive their test results.

Download & upgrade

To install Ruby Enterprise Edition, please visit the download page. To upgrade from a previous version, simply install into the same prefix that you installed to last time. Please also refer to the documentation for upgrade instructions.

UPDATE: Snow Leopard support

It looks like tcmalloc has some issues on Snow Leopard, causing the compiler to crash. We’ve just been able to obtain Snow Leopard last weekend (it had been sold out for several weeks) and we haven’t gotten around installing it yet.

To install REE on Snow Leopard, you should run the installer with --no-tcmalloc

Ruby Enterprise Edition | Comments (21)

Passenger Prefpane Update

By Ninh Bui September 2nd, 2009

Passenger Preference Pane provides a convenient GUI for OSX users to deploy their webapps with Phusion Passenger. However, some updates in Mac OS X Snow Leopard caused Passenger Preference Pane to stop working correctly. Thankfully, Alloy was quick to address these. Go check it out.

General | Comments (1)

Phusion Passenger 2.2.5 released

By Hongli Lai September 1st, 2009

Phusion Passenger is an Apache and Nginx module for deploying Ruby on Rails web applications, and is mainly focused on ease of use and stability.

Introducing the announcements mailing list

We now have an announcement-only mailing list for new Phusion Passenger releases. This mailing list can also be used for packagers to announce packaging updates. Future versions will no longer be announced on the discussion board, so discussion board subscribers who wish to be notified about new versions should subscribe to this announcements mailing list as well.

If you’re a packager and you’d like to have posting access to the announcements list, then please post a message on the discussion board.

Recent changes

Phusion Passenger is under constant maintenance and development. We are pleased to announce Phusion Passenger version 2.2.5. This is a bug fix release.

The following things have changed since 2.2.4:

[Apache] Small file uploads are now buffered; fixes potential DoS attack

Phusion Passenger buffers large file uploads to temp files so that it doesn’t block applications while an upload is in progress, but it sent small uploads directly to the application without buffering it. This could result in a potential DoS attack: the client can send many small, incomplete file uploads to the server, and this would block all application processes until a timeout occurs. In order to solve this problem, Phusion Passenger now buffers small file uploads in memory. Bug #356.

[Apache] Fixed support for mod_rewrite passthrough rules

Mod_rewrite passthrough rules were not properly supported because of a bug fix for supporting encoded slashes (%2f) in URLs. Unfortunately, due to bugs/limitations in Apache, we can support either encoded slashes or mod_rewrite passthrough rules, but not both; supporting one will break the other.

Support for mod_rewrite passthrough rules is now enabled by default; that is, support for encoded slashes is disabled by default. A new configuration option, PassengerAllowEncodedSlashes, has been added. Turning this option on will enable support for encoded slashes and disable support for mod_rewrite passthrough rules.

Issue #113 and issue #230.

[Apache] Added a configuration option for resolving symlinks in the document root path

Phusion Passenger 2.2.0 and higher no longer resolves symlinks in the document root path in order to properly support Capistrano-style directory structures. The exact behavior is documented in the Users Guide, section "How Phusion Passenger detects whether a virtual host is a web application".

However, some people relied on the old behavior. A new configuration option, PassengerResolveSymlinksInDocumentRoot, has been added to allow reverting back to the old behavior.

Patch contributed by Locaweb.

[Apache] mod_env variables are now also passed through CGI environment headers

Prior to version 2.2.3, environment variables set by mod_env are passed to the application as CGI environment headers, not through Ruby’s ENV variable. In the last release we introduced support for setting ENV environment variables with mod_env, and got rid of the code for setting CGI environment headers. It turns out that some people relied on the old behavior, we so now environment variables set with mod_env are set in both ENV and in the CGI environment.

Fixes bug #335.

[Apache] Fixed compilation problems on some Linux systems with older versions of Apache

If you used to see compilation errors like this:

ext/apache2/Configuration.cpp:554: error: expected primary-expression before '.' token

then this version should compile properly.

[Apache] Fixed I/O timeouts for communication with backend processes

Got rid of the code for enforcing I/O timeouts when reading from or writing to a backend process. This caused more problems than it solved.

[Nginx] Support for streaming responses (e.g. Comet or HTTP push)

Buffering of backend responses is now disabled. This fixes support for streaming responses, something which the Apache version has supported for a while now. One can generate streaming responses in Ruby on Rails like this:

render :text => lambda { |response, output|
    10_000.times do |i|
        output.write("hello #{i}!\n")
    end
}

[Nginx] Installer now installs Nginx 0.7.61 by default

Previously it installed 0.6.37 by default.

[Nginx] Fixed the installer’s –extra-configure-flags flag when combined with –auto-download

Arguments passed to –extra-configure-flags were not being passed to the Nginx configure script when –auto-download is given. This has been fixed: bug #349.

[Nginx] Fixed unnecessary download of PCRE

The installer now checks whether PCRE is installed in /opt/local (e.g. MacPorts) as well before concluding that it isn’t installed and going ahead with downloading PCRE.

Fixed STDERR capturing

While spawning an application, Phusion Passenger captures any output written to STDERR so that it can show them later if the application failed to start. This turns out to be much more difficult than expected, with all kinds of corner cases that can mess up this feature.

For example, if the Rails log file is not writable, then this can cause Rails to crash with a bizarre and unhelpful error message whenever it tries to write to STDERR:

/!\ FAILSAFE /!\  Thu Aug 20 14:58:39 +1000 2009
Status: 500 Internal Server Error
undefined method `[]' for nil:NilClass

Some applications reopen STDERR to a log file. This didn’t work.

Of all of these problems have been fixed now. (Bug #332)

Fixed some bugs in application sources preloading

Rails >= 2.2 already preloads the application sources, in which case Phusion Passenger wasn’t supposed to perform it’s own preloading, but the Rails >= 2.2 detection code was bugged. This has been fixed.

Rails < 2.2 doesn’t preload the application sources by itself, but there should be a certain order with which the sources are preloaded, otherwise preloading could fail in some applications. We now enforce a specific load order: first models, then controllers, then helpers.

Bug #359.

Fixed a few bugs in WSGI compliance

PATH_INFO is supposed to be set to the request URI, but without the query string and without the base URI. This has been fixed: bug #360.

Other things

Fixed some Ruby 1.9-specific crashes caused by encoding issues. Bug #354.

Fixed loading of config/environment.rb on Ruby 1.9.2, because Ruby 1.9.2 no longer has "." in the default load path. Patch by metaljastix, issue #368.

The Users Guide for Apache now mentions something about correct permissions for application directories.

Fixed compilation problems on IA-64 (bug #118). We also reduced the stack sizes for the threads by half, so Phusion Passenger should use even less virtual memory now.

Fixed compilation problems on Linux systems with ARM CPU.

Fixed a few compatibility problems with 64-bit OpenBSD.

Fixed a few typos and minor bugs.

Wait, what about Snow Leopard?

There are some known issues with Snow Leopard, though we do not yet know what these issues exactly are, what causes them or how to fix them. We haven’t upgraded yet because Snow Leopard is sold out in all local stores. Please check the discussion board for updates about Snow Leopard support.

How do I upgrade to 2.2.5?

Via a gem

Please install it with the following command:

gem install passenger

Next, run:

passenger-install-apache2-module

Or, if you’re an Nginx user:

passenger-install-nginx-module

Please don’t forget to copy & paste the Apache/Nginx config snippet that the installer gives you.

Via a native Linux package

John Leach from Brightbox has kindly provided an Ubuntu 8.04 package for Phusion Passenger. The package is available from the Brightbox repository which you can find at:

http://apt.brightbox.net

Add the following line to the Third Party Software Sources:

deb http://apt.brightbox.net hardy main

(The simplest way to do that is to create a file in /etc/apt/sources.list.d/ containing the deb instruction, and then run ‘apt-get update’).

Once you’ve done this then you can install Phusion Passenger by running:

sudo apt-get install libapache2-mod-passenger

-or-

sudo apt-get install nginx-brightbox

(Note that John is currently packaging 2.2.5, so it might take a while before this release shows up in the apt repository.)

Final

Phusion Passenger is provided to the community for free. If you like Phusion Passenger, please consider sending us a donation. Thank you!

Hongli Lai

Ninh Bui

Phusion Passenger | Comments (14)