Phusion white papers Phusion overview

Ruby Enterprise Edition 1.8.6-20081205 released, thank you sponsors

By Hongli Lai on December 5th, 2008

Ruby Enterprise Edition (REE) is a branch of the official Ruby interpreter which is capable of reducing your Rails applications’ memory usage by 33% on average, as well as improving your applications’ performance. This is possible because REE includes copy-on-write enhancements for the garbage collector, as well as an improved memory allocator (tcmalloc). REE has been out for several months now and is already used by many high-profile websites and organizations, such as New York Times, Shopify and 37signals.

“We switched to enterprise ruby to get the full benefit of the [copy-on-write] memory characteristics and we can absolutely confirm the memory savings of 30% some others have reported. This is many thousand dollars of savings even at today’s hardware prices.” – Tobias Lütke (Shopify)

And just like Phusion Passenger, Ruby Enterprise Edition is 100% open source.

Recent developments

REE has just become better. We had been talking with DHH from 37signals about a possible sponsorship campaign for supporting REE development. The campaign has recently ended, and so we’re presenting the world with Ruby Enterprise Edition version 1.8.6-20081205.

The sponsored improvements are:

Integration with the RailsBench garbage collector patches
These patches allow one to tweak various garbage collector settings. 37signals uses this to greatly improve server performance. The RailsBench GC patches did not work on REE in previous releases, but now we’ve properly integrated them into REE.
Better Mac OS X support
Previous versions sort-of support Mac OS X, but some people have reported problems. We’ve improved Mac OS X support as follows:

  • Installation should now work out-of-the-box on Mac OS X.
  • The tcmalloc memory allocator has been properly integrated with Mac OS X. Tcmalloc is what makes REE faster than the standard Ruby interpreter. In previous versions, tcmalloc was disabled when installing on OS X because it didn’t work properly on OS X. This not only meant that there were no performance gains compared to standard Ruby, it also meant that the copy-on-write optimizations were a bit less effective than they could be. By properly integrating tcmalloc on OS X, which was a tough task, OS X users can now fully enjoy the improved performance and copy-on-write optimizations.
  • Various OS X-related crashes have been fixed.

Special thanks to Stephen Heuer from Arux Software Inc. for providing access to an OS X testing environment.

Better 64-bit support
Even though previous versions support 64-bit platforms, many people experienced problems. We’ve spent some time making sure that installation works out-of-the-box on 64-bit platforms. We’ve tested against 64-bit FreeBSD 7 and 64-bit Ubuntu 8.10 Server.
Better Solaris support
We’ve made sure that installation works out-of-the-box.
Special thanks goes out to our friends at Sun for providing us with access to a Solaris testing environment.

In addition, this release also comes with various non-sponsored improvements and changes:

caller_for_all_threads
Philippe Hanrigou’s caller_for_all_threads patches have been integrated into REE. This feature allows one to dump the stack trace of all running threads in a Ruby application. This feature is a must-have for debugging multithreaded web applications.

Phusion Passenger’s development version has already been updated to take advantage of this feature, when available. It will be available starting from Phusion Passenger version 2.1.0 (which hasn’t been released yet at the moment of writing).

RubyGems updated to version 1.3.1.
The previous REE release ships with RubyGems 1.2. Version 1.3.1 is now included. FYI: Ruby on Rails 2.2 requires RubyGems 1.3.1.
REE’s RubyGems no longer makes use of the existing gems
One of the most often requested features is that REE’s RubyGems should be able to use the gems that are already installed (i.e. the gems that have been installed for the system’s Ruby interpreter). The previous REE release implemented this feature.

However, it turns out that this can cause problems with native extensions, e.g. RMagick and the Ruby MySQL library. If one uses REE to load a native extension that was originally compiled for the system’s Ruby, then REE might crash. So this feature has been removed. You should install all gems that you need with REE’s RubyGems.

ree-version
An ‘ree-version’ command is now provided. This command prints out the REE version number. This makes it easier for people to track REE updates.

The sponsors

Many thanks to the following people and organizations for sponsoring this release (list sorted in alphabetical order):

Officially tested platforms

We’ve tested this release of Ruby Enterprise Edition on the following platforms:

  • Ubuntu Linux 8.04 Desktop (x86)
  • Ubuntu Linux 8.10 Server (x86_64)
  • Mac OS X 10.5.5 Server (x86)
  • FreeBSD 5.0 (x86)
  • FreeBSD 7.0 (x86_64)
  • Sun Solaris (Sparc)

Download & upgrade

To install Ruby Enterprise Edition, please visit the download page. To upgrade from a previous version, simply install into the same prefix that you installed to last time.

  • http://www.phusion.nl/ hongli

    We had a MySQL problem earlier so we lost some comments. But to answer the people who asked why they get a 404 Not Found error on the download page: it’s because RubyForge is still updating their mirrors.

  • http://hungrymachine.com Aaron Batalion

    You mention better 64-bit support. does that mean the memory allocator now works on 64-bit platforms?

    http://www.rubyenterpriseedition.com/faq.html#thirty_three_percent_mem_reduction
    “this change the FAQ answer of “this memory allocator does not work on 64-bit platforms. As a result, on 64-bit platforms, Ruby Enterprise Edition is slightly slower than standard Ruby (by a few percent). “

  • http://www.phusion.nl/ hongli

    No, tcmalloc doesn’t work on 64-bit. But the copy-on-write patches still do, they just aren’t as effective without tcmalloc. So on 32-bit systems (with tcmalloc) you get about 33% memory savings, while on 64-bit systems (without tcmalloc) you get about 25% memory savings.

  • Keith

    Irony: you posted this literally 20 minutes after I finished installing passenger and the previous ruby-enterprise. Guess I should have taken the time to drink my coffee this morning after all :) Thanks for your efforts!

  • http://hungrymachine.com Aaron Batalion

    hongli, 25% is great…. but thats memory savings.. but the FAQ mentions performance… “slower”. Can you elaborate? Is that still the case? Beyond tcmalloc, what makes the 64bit slower?

  • http://www.phusion.nl/ hongli

    Yes. The copy-on-write patch makes the GC about 5% slower, but this is negated by the fact that tcmalloc makes everything about 25% faster. So in summary:

    On 32-bit:
    – 5% slowdown because of GC patches
    – 25% speedup because of tcmalloc
    – Net result: 20% speedup.

    On 64-bit:
    – 5% slowdown because of GC patches
    – No tcmalloc available
    – Net result: 5% slowdown.

  • Pingback: Apache + Passenger -vs- nginx + mongrel « Tom Mornini()

  • http://smartic.us bryanl

    This sounds like you are saying stay away if you using 64 bit linux.

  • http://joergbattermann.com Joerg Battermann

    Awesome.. Thanks!

    Quick Q: are there 64bit ubuntu/deb packages planned somewhere in the future?

  • Pingback: How REE and GC tuning reduced spec suite runtime to one third « Enjoying Rails()

  • http://www.phusion.nl/ hongli

    @bryanl: Well no, it just means that the tradeoff is different. If you’re on 64-bit, and you value memory usage more than raw CPU usage, then REE is the way to go.

  • http://www.akitaonrails.com AkitaOnRails

    Awesome release, I am installing it right now :-) And actually – I may be wrong about this – but it seems that Apache performance is worse on 64-bits. It seems related to the Linux kernel’s scheduler. For highly demanded web servers, it is better to use Linux 32-bits and if you need more RAM, use just PAE. Apache + Linux will perform better in 32-bit settings.

  • http://blog.viarails.net Wesley Moxam

    If you value memory consumption in general, you won’t use 64-bit linux for your app servers anyways. Ruby consumes much more memory on 64-bit platforms, and there isn’t any advantage to running on 64-bit.

  • http://jonraptor.com Jonathan

    Hi, ruby enterprise edition does not compile on Ubuntu Jaunty, using g++-4.3 or g++-4.2. Any info on that?

  • Pingback: Joakim Andersson » Blog Archive » Ruby Enterprise Edition 1.8.6-20081205 released()

  • http://betterlogic.com/roger roger

    I’d suggest submitting the ‘gc tweakability’ patches upstream–other projects would surely benefit, and I’d +1 them :)
    Or maybe run time tweakable would be better…hmm.

  • http://www.phusion.nl/ hongli

    @roger: Those patches had in fact been submitted upstream, but the Ruby core team didn’t accept them.

  • Gregg

    I can confirm that Ruby-Enterprise & Passenger works well on:

    CentOS 5.2 x86

  • Gregg

    But it does not work on

    CentOS 5.2 x86_64 *exits after make during installation*

    I’m surprised that there are no Red Hat flavored distros on that test list

  • Pingback: Ruby Enterprise Edition on Ubuntu 8.10 (x86_64) - sethbc’s (infrequent) blog…()

  • http://betterlogic.com/roger roger

    Have the all thread backtraces been posted upstream? That might be nice. Also do you have a link to the GC patches’ upstream post?
    Thanks!
    -=R

  • ste

    At line #73 in lib/passenger/spawn_manager.rb:

    # Commonly used libraries.
    ['mysql', 'sqlite3'].each do |lib|
    require lib
    end

    WTF????
    Can you explain the presence of this piece of code? I had to manually remove it since I use Postgres and I couldn’t install
    the Mysql gem!

    Also:

    $ sudo passenger-memory-stats
    /opt/ruby-enterprise-1.8.6-20081205/lib/ruby/gems/1.8/gems/passenger-2.0.5/bin/passenger-memory-stats:150:in `list_processes': Invalid options. (ArgumentError)
    from /opt/ruby-enterprise-1.8.6-20081205/lib/ruby/gems/1.8/gems/passenger-2.0.5/bin/passenger-memory-stats:112:in `start'
    from /opt/ruby-enterprise-1.8.6-20081205/lib/ruby/gems/1.8/gems/passenger-2.0.5/bin/passenger-memory-stats:236
    from /opt/ruby-ee/bin/passenger-memory-stats:19:in `load'
    from /opt/ruby-ee/bin/passenger-memory-stats:19

    $ uname -a
    Linux **** 2.6.18-xen #1 SMP Tue Feb 12 06:40:50 UTC 2008 x86_64 x86_64 x86_64 GNU/Linux

  • http://www.phusion.nl/ hongli

    ste: Please calm down and stay polite, there’s no reason to get upset.

    The code you pointed out is meant for improving copy-on-write efficiency, as the comment a few lines above your code piece explains:

    # Preload libraries for copy-on-write semantics.

    It doesn’t take into account that the library may be missing, but this has already been fixed.

  • ste

    Sorry for sounding impolite (and I assure you that I’m *always* calm :-), but this thing really caught me by surprise. Knowing the quality of your work, I find this “bug” really surprising, in a VERY bad way.

  • RyanR>

    Per the instructions in “REE’s RubyGems no longer makes use of the existing gems”, I installed the MySQL gem with:

    /opt/ruby-enterprise-1.8.6-20081205/bin/ruby /opt/ruby-enterprise-1.8.6-20081205/bin/gem install mysql — –with-mysql-dir=/usr/local/mysql

    But I am getting the following error in /var/log/apache2/error_log:

    [Tue Dec 09 22:14:28 2008] [notice] Apache/2.2.9 (Unix) mod_ssl/2.2.9 OpenSSL/0.9.7l DAV/2 Phusion_Passenger/2.0.5 configured — resuming normal operations
    dyld: NSLinkModule() error
    dyld: Library not loaded: /usr/local/mysql/lib/mysql/libmysqlclient.15.dylib
    Referenced from: /opt/ruby-enterprise-1.8.6-20081205/lib/ruby/gems/1.8/gems/mysql-2.7/lib/mysql.bundle
    Reason: image not found

    I am running Leopard 10.5.5 and a local build of MySQL 5.0.45.

    Thanks for your help. This is the first time I have tried to get this running.

  • Pingback: Ruby Enterprise Edition versão 1.8.6-20081205 | Ruby Brasil()

  • RyanR>

    Update to previous comment: I tried correcting the reference in mysql.bundle to reflect the correct location of the mysqlclient.15.dylib file, and I got seg faults instead.

  • http://betterlogic.com/roger roger

    @ryanr — could try using it –with_mysql_config or the like
    GL!
    -=R

  • http://betterlogic.com/roger roger

    Wondering–which part(s) of the gcpatch were applied?
    Thanks!
    -=R

  • Just Wondering

    I’m just wondering, why is it assumed by REE developers that reinstalling all Ruby Gems on a system is ok? I have over 50 gems used across 4 Rails applications, and this needs to be done on 5 servers.

    I was hoping to use GEM_PATH to continue using the already-installed gems.

  • http://www.phusion.nl/ hongli

    @Just Wondering: Because it is not possible to use your existing gems reliably. We tried this in the past and it turns out that on some platforms, native extensions compiled against the existing Ruby installation can cause REE to crash, and the only way to prevent this is by reinstalling said gems. If there is a silver bullet we would have used it a long time ago.

  • http://betterlogic.com/roger roger

    still wondering…Wondering–which part(s) of the gcpatch were applied?

    Thanks!
    -=R

  • http://www.phusion.nl/ hongli

    @roger: Everything.