Ruby Enterprise Edition 1.8.7-20090928 released

By Hongli Lai September 29th, 2009

In the not so distant past we launched a Ruby Enterprise Edition sponsorship campaign. We’ve been working hard on the code since that time. And although the campaign target hasn’t been reached, we’ve decided to release it anyway because we believe that this release is of great importance to the community.

Just how important is this release? Well, Twitter has beta tested this release and they’re reporting roughly 30% improvement in throughput. Keep reading to learn more and follow us on @phusion_nl for the latest updates.

Summary of changes

It’s now based on Ruby 1.8.7-p174

Previous REE releases were based on 1.8.6. We were initially reluctant about switching to 1.8.7: many people reported various incompatibility problems introduced by 1.8.7, and it seemed that alternative Ruby implementors were boycotting 1.8.7. However, the tide has now changed. Rails 3.0 will drop support for Ruby versions earlier than 1.8.7, JRuby had recently decided to support 1.8.7 and OS X Snow Leopard as well as all the Linux distros ship 1.8.7. We’ve received a lot of requests from the community for a 1.8.7-based version.

In short, everybody is moving towards 1.8.7, and we can’t afford to stay behind.

We were initially planning to release a final 1.8.6-based version, but after carefully considering the current state of things, we decided to skip the 1.8.6 release and jump directly to 1.8.7. 1.8.7 is also more compatible with 1.9, allowing a smoother transition to future Ruby versions.

The MBARI patches have been integrated
Brent Roman’s MBARI patch set uses a variety of techniques to significantly reduce Ruby’s memory usage. It also adds a bunch of performance optimizations, useful features and bug fixes.

  • Ruby’s garbage collector scans the C stack conservatively. This means that any data on the C stack that might look like a pointer pointing to a valid Ruby object will be treated like a real pointer, causing the object to be retained during garbage collection. Because of the way Ruby is written and because of compiler optimizations, there may be a lot of such “pointers” on the stack, even when they’re not really pointers but just garbage data that like look pointers.

    Brent has refactored and modified many parts of Ruby in order to reduce the possibility of such “garbage pointers” occurring on the stack. As a result, garbage collection will be much more effective. Many objects that would otherwise not have been freed will be freed thanks to the MBARI patches.

    As a side-effect of the reduced memory usage, applications typically run faster as well. Not only because applications are less likely to be swapped out, but also because modern hardware’s performance depends greatly on the amount of memory traffic.

  • Ruby uses an inefficient, linear time algorithm for extending the stack space when switching context between threads and/or continuations. This is mostly noticeable when you have a lot of threads. Brent replaces this with a constant time algorithm, which is much faster.
  • It adds the methods __file__ and __line__ to the Method, UnboundMethod and Proc classes. Very useful for inspecting large code bases that make heavy use of metaprogramming. These methods are documented in the Ruby Enterprise Edition manual.

We’ve ported his patch set to 1.8.7-p174. Especially Twitter benefited a lot from these patches.

The zero-copy context switching patch have been integrated as an optional feature
Joe Damato and Aman Gupta have created a patch which implements zero-copy context switching in Ruby’s threading system. In Ruby 1.8, whenever Ruby switches context to another thread, it has to copy over the entire thread stack. This is very expensive, and the deeper your function calls are nested, the slower it becomes. The zero-copy context switching patch removes the need to copy any stack data; instead, it keeps all thread stacks at different areas in memory and simply changes the stack pointer. In extreme situations, this patch can increase threading performance by as much as 10 times!

Phusion Passenger is not really affected by this patch because its request handling core is not multithreaded, at least not at this moment. But Mongrel is heavily multithreaded and will benefit tremendously from this patch.

We’ve made this patch compatible with the MBARI patch set, but there are currently some issues with 64-bit platforms, so we’ve disabled it by default. You can enable it by running the REE installer with --fast-threading

Various other minor changes
  • When compiling REE, binaries are optimized for size (-Os), not speed. See below for an explanation.
  • The tcmalloc memory allocator has been upgraded to version 1.3
  • RubyGems upgraded to version 1.3.5.
  • The Debian package now installs files to /usr/local, so that one doesn’t have to mess with $PATH to get things working.

In practice

This release has been beta tested with Twitter. Evan Weaver reported significant improvements in Twitter’s throughput after having switched to this REE release.

REE benchmark by Twitter

Twitter especially benefited from the MBARI patch set. We’ve found a few unexpected things during the beta testing phase:

  • We’ve found that optimizing Ruby for code size (compilation with -Os) yields better results that optimizing Ruby for speed (compiling with -O2). Ruby is a very unconventional program and we suspect that optimizing for code size allows more instructions to fit in the CPU instruction cache, resulting in better performance.
  • Microbenchmarks and synthetic benchmarks such as the Ruby benchmark suite do not show any performance improvements at all, although Twitter was able to measure a significant performance improvement. As always, benchmark results should be taken with a grain of salt.

Other major beta testers include 37signals and Novell. They’re still in the progress of testing, and we’re eager to receive their test results.

Sponsors

Special thanks go out to LVS for acting as a gold sponsor for this release:

LVS has been building and supplying enterprise platforms for the betting and gaming industry in South East Asia, Europe and the UK for more than 10 years. The team at LVS is a balanced mix of gaming industry specialists, technical experts and entrepreneurial leaders. With offices in London and Gibraltar we directly employ over 40 staff.

Our Advanced Betting Platform (a fully internationalized and localized high-availability platform based on Java and Ruby on Rails) is now regarded as the most technically advanced sports book system available in the World, and has been customised to fit the requirements of companies like Victor Chandler in Gibraltar and La Française des Jeux in France.

LVSLogo

We also thank the following people and organizations for sponsoring this release:

Download & upgrade

To install Ruby Enterprise Edition, please visit the download page. To upgrade from a previous version, simply install into the same prefix that you installed to last time. Please also refer to the documentation for upgrade instructions.

UPDATE: Snow Leopard support

It looks like tcmalloc has some issues on Snow Leopard, causing the compiler to crash. We’ve just been able to obtain Snow Leopard last weekend (it had been sold out for several weeks) and we haven’t gotten around installing it yet.

To install REE on Snow Leopard, you should run the installer with --no-tcmalloc

Ruby Enterprise Edition

Comments

  1. Peter Hellberg says:
    September 29, 2009 at 6:51 am

    This is great news guys! I’m particularly interested in whether or not this will improve the performance of our backend services. (REST/JSON-P) Not that it was any problems with the last release :)

    Thanks again.

    /P

  2. Michael Guterl says:
    September 29, 2009 at 7:34 am

    Is this supposed to work on Snow Leopard?

    It looks like the source is already configured.
    Skipping configure script…
    /usr/bin/gcc -dynamiclib system_allocator.c -install_name @rpath/libsystem_allocator.dylib -o libsystem_allocator.dylib
    mkdir -p .ext/common
    make PRELIBS=’-Wl,-rpath,/usr/local/ruby-ee/lib -L/usr/local/ruby-ee/usr/local/ruby-ee/lib -ltcmalloc_minimal -lsystem_allocator’
    make: *** [.rbconfig.time] Bus error

  3. Attila says:
    September 29, 2009 at 8:37 am

    @Michael: The problem source is that the tcmalloc garbage collector does not seem to be compatible with Snow Leopard as yet. You can try installing REE with –no-tcmalloc.

    If the problem still occurs than please see the relevant thread on the support mailing list:
    http://groups.google.com/group/emm-ruby/browse_thread/thread/ff79c2d140c26b42

  4. TwittLink - Your headlines on Twitter says:
    September 29, 2009 at 8:44 am

    [...] Ruby Enterprise Edition 1.8.7-20090928 released « Phusion Corporate Blog [...]

  5. MySchizoBuddy says:
    September 29, 2009 at 9:31 am

    Regarding good unicode support. what are your plans for Ruby 1.9.1
    Is it on the roadmap somewhere

  6. Ruby Enterprise Edition 1.8.7 Released – Lower Memory Usage, Increased Speed says:
    September 29, 2009 at 3:07 pm

    [...] to subscribe to the RSS feed for updates on this topic.Powered by WP Greet BoxToday Phusion has announced the release of Ruby Enterprise Edition (REE) 1.8.7 (more specifically, 1.8.7-20090928). Once [...]

  7. Ruby Enterprise Edition 1.8.7 Released – Lower Memory Usage, Increased Speed | Webs Developer says:
    September 29, 2009 at 5:01 pm

    [...] Phusion has announced the release of Ruby Enterprise Edition (REE) 1.8.7 (more specifically, 1.8.7-20090928). Once [...]

  8. Edvin Aghanian says:
    September 29, 2009 at 5:53 pm

    Awesome work guys. Thank you for this valuable contribution to the community.

  9. Ryan Sobol says:
    September 30, 2009 at 3:52 am

    I think dropping ruby 1.8.6 from receiving future enhancements is a wise strategy. Keep up for great work, Phusion team!

  10. Double Shot #551 « A Fresh Cup says:
    September 30, 2009 at 5:58 am

    [...] Ruby Enterprise Edition 1.8.7-20090928 released – And apparently with a good speed boost. [...]

  11. Ruby on Rails : Ruby Enterprise Edition 1.8.7-20090928 erschienen says:
    September 30, 2009 at 8:36 am

    [...] Mehr Details zu diesem Release kann im Changelog nachgelesen werden: Link zum Changelog [...]

  12. leopard says:
    September 30, 2009 at 9:45 am

    Thanks, guys. But I have one problem in Ubuntu Linux. Your deb not put in opt dir filse with ruby, gem, ets. Why?
    …
    usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/doc/rational.rdoc
    usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/doc/command_line_usage.rdoc
    usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/doc/rake.1.gz
    usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/doc/proto_rake.rdoc
    usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/doc/example/
    usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/doc/example/b.c
    usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/doc/example/Rakefile1
    usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/doc/example/a.c
    usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/doc/example/Rakefile2
    usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/doc/example/main.c
    usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/doc/glossary.rdoc
    usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/doc/rakefile.rdoc
    usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/install.rb
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/.require_paths
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/lib/
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/lib/mysql_api.so
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/lib/mysql.rb
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/README.txt
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/Rakefile
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/test/
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/test/test_mysql.rb
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/ext/
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/ext/mysql_api/
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/ext/mysql_api/mysql.o
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/ext/mysql_api/extconf.rb
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/ext/mysql_api/error_const.h
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/ext/mysql_api/mysql_api.so
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/ext/mysql_api/mysql.c
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/ext/mysql_api/mkmf.log
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/ext/mysql_api/Makefile
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/tasks/
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/tasks/gem.rake
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/tasks/vendor_mysql.rake
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/tasks/native.rake
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/History.txt
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/Manifest.txt
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/COPYING
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/extra/
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/extra/README.html
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/extra/README_ja.html
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/extra/tommy.css
    usr/local/lib/ruby/gems/1.8/gems/mysql-2.8.1/COPYING.ja
    usr/local/lib/ruby/gems/1.8/doc/
    usr/local/lib/ruby/vendor_ruby/
    usr/local/lib/ruby/vendor_ruby/1.8/
    usr/local/lib/ruby/vendor_ruby/1.8/i686-linux/
    usr/local/bin/
    usr/local/bin/gem
    usr/local/bin/passenger-stress-test
    usr/local/bin/ree-version
    usr/local/bin/passenger-install-nginx-module
    usr/local/bin/rackup
    usr/local/bin/passenger-make-enterprisey
    usr/local/bin/passenger-install-apache2-module
    usr/local/bin/passenger-memory-stats
    usr/local/bin/irb
    usr/local/bin/erb
    usr/local/bin/passenger-status
    usr/local/bin/rdoc
    usr/local/bin/passenger-spawn-server
    usr/local/bin/passenger-config
    usr/local/bin/rake
    usr/local/bin/testrb
    usr/local/bin/ri
    usr/local/bin/ruby
    usr/local/lib/libtcmalloc_minimal.so.0
    usr/local/lib/libtcmalloc_minimal.so
    …

    You change system ruby?

  13. Brian Landers says:
    September 30, 2009 at 11:41 am

    Something is broken with the new release on OpenSolaris (Joyent’s accelerators)

    ./configure –prefix=/opt/ree –enable-mbari-api CFLAGS=’-g -Os -fno-stack-protector ‘
    checking build system type… i386-pc-solaris2.11
    checking host system type… i386-pc-solaris2.11
    checking target system type… i386-pc-solaris2.11
    checking for gcc… gcc
    checking for C compiler default output file name…
    configure: error: in `/root/ruby-enterprise-1.8.7-20090928/source’:
    configure: error: C compiler cannot create executables
    See `config.log’ for more details.

    Full config.log is at http://www.packetslave.com/tmp/config.log

  14. Lançado o Ruby Enterprise Edition 1.8.7 - Consome menos memória e é mais rápido says:
    October 1, 2009 at 8:02 am

    [...] alguns dias a Phusion anunciou o lançamento do Ruby Enterprise Edition (REE) 1.8.7 (mais especificamente, 1.8.7-20090928). Embora [...]

  15. Chris Williams says:
    October 1, 2009 at 8:16 am

    Where should we file bugs for this specific version. I am getting a segfault in hpricot/tag.rb with a register state dump. Using hpricot for sanitization, point me to the preferred ticketing platform and will expound further there. Thanks for the great job on everything!

  16. Andrea says:
    October 2, 2009 at 4:53 am

    Thanks for your great work guys! Aniway i think is now time to drop even 1.8.7 in favor of 1.9 :)

  17. TheR says:
    October 5, 2009 at 3:05 pm

    Regarding optimization for size. I think I read somewhere an article where someone has profiled that Rails is using String object 80% of time.

    My idea was if code could be optimized in a way that String methods would occupy single part of memory and therefore be kept all the time in processors cache.

    Could ruby be compiled (optimized) in a way that often called parts would be kept in processors cache most of the time and to minimize expensive calls to system RAM.

    by
    TheR

  18. Mark says:
    October 9, 2009 at 8:51 am

    I had to trash the source dir and recreate after getting the Bus error on SL. After this the –no-tcmalloc flag let me build it.

    There should be an option to “make clean”.

  19. Now with all new REE + Phusion. at Booko’s Blogo says:
    October 11, 2009 at 6:41 am

    [...] excellent people at Phusion have released a 1.8.7 based version of their fantastic Ruby Enterprise Edition. I’ve just updated to it and Booko sure feels [...]

  20. Ruby 发布 Ruby 1.8.7 « 燕之庐黑板报 says:
    October 16, 2009 at 1:07 am

    [...] Phusion发布了新的Ruby Enterprise Edition，版本号为1.8.7-20090928。之前的版本都基于Ruby 1.8.6，该版本开始基于Ruby 1.8.7。Ruby 1.8.7发布已经有段时间了，为什么现在才切换到1.8.7呢？Phusion在发布声明中解释了这个问题： 我 们起先并不愿意切换到1.8.7：很多人报告说在引入1.8.7后遇到了各种不兼容的情况，而且其他几个Ruby实现都抵制 1.8.7。然而，现在的情况不同了。Rails 3.0将不再支持Ruby 1.8.7之前的版本，JRuby最近也决定支持1.8.7，并且OS X Snow Leopard和所有Linux发行版都自带了1.8.7。我们已经收到了来自社区的很多请求，希望能有一个基于1.8.7的版本。 [...]

  21. tvw says:
    October 20, 2009 at 7:26 am

    Hi,

    I would sponsor too, if I could do it without registering for paypal. I am not using paypal at all and therefore I do not want to have an account I need to pay attention to.

    Anyway: really good work, thanks.

    TVW

Leave a comment