Phusion white papers Phusion overview

Phusion Passenger 4.0.41 released, OpenSSL Heartbleed security update

By Hongli Lai on April 8th, 2014


Phusion Passenger is a fast and robust web server and application server for Ruby, Python, Node.js and Meteor. Passenger takes a lot of complexity out of deploying web apps, and adds powerful enterprise-grade features that are useful in production. High-profile companies such as Apple, New York Times, AirBnB, Juniper, American Express, etc are already using it, as well as over 350.000 websites.

Phusion Passenger 4.0.41 has been released ahead of time in order to address the OpenSSL heartbleed security issue (CVE-2014-0160). This is an extremely serious vulnerability in OpenSSL which can completely negate the security that it provides. Users are advised to upgrade as soon as possible.

Phusion Passenger’s relationship with the OpenSSL heartbleed vulnerability is as follows.

We provide precompiled binaries for Passenger Standalone. These binaries are statically linked to OpenSSL in order to make them useable on a wide range of operating systems. With 4.0.41, the binaries have been upgraded to link against OpenSSL 1.0.1g, which fixes the heartbleed vulnerability.

You are vulnerable if:

  • You are using Passenger Standalone, with SSL enabled inside Passenger Standalone (that is, passenger start --ssl).

You are not vulnerable (to the Passenger Standalone static linking issue) if:

  • You are not using Passenger Standalone (e.g. if you’re using Phusion Passenger through the Apache or Nginx integration mode).
  • You are using Passenger Standalone, but without SSL.
  • Your Passenger Standalone is behind another SSL-enabled reverse proxy.

Update: Please note that the only thing this Phusion Passenger update fixes, is any potential vulnerabilities in the Passenger Standalone binaries that we provide. Your system as a whole may still be vulnerable because you’re running a vulnerable OpenSSL version. Please check with your vendor for system updates.

There aren’t many other changes in this release:

  • Fixed some issues with printing UTF-8 log files on Heroku.
  • Added a new flag --ignore-app-not-running to passenger-config restart-app.
    When this flag is given, passenger-config restart-app will exit successfully
    when the specified application is not running, instead of exiting with
    an error.

Installing or upgrading to 4.0.41

OS X OS X Debian Debian Ubuntu Ubuntu
Heroku Heroku Ruby gem Ruby gem Tarball Tarball

Final

Phusion Passenger’s core is open source. Please fork or watch us on Github. :)

Phusion Passenger 4.0.40 released, Nginx 1.4.7 with buffer overflow fix

By Hongli Lai on March 19th, 2014

Phusion Passenger 4.0.40 has been released. The only change in this version is that the preferred Nginx version has been bumped to 1.4.7, because of a buffer overflow exploit in Nginx (CVE-2014-0133). Nginx users are strongly encouraged to upgrade.

Phusion Passenger 4.0.39 released

By Hongli Lai on March 18th, 2014


Phusion Passenger is a fast and robust web server and application server for Ruby, Python, Node.js and Meteor. Passenger takes a lot of complexity out of deploying web apps, and adds powerful enterprise-grade features that are useful in production. High-profile companies such as Apple, New York Times, AirBnB, Juniper, American Express, etc are already using it, as well as over 350.000 websites.

Phusion Passenger is under constant maintenance and development. Version 4.0.39 is a bugfix release.

Phusion Passenger also has an Enterprise version which comes with a wide array of additional features. By buying Phusion Passenger Enterprise you will directly sponsor the development of the open source version.

Recent changes

  • Fixed a crash that could happen if the client disconnects while a chunked response is being sent. Fixes issue #1062.
  • In Phusion Passenger Standalone, it is now possible to customize the Nginx configuration file on Heroku. It is now also possible to permanently apply changes to the Nginx configuration file, surviving upgrades. Please refer to the "Advanced configuration" section of the Phusion Passenger Standalone manual for more information.
  • The programming language selection menu in passenger-install-apache2-module and passenger-install-nginx-module only works on terminals that support UTF-8 and that have a UTF-8 capable font. To cater to users who cannot meet these requirements (e.g. PuTTY users using any of the default Windows fonts), it is now possible to switch the menu to a plain text mode by pressing ‘!’. Fixes issue #1066.
  • Fixed printing UTF-8 characters in log files in Phusion Passenger Standalone.
  • It is now possible to dump live backtraces of Python apps through the ‘SIGABRT’ signal.
  • Fixed closing of file descriptors on OS X 10.9.
  • Fixed compilation of native_support on Rubinius.

Installing or upgrading to 4.0.39

OS X OS X Debian Debian Ubuntu Ubuntu
Heroku Heroku Ruby gem Ruby gem Tarball Tarball

Final

Phusion Passenger’s core is open source. Please fork or watch us on Github. :)

If you would like to stay up to date with Phusion news, please fill in your name and email address below and sign up for our newsletter. We won’t spam you, we promise.



Phusion Passenger 4.0.38 released

By Hongli Lai on March 10th, 2014


Phusion Passenger is a fast and robust web server and application server for Ruby, Python, Node.js and Meteor. Passenger takes a lot of complexity out of deploying web apps, and adds powerful enterprise-grade features that are useful in production. High-profile companies such as Apple, New York Times, AirBnB, Juniper, American Express, etc are already using it, as well as over 350.000 websites.

Phusion Passenger is under constant maintenance and development. Version 4.0.38 is a bugfix release.

Phusion Passenger also has an Enterprise version which comes with a wide array of additional features. By buying Phusion Passenger Enterprise you will directly sponsor the development of the open source version.

Recent changes

  • Fixed a symlink-related security vulnerability.

    Urgency: low
    Scope: local exploit
    Summary: writing files to arbitrary directory by hijacking temp directories
    Affected versions: 4.0.37
    Fixed versions: 4.0.38
    CVE-2014-1832

    Description: This issue is related to CVE-2014-1831 (the security issue as mentioned in the 4.0.37 release notes). The previous fix was incomplete, and still has a (albeit smaller) small attack time window in between two filesystem checks. This attack window is now gone.

  • Added support for the new Ruby 2.1.0 out-of-band garbage collector. This can much improve garbage collection performance, and drastically reduce request times.
  • Passenger Standalone is now compatible with IPv6.
  • Fixed some compilation problems on Solaris. See issue #1047.
  • passenger-install-apache2-module and passenger-install-nginx-module now automatically run in `–auto` mode if stdin is not a TTY. Fixes issue #1030.
  • Fixed an issue with non-bundled Meteor apps not correctly running in production mode.
  • The `PassengerPreStart` option is now compatible with IPv6 server sockets.
  • When running Python WSGI apps, `wsgi.run_once` is now set to False. This should improve the performance of certain apps and frameworks.
  • When handling HTTP requests with chunked transfer encoding, the ‘Transfer-Encoding’ header is no longer passed to the application. This is because the web server already buffers and dechunks the request body.
  • Fixed a possible hang in Phusion Passenger for Nginx when Nginx is instructed to reload or reopen log files. Thanks to Feng Gu, pull request #97.
  • The preferred Nginx version has been upgraded to 1.4.6.
  • Fixed a problem with running passenger-install-apache2-module and passenger-install-nginx-module on JRuby. They were not able to accept any terminal input after displaying the programming language menu.

Installing or upgrading to 4.0.38

OS X OS X Debian Debian Ubuntu Ubuntu
Heroku Heroku Ruby gem Ruby gem Tarball Tarball

Final

Phusion Passenger’s core is open source. Please fork or watch us on Github. :)

If you would like to stay up to date with Phusion news, please fill in your name and email address below and sign up for our newsletter. We won’t spam you, we promise.



Ruby Rogues 143: Phusion Passenger Enterprise with Hongli Lai and Tinco Andringa

By Hongli Lai on February 12th, 2014

We’ve been invited by Ruby Rogues to participate in a podcast about Phusion Passenger Enterprise. This podcast covers the following topics:

  • Hongli Lai and Tinco Andringa Introductions
  • Phusion Passenger introduction
  • Rack
  • Node.js, MeteorJS, Python Support
  • Processes and Threads
    • Ruby Rogues Episode #58 – Book Club: Working with Unix Processes with Jesse Storimer
    • Ruby Enterprise Edition
    • Smart Spawning
  • Advantages of Phusion Passenger Enterprise
    • Rolling Restarts
    • Mass Deployment
  • Passenger vs Unicorn
  • Error Resistant Deploys
  • Hosting
    • DreamHost
  • Apache, Nginx support
  • Stability Issues
  • Documentation and Support

Listen to the podcast at the Ruby Rogues website

Thanks Ruby Rogues for hosting us!

Phusion Passenger now supports the new Ruby 2.1 Out-Of-Band GC

By Hongli Lai on January 31st, 2014


Phusion Passenger is a fast and robust web server and application server for Ruby, Python, Node.js and Meteor. Passenger takes a lot of complexity out of deploying web apps, and adds powerful enterprise-grade features that are useful in production. High-profile companies such as Apple, New York Times, AirBnB, Juniper, American Express, etc are already using it, as well as over 350.000 websites.

Yesterday, Aman Gupta announced an Out-Of-Band garbage collector for Ruby 2.1. What is an “Out-Of-Band garbage collector”? It means that the garbage collector is run in between requests, instead of during a request. While the garbage collector is running, other processes can serve requests. This way visitors will never experience the slight delay that is caused by the garbage collector, much improving response times. We just implemented support for this in Phusion Passenger.

Classical out-of-band garbage collection

Out-of-band garbage collection is not a new idea. The idea was first introduced in Unicorn several years ago. Its mechanism was simple: disable automatic garbage collection, and manually run the garbage collector at the end of every N requests.

Last year, we introduced a much improved version of Unicorn’s out-of-band garbage collection mechanism in Phusion Passenger 4. This feature was originally contributed by AppFolio, and since then many further improvements have been made.

  • Unicorn’s out-of-band garbage collection doesn’t work well with multithreaded servers, because a garbage collection is a stop-the-world action that blocks all threads. Phusion Passenger’s mechanism lifts this limit by intelligently coordinating the garbage collector and threads.
  • Unicorn’s mechanism is susceptible to occasional pathological behavior. It is possible for many, or all, Unicorn processes to be simultaneously running the out-of-band garbage collector. During that short period of time, no requests can be served, and visitors will experience a delay. Phusion Passenger’s mechanism ensures that only 1 process can run the garbage collector at a time, thereby eliminating this problem.
  • Phusion Passenger’s mechanism is more flexible, and can be used for work other than just garbage collection.

The results were impressive. AppFolio observed an average response time reduction of 100 ms.

Before Out-Of-Band GC After Out-Of-Band GC
Before and after applying out of band GC at AppFolio

Unfortunately, this Out-Of-Band GC mechanism (running the GC every few requests) is still suboptimal, as we had pointed out last year:

  • If a request generates a lot of garbage, then delaying the GC for a few more request can result in high peak memory usage.
  • If the garbage collector doesn’t actually need to be run yet, then running it anyway wastes CPU cycles.

The new Ruby 2.1 Out-Of-Band GC

Fortunately, Ruby has recently introduced many useful garbage collector improvements. Ruby 2.1 introduced a semi-generational garbage collector. Aman Gupta built an Out-Of-Band GC mechanism on top of the recent improvements, which is almost exactly what we needed in Phusion Passenger.

Today, we’ve submitted a pull request to make his Out-Of-Band GC mechanism work well with Phusion Passenger, and we’ve introduced support in Phusion Passenger for his Out-Of-Band GC mechanism. Using it is very easy. First, add our patched gctools to your Gemfile:

gem "gctools", :github => "FooBarWidget/gctools", :require => false

Next, put this in your config.ru:

require "gctools/oobgc"
GC::OOB.setup

Now you can enable the Out-Of-Band GC feature in Phusion Passenger by using the :gctools_oobgc strategy:

PhusionPassenger.require_passenger_lib 'rack/out_of_band_gc'
use PhusionPassenger::Rack::OutOfBandGc, :strategy => :gctools_oobgc

Aman Gupta’s improvements are very impressive indeed. With Ruby 2.1, his average OOBGC pause time (oobgc.mean) “went from 125ms to 50ms thanks to RGenGC”. And “the number of out-of-band collections (oobgc.count) also went down, since the new OOBGC only runs when necessary”.

From Aman Gupta’s blog: “The overall result is much less CPU time (oobgc.sum) spent doing GC work between requests”:

Conclusion

Support for Aman Gupta’s Ruby 2.1 Out-Of-Band GC mechanism will be included in the next release of Phusion Passenger. It’s not completely finished yet: as seen in our pull request, there are still some issues to be fleshed out. But the results are very promising indeed. Although the Ruby hype is over, there are still plenty of improvements going on.

Discuss this on Hacker News.

Like our articles? Consider subscribing to our newsletter. You can unsubscribe any time.



Phusion Passenger 4.0.37 released

By Hongli Lai on January 29th, 2014


Phusion Passenger is a fast and robust web server and application server for Ruby, Python, Node.js and Meteor. Passenger takes a lot of complexity out of deploying web apps, and adds powerful enterprise-grade features that are useful in production. High-profile companies such as Apple, New York Times, AirBnB, Juniper, American Express, etc are already using it, as well as over 350.000 websites.

Phusion Passenger is under constant maintenance and development. Version 4.0.37 is a bugfix release.

Phusion Passenger also has an Enterprise version which comes with a wide array of additional features. By buying Phusion Passenger Enterprise you will directly sponsor the development of the open source version.

Recent changes

  • Improved Node.js compatibility. Calling on() on the request object now returns the request object itself. This fixes some issues with Express, Connect and Formidable. Furthermore, some WebSocket-related issues have been fixed.
  • Improved Meteor support. Meteor application processes are now shut down quicker. Previously, they linger around for 5 seconds while waiting for all connections to terminate, but that didn’t work well because WebSocket connections were kept open indefinitely. Also, some WebSocket-related issues have been fixed.
  • Introduced a new tool `passenger-config detach-process` for gracefully detaching an application process from the process pool. Has a similar effect to killing the application process directly with `kill <PID>`, but killing directly may cause the HTTP client to see an error, while using this command guarantees that clients see no errors.
  • Fixed a crash that occurs when an application fails to spawn, but the HTTP client disconnects before the error page is generated. Fixes issue #1028.
  • Fixed a symlink-related security vulnerability.

    Urgency: low
    Scope: local exploit
    Summary: writing files to arbitrary directory by hijacking temp directories
    Affected versions: 4.0.5 and later
    Fixed versions: 4.0.37
    CVE-2014-1831

    Description: Phusion Passenger creates a "server instance directory" in /tmp during startup, which is a temporary directory that Phusion Passenger uses to store working files. This directory is deleted after Phusion Passenger exits. For various technical reasons, this directory must have a semi-predictable filename. If a local attacker can predict this filename, and precreates a symlink with the same filename that points to an arbitrary directory with mode 755, owner root and group root, then the attacker will succeed in making Phusion Passenger write files and create subdirectories inside that target directory. The following files/subdirectories are created:

    • control_process.pid
    • generation-X, where X is a number.

    If you happen to have a file inside the target directory called `control_process.pid`, then that file’s contents are overwritten. These files and directories are deleted during Phusion Passenger exit. The target directory itself is not deleted, nor are any other contents inside the target directory, although the symlink is.

    Thanks go to Jakub Wilk for discovering this issue.

Installing or upgrading to 4.0.37

OS X OS X Debian Debian Ubuntu Ubuntu
Heroku Heroku Ruby gem Ruby gem Tarball Tarball

Final

Phusion Passenger’s core is open source. Please fork or watch us on Github. :)

If you would like to stay up to date with Phusion news, please fill in your name and email address below and sign up for our newsletter. We won’t spam you, we promise.



Phusion Passenger 4.0.36 released

By Hongli Lai on January 25th, 2014


Phusion Passenger is a fast and robust web server and application server for Ruby, Python, Node.js and Meteor. Passenger takes a lot of complexity out of deploying web apps, and adds powerful enterprise-grade features that are useful in production. High-profile companies such as Apple, New York Times, AirBnB, Juniper, American Express, etc are already using it, as well as over 350.000 websites.

Phusion Passenger is under constant maintenance and development. Version 4.0.36 is a bugfix release.

Phusion Passenger also has an Enterprise version which comes with a wide array of additional features. By buying Phusion Passenger Enterprise you will directly sponsor the development of the open source version.

Recent changes

  • [Enterprise] Fixed some Mass Deployment bugs.
  • [Enterprise] Fixed a bug that causes an application group to be put into Deployment Error Resistance Mode if rolling restarting fails while deployment error resistance is off. Deployment Error Resistance Mode is now only activated if it’s explicitly turned on.
  • Passenger Standalone now gzips JSON responses.
  • Fixed some cases in which Passenger Standalone does not to properly cleanup its temporary files.

Installing or upgrading to 4.0.36

OS X OS X Debian Debian Ubuntu Ubuntu
Heroku Heroku Ruby gem Ruby gem Tarball Tarball

Final

Phusion Passenger’s core is open source. Please fork or watch us on Github. :)

If you would like to stay up to date with Phusion news, please fill in your name and email address below and sign up for our newsletter. We won’t spam you, we promise.



Phusion Passenger 4.0.35 released

By Hongli Lai on January 16th, 2014

Version 4.0.34 has been skipped because it was an non-public release for QA purposes. The changes in 4.0.34 and 4.0.35 combined are:

  • The Node.js loader code now sets the isApplicationLoader attribute on the
    bootstrapping module. This provides a way for apps and frameworks that check
    for module.parent to check whether the current file is loaded by Phusion
    Passenger, or by other software that work in a similar way.

    This change has been introduced to solve a compatibility issue with CompoundJS.
    CompoundJS users should modify their server.js, and change the following:

    if (!module.parent) {
    

    to:

    if (!module.parent || module.parent.isApplicationLoader) {
    
  • Improved support for Meteor in development mode. Terminating Phusion Passenger
    now leaves less garbage Meteor processes behind.

  • It is now possible to disable the usage of the Ruby native extension by setting
    the environment variable PASSENGER_USE_RUBY_NATIVE_SUPPORT=0.
  • Fixed incorrect detection of the Apache MPM on Ubuntu 13.10.
  • When using RVM, if you set PassengerRuby/passenger_ruby to the raw Ruby binary
    instead of the wrapper script, Phusion Passenger will now print an error.
  • Added support for RVM >= 1.25 wrapper scripts.
  • Fixed loading passenger_native_support on Ruby 1.9.2.
  • The Union Station analytics code now works even without native_support.
  • Fixed passenger-install-apache2-module and passenger-install-nginx-module in
    Homebrew.
  • Binaries are now downloaded from an Amazon S3 mirror if the main binary server is unavailable.
  • And finally, although this isn’t really a change in 4.0.34, it should be noted.
    In version 4.0.33 we changed the way Phusion Passenger’s own Ruby source files
    are loaded, in order to fix some Debian and RPM packaging issues. The following
    doesn’t work anymore:

    require 'phusion_passenger/foo'
    

    Instead, it should become:

    PhusionPassenger.require_passenger_lib 'foo'
    

    However, we overlooked the fact that this change breaks Ruby apps which use
    our Out-of-Band GC feature, because such apps had to call
    require 'phusion_passenger/rack/out_of_band_gc'. Unfortunately we’re not able
    to maintain compatibility without reintroducing the Debian and RPM packaging
    issues. Users should modify the following:

    require 'phusion_passenger/rack/out_of_band_gc'
    

    to:

    if PhusionPassenger.respond_to?(:require_passenger_lib)
      # Phusion Passenger >= 4.0.33
      PhusionPassenger.require_passenger_lib 'rack/out_of_band_gc'
    else
      # Phusion Passenger < 4.0.33
      require 'phusion_passenger/rack/out_of_band_gc'
    end
    

Installing or upgrading to 4.0.35

OS X OS X Debian Debian Ubuntu Ubuntu
Heroku Heroku Ruby gem Ruby gem Tarball Tarball

About the recent oss-binaries.phusionpassenger.com outages

By Hongli Lai on January 14th, 2014

Update January 15 2013: This issue has been fixed in version 4.0.35.

As some of you might have noticed, there were some problems recently with the server oss-binaries.phusionpassenger.com, where we host our APT repository and precompiled binaries for Phusion Passenger. Although it was originally meant to be a simple file server meant for speeding up installation (by avoiding the need to compile Phusion Passenger), it has grown a lot in importance in the past few Phusion Passenger releases, so that any down time causes major problems for many users:

  • Our APT repository has grown more popular than we thought.
  • Many Heroku users are unable to start new dynos as long as the server is down. The Heroku dyno environment does not provide the necessary compiler toolchain, nor the hardware resources, to compile Phusion Passenger. Which is why when run on Heroku, Phusion Passenger downloads binaries from our server.

The server had first gone down on Sunday and was fixed later that day. Unfortunately it had gone down again on Tuesday morning, which we fixed soon after.

We sincerely apologize for this problem. But of course, apologies are not going to cut it. Since the first outage on Sunday we realized just how important this — originally minor — server became. Since Sunday we’ve begun work to solve this issue permanently. It’s clear that relying on a single server is a mistake, so we’re taking the following actions:

  • We’re adjusting the download timeouts in Phusion Passenger so that server problems don’t freeze it indefinitely. This allows Phusion Passenger to detect server problems quicker, and to fall back to compilation, without triggering any timeouts that may abort Phusion Passenger entirely. This work has been implemented yesterday.
  • Instead of trying to download the native_support binary, Phusion Passenger should try to compile it first, because compiling native_support takes less than 1 second. If the correct compiler toolchain is
    installed on the server then it will avoid using the network entirely, so that it’s unaffected by any server outages of ours. This has also been implemented yesterday. (The rest of Phusion Passenger takes longer to compile so we can’t apply the same strategy there.)
  • For Heroku users: having the binaries downloaded at Heroku deploy time, not at dyno boot time, so that Heroku users are less susceptible to download problems. This has been implemented yesterday.
  • Reverting any server changes that we’ve made recently to oss-binaries.phusionpassenger.com, in the hope that it would increase the server’s uptime. The true reason for the downtime is still under investigation, but we’re giving the other items in this list more priority because they have more potential to fix the problem permanently. This has been implemented today.
  • Setting up an Amazon S3 mirror for high availability. If the main server is down, Phusion Passenger should automatically download from the mirror instead. We’re currently working on this. This has been implemented.

The goal is to finish all these items this week and to release a new version that includes these fixes. We’re working around the clock on this.

Workarounds for now

Users can apply the following workaround for now in order to prevent Phusion Passenger from freezing during downloading of binaries:

Edit /etc/hosts and add “127.0.0.1 oss-binaries.phusionpassenger.com”

Phusion Passenger will automatically fall back to compiling if it can’t download binaries.

Unfortunately, this workaround will not be useful for users who rely on our APT repository, or Heroku users. We’re working on a true fix as quickly as we can.