Phusion white papers Phusion overview

Phusion Passenger 4.0.39 released

By Hongli Lai on March 18th, 2014


Phusion Passenger is a fast and robust web server and application server for Ruby, Python, Node.js and Meteor. Passenger takes a lot of complexity out of deploying web apps, and adds powerful enterprise-grade features that are useful in production. High-profile companies such as Apple, New York Times, AirBnB, Juniper, American Express, etc are already using it, as well as over 350.000 websites.

Phusion Passenger is under constant maintenance and development. Version 4.0.39 is a bugfix release.

Phusion Passenger also has an Enterprise version which comes with a wide array of additional features. By buying Phusion Passenger Enterprise you will directly sponsor the development of the open source version.

Recent changes

  • Fixed a crash that could happen if the client disconnects while a chunked response is being sent. Fixes issue #1062.
  • In Phusion Passenger Standalone, it is now possible to customize the Nginx configuration file on Heroku. It is now also possible to permanently apply changes to the Nginx configuration file, surviving upgrades. Please refer to the "Advanced configuration" section of the Phusion Passenger Standalone manual for more information.
  • The programming language selection menu in passenger-install-apache2-module and passenger-install-nginx-module only works on terminals that support UTF-8 and that have a UTF-8 capable font. To cater to users who cannot meet these requirements (e.g. PuTTY users using any of the default Windows fonts), it is now possible to switch the menu to a plain text mode by pressing ‘!’. Fixes issue #1066.
  • Fixed printing UTF-8 characters in log files in Phusion Passenger Standalone.
  • It is now possible to dump live backtraces of Python apps through the ‘SIGABRT’ signal.
  • Fixed closing of file descriptors on OS X 10.9.
  • Fixed compilation of native_support on Rubinius.

Installing or upgrading to 4.0.39

OS X OS X Debian Debian Ubuntu Ubuntu
Heroku Heroku Ruby gem Ruby gem Tarball Tarball

Final

Phusion Passenger’s core is open source. Please fork or watch us on Github. :)

If you would like to stay up to date with Phusion news, please fill in your name and email address below and sign up for our newsletter. We won’t spam you, we promise.



Phusion Passenger 4.0.38 released

By Hongli Lai on March 10th, 2014


Phusion Passenger is a fast and robust web server and application server for Ruby, Python, Node.js and Meteor. Passenger takes a lot of complexity out of deploying web apps, and adds powerful enterprise-grade features that are useful in production. High-profile companies such as Apple, New York Times, AirBnB, Juniper, American Express, etc are already using it, as well as over 350.000 websites.

Phusion Passenger is under constant maintenance and development. Version 4.0.38 is a bugfix release.

Phusion Passenger also has an Enterprise version which comes with a wide array of additional features. By buying Phusion Passenger Enterprise you will directly sponsor the development of the open source version.

Recent changes

  • Fixed a symlink-related security vulnerability.

    Urgency: low
    Scope: local exploit
    Summary: writing files to arbitrary directory by hijacking temp directories
    Affected versions: 4.0.37
    Fixed versions: 4.0.38
    CVE-2014-1832

    Description: This issue is related to CVE-2014-1831 (the security issue as mentioned in the 4.0.37 release notes). The previous fix was incomplete, and still has a (albeit smaller) small attack time window in between two filesystem checks. This attack window is now gone.

  • Added support for the new Ruby 2.1.0 out-of-band garbage collector. This can much improve garbage collection performance, and drastically reduce request times.
  • Passenger Standalone is now compatible with IPv6.
  • Fixed some compilation problems on Solaris. See issue #1047.
  • passenger-install-apache2-module and passenger-install-nginx-module now automatically run in `–auto` mode if stdin is not a TTY. Fixes issue #1030.
  • Fixed an issue with non-bundled Meteor apps not correctly running in production mode.
  • The `PassengerPreStart` option is now compatible with IPv6 server sockets.
  • When running Python WSGI apps, `wsgi.run_once` is now set to False. This should improve the performance of certain apps and frameworks.
  • When handling HTTP requests with chunked transfer encoding, the ‘Transfer-Encoding’ header is no longer passed to the application. This is because the web server already buffers and dechunks the request body.
  • Fixed a possible hang in Phusion Passenger for Nginx when Nginx is instructed to reload or reopen log files. Thanks to Feng Gu, pull request #97.
  • The preferred Nginx version has been upgraded to 1.4.6.
  • Fixed a problem with running passenger-install-apache2-module and passenger-install-nginx-module on JRuby. They were not able to accept any terminal input after displaying the programming language menu.

Installing or upgrading to 4.0.38

OS X OS X Debian Debian Ubuntu Ubuntu
Heroku Heroku Ruby gem Ruby gem Tarball Tarball

Final

Phusion Passenger’s core is open source. Please fork or watch us on Github. :)

If you would like to stay up to date with Phusion news, please fill in your name and email address below and sign up for our newsletter. We won’t spam you, we promise.



Farewell, Jim.

By Ninh Bui on February 20th, 2014

Today, the sad news has reached us that Jim Weirich has passed away. We’re incredibly sad about this
as Jim was one of the nicest people we’ve got to know in the Ruby/Rails community when we first started Phusion.
In keeping his memory alive, I’d like to reflect on a particular anecdote that made Jim especially awesome to us
and most likely to you as well. I’m sure many of you who were fortunate enough to get to know him can relate to his
kindness.

Back in 2008 when Hongli, Tinco and I set out to go to RailsConf to give our very first talk abroad, we
met Jim in the lobby of the conference space. We had just attended a talk of his where he had gone through
a myriad of valuable do’s and don’ts one should be aware of when giving a talk. These tips proved to be
incredibly valuable to us in years to come, and we hope Jim knows how grateful we are for this.

Our talk was scheduled to be held the day after, and seeing Jim’s do’s and don’ts, we were suddenly confronted
with how many embarassing “don’ts” we had in our slides. As Jim told the audience that it’s generally a good idea to avoid
cliches such as having bulletpoint hell, stock images of “the world” and “business people shaking hands”, we felt
more and more uncomfortable. Not only did we have a lot of bulletpoints, we even had an image of “business people
shaking hands”… in front of “the world”. We basically tripped over every possible cliche in the book!

But hey, we still had 24 hours, surely we’d be able to fix this right? Luckily, Jim had the demeanor of a big
kind cuddly bear, so we felt compelled to walk up to him after his talk to ask for some help with our slides.
Instead of brushing us off, Jim graciously sat down with us for about 2 hours in pointing out the things that
could use improvement in the delivery of our talk. And understandibly laughed out loud at our slide with the business people
shaking hands in front of the world. ;)

The next day, after giving our talk, we had people walking up to us saying that we killed it. In reality, it was
Jim’s tips and kindness in sharing these tips that “killed it”.

We will miss you buddy.

Your friends,
Tinco, Hongli and Ninh.

Your Docker image might be broken without you knowing it

By Hongli Lai on February 18th, 2014

Docker is an awesome new technology for the creation of lightweight containers. It has many purposes and serves as a good building block for PaaS, application deployments, continuous integration systems and more. No wonder it’s becoming more and more popular every day.

However what a lot of people may not realize is that the operating system inside the container must be configured correctly, and that this is not easy to do. Unix has many strange corner cases that are hard to get right if you are not intimately familiar with the Unix system model. This can cause a lot of strange problems. In other words, your Docker image might be broken, without you knowing it.

To raise awareness about this problem, we’ve published a website which explains the problem in detail. It explains what exactly could be broken, why it’s broken that way, and what can be done to fix them.

And to make it easier for other Docker users to get things right, we’ve published a preconfigured image — baseimage-docker — which does get everything right. This potentially saves you a lot of time.

Learn the right way to build your Dockerfile.

One of the gripes we have with random Docker images on the Docker registry is that they’re often very poorly documented, and do not provide the original Dockerfile from which they’re created. With baseimage-docker, we’re breaking that tradition by providing excellent documentation and by making the entire build reproducible. The Dockerfile and all other sources are available at Github, for everyone to see and to modify.

Happy Docking!

Ruby Rogues 143: Phusion Passenger Enterprise with Hongli Lai and Tinco Andringa

By Hongli Lai on February 12th, 2014

We’ve been invited by Ruby Rogues to participate in a podcast about Phusion Passenger Enterprise. This podcast covers the following topics:

  • Hongli Lai and Tinco Andringa Introductions
  • Phusion Passenger introduction
  • Rack
  • Node.js, MeteorJS, Python Support
  • Processes and Threads
    • Ruby Rogues Episode #58 – Book Club: Working with Unix Processes with Jesse Storimer
    • Ruby Enterprise Edition
    • Smart Spawning
  • Advantages of Phusion Passenger Enterprise
    • Rolling Restarts
    • Mass Deployment
  • Passenger vs Unicorn
  • Error Resistant Deploys
  • Hosting
    • DreamHost
  • Apache, Nginx support
  • Stability Issues
  • Documentation and Support

Listen to the podcast at the Ruby Rogues website

Thanks Ruby Rogues for hosting us!

Phusion Passenger now supports the new Ruby 2.1 Out-Of-Band GC

By Hongli Lai on January 31st, 2014


Phusion Passenger is a fast and robust web server and application server for Ruby, Python, Node.js and Meteor. Passenger takes a lot of complexity out of deploying web apps, and adds powerful enterprise-grade features that are useful in production. High-profile companies such as Apple, New York Times, AirBnB, Juniper, American Express, etc are already using it, as well as over 350.000 websites.

Yesterday, Aman Gupta announced an Out-Of-Band garbage collector for Ruby 2.1. What is an “Out-Of-Band garbage collector”? It means that the garbage collector is run in between requests, instead of during a request. While the garbage collector is running, other processes can serve requests. This way visitors will never experience the slight delay that is caused by the garbage collector, much improving response times. We just implemented support for this in Phusion Passenger.

Classical out-of-band garbage collection

Out-of-band garbage collection is not a new idea. The idea was first introduced in Unicorn several years ago. Its mechanism was simple: disable automatic garbage collection, and manually run the garbage collector at the end of every N requests.

Last year, we introduced a much improved version of Unicorn’s out-of-band garbage collection mechanism in Phusion Passenger 4. This feature was originally contributed by AppFolio, and since then many further improvements have been made.

  • Unicorn’s out-of-band garbage collection doesn’t work well with multithreaded servers, because a garbage collection is a stop-the-world action that blocks all threads. Phusion Passenger’s mechanism lifts this limit by intelligently coordinating the garbage collector and threads.
  • Unicorn’s mechanism is susceptible to occasional pathological behavior. It is possible for many, or all, Unicorn processes to be simultaneously running the out-of-band garbage collector. During that short period of time, no requests can be served, and visitors will experience a delay. Phusion Passenger’s mechanism ensures that only 1 process can run the garbage collector at a time, thereby eliminating this problem.
  • Phusion Passenger’s mechanism is more flexible, and can be used for work other than just garbage collection.

The results were impressive. AppFolio observed an average response time reduction of 100 ms.

Before Out-Of-Band GC After Out-Of-Band GC
Before and after applying out of band GC at AppFolio

Unfortunately, this Out-Of-Band GC mechanism (running the GC every few requests) is still suboptimal, as we had pointed out last year:

  • If a request generates a lot of garbage, then delaying the GC for a few more request can result in high peak memory usage.
  • If the garbage collector doesn’t actually need to be run yet, then running it anyway wastes CPU cycles.

The new Ruby 2.1 Out-Of-Band GC

Fortunately, Ruby has recently introduced many useful garbage collector improvements. Ruby 2.1 introduced a semi-generational garbage collector. Aman Gupta built an Out-Of-Band GC mechanism on top of the recent improvements, which is almost exactly what we needed in Phusion Passenger.

Today, we’ve submitted a pull request to make his Out-Of-Band GC mechanism work well with Phusion Passenger, and we’ve introduced support in Phusion Passenger for his Out-Of-Band GC mechanism. Using it is very easy. First, add our patched gctools to your Gemfile:

gem "gctools", :github => "FooBarWidget/gctools", :require => false

Next, put this in your config.ru:

require "gctools/oobgc"
GC::OOB.setup

Now you can enable the Out-Of-Band GC feature in Phusion Passenger by using the :gctools_oobgc strategy:

PhusionPassenger.require_passenger_lib 'rack/out_of_band_gc'
use PhusionPassenger::Rack::OutOfBandGc, :strategy => :gctools_oobgc

Aman Gupta’s improvements are very impressive indeed. With Ruby 2.1, his average OOBGC pause time (oobgc.mean) “went from 125ms to 50ms thanks to RGenGC”. And “the number of out-of-band collections (oobgc.count) also went down, since the new OOBGC only runs when necessary”.

From Aman Gupta’s blog: “The overall result is much less CPU time (oobgc.sum) spent doing GC work between requests”:

Conclusion

Support for Aman Gupta’s Ruby 2.1 Out-Of-Band GC mechanism will be included in the next release of Phusion Passenger. It’s not completely finished yet: as seen in our pull request, there are still some issues to be fleshed out. But the results are very promising indeed. Although the Ruby hype is over, there are still plenty of improvements going on.

Discuss this on Hacker News.

Like our articles? Consider subscribing to our newsletter. You can unsubscribe any time.



Phusion Passenger 4.0.37 released

By Hongli Lai on January 29th, 2014


Phusion Passenger is a fast and robust web server and application server for Ruby, Python, Node.js and Meteor. Passenger takes a lot of complexity out of deploying web apps, and adds powerful enterprise-grade features that are useful in production. High-profile companies such as Apple, New York Times, AirBnB, Juniper, American Express, etc are already using it, as well as over 350.000 websites.

Phusion Passenger is under constant maintenance and development. Version 4.0.37 is a bugfix release.

Phusion Passenger also has an Enterprise version which comes with a wide array of additional features. By buying Phusion Passenger Enterprise you will directly sponsor the development of the open source version.

Recent changes

  • Improved Node.js compatibility. Calling on() on the request object now returns the request object itself. This fixes some issues with Express, Connect and Formidable. Furthermore, some WebSocket-related issues have been fixed.
  • Improved Meteor support. Meteor application processes are now shut down quicker. Previously, they linger around for 5 seconds while waiting for all connections to terminate, but that didn’t work well because WebSocket connections were kept open indefinitely. Also, some WebSocket-related issues have been fixed.
  • Introduced a new tool `passenger-config detach-process` for gracefully detaching an application process from the process pool. Has a similar effect to killing the application process directly with `kill <PID>`, but killing directly may cause the HTTP client to see an error, while using this command guarantees that clients see no errors.
  • Fixed a crash that occurs when an application fails to spawn, but the HTTP client disconnects before the error page is generated. Fixes issue #1028.
  • Fixed a symlink-related security vulnerability.

    Urgency: low
    Scope: local exploit
    Summary: writing files to arbitrary directory by hijacking temp directories
    Affected versions: 4.0.5 and later
    Fixed versions: 4.0.37
    CVE-2014-1831

    Description: Phusion Passenger creates a "server instance directory" in /tmp during startup, which is a temporary directory that Phusion Passenger uses to store working files. This directory is deleted after Phusion Passenger exits. For various technical reasons, this directory must have a semi-predictable filename. If a local attacker can predict this filename, and precreates a symlink with the same filename that points to an arbitrary directory with mode 755, owner root and group root, then the attacker will succeed in making Phusion Passenger write files and create subdirectories inside that target directory. The following files/subdirectories are created:

    • control_process.pid
    • generation-X, where X is a number.

    If you happen to have a file inside the target directory called `control_process.pid`, then that file’s contents are overwritten. These files and directories are deleted during Phusion Passenger exit. The target directory itself is not deleted, nor are any other contents inside the target directory, although the symlink is.

    Thanks go to Jakub Wilk for discovering this issue.

Installing or upgrading to 4.0.37

OS X OS X Debian Debian Ubuntu Ubuntu
Heroku Heroku Ruby gem Ruby gem Tarball Tarball

Final

Phusion Passenger’s core is open source. Please fork or watch us on Github. :)

If you would like to stay up to date with Phusion news, please fill in your name and email address below and sign up for our newsletter. We won’t spam you, we promise.



Phusion Passenger 4.0.36 released

By Hongli Lai on January 25th, 2014


Phusion Passenger is a fast and robust web server and application server for Ruby, Python, Node.js and Meteor. Passenger takes a lot of complexity out of deploying web apps, and adds powerful enterprise-grade features that are useful in production. High-profile companies such as Apple, New York Times, AirBnB, Juniper, American Express, etc are already using it, as well as over 350.000 websites.

Phusion Passenger is under constant maintenance and development. Version 4.0.36 is a bugfix release.

Phusion Passenger also has an Enterprise version which comes with a wide array of additional features. By buying Phusion Passenger Enterprise you will directly sponsor the development of the open source version.

Recent changes

  • [Enterprise] Fixed some Mass Deployment bugs.
  • [Enterprise] Fixed a bug that causes an application group to be put into Deployment Error Resistance Mode if rolling restarting fails while deployment error resistance is off. Deployment Error Resistance Mode is now only activated if it’s explicitly turned on.
  • Passenger Standalone now gzips JSON responses.
  • Fixed some cases in which Passenger Standalone does not to properly cleanup its temporary files.

Installing or upgrading to 4.0.36

OS X OS X Debian Debian Ubuntu Ubuntu
Heroku Heroku Ruby gem Ruby gem Tarball Tarball

Final

Phusion Passenger’s core is open source. Please fork or watch us on Github. :)

If you would like to stay up to date with Phusion news, please fill in your name and email address below and sign up for our newsletter. We won’t spam you, we promise.



Phusion Passenger 4.0.35 released

By Hongli Lai on January 16th, 2014

Version 4.0.34 has been skipped because it was an non-public release for QA purposes. The changes in 4.0.34 and 4.0.35 combined are:

  • The Node.js loader code now sets the isApplicationLoader attribute on the
    bootstrapping module. This provides a way for apps and frameworks that check
    for module.parent to check whether the current file is loaded by Phusion
    Passenger, or by other software that work in a similar way.

    This change has been introduced to solve a compatibility issue with CompoundJS.
    CompoundJS users should modify their server.js, and change the following:

    if (!module.parent) {
    

    to:

    if (!module.parent || module.parent.isApplicationLoader) {
    
  • Improved support for Meteor in development mode. Terminating Phusion Passenger
    now leaves less garbage Meteor processes behind.

  • It is now possible to disable the usage of the Ruby native extension by setting
    the environment variable PASSENGER_USE_RUBY_NATIVE_SUPPORT=0.
  • Fixed incorrect detection of the Apache MPM on Ubuntu 13.10.
  • When using RVM, if you set PassengerRuby/passenger_ruby to the raw Ruby binary
    instead of the wrapper script, Phusion Passenger will now print an error.
  • Added support for RVM >= 1.25 wrapper scripts.
  • Fixed loading passenger_native_support on Ruby 1.9.2.
  • The Union Station analytics code now works even without native_support.
  • Fixed passenger-install-apache2-module and passenger-install-nginx-module in
    Homebrew.
  • Binaries are now downloaded from an Amazon S3 mirror if the main binary server is unavailable.
  • And finally, although this isn’t really a change in 4.0.34, it should be noted.
    In version 4.0.33 we changed the way Phusion Passenger’s own Ruby source files
    are loaded, in order to fix some Debian and RPM packaging issues. The following
    doesn’t work anymore:

    require 'phusion_passenger/foo'
    

    Instead, it should become:

    PhusionPassenger.require_passenger_lib 'foo'
    

    However, we overlooked the fact that this change breaks Ruby apps which use
    our Out-of-Band GC feature, because such apps had to call
    require 'phusion_passenger/rack/out_of_band_gc'. Unfortunately we’re not able
    to maintain compatibility without reintroducing the Debian and RPM packaging
    issues. Users should modify the following:

    require 'phusion_passenger/rack/out_of_band_gc'
    

    to:

    if PhusionPassenger.respond_to?(:require_passenger_lib)
      # Phusion Passenger >= 4.0.33
      PhusionPassenger.require_passenger_lib 'rack/out_of_band_gc'
    else
      # Phusion Passenger < 4.0.33
      require 'phusion_passenger/rack/out_of_band_gc'
    end
    

Installing or upgrading to 4.0.35

OS X OS X Debian Debian Ubuntu Ubuntu
Heroku Heroku Ruby gem Ruby gem Tarball Tarball

About the recent oss-binaries.phusionpassenger.com outages

By Hongli Lai on January 14th, 2014

Update January 15 2013: This issue has been fixed in version 4.0.35.

As some of you might have noticed, there were some problems recently with the server oss-binaries.phusionpassenger.com, where we host our APT repository and precompiled binaries for Phusion Passenger. Although it was originally meant to be a simple file server meant for speeding up installation (by avoiding the need to compile Phusion Passenger), it has grown a lot in importance in the past few Phusion Passenger releases, so that any down time causes major problems for many users:

  • Our APT repository has grown more popular than we thought.
  • Many Heroku users are unable to start new dynos as long as the server is down. The Heroku dyno environment does not provide the necessary compiler toolchain, nor the hardware resources, to compile Phusion Passenger. Which is why when run on Heroku, Phusion Passenger downloads binaries from our server.

The server had first gone down on Sunday and was fixed later that day. Unfortunately it had gone down again on Tuesday morning, which we fixed soon after.

We sincerely apologize for this problem. But of course, apologies are not going to cut it. Since the first outage on Sunday we realized just how important this — originally minor — server became. Since Sunday we’ve begun work to solve this issue permanently. It’s clear that relying on a single server is a mistake, so we’re taking the following actions:

  • We’re adjusting the download timeouts in Phusion Passenger so that server problems don’t freeze it indefinitely. This allows Phusion Passenger to detect server problems quicker, and to fall back to compilation, without triggering any timeouts that may abort Phusion Passenger entirely. This work has been implemented yesterday.
  • Instead of trying to download the native_support binary, Phusion Passenger should try to compile it first, because compiling native_support takes less than 1 second. If the correct compiler toolchain is
    installed on the server then it will avoid using the network entirely, so that it’s unaffected by any server outages of ours. This has also been implemented yesterday. (The rest of Phusion Passenger takes longer to compile so we can’t apply the same strategy there.)
  • For Heroku users: having the binaries downloaded at Heroku deploy time, not at dyno boot time, so that Heroku users are less susceptible to download problems. This has been implemented yesterday.
  • Reverting any server changes that we’ve made recently to oss-binaries.phusionpassenger.com, in the hope that it would increase the server’s uptime. The true reason for the downtime is still under investigation, but we’re giving the other items in this list more priority because they have more potential to fix the problem permanently. This has been implemented today.
  • Setting up an Amazon S3 mirror for high availability. If the main server is down, Phusion Passenger should automatically download from the mirror instead. We’re currently working on this. This has been implemented.

The goal is to finish all these items this week and to release a new version that includes these fixes. We’re working around the clock on this.

Workarounds for now

Users can apply the following workaround for now in order to prevent Phusion Passenger from freezing during downloading of binaries:

Edit /etc/hosts and add “127.0.0.1 oss-binaries.phusionpassenger.com”

Phusion Passenger will automatically fall back to compiling if it can’t download binaries.

Unfortunately, this workaround will not be useful for users who rely on our APT repository, or Heroku users. We’re working on a true fix as quickly as we can.