Phusion white papers Phusion overview

Your Docker image might be broken without you knowing it

By Hongli Lai on February 18th, 2014

Docker is an awesome new technology for the creation of lightweight containers. It has many purposes and serves as a good building block for PaaS, application deployments, continuous integration systems and more. No wonder it’s becoming more and more popular every day.

However what a lot of people may not realize is that the operating system inside the container must be configured correctly, and that this is not easy to do. Unix has many strange corner cases that are hard to get right if you are not intimately familiar with the Unix system model. This can cause a lot of strange problems. In other words, your Docker image might be broken, without you knowing it.

To raise awareness about this problem, we’ve published a website which explains the problem in detail. It explains what exactly could be broken, why it’s broken that way, and what can be done to fix them.

And to make it easier for other Docker users to get things right, we’ve published a preconfigured image — baseimage-docker — which does get everything right. This potentially saves you a lot of time.

Learn the right way to build your Dockerfile.

One of the gripes we have with random Docker images on the Docker registry is that they’re often very poorly documented, and do not provide the original Dockerfile from which they’re created. With baseimage-docker, we’re breaking that tradition by providing excellent documentation and by making the entire build reproducible. The Dockerfile and all other sources are available at Github, for everyone to see and to modify.

Happy Docking!

Ruby Rogues 143: Phusion Passenger Enterprise with Hongli Lai and Tinco Andringa

By Hongli Lai on February 12th, 2014

We’ve been invited by Ruby Rogues to participate in a podcast about Phusion Passenger Enterprise. This podcast covers the following topics:

  • Hongli Lai and Tinco Andringa Introductions
  • Phusion Passenger introduction
  • Rack
  • Node.js, MeteorJS, Python Support
  • Processes and Threads
    • Ruby Rogues Episode #58 – Book Club: Working with Unix Processes with Jesse Storimer
    • Ruby Enterprise Edition
    • Smart Spawning
  • Advantages of Phusion Passenger Enterprise
    • Rolling Restarts
    • Mass Deployment
  • Passenger vs Unicorn
  • Error Resistant Deploys
  • Hosting
    • DreamHost
  • Apache, Nginx support
  • Stability Issues
  • Documentation and Support

Listen to the podcast at the Ruby Rogues website

Thanks Ruby Rogues for hosting us!

Phusion Passenger now supports the new Ruby 2.1 Out-Of-Band GC

By Hongli Lai on January 31st, 2014


Phusion Passenger is a fast and robust web server and application server for Ruby, Python, Node.js and Meteor. Passenger takes a lot of complexity out of deploying web apps, and adds powerful enterprise-grade features that are useful in production. High-profile companies such as Apple, New York Times, AirBnB, Juniper, American Express, etc are already using it, as well as over 350.000 websites.

Yesterday, Aman Gupta announced an Out-Of-Band garbage collector for Ruby 2.1. What is an “Out-Of-Band garbage collector”? It means that the garbage collector is run in between requests, instead of during a request. While the garbage collector is running, other processes can serve requests. This way visitors will never experience the slight delay that is caused by the garbage collector, much improving response times. We just implemented support for this in Phusion Passenger.

Classical out-of-band garbage collection

Out-of-band garbage collection is not a new idea. The idea was first introduced in Unicorn several years ago. Its mechanism was simple: disable automatic garbage collection, and manually run the garbage collector at the end of every N requests.

Last year, we introduced a much improved version of Unicorn’s out-of-band garbage collection mechanism in Phusion Passenger 4. This feature was originally contributed by AppFolio, and since then many further improvements have been made.

  • Unicorn’s out-of-band garbage collection doesn’t work well with multithreaded servers, because a garbage collection is a stop-the-world action that blocks all threads. Phusion Passenger’s mechanism lifts this limit by intelligently coordinating the garbage collector and threads.
  • Unicorn’s mechanism is susceptible to occasional pathological behavior. It is possible for many, or all, Unicorn processes to be simultaneously running the out-of-band garbage collector. During that short period of time, no requests can be served, and visitors will experience a delay. Phusion Passenger’s mechanism ensures that only 1 process can run the garbage collector at a time, thereby eliminating this problem.
  • Phusion Passenger’s mechanism is more flexible, and can be used for work other than just garbage collection.

The results were impressive. AppFolio observed an average response time reduction of 100 ms.

Before Out-Of-Band GC After Out-Of-Band GC
Before and after applying out of band GC at AppFolio

Unfortunately, this Out-Of-Band GC mechanism (running the GC every few requests) is still suboptimal, as we had pointed out last year:

  • If a request generates a lot of garbage, then delaying the GC for a few more request can result in high peak memory usage.
  • If the garbage collector doesn’t actually need to be run yet, then running it anyway wastes CPU cycles.

The new Ruby 2.1 Out-Of-Band GC

Fortunately, Ruby has recently introduced many useful garbage collector improvements. Ruby 2.1 introduced a semi-generational garbage collector. Aman Gupta built an Out-Of-Band GC mechanism on top of the recent improvements, which is almost exactly what we needed in Phusion Passenger.

Today, we’ve submitted a pull request to make his Out-Of-Band GC mechanism work well with Phusion Passenger, and we’ve introduced support in Phusion Passenger for his Out-Of-Band GC mechanism. Using it is very easy. First, add our patched gctools to your Gemfile:

gem "gctools", :github => "FooBarWidget/gctools", :require => false

Next, put this in your config.ru:

require "gctools/oobgc"
GC::OOB.setup

Now you can enable the Out-Of-Band GC feature in Phusion Passenger by using the :gctools_oobgc strategy:

PhusionPassenger.require_passenger_lib 'rack/out_of_band_gc'
use PhusionPassenger::Rack::OutOfBandGc, :strategy => :gctools_oobgc

Aman Gupta’s improvements are very impressive indeed. With Ruby 2.1, his average OOBGC pause time (oobgc.mean) “went from 125ms to 50ms thanks to RGenGC”. And “the number of out-of-band collections (oobgc.count) also went down, since the new OOBGC only runs when necessary”.

From Aman Gupta’s blog: “The overall result is much less CPU time (oobgc.sum) spent doing GC work between requests”:

Conclusion

Support for Aman Gupta’s Ruby 2.1 Out-Of-Band GC mechanism will be included in the next release of Phusion Passenger. It’s not completely finished yet: as seen in our pull request, there are still some issues to be fleshed out. But the results are very promising indeed. Although the Ruby hype is over, there are still plenty of improvements going on.

Discuss this on Hacker News.

Like our articles? Consider subscribing to our newsletter. You can unsubscribe any time.



Phusion Passenger 4.0.37 released

By Hongli Lai on January 29th, 2014


Phusion Passenger is a fast and robust web server and application server for Ruby, Python, Node.js and Meteor. Passenger takes a lot of complexity out of deploying web apps, and adds powerful enterprise-grade features that are useful in production. High-profile companies such as Apple, New York Times, AirBnB, Juniper, American Express, etc are already using it, as well as over 350.000 websites.

Phusion Passenger is under constant maintenance and development. Version 4.0.37 is a bugfix release.

Phusion Passenger also has an Enterprise version which comes with a wide array of additional features. By buying Phusion Passenger Enterprise you will directly sponsor the development of the open source version.

Recent changes

  • Improved Node.js compatibility. Calling on() on the request object now returns the request object itself. This fixes some issues with Express, Connect and Formidable. Furthermore, some WebSocket-related issues have been fixed.
  • Improved Meteor support. Meteor application processes are now shut down quicker. Previously, they linger around for 5 seconds while waiting for all connections to terminate, but that didn’t work well because WebSocket connections were kept open indefinitely. Also, some WebSocket-related issues have been fixed.
  • Introduced a new tool `passenger-config detach-process` for gracefully detaching an application process from the process pool. Has a similar effect to killing the application process directly with `kill <PID>`, but killing directly may cause the HTTP client to see an error, while using this command guarantees that clients see no errors.
  • Fixed a crash that occurs when an application fails to spawn, but the HTTP client disconnects before the error page is generated. Fixes issue #1028.
  • Fixed a symlink-related security vulnerability.

    Urgency: low
    Scope: local exploit
    Summary: writing files to arbitrary directory by hijacking temp directories
    Affected versions: 4.0.5 and later
    Fixed versions: 4.0.37
    CVE-2014-1831

    Description: Phusion Passenger creates a "server instance directory" in /tmp during startup, which is a temporary directory that Phusion Passenger uses to store working files. This directory is deleted after Phusion Passenger exits. For various technical reasons, this directory must have a semi-predictable filename. If a local attacker can predict this filename, and precreates a symlink with the same filename that points to an arbitrary directory with mode 755, owner root and group root, then the attacker will succeed in making Phusion Passenger write files and create subdirectories inside that target directory. The following files/subdirectories are created:

    • control_process.pid
    • generation-X, where X is a number.

    If you happen to have a file inside the target directory called `control_process.pid`, then that file’s contents are overwritten. These files and directories are deleted during Phusion Passenger exit. The target directory itself is not deleted, nor are any other contents inside the target directory, although the symlink is.

    Thanks go to Jakub Wilk for discovering this issue.

Installing or upgrading to 4.0.37

OS X OS X Debian Debian Ubuntu Ubuntu
Heroku Heroku Ruby gem Ruby gem Tarball Tarball

Final

Phusion Passenger’s core is open source. Please fork or watch us on Github. :)

If you would like to stay up to date with Phusion news, please fill in your name and email address below and sign up for our newsletter. We won’t spam you, we promise.



Phusion Passenger 4.0.36 released

By Hongli Lai on January 25th, 2014


Phusion Passenger is a fast and robust web server and application server for Ruby, Python, Node.js and Meteor. Passenger takes a lot of complexity out of deploying web apps, and adds powerful enterprise-grade features that are useful in production. High-profile companies such as Apple, New York Times, AirBnB, Juniper, American Express, etc are already using it, as well as over 350.000 websites.

Phusion Passenger is under constant maintenance and development. Version 4.0.36 is a bugfix release.

Phusion Passenger also has an Enterprise version which comes with a wide array of additional features. By buying Phusion Passenger Enterprise you will directly sponsor the development of the open source version.

Recent changes

  • [Enterprise] Fixed some Mass Deployment bugs.
  • [Enterprise] Fixed a bug that causes an application group to be put into Deployment Error Resistance Mode if rolling restarting fails while deployment error resistance is off. Deployment Error Resistance Mode is now only activated if it’s explicitly turned on.
  • Passenger Standalone now gzips JSON responses.
  • Fixed some cases in which Passenger Standalone does not to properly cleanup its temporary files.

Installing or upgrading to 4.0.36

OS X OS X Debian Debian Ubuntu Ubuntu
Heroku Heroku Ruby gem Ruby gem Tarball Tarball

Final

Phusion Passenger’s core is open source. Please fork or watch us on Github. :)

If you would like to stay up to date with Phusion news, please fill in your name and email address below and sign up for our newsletter. We won’t spam you, we promise.



Phusion Passenger 4.0.35 released

By Hongli Lai on January 16th, 2014

Version 4.0.34 has been skipped because it was an non-public release for QA purposes. The changes in 4.0.34 and 4.0.35 combined are:

  • The Node.js loader code now sets the isApplicationLoader attribute on the
    bootstrapping module. This provides a way for apps and frameworks that check
    for module.parent to check whether the current file is loaded by Phusion
    Passenger, or by other software that work in a similar way.

    This change has been introduced to solve a compatibility issue with CompoundJS.
    CompoundJS users should modify their server.js, and change the following:

    if (!module.parent) {
    

    to:

    if (!module.parent || module.parent.isApplicationLoader) {
    
  • Improved support for Meteor in development mode. Terminating Phusion Passenger
    now leaves less garbage Meteor processes behind.

  • It is now possible to disable the usage of the Ruby native extension by setting
    the environment variable PASSENGER_USE_RUBY_NATIVE_SUPPORT=0.
  • Fixed incorrect detection of the Apache MPM on Ubuntu 13.10.
  • When using RVM, if you set PassengerRuby/passenger_ruby to the raw Ruby binary
    instead of the wrapper script, Phusion Passenger will now print an error.
  • Added support for RVM >= 1.25 wrapper scripts.
  • Fixed loading passenger_native_support on Ruby 1.9.2.
  • The Union Station analytics code now works even without native_support.
  • Fixed passenger-install-apache2-module and passenger-install-nginx-module in
    Homebrew.
  • Binaries are now downloaded from an Amazon S3 mirror if the main binary server is unavailable.
  • And finally, although this isn’t really a change in 4.0.34, it should be noted.
    In version 4.0.33 we changed the way Phusion Passenger’s own Ruby source files
    are loaded, in order to fix some Debian and RPM packaging issues. The following
    doesn’t work anymore:

    require 'phusion_passenger/foo'
    

    Instead, it should become:

    PhusionPassenger.require_passenger_lib 'foo'
    

    However, we overlooked the fact that this change breaks Ruby apps which use
    our Out-of-Band GC feature, because such apps had to call
    require 'phusion_passenger/rack/out_of_band_gc'. Unfortunately we’re not able
    to maintain compatibility without reintroducing the Debian and RPM packaging
    issues. Users should modify the following:

    require 'phusion_passenger/rack/out_of_band_gc'
    

    to:

    if PhusionPassenger.respond_to?(:require_passenger_lib)
      # Phusion Passenger >= 4.0.33
      PhusionPassenger.require_passenger_lib 'rack/out_of_band_gc'
    else
      # Phusion Passenger < 4.0.33
      require 'phusion_passenger/rack/out_of_band_gc'
    end
    

Installing or upgrading to 4.0.35

OS X OS X Debian Debian Ubuntu Ubuntu
Heroku Heroku Ruby gem Ruby gem Tarball Tarball

About the recent oss-binaries.phusionpassenger.com outages

By Hongli Lai on January 14th, 2014

Update January 15 2013: This issue has been fixed in version 4.0.35.

As some of you might have noticed, there were some problems recently with the server oss-binaries.phusionpassenger.com, where we host our APT repository and precompiled binaries for Phusion Passenger. Although it was originally meant to be a simple file server meant for speeding up installation (by avoiding the need to compile Phusion Passenger), it has grown a lot in importance in the past few Phusion Passenger releases, so that any down time causes major problems for many users:

  • Our APT repository has grown more popular than we thought.
  • Many Heroku users are unable to start new dynos as long as the server is down. The Heroku dyno environment does not provide the necessary compiler toolchain, nor the hardware resources, to compile Phusion Passenger. Which is why when run on Heroku, Phusion Passenger downloads binaries from our server.

The server had first gone down on Sunday and was fixed later that day. Unfortunately it had gone down again on Tuesday morning, which we fixed soon after.

We sincerely apologize for this problem. But of course, apologies are not going to cut it. Since the first outage on Sunday we realized just how important this — originally minor — server became. Since Sunday we’ve begun work to solve this issue permanently. It’s clear that relying on a single server is a mistake, so we’re taking the following actions:

  • We’re adjusting the download timeouts in Phusion Passenger so that server problems don’t freeze it indefinitely. This allows Phusion Passenger to detect server problems quicker, and to fall back to compilation, without triggering any timeouts that may abort Phusion Passenger entirely. This work has been implemented yesterday.
  • Instead of trying to download the native_support binary, Phusion Passenger should try to compile it first, because compiling native_support takes less than 1 second. If the correct compiler toolchain is
    installed on the server then it will avoid using the network entirely, so that it’s unaffected by any server outages of ours. This has also been implemented yesterday. (The rest of Phusion Passenger takes longer to compile so we can’t apply the same strategy there.)
  • For Heroku users: having the binaries downloaded at Heroku deploy time, not at dyno boot time, so that Heroku users are less susceptible to download problems. This has been implemented yesterday.
  • Reverting any server changes that we’ve made recently to oss-binaries.phusionpassenger.com, in the hope that it would increase the server’s uptime. The true reason for the downtime is still under investigation, but we’re giving the other items in this list more priority because they have more potential to fix the problem permanently. This has been implemented today.
  • Setting up an Amazon S3 mirror for high availability. If the main server is down, Phusion Passenger should automatically download from the mirror instead. We’re currently working on this. This has been implemented.

The goal is to finish all these items this week and to release a new version that includes these fixes. We’re working around the clock on this.

Workarounds for now

Users can apply the following workaround for now in order to prevent Phusion Passenger from freezing during downloading of binaries:

Edit /etc/hosts and add “127.0.0.1 oss-binaries.phusionpassenger.com”

Phusion Passenger will automatically fall back to compiling if it can’t download binaries.

Unfortunately, this workaround will not be useful for users who rely on our APT repository, or Heroku users. We’re working on a true fix as quickly as we can.

Phusion Passenger 4.0.33 released

By Hongli Lai on January 2nd, 2014


Phusion Passenger is a fast and robust web server and application server for Ruby, Python, Node.js and Meteor. Passenger takes a lot of complexity out of deploying web apps, and adds powerful enterprise-grade features that are useful in production. High-profile companies such as Apple, New York Times, AirBnB, Juniper, American Express, etc are already using it, as well as over 350.000 websites.

Phusion Passenger is under constant maintenance and development. Version 4.0.33 is a bugfix release.

Phusion Passenger also has an Enterprise version which comes with a wide array of additional features. By buying Phusion Passenger Enterprise you will directly sponsor the development of the open source version.

Recent changes

4.0.31 and 4.0.32 have been skipped because an incompatibility problem with very old Ruby versions was found by our build server shortly after tagging the releases. 4.0.32 fixes all those problems. The changes in 4.0.31, 4.0.32 and 4.0.33 combined are:

  • Introduced a new tool: passenger-config restart-app. With this command you can initiate an application restart without touching restart.txt. Unlike touching restart.txt, this tool initiates the restart immediately instead of on the next request.
  • Fixed some problems in process spawning and request handling.
  • Fixed some problems with the handling of HTTP chunked transfer encoding bodies. These problems only occurred in Ruby.
  • Fixed a compatibility problem in passenger-install-apache2-module with Ruby 1.8. The language selection menu didn’t work properly.
  • Fixed the HelperAgent, upon shutdown, not correctly waiting 5 seconds until all clients have disconnected. Fixes issue #884.
  • Fixed compilation problems on FreeBSD 10.
  • Fixed some C++ strict aliasing problems.
  • Fixed some problems with spawning applications that print messages without newline during startup. Fixes issue #1039.
  • Fixed potential hangs on JRuby when Ctrl-C is used to shutdown the server. Fixes issue #1035.
  • When Phusion Passenger is installed through the Debian package, passenger-install-apache2-module now checks whether the Apache module package (libapache2-mod-passenger) is properly installed, and installs it using apt-get if it’s not installed. Fixes issue #1031.
  • The passenger-status --show=xml command no longer prints the non-XML preamble, such as the version number and the time. Fixes issue #1037.
  • The Ruby native extension check whether it’s loaded against the right Ruby version, to prevent problems when people upgrade Ruby without recompiling their native extensions.
  • Various other minor Debian packaging improvements.

Installing or upgrading to 4.0.33

OS X OS X Debian Debian Ubuntu Ubuntu
Heroku Heroku Ruby gem Ruby gem Tarball Tarball

Final

Phusion Passenger’s core is open source. Please fork or watch us on Github. :)

If you would like to stay up to date with Phusion news, please fill in your name and email address below and sign up for our newsletter. We won’t spam you, we promise.



Phusion Passenger 4.0.30 released, fixes Date header bug that causes cookies to expire prematurely

By Hongli Lai on December 30th, 2013

A few hours ago we’ve been notified of a serious bug in Phusion Passenger 4. If the web app does not supply a Date header, then Phusion Passenger normally adds one in order to comply to the HTTP standard. Unfortunately due to the use of the wrong date format string, December 30 2013 and December 31 2013 are formatted as December 30 2014 and December 31 2014, respectively. As a result, cookies that expire before 2014 would expire on December 30 2013 and December 31 2013. Details can be found at Github pull request 93.

This issue only affects Phusion Passenger for Nginx and Phusion Passenger Standalone. The following are not affected:

  • Phusion Passenger for Apache.
  • Phusion Passenger versions older than 4.0.0 are also not affected because those versions did not try to set the Date header.
  • Web apps that set a Date header themselves.
  • Cookies that expire after 2014.

We’ve taken immediate action and we’ve released version 4.0.30 which addresses this issue. You should upgrade immediately.

You can work around this problem in your application by setting a Date header. For example, in Rails you can do:

before_filter { response.date = Time.now.utc }

Many thanks to Jeff Michael Dean (zilkey), Adam Becker and many others for bringing this to our attention and for providing suggestions, workarounds and feedback.

Upgrading to 4.0.30

OS X OS X Debian Debian Ubuntu Ubuntu
Heroku Heroku Ruby gem Ruby gem Tarball Tarball

Phusion Passenger 4.0.29 released

By Hongli Lai on December 13th, 2013


Phusion Passenger is a fast and robust web server and application server for Ruby, Python, Node.js and Meteor. Passenger takes a lot of complexity out of deploying web apps, and adds powerful enterprise-grade features that are useful in production. High-profile companies such as Apple, New York Times, AirBnB, Juniper, American Express, etc are already using it, as well as over 350.000 websites.

Phusion Passenger is under constant maintenance and development. Version 4.0.29 is a bugfix release.

Phusion Passenger also has an Enterprise version which comes with a wide array of additional features. By buying Phusion Passenger Enterprise you will directly sponsor the development of the open source version.

Recent changes

4.0.28 has been skipped because a compilation problem on OS X Mavericks as discovered shortly after uploading the tarball. The changes in 4.0.28 and 4.0.29 combined are as follows:

  • Introduced a workaround for a GCC 4.6 bug. This bug could cause Phusion Passsenger to crash during startup. Affected operating systems include Ubuntu 12.04 and Amazon Linux 2013.09.01, though not every machine with this OS installed exhibits the problem. See issue #902.
  • Improved Node.js support: the Sails framework is now supported.
  • Improved Node.js support: the streams2 API is now supported.
  • Introduced support for hooks, allowing users to easily extend Phusion Passenger’s behavior.
  • Fixed a bug in the `passenger start -R` option. It was broken because of a change introduced in 4.0.25.
  • Fixed a bug in PassengerMaxInstancesPerApp. Fixes issue #1016.
  • Fixed compilation problems on Solaris.
  • Fixed compilation problems on OS X Mavericks.
  • Fixed an encoding problem in the Apache autodetection code. Fixes issue #1026.
  • The Debian packages no longer depend on libruby.
  • Application stdout and stderr are now printed without normal Phusion Passenger debugging information, making them easier to read.

Installing or upgrading to 4.0.29

OS X OS X Debian Debian Ubuntu Ubuntu
Heroku Heroku Ruby gem Ruby gem Tarball Tarball

Final

Phusion Passenger’s core is open source. Please fork or watch us on Github. :)

If you would like to stay up to date with Phusion news, please fill in your name and email address below and sign up for our newsletter. We won’t spam you, we promise.