On october 12th 2012, we had the pleasure of co-organizing BubbleConf with our friends from Nedap and Teixido: the very first conference in Amsterdam aimed at designers, developers and entrepreneurs that was actually affordable for (bootstrapped) startups to attend[1]. We were blown away by the positive reception for our very first conference: 400 people put their trust in us in delivering an unforgettable day full of ideas and inspiration to apply to their own worklife. After reviewing the post conference survey, we were super excited to see so many people demanding a 2013 edition.

So without further ado, we’re pleased to announce that we’ll be doing a 2013 edition on September 27th 2013 so be sure to save the date! It will be held in Amsterdam once more, but we’ll be switching venues this time around to keep things exciting and fresh. The Tuschinski Theater had already set the bar quite high as we had been able to infer from our post conference survey so it took a while for us to find a venue that was at least as awesome. With the Beurs van Berlage, we think we have found just that:

© Beurs van Berlage Yakult Zaal

© Beurs van Berlage “Glazen Zaal”

The Beurs van Berlage will allow us to implement a few ideas we were unable to implement in our first edition due to time and space constraints. We look forward to sharing these ideas with you over the next couple of months leading up to BubbleConf! Many of these ideas align with what you’ve requested during last year’s survey, e.g. more opportunities to (better) meet up with one another and possibly showing each other their projects. Without saying too much just yet, please keep an eye out on our blog, twitter and newsletter to stay up to date about these developments.

© Beurs van Berlage terrace

© Beurs van Berlage cafe

Like last year, BubbleConf 2013 will be organized by Phusion and Nedap, and the design will be taken care of by Teixido. As for speakers, we’re once again working on securing some of the best in the business. We can already tell you that designers will be in for a huge treat. One might even say that this bubbleconf should be named bubbbleconf (typo intended ).

We started BubbleConf early this year by setting up a teaser site where people are able to purchase early believer tickets: these tickets are offered at a huge discount to people who were adventurous enough to buy them upfront without any beforehand knowledge of the date and location. We were super excited to see the first few tickets already sold this way, and want to thank these folks in particular for putting their blind-trust in us.

Now that the venue and date are known, the early believer discount is no longer applicable. The early bird discount however is still applicable so be sure to get your tickets asap to enjoy this discount.

Oh and one more thing: even though this is not yet set in stone, we’re working with Martijn Stegeman from the University of Amsterdam to see if we can make this into a two day event! More specifically, we’re working with him to see if we can kick off BubbleConf with an unconference on September 26th 2013. Details regarding tickets to this unconference will be made available in the near future.

The App University will bring together BubbleConf attendees as well as students who want to learn and teach more about programming, design or entrepreneurship. It will be targeted towards novices, intermediates and experts: we’re working hard on making it as accessible as possible to a broad range of people. So if you want to learn, or teach, keep an eye out for this one.

Well, that wraps it up for this week’s update. Be sure to keep an eye out for our blog, twitter and newsletter to stay in the loop!

[1] This is only possible due to the fact that we are personally sponsoring this conference for now. Like last year, all tickets are sold below cost price. Who knew that there are actually valid reasons as to why tickets are usually as expensive as they are?! Anyway, we’ll hope to get enough sponsors one day to cover the costs but at this time, we’re just super excited to have you over and contributing to the startup cause!

P.S. Interested in sponsoring this event? Please feel encouraged to contact us at info@bubbleconf.com! We’d love to talk!

We are releasing an emergency release in response to a recently discovered remote code execution vulnerability in Nginx (CVE-2013-2028). Many versions of Nginx 1.3, as well as Nginx 1.4.0, are affected. Phusion Passenger 4.0.2 installs Nginx 1.4.1 by default. There are no other code changes.

Installing 4.0.2

Quick install/upgrade

Phusion Passenger Enterprise users can download the Enterprise version of 4.0.2 from the Customer Area.

Open source users can install the open source version of 4.0.2 with the following commands:

gem install passenger
passenger-install-apache2-module
passenger-install-nginx-module

You can also download the tarball at Google Code. We strongly encourage you to cryptographically verify files after downloading them.

In-depth instructions

In-depth installation and upgrade instructions can be found in the Installation section of the documentation. The documentation has been updated to cover 4.0 changes, including Enterprise features. You can view them online here:

• Apache manual
• Nginx manual

Final

If you would like to stay up to date with Phusion news, please fill in your name and email address below and sign up for our newsletter. We won’t spam you, we promise.

Dear users and customers,

About 3 weeks ago, our web host Linode issued several public statements[1][2] claiming that one of their customers was the subject of an attack by a group called HTP. From what we’ve been able to read from HTP[3] a few hours ago, we believe that SwiftIRC and/or nmap was the target Linode was referring to.

In Linode’s initial statement[1], they also mentioned that law officials were aware of the attack and that Linode had found no evidence of other customer data being compromised. We too hadn’t noticed any suspicious activity on our servers and weren’t notified by Linode about being the attacked target which led us to believe that this initial statement held true.

A few hours ago however, a statement released by HTP was brought to our attention wherein they claimed otherwise[3]. In particular, the statement appears to claim that HTP has had root access to one of the Phusion servers and this immediately prompted us to start a new investigation of our own. Up to this point, we have found no evidence that they have had access to our data, but we are checking our systems several times over to minimize the possibility of having missed a potential attack vector on the first few passes. We have also contacted Linode to get a clarification on their first statement[1] in light of new events that seem to point to nmap’s server to have indeed been compromised. Pending this response, we didn’t want to take any risks in waiting to notify our customers of the current situation.

The absence of evidence after all doesn’t necessarily mean that the server has not been accessed: even though we feel we have taken all the necessary steps to ensure maximum security on our servers, we remain scrutinous of our systems’ integrity at all times. There are after all a myriad of components that comprise a server, and each of them could be a potential attack vector as long as fault free software is something developers in general can only hope to aspire to. More specifically, as long as erring is human, we can only hope to minimize these chances rather than believing we can prevent them completely 100% of the time. Zero day exploits can always occur at any time and the best thing we can do is to be as transparent about this to our customers as we can. To that end, we’d like to notify our customers that we are moving our services to another web host and will be reinstalling our servers as a precaution.

If HTP has indeed compromised our systems without us being able to tell, then we would be interested in learning how and would encourage them to contact us (info@phusion.nl). We value security and transparency over pride and are extremely committed towards serving our customers. It is also the reason why we are informing our customers about this in an open manner several hours after seeing HTP’s claim despite not being able to verify this claim to be accurate ourselves.

We would also like to take this opportunity to encourage all Phusion Passenger users – that is, open source users and Enterprise customers alike – to make use of the PGP digital signatures that we employed since February this year.[4] Checking the signature of your Phusion Passenger download against the corresponding key helps minimize the chances of the downloaded software being tampered with. We have already manually reviewed the Phusion Passenger Enterprise source code and have found no evidence of suspicious activity. For your own safety however, we would always recommend you to take proper caution when downloading and installing software from the internet. The PGP digital signatures are provided to aid in that aspect and we would highly recommend you to use this at all times.

Having said this, if our servers actually were accessed, then it’s possible that the attackers temporarily inserted compromised gems and tarballs and removed them later. We therefore urge our Enterprise customers to verify the integrity of their Phusion Passenger Enterprise installations. Instructions can be found at the end of this message.

In any case, Phusion has not, does not and will not store customer creditcard information on any of its servers. All credit card information is stored on servers of third party, PCI-DSS compliant payment gateways, e.g. FastSpring and Paypal. Phusion also does not store customer passwords in plain text; all customer passwords are stored in BCrypt format.

The open source version of Phusion Passenger is hosted on another server, namely GitHub, and we have also found no suspicious activity in its repository.

We understand that after reading all this, you might have concerns with regards to your own server’s integrity. Even though we have found no evidence of suspicious activity on our own servers or in Phusion Passenger’s code base, we feel that we should still encourage you to remain scrutinous of your own servers’ integrity and take the steps you deem necessary in maximizing its security.

Needless to say, we remain committed in being transparent towards our customers and will continue in keeping them up to date of any of our findings concerning this matter. If you have any questions, please feel encouraged to contact support@phusion.nl.

With warm regards,
Hongli Lai
Ninh Bui

References:

1. https://blog.linode.com/2013/04/12/security-notice-linode-manager-password-reset/
2. https://blog.linode.com/2013/04/16/security-incident-update/
3. http://straylig.ht/zines/HTP5/0x02_Linode.txt
4. http://www.modrails.com/documentation/Users%20guide%20Apache.html#_cryptographic_verification_of_installation_files

Instructions for verifying Phusion Passenger Enterprise installations

We have generated SHA-1 hashes of all Phusion Passenger Enterprise files inside the gems and tarballs. You can use these hashes to verify your installed Phusion Passenger files. If anything is amiss or if you require further assistance, please contact support@phusion.nl.

1. Install GnuPG. Debian users can apt-get install gnupg, OS X users can use GPG Tools: https://gpgtools.org/
2. Login to the Customer Area: https://www.phusionpassenger.com/orders
3. Scroll down to the “Files” section.
4. Download the “sha1sums.txt” and “sha1sums.txt.asc” files that pertain to the version of Phusion Passenger Enterprise that you’re currently running. Ensure that both files are in the same directory.
5. Import the Phusion Software Signing PGP key: http://www.modrails.com/documentation/Users%20guide%20Apache.html#_importing_the_phusion_software_signing_key Name: Phusion Software Signing (software-signing@phusion.nl) Short key ID: 0x0A212A8C Long key ID: 0x2AC745A50A212A8C Fingerprint: D5F0 8514 2693 9232 F437 AB72 2AC7 45A5 0A21 2A8C
6. Set this key to trusted: gpg –edit-key software-signing@phusion.nl Then in the GPG prompt, type: trust Choose: 5 = I trust ultimately In the GPG prompt, type: save
7. Verify the downloaded sha1sums.txt against its signature: gpg –verify sha1sums.txt.asc You should see: Good signature from “Phusion Software Signing software-signing@phusion.nl“
8. Copy sha1sums.txt to your server.
9. On your server, find out where the Phusion Passenger files are by running: passenger-config –root
10. Run: cd
11. Run: sha1sum -c /path-to/sha1sums.txt –quiet

Phusion Passenger is software that deploys Ruby and Python web apps, by integrating into Apache and Nginx and turning them into a fully-featured application server. It is very fast, stable and robust and thus used by the likes of New York Times, AirBnB, Symantec, Pixar, etc. It comes with many features that makes your life easier and your application perform better.

After a period of being in beta, we’re proud to announce the first stable release of the Phusion Passenger 4 series. The 4.x series is a huge improvement over the 3.x series: during the development of 4.0, we’ve introduced a myriad of changes which we’ve covered in past beta preview articles:

• Support for multiple Ruby versions, support for Python WSGI, multithreading, evented core similar to Nginx and Node.js, real-time response buffering, improved zero-copy architecture, better error diagnostics.
• JRuby and Rubinius support.
• Out-of-Band Work.
• Support for the Rack socket hijacking API.
• Improved stability and test suite.
• Ruby 2.0 support.

The beta period took a while because we wanted to ensure that the first stable release is indeed rock solid. People tend to say that one should skip “x.0.0″ releases and wait until “x.0.1″ for the first bug fixes. But we’re confident enough about the stability of the 4.x series that we gave this first release the version number 4.0.1.

Changes in 4.0.1

Compared to 4.0.0 RC 6, the following changes have been introduced:

• Fixed a crasher bug in the Deployment Error Resistance feature.
• Fixed a bug in PassengerDefaultUser and PassengerDefaultGroup.
• Fixed a bug which could cause application processes to exit before they’ve finished their request.
• Fixed some small file descriptor leaks.
• Bumped the preferred Nginx version to 1.4.0.
• Editing the Phusion Passenger Standalone Nginx config template is no longer discouraged.
• Improved documentation.

Installing and testing 4.0.1

Quick install/upgrade

Phusion Passenger Enterprise users can download the Enterprise version of 4.0.1 from the Customer Area.

Open source users can install the open source version of 4.0.1 with the following commands:

gem install passenger
passenger-install-apache2-module
passenger-install-nginx-module

You can also download the tarball at Google Code. All our gems and tarballs can be cryptographically verified.

In-depth instructions

In-depth installation and upgrade instructions can be found in the Installation section of the documentation. The documentation has been updated to cover 4.0 changes, including Enterprise features. You can view them online here:

• Apache manual
• Nginx manual

Final

We would like to thank everybody who has helped with testing the betas and release candidates so far, and we would like to thank our Enterprise customers. We couldn’t have done it without you!

4.0.1 is just the beginning though. We have many excited changes on the pipeline. Want to stay up to date? Fill in your name and email address below and sign up for our newsletter. We won’t spam you, we promise.

Today we are pleased to announce Release Candidate 6 of Phusion Passenger 4.0. The 4.x series is a huge improvement over the 3.x series: during the development of 4.0, we’ve introduced a myriad of changes which we’ve covered in past beta preview articles:

• Support for multiple Ruby versions, support for Python WSGI, multithreading, evented core similar to Nginx and Node.js, real-time response buffering, improved zero-copy architecture, better error diagnostics.
• JRuby and Rubinius support.
• Out-of-Band Work.
• Support for the Rack socket hijacking API
• Improved stability and test suite.

Release Candidate 5 was a private interim release for Phusion Passenger Enterprise customers only.

Changes in 4.0 RC 5 and RC 6

The most important changes in RC 5 and RC 6 are as follows:

• The default config snippet for Apache has changed! It must now contain a PassengerDefaultRuby option. The installer has been updated to output this option. The PassengerRuby option still exists, but it’s only used for configuring different Ruby interpreters in different contexts. Please refer to the manual for more information.
• We now provide GPG digital signatures for all file releases by Phusion. More information can be found in the manual.
• WebSocket support on Nginx. Requires Nginx >= 1.3.15.
• passenger-status now displays process memory usage and time when it was last used. The latter fixes issue #853.
• Exceptions in Rack application objects are now caught to prevent application processes from exiting.
• The passenger-config tool now supports the --ruby-command argument, which helps the user with figuring out the correct Ruby command to use in case s/he wants to use multiple Ruby interpreters. The manual has also been updated to mention this tool.
• Fixed streaming responses on Apache.
• Worked around an OS X Unix domain socket bug. Fixes issue #854.
• Out-of-Band Garbage Collection now works properly when the application has disabled garbage collection. Fixes issue #859.
• Fixed support for /usr/bin/python on OS X. Fixes issue #855.
• Fixed looping-without-sleeping in the ApplicationPool garbage collector if PassengerPoolIdleTime is set to 0. Fixes issue #858.
• Fixed some process memory usage measurement bugs.
• Fixed process memory usage measurement on NetBSD. Fixes issue #736.
• Fixed a file descriptor leak in the Out-of-Band Work feature. Fixes issue #864.
• The PassengerPreStart helper script now uses the default Ruby interpreter specified in the web server configuration, and no longer requires a ruby command to be in $PATH.
• Updated preferred PCRE version to 8.32.
• Worked around some RVM bugs and generally improved RVM support.
• The ngx_http_stub_status_module is now enabled by default.
• Performance optimizations.

Installing and testing 4.0.0 Release Candidate 6

Quick install/upgrade

Phusion Passenger Enterprise users can download the Enterprise version of 4.0 RC 6 from the Customer Area.

Open source users can install the open source version of 4.0 RC 6 with the following commands:

gem install passenger --pre
passenger-install-apache2-module
passenger-install-nginx-module

You can also download the tarball at Google Code.

In-depth instructions

In-depth installation and upgrade instructions can be found in the Installation section of the documentation. The documentation has been updated to cover 4.0 changes, including Enterprise features. You can view them online here:

• Apache manual
• Nginx manual

Final

We are excited about the final release. You can help us by testing RC 6 and reporting any bugs. Please submit bug reports to our bug tracker.

Phusion Passenger turns Apache and Nginx into a full-featured application server for Ruby and Python web apps. It has a strong focus on ease of use, stability and performance. Phusion Passenger is built on top of tried-and-true, battle-hardened Unix technologies, yet at the same time introduces innovations not found in most traditional Unix servers. Since mid-2012, it aims to be the ultimate polyglot application server.

We recently received a support inquiry from a Phusion Passenger Enterprise customer regarding excessive process creation activity. During peak times, Phusion Passenger would suddenly create a lot of processes, making the server slow or unresponsive for a period of time. This is because Phusion Passenger spawns and shuts down application processes according to traffic, but they apparently had irregular traffic patterns during peak times. Since their servers were dedicated for 1 application only, the solution was to make the number of processes constant regardless of traffic. This could be done by setting PassengerMinInstances to a value equal to PassengerMaxPoolSize.

The customer then raised the question: what is the best value for PassengerMaxPoolSize? This is a non-trivial question, and the answer encompasses more than just PassengerMaxPoolSize. In this article we’re going to shed more light on this topic.

For simplicity reasons, we assume that your server only hosts 1 web application. Things become more complicated when more web applications are involved, but you can use the principles in this article to apply to multi-application server environments.

Aspects of concurrency tuning

The goal of tuning is usually to maximize throughput. Increasing the number of processes or threads increases the maximum throughput and concurrency, but there are several factors that should be kept in mind.

• Memory. More processes implies a higher memory usage. If too much memory is used then the machine will hit swap, which slows everything down. You should only have as many processes as memory limits comfortably allow. Threads use less memory, so prefer threads when possible. You can create tens of threads in place of one process.

• Number of CPUs. True (hardware) concurrency cannot be higher than the number of CPUs. In theory, if all processes/threads on your system use the CPUs constantly, then:

  • You can increase throughput up to NUMBER_OF_CPUS processes/threads.
  • Increasing the number of processes/threads after that point will increase virtual (software) concurrency, but will not increase true (hardware) concurrency and will not increase maximum throughput.

  Having more processes than CPUs may decrease total throughput a little thanks to context switching overhead, but the difference is not big because OSes are good at context switching these days.

  On the other hand, if your CPUs are not used constantly, e.g. because they’re often blocked on I/O, then the above does not apply and increasing the number of processes/threads does increase concurrency and throughput, at least until the CPUs are saturated.

• Blocking I/O. This covers all blocking I/O, including hard disk access latencies, database call latencies, web API calls, etc. Handling input from the client and output to the client does not count as blocking I/O, because Phusion Passenger has buffering layers that relief the application from worrying about this.

  The more blocking I/O calls your application process/thread makes, the more time it spends on waiting for external components. While it’s waiting it does not use the CPU, so that’s when another process/thread should get the chance to use the CPU. If no other process/thread needs CPU right now (e.g. all processes/threads are waiting for I/O) then CPU time is essentially wasted. Increasing the number processes or threads decreases the chance of CPU time being wasted. It also increases concurrency, so that clients do not have to wait for a previous I/O call to be completed before being served.

With these in mind, we give the following tuning recommendations. These recommendations assume that your machine is dedicated to Phusion Passenger. If your machine also hosts other software (e.g. a database) then you should look at the amount of RAM that you’re willing to reserve for Phusion Passenger and Phusion Passenger-served applications.

Tuning the application process and thread count

In our experience, a typical single-threaded Rails application process uses 100 MB of RAM on a 64-bit machine, and by contrast, a thread would only consume 10% as much. We use this fact in determining a proper formula.

Step 1: determine the system’s limits

First, let’s define the maximum number of (single-threaded) processes, or the number of threads, that you can comfortably have given the amount of RAM you have. This is a reasonable upper limit that you can reach without degrading system performance. We use the following formulas.

In purely single-threaded multi-process deployments, the formula is as follows:

max_app_processes = (TOTAL_RAM * 0.75) / RAM_PER_PROCESS

This formula is derived as follows:

• (TOTAL_RAM * 0.75): We can assume that there must be at least 25% of free RAM that the operating system can use for other things. The result of this calculation is the RAM that is freely available for applications.
• / RAM_PER_PROCESS: Each process consumes a roughly constant amount of RAM, so the maximum number of processes is a single devision between the aforementioned calculation and this constant.

In multithreaded deployments, the formula is as follows:

max_app_threads_per_process =
  ((TOTAL_RAM * 0.75) - (NUMBER_OF_PROCESSES * RAM_PER_PROCESS * 0.9)) /
  (RAM_PER_PROCESS / 10)

Here, NUMBER_OF_PROCESSES is the number of application process you want to use. In case of Ruby or Python, this should be equal to NUMBER_OF_CPUS. This is because both Ruby and Python have a Global Interpreter Lock so that they cannot utilize multicore no matter how many threads they’re using. By using multiple processes, you can utilize multicore. If you’re using a language runtime that does not have a Global Interpreter Lock, e.g. JRuby or Rubinius, then NUMBER_OF_PROCESSES can be 1.

This formula is derived as follows:

• (TOTAL_RAM * 0.75): The same as explained earlier.
• - (NUMBER_OF_PROCESSES * RAM_PER_PROCESS): In multithreaded deployments, the application processes consume a constant amount of memory, so we deduct this from the RAM that is available to applications. The result is the amount of RAM available to application threads.
• / (RAM_PER_PROCESS / 10): A thread consumes about 10% of the amount of memory a process would, so we divide the amount of RAM available to threads with this number. What we get is the number of threads that the system can handle.

On 32-bit systems, max_app_threads_per_process should not be higher than about 200. Assuming an 8 MB stack size per thread, you will run out of virtual address space if you go much further. On 64-bit systems you don’t have to worry about this problem.

Step 2: derive the applications’ needs

The earlier two formulas were not for calculating the number of processes or threads that application needs, but for calculating how much the system can handle without getting into trouble. Your application may not actually need that many processes or threads! If your application is CPU-bound, then you only need a small multiple of the number of CPUs you have. Only if your application performs a lot of blocking I/O (e.g. database calls that take tens of milliseconds to complete, or you call to Twitter) do you need a large number of processes or threads.

Armed with this knowledge, we derive the formulas for calculating how many processes or threads we actually need.

• If your application performs a lot of blocking I/O then you should give it as many processes and threads as possible:

  # Use this formula for purely single-threaded multi-process deployments.
  desired_app_processes = max_app_processes
  
  # Use this formula for multithreaded deployments.
  desired_app_threads_per_process = max_app_threads_per_process
  

• If your application doesn’t perform a lot of blocking I/O, then you should limit the number of processes or threads to a multiple of the number of CPUs to minimize context switching:

  # Use this formula for purely single-threaded multi-process deployments.
  desired_app_processes = min(max_app_processes, NUMBER_OF_CPUS)
  
  # Use this formula for multithreaded deployments.
  desired_app_threads_per_process = min(max_app_threads_per_process, 2 * NUMBER_OF_CPUS)
  

Step 3: configure Phusion Passenger

You should put the number for desired_app_processes into the PassengerMaxPoolSize option. Whether you want to make PassengerMinInstances equal to that number or not is up to you: doing so will make the number of processes static, regardless of traffic. If your application has very irregular traffic patterns, response times could drop while Passenger spins up new processes to handle peak traffic. Setting PassengerMinInstances as high as possible prevents this problem.

If desired_app_processes is 1, then you should set PassengerSpawnMethod conservative (on Phusion Passenger 3 or earlier) or PassengerSpawnMethod direct (on Phusion Passenger 4 or later). By using direct/conservative spawning instead of smart spawning, Phusion Passenger will not keep an ApplicationSpawner/Preloader process around. This is because an ApplicationSpawner/Preloader process is useless when there’s only 1 application process.

In order to use multiple threads you must use Phusion Passenger Enterprise 4. The open source version of Phusion Passenger does not support multithreading, and neither does version 3 of Phusion Passenger Enterprise. At the time of writing, Phusion Passenger Enterprise 4.0 is on its 4th Release Candidate. You can download it from the Customer Area.

You should put the number for desired_app_threads_per_process into the PassengerThreadCount option. If you do this, you also need to set PassengerConcurrencyModel thread in order to turn on multithreading support.

Possible step 4: configure Rails

Only if you’re on a multithreaded deployment do you need to configure Rails.

Rails is thread-safe since version 2.2, but you need to enable thread-safety by setting config.thread_safe! in config/environments/production.rb.

You should also increase the ActiveRecord pool size because it limits concurrency. You can configure it in config/database.yml. Set the pool size to the number of threads. But if you believe your database cannot handle that much concurrency, keep it at a low value.

Example 1: purely single-threaded multi-process deployment with lots of blocking I/O

Suppose you have 1 GB of RAM and lots of blocking I/O, and you’re on a purely single-threaded multi-process deployment.

# Use this formula for purely single-threaded multi-process deployments.
max_app_processes = (1024 * 0.75)  / 100 = 7.68
desired_app_processes = max_app_processes = 7.68

Conclusion: you should use 7 or 8 processes. Phusion Passenger should be configured as follows:

PassengerMaxPoolSize 7

However a concurrency of 7 or 8 is way too low if your application performs a lot of blocking I/O. You should use a multithreaded deployment instead, or you need to get more RAM so you can run more processes.

Example 2: multithreaded deployment with lots of blocking I/O

Consider the same machine and application (1 GB RAM, lots of blocking I/O), but this time you’re on a multithreaded deployment with 2 application processes. How many threads do you need per process?

Let’s assume that we’re using Ruby and that we have 4 CPUs. Then:

# Use this formula for multithreaded deployments.
max_app_threads_per_process
= ((1024 * 0.75) - (4 * 100)) / (100 / 10)
= 368 / 10
= 36.8

Conclusion: you should use 4 processes, each with 36-37 threads, so that your system ends up with . Phusion Passenger Enterprise should be configured as follows:

PassengerMaxPoolSize 4
PassengerConcurrencyModel thread
PassengerThreadCount 36

Configuring the web server

If you’re using Nginx then it does not need configuring. Nginx is evented and already supports a high concurrency out of the box.

If you’re using Apache, then prefer the worker MPM (which uses a combination of processes and threads) or the event MPM (which is similar to the worker MPM, but better) over the prefork MPM (which only uses processes) whenever possible. PHP requires prefork, but if you don’t use PHP then you can probably use one of the other MPMs. Make sure you set a low number of processes and a moderate to high number of threads.

Because Apache performs a lot of blocking I/O (namely HTTP handling), you should give it a lot of threads so that it has a lot of concurrency. The number of threads should be at least the number of concurrent clients that you’re willing to serve with Apache. A small website can get away with 1 process and 100 threads. A large website may want to have 8 processes and 200 threads per process (resulting in 1600 threads in total).

If you cannot use the event MPM, consider putting Apache behind an Nginx reverse proxy, with response buffering turned on on the Nginx side. This reliefs a lot of concurrency problems from Apache. If you can use the event MPM then adding Nginx to the mix does not provide many advantages.

Conclusion

• If your application performs a lot of blocking I/O, use lots of processes/threads. You should move away from single-threaded multiprocessing in this case, and start using multithreading.
• If your application is CPU-bound, use a small multiple of the number of CPUs.
• Do not exceed the number of processes/threads your system can handle without swapping.

• Environments that host arbitrary untrusted applications, e.g. shared hosting environments.
• Applications which contain vulnerabilities that allow their own code to be modified.
• Environments in which untrusted non-root users can modify application code.

Affected users are advised to upgrade to 4.0.0 RC 4.

Affected versions

• Phusion Passenger open source 4.0.0 beta 1
• Phusion Passenger open source 4.0.0 beta 2
• Phusion Passenger Enterprise 4.0.0 beta 1
• Phusion Passenger Enterprise 4.0.0 beta 2

Unaffected versions

• Phusion Passenger open source 3.x and earlier
• Phusion Passenger open source 4.0.0 RC 1 and later
• Phusion Passenger Enterprise 3.x and earlier
• Phusion Passenger Enterprise 4.0.0 RC 1 and later

Today we are pleased to announce Release Candidate 4 of Phusion Passenger 4.0. Last week we said that the open source release of Release Candidate 1 will be out today. However because of the helpful feedback and bug reports we’ve received from Enterprise customers, we’ve decided to push out these bug fixes to the open source version earlier. Release Candidate 3 was only available for Enterprise customers in order to test bug fixes, so it hasn’t been announced publicly.

The 4.x series is a huge improvement over the 3.x series: during the development of 4.0, we’ve introduced a myriad of changes which we’ve covered in past beta preview articles:

• Support for multiple Ruby versions, support for Python WSGI, multithreading, evented core similar to Nginx and Node.js, real-time response buffering, improved zero-copy architecture, better error diagnostics.
• JRuby and Rubinius support.
• Out-of-Band Work.
• Support for the Rack socket hijacking API
• Improved stability and test suite.

Changes in 4.0 RC 3 and RC 4

The focus of RC 3 and RC 4 have yet again been on improving stability. We’ve closed over 50 issues in our issue tracker.

The most important changes in RC 3 and RC 4 are as follows:

• Fixed Rake autodetection.
• Fixed compilation on systems where /tmp is mounted noexec.
• Fixed some memory corruption bugs.
• Phusion Passenger Standalone now sets underscores_in_headers. Fixes issue #708.
• Fixed some process spawning compatibility problems, as reported in issue #842.
• The Python WSGI loader now correctly shuts down client sockets even when there are child processes that keep the socket open.
• A new configuration option PassengerPython (Apache) and passenger_python (Nginx) has been added so that users can customize the Python interpreter on a per-application basis. Fixes issue #852.
• The Apache module now supports file uploads larger than 2 GB when on 32-bit systems. Fixes issue #838.
• The Nginx version now supports the passenger_temp_dir option.
• Environment variables set in the Nginx configuration file (through the env config option) are now correctly passed to all application processes. Fixes issue #371.
• Fixed support for RVM mixed mode installations. Fixes issue #828.
• Phusion Passenger now outputs the Date HTTP header in case the application didn’t already do that (and was violating the HTTP spec). Fixes issue #485.
• Phusion Passenger now checks whether /dev/urandom isn’t broken. Fixes issue #516.
• Improved debugging messages.

Installing and testing 4.0.0 Release Candidate 4

Quick install

Phusion Passenger Enterprise users can download the Enterprise version of 4.0 RC 4 from the Customer Area.

Open source users can install the open source version of 4.0 RC 4 with the following commands:

gem install passenger --pre
passenger-install-apache2-module
passenger-install-nginx-module

You can also download the tarball at Google Code.

In-depth

In-depth installation and upgrade instructions can be found in the Installation section of the documentation. The documentation has been updated to cover 4.0 changes, including Enterprise features. You can view them online here:

• Apache manual
• Nginx manual

Final

We are excited about the final release. You can help us by testing RC 4 and reporting any bugs. Please submit bug reports to our bug tracker.

We know many users are eagerly awaiting the final release of Phusion Passenger 4.0. The 4.x series is a huge improvement over the 3.x series: during the development of 4.0, we’ve introduced a myriad of changes which we’ve covered in past beta preview articles:

• Beta 1: support for multiple Ruby versions, support for Python WSGI, multithreading (an Enterprise only feature), evented core similar to Nginx and Node.js, real-time response buffering, improved zero-copy architecture, better error diagnostics.
• Beta 2:
  • JRuby and Rubinius support.
  • Out-of-Band Work.
  • Support for the Rack socket hijacking API
  • Improved stability and test suite.

Today we are proud to announce Release Candidate 2 of Phusion Passenger 4.0. Release Candidate 1 has been skipped because a few bug fixes were applied right after RC 1 was tagged.

Changes in 4.0 RC 1 and RC 2

The focus of RC 1 and RC 2 have been on improving stability and on refining previously introduced features. We’ve closed over 100 issues in our issue tracker. We couldn’t have done this without the fantastic feedback from our users, especially those from many Phusion Passenger Enterprise customers who have beta tested the RC previews in their staging environments.

The changes in RC 1 and RC 2 are as follows:

• The Nginx version now supports the passenger_app_root configuration option.
• The Enterprise memory limiting feature has been extended to work with non-Ruby applications as well.
• Application processes that have been killed are now automatically detected within 5 seconds. Previously Phusion Passenger needed to send a request to the process before detecting that it’s gone. This change means that when you kill a process by sending it a signal, Phusion Passenger will automatically respawn it within 5 seconds (provided that the process limit settings allow respawning).
• Phusion Passenger Standalone’s HTTP client body limit has been raised from 50 MB to 1 GB.
• Python 3 support has been added.
• The build system has been made compatible with JRuby and Ruby 2.0. This does not mean that Phusion Passenger works on Ruby 2.0; please read on for more about this subject.
• The installers now print a lot more information about detected system settings so that the user can see whether something has been wrongly detected.
• Some performance optimizations. These involve further extending the zero-copy architecture, and the use of hash table maps instead of binary tree maps.
• Many potential crasher and freezer bugs have been fixed.
• Error diagnostics have been further improved.
• Many documentation improvements.

What about Ruby 2.0?

We are just as excited about Ruby 2.0 as many of you are. Since 2.0 was released a few days ago, we’ve been testing Phusion Passenger on it. We really wanted to release RC 2 with Ruby 2.0 support, but a few things stood in our way so we had to postpone this goal.

• We couldn’t get Ruby 2.0.0 installed on OS X Mountain Lion. The compiled Ruby crashes during Ruby 2.0.0′s build process with a low-level error ([BUG] Stack consistency error). Apparently we aren’t the only ones.
• We were able to get it installed on a Debian VM, but it does not pass all the Phusion Passenger unit tests. It fails on some tests with obscure errors that seem to indicate bugs in Ruby, e.g. errors in which Ruby cannot figure out where the exception came from.

We recommend sticking with 1.9.3 in the mean time until the next Ruby 2.0 patchlevel release.

Release Candidate 2 timeline & download

Phusion Passenger Enterprise customers are given priority access to Release Candidate 2. They can download RC 2 from the Customer Area immediately.

The release of the open source version will follow in one week, on March 5 2013. Of course, open source users who want to stay on the bleeding edge are free to obtain the latest sources from the open source Phusion Passenger git repository at any time.

When the open source version is released, users can install it by following the in-depth installation and upgrade instructions in the Installation section of the documentation. The manual also covers installation of beta releases.

• Apache manual
• Nginx manual

Final

We are excited about the final release. You can help us by testing RC 2 and reporting any bugs. Please submit bug reports to our bug tracker.

Development of the Phusion Passenger 4.x series is progressing steadily. The 4.x series is a huge improvement over the 3.x series: in the announcement for Phusion Passenger 4.0.0 beta 1, we introduced a myriad of changes such as support for multiple Ruby versions, Python WSGI support, multithreading (Enterprise only), improved zero-copy architecture, better error diagnostics and more. That was just the beginning, because soon after we announced JRuby and Rubinius support, Out-of-Band Work and the Rack socket hijacking API.

Today we are proud to announce Phusion Passenger 4.0 beta 2, which brings us closer to a final release.

Better stability, documentation, test coverage

Beta 1 was usable, but not yet production-ready. While it worked well most of the time, there were some bugs that could cause crashes. So for beta 2 we haven’t introduced too many new features. Instead we’ve been focussing a lot on fixing bugs, improving stability, improving documentation and improving test coverage. It is easy to fall into the trap of constantly adding features, but we want a rock-solid product that our users and customers can rely on.

How do we ensure quality? There are a few tools and techniques that we use:

System call failure simulation framework

Our inspiration

The system call failure simulation framework is a new developer feature in 4.0 beta 2 and allows us to simulate random system call failures so that we can test whether error handling in Phusion Passenger is done correctly. Although we already test for many error handling scenarios in our unit tests, test coverage is not perfect. This framework gives us another tool to ensure quality.

A few months ago we sat down with a customer who was experiencing seemingly random crashes with Phusion Passenger. These crashes could not be reproduced on any of our systems, but could be reliably reproduced on theirs. The crashes would only manifest under high concurrency scenarios. After a day of intensive investigation, we found that the crash was caused because their systems’ file descriptor limit is much lower than any of our systems’. Phusion Passenger did not always catch out-of-file-descriptors errors, so those errors caused Phusion Passenger to crash. Due to other unrelated issues, relevant error messages could not be printed to the log file and were lost.

All of those issues have since been fixed, but it made us realize that our testing tools were not adequate. That situation could and should have been prevented. Thus, the system call failure simulation framework was born. This framework allows us to specify which system calls should fail, and with what probability. For example, the following configuration simulates the “out of file descriptors” error in the helper agent with a probability of 1%.

export PASSENGER_SIMULATE_SYSCALL_FAILURES=PassengerHelperAgent=ENFILES=0.01

Different runs will produce different errors, but you can force determinism by specifying the same random seed that was used in the last run. The random seed is printed as a debugging message during startup and during crash.

export PASSENGER_RANDOM_SEED=...

The system call failure simulation framework was inspired by SQLite’s testing process. Real hardware, network or OS-level errors are difficult to create, so simulating them is the next best thing. SQLite has an internal virtual filesystem layer, and it is in that layer that they simulate failures. In our case we have a similar layer, namely the system call interruption framework which was originally written to facilitate interrupting threads that are blocked on blocking system calls.

Continuously expanding and improving our test suite

We already had an extensive test suite which consists of a hybrid of C++ and Ruby RSpec code. In 4.0 beta 2 we’ve improved the test suite by modernizing some dependencies, testing more edge cases, testing more failure conditions, etc.

Setting up Continuous integration

Before today our extensive test suite was run on our development machines as well as an army of virtual machines with different OSes. We have now setup Travis CI so that we would have an additional quality assurance tool. The test suite has also been extended to cover more cases.

Ruby 1.8 is now considered legacy

Ruby 1.8 is no longer supported by its authors, and Ruby Enterprise Edition has been End-Of-Lifed a while ago. Many gems these days are Ruby 1.9-only. It is more than apparent that Ruby 1.8 is considered legacy by the community, and for good reasons. We too are joining the community by considering Ruby 1.8 legacy. This has the following implications:

• Phusion Passenger 4.x will continue to support Ruby 1.8. Our support goes as far back as Ruby 1.8.5.
• We will optimize performance for Ruby 1.9. Phusion Passenger will still work on Ruby 1.8, but we will no longer put in any effort to make it work fast on Ruby 1.8.

Installing and testing 4.0.0 beta 2

Quick install

Phusion Passenger Enterprise users can download the Enterprise version of 4.0 beta 2 from the Customer Area.

Open source users can install the open source version of 4.0 beta 2 with the following commands:

gem install passenger --pre
passenger-install-apache2-module
passenger-install-nginx-module

You can also download the tarball at Google Code.

In-depth

In-depth installation and upgrade instructions can be found in the Installation section of the documentation. The documentation has been updated to cover 4.0 changes, including Enterprise features. You can view them online here:

• Apache manual
• Nginx manual

Final

We are excited about the final release. You can help us by testing beta 2 and reporting any bugs. Please submit bug reports to our bug tracker.