Phusion white papers Phusion overview

Re: EngineYard’s recent post about Phusion Passenger

By Ninh Bui on December 6th, 2008

At Phusion, we’ve received some comments from Engine Yard customers who have mentioned that upon requesting support for Passenger, they were advised to stick with an Nginx + Mongrel setup instead. Even though Passenger is our flagship product, I also have the utmost respect for both Mongrel and Nginx, and believe that people should choose the solution they deem most suitable and pragmatic for the job. In light of this, I believe Passenger and Apache are just as viable a solution for “the deployment problem” as the aforementioned solutions. Unfortunately, after inquiring a bit further, I was unable to get a clear answer as to why they were advised by Engine Yard to not use Passenger specifically.

After reading some responses on Ezra’s talk on Rails deployment, and in particular, after reading this summary by an attendee of this talk, the impression seemed to be made that Passenger was only, if not more suitable, for small VPS and/or shared hosting environments. Unfortunately, I was unable to attend this talk myself but luckily, Ezra has made his slides publicly available on his blog.

Here, slide 13 states “Passenger(with caveats)” and “State of art now: Passenger for shared hosting/small VPS”. Clearly, we’re unable to infer what is being said here without the proper narrative/annotations which unfortunately isn’t available. This leaves room for interpretation and the slides on their own seem to suggest that Passenger “has caveats” and is the best solution for shared hosting/small VPS. Even though it seems to do a great job in these fields, it’s certainly not the only goal we had in mind when we were designing and engineering Passenger.

Taking this into consideration, I felt inclined to further investigate and asked Ezra for some elaboration in the comments section of the aforementioned blogs. Here it soon became apparent that Ezra seemed to have problems with the memory consumption of Apache and Ruby Enterprise Edition(REE) not giving them memory savings on 64bit platforms but segfaulting instead.

As I’ve pointed out earlier in the comments section of Ezra’s blog, I believe it’s one thing (and very unfortunate) that saying one solution isn’t working out for Engine Yard, but it’s an entirely different matter to give the impression that is therefore more suitable for VPS/small hosting. Especially if no specifics are or have ever been given about this potential problem in order for people like us to try to reproduce it. After asking for these specifics, Ezra commented that he would contact us on this, but Ezra is probably a super busy man so we unfortunately didn’t receive anything on this as of yet. The just recently released Ruby Enterprise Edition however, contains fixes that should increase the support for 64bit and we hope that they will solve any problems people might have with this on their 64bit platform.

Update:
It seems Ezra sent us an email at around the same time this reply was posted which contents now seems to indicate a working REE setup on their 64 bit platform! :-) I’d still be interested in learning more about their test suite though which would allow us verify a working setup through reproduction, and I’m sure the community would benefit from having this kind of knowledge as well.

Many thanks to the following people and organizations for making this possible through sponsorship (list sorted in alphabetical order):

After releasing both Passenger 2.0.5 and Ruby Enterprise Edition, Tom Mornini from Engine Yard elaborated a bit more on the reasons why Engine Yard is currently favoring Nginx + Mongrel over Passenger + Apache. Seeing as some reasons are given with regards to Passenger and Apache, I’d like to go over these to try to get the entire story across for you to decide whether or not it is suitable for you.

First off, Tom writes:

“If you’re running multiple Rails applications, and are using Passenger’s ability to serve more applications than applications processes, you need to be aware of the performance of requests requiring an application process switch from one application to another. First requests are always slow, as a *lot* of code needs to be heated up, etc.”

Here, I believe the very first thing to note is the first word Tom uses in this paragraph, being “if”. “If” you are running multiple Rails applications and are using Passenger’s ability to serve more applications than application processes, then yes, you need to be aware of the possibility of the performance of requests requiring an application switch from one application to another. The reason why I’ve emphasized possibility here is that you need to have more concurrent requests in progress than application processes in the first place in order to be able to get into such a situation. Having said that, one could just as easily increase the allowed application instances Passenger is allowed to spawn to the number of applications we have and increase the PassengerPoolIdleTime to a sufficiently large number so that they will virtually live as long as you’ll ever need. In git master, we’ve made this solution a bit more elegant by providing a sugar that allows you to assign a negative value to PassengerPoolIdleTime, which will result in it “living infinitely long” and you can expect this to be in 2.0.6. Doing something like this will essentially give you the same behavior as having and keeping multiple mongrel instances alive behind a proxy balancer. The key thing to note here however, is that we actually allow you to choose for an alternative setup as well to accommodate your needs even better. Just as there are situations in which you want the behavior Tom is describing, there are situations where it is perhaps deemed more suitable to have less application processes than actual applications (e.g. where memory is limited). The important thing to remember here however, is that Passenger allows you to do both (very easily). Doing the aforementioned with Passenger requires you to just modify two numeric values in the Apache configuration (PassengerPoolIdleTime and PassengerMaxPoolSize) and doesn’t involve manual port management for example which in general, is necessary for proxy-load-balanced-mongrel setups. Using Passenger to establish the same setup as proxy-load-balanced-mongrel setups involves significantly less steps, making it arguably less tedious and less error prone to maintain. Unfortunately, this part isn’t elaborated on in Tom’s post, which could result in people being left only half informed.

Also, Tom has made mention of Passenger possibly needing to load a “lot” of code. Even though Tom acknowledges that Ruby Enterprise Edition *does* offer the additional advantage of memory savings, there doesn’t seem to be an explicit mention of the fact that Ruby Enterprise Edition negates “this need to load a lot of code” dramatically/entirely by leveraging a technique called Copy-on-Write which is available on a myriad of *nix variants. Copy-on-Write works on an operating systems level so if you already have a running Rails process, by the time you need to spawn a new one, the operating system is able to infer that most –if not all– memory can be pooled and doesn’t need to be replicated. This in particular is the case for identical applications, where not only the Rails framework will be shared among processes but also the application code.

Moving on, Tom writes:

“The issues I’m discussing are generally acceptable in a situation where you have many low traffic applications, and you need to run those applications in a very limited environment. At Engine Yard, this is a very rare need for our customers, so this makes us less anxious to switch to something new -vs- something that is proven and works well.”

Even though just shy of 8 months in production status, a growing number of very large sites (see here and here) are using REE and Passenger to successfully serve millions upon millions of request per day respectively. I’m not sure what one would need to do to get it to a point of being considered “proven and working well”, but what I do however know is that sites by MTV, Yammer, Geni, Shopify and 37signals are powered by Passenger and seem to have no problems dealing with high-volume requests.

Moving on, Tom writes:

“With the additional of Ruby Enterprise Edition, however, it *does* offer the additional advantage of memory savings. Some of those advantages are lost by having to run Apache, which as commonly delivered, is considerably larger and less efficient than nginx. Of course, you can reduce these disadvantages by cuting down a custom build and tweaking the included modules, but once you’re headed down that road, the simplicity advantage has left the building. :-)”

Even though we’ve also inferred that an “out-of-the-box” Apache installation incurs a larger overhead than an “out-of-the-box” Nginx, it begs the question if this is really relevant in this article. After all, I was under the impression that Tom was elaborating about why Engine Yard isn’t using Passenger + Apache, where he also mentions that one can configure Apache to get comparable results to Nginx, effectively giving the answer to Engine Yard’s own “problems” with Apache.

From our experience, configuring an Apache setup to “perform on par with Nginx” isn’t trivial, but it’s very doable. Also note that this initial time one spends on setting up such an Apache installation is being rewarded by not having to maintain a relatively more difficult setup with nginx+mongrel. Here, you need to ask yourself how much time you spend on doing an Apache setup as opposed to maintaining the actual installation, i.e. how many times you update Apache as opposed to maintaining and updating Passenger. In general, from our experience, the former doesn’t take as nearly as much time as the latter. To get a sense of what I’m trying to get at, please take a look at the Apache site where you can see that the previous and most current version were released about 9 months apart from each other by the time of this writing. If we also take into account that in general, your old Apache configuration files will probably still work after doing an Apache upgrade, then I’m not sure if this really is such a big problem as it is made out to be. Especially with the solution we’re currently working on, which I hope to be able to elaborate a bit more about soon.

One thing is for sure however, heavily visited Passenger powered sites like Shopify which processes millions upon millions of requests per day
certainly don’t seem to have a problem with neither memory or performance and benefit greatly from this approach.

In particular, you might find the following quote from Tobias Lütke of Shopify fame
to be of particular interest:

“We switched to enterprise ruby to get the full benefit of the [copy-on-write] memory characteristics and we can absolutely confirm the memory savings of 30% some others have reported. This is many thousand dollars of savings even at today’s hardware prices.”

Moving on, Tom writes:

“Second, our customers choose Engine Yard for experience and expertise in hosting Rails applications. At this moment, we have no experience or expertise in Passenger™.
That is a fact.”

Well fortunately, Phusion offers commercial support for Passenger so this doesn’t necessarily have to be a problem. ;-) More interestingly however, is that even though Engine Yard claims to not have experience or expertise in Passenger, they seem to be perfectly capable of making some strong statements about it anyway as found on Ezra’s slides and Tom’s blog post. This also, is a fact.

In closing, somewhere in the middle of the article, Tom writes:

“The reason that I felt compelled to respond to Pratik’s Twitter post is that it entirely ignores a few issues that prevent Engine Yard, and no doubt other companies from immediately supporting Passenger™ and/or Ruby Enterprise Edition.”

In a similar fashion, I felt inclined to respond to Tom’s article seeing as it completely seemed to ignore a few (easy) solutions as well.

The morale of the story: feel encouraged to try out different solutions to find out which one suits you best, regardless of its current life-span, because that’s the only way of helping a product mature. In any case, please remain critical of information you receive, and yes, that even means this blog post. Seeing is believing right? Whatever the case may be, stay tuned for more exciting news from Phusion. :-)

  • http://trevorturk.com Trevor Turk

    I was a bit surprised to hear that Engine Yard hasn’t spent time with Passenger yet, especially considering the amount of attention it’s been getting from 37signals lately. I trust that they’ll do their due diligence, though, and I’m looking forward to hearing what they have to say about their experience!

  • Scott Becker

    “From our experience, configuring an Apache setup to ‘perform on par with Nginx’ isn’t trivial, but it’s very doable….Especially with the solution we’re currently working on, which I hope to be able to elaborate a bit more about soon.” – What’s that, do I hear an Apache Enterprise Edition coming on? :)

  • Rich Kilmer

    Ninh,

    Thanks for this post. We have been using Passenger for 6 months and are very happy with it for a few sites/services that we run. We also have several customers either using or moving to use Passenger. One customer in particular (because the the usage pattern of their SaaS) would have been a total pain to build a production deployment solution for without Passenger. This is the reason we supported you guys with the latest release, and will continue to do so.

    Ultimately, if hosting providers don’t provide the deployment options that their customers ask for, those customers will churn to other hosting providers that do. It’s going to be their loss at the end of the day. For anything we work on there has to be a darned good reason to use something other than Passenger. Keep up the great work.

  • http://tomcopeland.blogs.com Tom Copeland

    mongrel was a big improvement over mod_fcgi, but after using Passenger there’s no way I’m going back to mucking around with Apache ProxyBalancers, port ranges, stale pid files, symlinking mongrel_cluster.yml from /etc/mongrel_cluster/myapp/blah, separate init scripts for each app, monit, and all that. Passenger is awesome – it dramatically reduces the number of moving parts. And it just keeps getting better. Keep up the great work, guys!

  • http://cligs.ee Egze

    Great post! I was very disappointed that Engine Yard was bashing Passenger just because it doesn’t fit in their business model.

  • http://sikachu.com Sikachu!

    I belive in one thing: Time will tell :)

  • Richy

    People using nginx are not willing to change back to apache. Howerver, people are forced to use apache if they choose passenger instead of mongrel cluster.
    Having to give up nginx is really unfortunate.

  • Fadhli Rahim

    I’m really interested in seeing the EngineYard battled tested test suit for hosting rails app. I have one application running on a 64bit unix ubuntu and Passenger 2.0.5 equipped with the latest REE. So far no segment fault issues.

  • Riptie

    I’m hosted with EY and I’m disappointed by their entire approach to this. Clearly Passenger is something they, as the current leading Rails host should support (in many ways). I can only speculate that it’s because Passenger beat them to the punch in what they hope to accomplish with mod_rubinius. Things of the future just don’t solve current problems.

  • Pingback: Weekly Digest, 12-7-08 - almost effortless()

  • http://engineyard.com Ezra

    Guys I think everyone needs to calm down here. We have nothing against passenger and in fact my testing with the latest release looks very promising on our 64 bit gentoo servers. The fact of the matter is that until yesterday’s release it was not really a viable option for us due to the 64 bit issues.

    But with the new release I have passenger in a set of rigorous testing right now looking toward supporting it at engineyard in the very near future. People need to realize that we are running thousands of servers right now and it is not an easy overnight switch to move to Passenger. Also we have served *billions* of requests on our current stack and so it requires a lot of testing before we can make a dramatic switch like this.

    Rest assured EngineYard *will* be supporting passenger in the near term future. I already have support for it baked into my new EY on AWS offering that is in beta.

    Passenger guys, you have done great work but please calm down a bit. No one is attacking you or your awesome contribution to the community. And in fact I have been emailing you guys about my testing of passenger on 64 bit as well as some other questions I have, still haven’t gotten a response, do you guys check info@phusion.nl emails?

    So I think the whole community needs to settle down about this. Passenger will be supported by EY in the future, it is just a big transition for us and we will also continue to support our battle tested nginx stack for the forseeable future.

    I think there is room for both stacks as they both have strengths and weaknesses. There are some very compelling nginx modules that passenger has no answer to like the upload module that really help out making scalable upload handlers. There are also a very few minor issues with passenger+REE and some C extensions not working as well as class load issues that we have seen on occasion so nothing is perfect.

    In the end I think everyone is getting upset over nothing. I expect some folks to stick with nginx+thin/mongrel and I fully expect we will support passenger in the near future so folks can make their own choices. For now I;d love to see more folks making decisions based on their own testing and benchmakring rather then on what DHH or I say on the internets.

    Trust but verify. Don;t follow someone’s suggestions without weighing the options and truly understanding the pros and cons of the various approaches.

    That’s all I got to say for now.

  • http://engineyard.com Ezra

    How much money would the phusion guys need to make an nginx version of passenger? I’d happily pitch in a good chunk of cash to make that happen. Contact me offline at ez@engineyard.com if you want to talk further about this one, I’m really interested.

  • http://chopine.be/lrz Laurent Sansonetti

    Ezra: I am curious, what kind of problems did you get when running Passenger on 64-bit? I have been testing it successfully here on IA64 with a 64-bit version of Apache also. It is my understanding that REE was lacking some 64-bit support (until a couple of days ago), but not Passenger.

  • http://engineyard.com Ezra

    Laurent- Maybe I’ve conflated two issues. I have no issues with passenger on 64 bit, its with REE and it appears to be fixed with the latest release. And without REE i think passenger is less of a win memory wise compared to nginx+mongrel on 64 bit, that’s all. But it is definitely more compelling now that latest ree appears to not segfault on 64bit so far.

  • http://www.phusion.nl/ Ninh Bui

    @Ezra:
    I apologize if this blog post comes off as being offensive in any way, that really wasn’t my intention. As I’ve stated in the first paragraph of my soliloquy, I believe people should choose the right tool for the right job, and this doesn’t necessarily have to be from a technical point of view alone and can involve diverse reasons such as “the software you’re most comfortable with”. So don’t get me wrong, I’m not trying to get people to think that Passenger is the silver bullet to everyone’s problems. If anything, I just wanted to point out some simple solutions and supporting examples to some of the problems and reasons Tom mentioned on his blog, and have done this in a point-wise manner. As can be seen in the last paragraph of my post, my intent was to encourage people to try things out for themselves instead of judge based solely on opinions.

    Also, I’ve just checked my inbox and it seems your mail has crossed this scheduled posting several hours ago at around saturdaynight local time. This was also the only email I’ve received from you by the way (kept a good eye out for it the past few weeks and decided to post this after reading Tom’s post). Even though I’m glad to read that it seems to work now for you guys, I’m still very interested in receiving any kind of information about your test suite and on what points it used to fail so that it could help us refine the releases even better through the possibility of fault reproduction. I’m sure the community would benefit from having this kind of information as well. Regardless, I will get to your mail first thing tomorrow seeing as it’s currently 11:00PM local time.

  • Olga

    @Ezra If you guys are so concerned about memory usage, why use 64 platform at first place ? It’s a known fact that Ruby isn’t optimized for 64 bit and will use A LOT more memory compared to a few MBs of difference you might see with nginx/mongrel v/s passenger – http://www.ruby-forum.com/topic/137794

    I understand there are other factors involved and I don’t have any experience with Linux. But I know that you can easily compile kernel to support more than 4 GB of memory. So I’m just curious to know why 64 bit when you guys are SO concerned about memory usage.

  • Luiz

    @ezra you probably tested this thing out more than anyone else, but it is known to me that 64-bit webservers are not as performant as 32-bit PAE web servers. People always assume that 64-bits is necessarily better, but not always and web servers is one such scenario. 64-bits is just if you need lots of memory and PAE can solve this partially on 32-bits. Mind to share your conclusions on this?

  • http://engineyard.com Tom Mornini

    Once again, I urge the community to put down its weapons.

    To be 100% clear, my blog post was in response to the referenced Twitter post. It was my intention to explain why we’re not currently offering Passenger support by exposing the decidedly non-draconian reasons.

    The reason is fairly simple: We see most of the advantage of Passenger to be simplified configuration, which we don’t need, and that comes at the expense of Apache -vs- Nginx. We like Nginx *a lot* and have over 30 support personnel and hundreds of customers running production environments, and thousands of lines of automation code written around our current stack.

    As Ezra responded, we see the majority of benefit to our customers coming from Ruby Enterprise Edition, which as noted in my blog post, would not work for us until the very same day as my post. I also agree that we’d *love* to see a Passenger module for Nginx, and would consider supporting such a project.

    It’s distressing to see commenters here suggesting that Passenger is somehow at odds with our business model, for instance, unless it simply means that our business model is to focus our attention on the stack that we best feel meets the needs of our customers. Our business model is very simple: provide what we feel is the best production environments available to host Ruby applications, and back up that stack 24x7x365 with support people. In order to do this, we must *not* support every possible combination of components, as that would drastically dilute the expertise that we can provide.

    Finally, as for 32 bit -vs- 64 bit environments: Yes, 64 bit environments consume a lot more memory than 32 bit environments. This is the reason that we’re more focused on Ruby Enterprise Edition -vs- Passenger! 64 bit environments, when virtualized via Xen, have performance advantages compared to 32 bit environments. Additionally, the ability to run processes of greater than 2 GB of RAM has proven to be a very useful capability to have in our toolbox.

    Trevor Turk: Ezra’s statements make it clear that we have spent time with Passenger. My post said that we did not have expertise in it, which is quite a different subject!

    Egze: I’m sorry that you felt we were bashing Phusion’s products. I don’t believe there was anything in my post, or anything else from Engine Yard, that qualifies as bashing any of Phusion’s products, so I’m very sorry if it came across that way.

    Riptie: please send me an email or call me at 866-518-9273 extension 201. I’d really like to understand how we’re underperforming your expectations on this issue. It’s tremendously import for me to understand, because I feel the path we’re taking is 100% in your best interests, but if you don’t agree, then I’m wrong.

  • http://trevorturk.com Trevor Turk

    Tom, my apologies for misunderstanding you guys. Your comments are very encouraging, and I’m excited to hear more about your experiences with Passenger going forward!

  • http://betterlogic.com/roger roger

    I’d imagine engine yard doesn’t need passenger since their biggest concern is not setup. Just my $0.02

  • http://m.onkey.org Pratik

    @Tom : It’s you who need to put down your weapons when *YOU* felt the need to reply to my simple twits ( where there was no mention of you/your employees/your comany ) with a gigantic blog post. When you have to reply to me saying “PEOPLE spreading FUD about passenger”, the shoe must fit. Even your employee had to make a direct personal attack, which is a shame.

    But it’s very apparent you weren’t replying to me, but you were replying to your customers justifying your position against the growing popularity of Passenger. And that’s fine. Just don’t pretend as if it’s all in a response to my twitter messages. My twit, clearly, was just a catalyst.

    But forget all that. Let’s have a little history lesson, shall we :

    ** Feb 12 **
    – Ezra Announces mod_rubinius http://brainspl.at/articles/2008/02/12/what-do-you-want-to-see-in-mod_rubinius.
    – Comment #12 is EXACTLY what passenger is.
    – And now mod_rubinius has been officially killed ? ( correct me if I’m wrong http://www.google.com/search?q=github+mod_rubinius )

    ** April 4 **
    – Passenger is released to the public http://ninh.nl/blog/2008/04/04/the-moment-youve-probably-been-waiting-for-so-have-we
    – If you notice the announcement, “Lastly, we’ve been approached by EngineYard..”. And we never heard about that ever again. Clearly, something went wrong there.

    After that, you guys CONSTANTLY tried to ignore/sideline Passenger, by saying it’s suitable only for shared hosting/VPS. COMPLETELY BASELESS as pointed out 100 times before. That’s pure FUD.

    Here’s an interesting part :

    Tom says “Of course, you can reduce these disadvantages by cuting down a custom build and tweaking the included modules”
    Ezra says “I think i;ve scaled a few ruby websites and I use nginx cuz is beats apache’s ass”

    Clearly, those are contradictory statements and one of you doesn’t know what you’re talking about. Pick one. Period. When you start putting Apache/nginx and “Scaling” in the same sentence, you’re just trolling/spreading FUD.

  • http://www.phusion.nl/ Ninh Bui

    Guys, I think that’s enough, please continue this conversation elsewhere. This post and its comments are only meant to elaborate on some solutions and to encourage people to try stuff for themselves instead of solely basing it on opinions of others.

    Ezra, as promised I’ve sent you a mail a few hours ago and am eagerly awaiting your response.

  • Pingback: Response at a distance « Tom Mornini()

  • colt1x

    I cannot stand apache and because passenger is a fan boy of apache I will never use passenger. I rather deal with mongrel so I can keep my nginx setup. Also I have no clue why anyone would want to waste memory on a apache configuration especially if your hosting with a vps plan.

  • Pingback: Phusion’s One Year Anniversary Gift: Phusion Passenger 2.2.0 « Phusion Corporate Blog()