And keep up the good work improving passenger, I like where things are going in general… It shows you do listen to what people want and need, once you are convinced. Which is exactly the way you should be doing it as keeper of your own open codebase.

And if anyone thinks the work should be done very differently or at a very different pace, they are free to do better. And while that is an appropriate response to a troller, thanks for not using that as a pat answer to every helpful suggestion and discourse about possible ways to improve the product! You guys rock!

In regards to the whole “watchdog of the watchdog” thing… I would expect a reasonable two-watchdog implementation to be like a master-slave thing, where the slave watchdog turns into a new master when it notices its master died (i.e. launching a new slave, etc).

However, I can also see it as a logical opinion that many people underestimate the increased complexity of such a system, and the negative impact that can have on the stability of the watchdogs in the first place. Of course the only way to truly tell is to build both in a modular fashion and run the same site with half the server farm one way and half the other way… for.. um.. probably years? Anyone interested in building this and running it to see? … I thought so

Instead zero-downtime web server restart is for when you’ve modified the web server configuration file and needs the web server to reload it.

First, what failure mode does this new feature prevent? And in what scenario would I use it? For example, when I restart a web server, I usually do it for one of two reasons:

1) The web server has crashed or is not responsive
2) I have deployed new code or changed a properties file

Obviously, the first scenario would be handled by the watchdog process. So is scenario #2 handled by this “Zero-downtime web server restart” feature? Can I make an arbitrary change to my application, and have the old and new versions hosted simultaneously?

Having said that, below you’ll find my reply to your “points”:

1. First off, as repeatedly stated to you on twitter and something I’d like to underline in this comment, we never remotely claimed we invented these concepts, nor would we dare to claim such a thing as having invented signaling. Not only because that would be a gross falsity but just also for the fact that we respect our audience in having the knowledge in something like signal handling.

Furthermore, we’ve named the concepts by their proper name so anyone who would have the scrutiny to be interested in these concepts would easily be able to perform a google search to find scientific papers on these matters in all sorts of forms, should they desire to do so. In fact, they are so numerous, I wouldn’t even know where to begin with regards to referencing. Call me a fool, but I believe our audience is scrutinous and smart enough to do their own part of research as well .

Now, let’s flip the burger, should one who refers to the observer pattern also make reference to Gamma et al’s Gang of Four design pattern book as well as their first scientific paper on these things in an effort to not come across as being pompous? Am I pompous because I don’t do:

// Oh this is a derivative of the observer pattern, which was originally created
// by Erich Gamma et al. or at least formally described by them  in their excellent
// Gang of Four design pattern book. I didn't invent this!
// ISBN 0-201-63361-2
class Foo : Observer {

};

Should “man kill” also be called pompous because it does not reference to the theory behind signal handling in processes? If you think so too, then I guess there is no point in discussing this any further as we’ll fundamentally differ in opinions on these matters. By that logic, you’d probably find nearly all developers pompous then I guess

Furthermore, you make mention of us not reading enough books, yet you claim we are the ones being pompous… I find that kind of interesting especially when you referenced a paper by Philips from 2006 as proof of fault tolerant systems predating our blog post, to which I replied that Joe Armstrong implemented these concepts as early as in 1986 in Erlang. Undoubtedly, the latter was preceded by other efforts as well as I’ve been able to find papers going back to the 70s on this topic on ACM with ease.

2. Let me just reply to the sentence you quoted and try to elaborate on that. In fact, our blog has a commenting system just for that purpose. There are a few factors why we make this claim:

The watchdog is extremely lightweight, i.e. less than 1000 LOC, most of it being C++ boilerplate and error checking. With code that is this lightweight it’s very feasible to test this extremely well, which we have done indeed. In fact, most of the code of this watchdog is actually to test that conditions pass as expected and get handled correctly as well. There are of course a myriad of anomalies that could lead applications to crash, but we’ve made an extremely big effort in getting this minimized when it comes to code. The use of idioms and libraries that have been integrated in far larger projects than Passenger and have had a significant effect with regards to stability (e.g. reducing the possibility of segfaults by using smart pointers and preventing memory leaks by using RAII) is something we wish to underline. To say it is only speculation however would be doing it a disservice, as I’ve mentioned in one of the earlier comments, we’re currently beta testing Passenger not only in our environments but also at some of the most demanding Ruby environments out there to make sure that it is holding its own in those environments as well.

It is for this reason that I don’t entirely agree with your paraphrasing as it seems to be overseeing the fact that we’re actively battle testing this in various live production environments (some of the most demanding in fact) right now as we speak and have done so for quite some time now. In fact, we’ve been testing well over half a year now, and we firmly believe that by using these techniques and doing intensive testing we’ve reduced the possibilities significantly of allowing the watchdog to contain crash bugs especially in contrast to the alternate scenario that we would NOT have used any of these techniques.

Beta test results seem to back this up for the moment, but I understand what you’re trying to get at: indeed, I suppose I can’t give you a probability such that P(X=”watchdog crashes”)<0.0001 as a definition of “extremely low”, so I will edit this in the blog post to reflect that: instead of “is” I’ll use “expect” in the sentence with regards to the watchdog. However, I find it interesting that you seem to take such extreme offense of this as you’re quoting Apple who are well known for making far bolder claims like “reinventing the phone”. I’d only think it would be fair if you’d email Steve Jobs right now too to get a scientific backing on that claim.

3. It’s funny that you first mention that you find Phusion Passenger a good product, but at the end of your comment call it ancient and are apparently capable of speaking of behalf of all our users/clients. That’s an indication of a troll I guess, and shame on me for indulging you with this comment .

Now, in particular, you mention “Their bars are actually a lot higher, they consider your technology ancient, and – like me – wonder why you didn’t put it in in the first place”. Would you like me to find a cure for cancer too while I’m at it? (Trust me, if I could, I would but the fact remains that we’re mere mortals, not wizards).

All kidding aside, I could go on for a while, but you know what I’m thinking instead? Phusion Passenger has been an open source project from the very beginning, for over two years now in fact. If there were particular features that you needed in there, there would’ve been plenty of opportunity for you to implement these yourself. So what’s your excuse for not doing so?

I will explain my wording.

But first, you need to know that I think Passenger is a good product, a valuable addition to the open source world. It is by far the best way to serve up a rails app. I use it myself.

I’ve seen you guys presenting, I read some blog posts, so I tend to generalize. Again, that’s unfair, so I’ll stick to just this one post, technology preview 2.
Pompous means making yourself look ‘annoyingly self-important’, but it’s of course highly opinionated. Henceforth, I can only give two key pieces of constructive feedback that would make your posts/presentations seem LESS POMPOUS to me (I.E. less annoying).

1. Point readers to the good ideas you borrowed, give credit.
(You did not invent the concept Watchdog, nor the SIGTERM/SIGKILL signalling, nor the rolling updates for multiple server instances. You just re-implemented these. )

BECAUSE:
- If you provide the reference you can explain it in the “for dummies” style like you do so well, but the references earn trust from serious (“enterprisy”?) people. You need the serious people to grow the community.
-If you provide the reference, maybe you will inspire someone to read them, who can actually apply the same concepts, AS WELL AS OTHER CONCEPTS FOUND IN THE SAME BOOK/SOURCE. While looking up the sources of the stuff you borrowed, they will find solutions to their OTHER PROBLEMS and – again, implement them for the greater good.

By leaving out references you imply that you ‘invented’ stuff, which you clearly haven’t, which looks self-important (and ignorant) to people who know this stuff.
BTW: If you are now saying: “But really, I never read about this, I just built it”, then – indeed you are re-inventing the wheel and you really should read more books.

2. Separate your marketing fluff from your educational writing
(If you are making time to educate people, don’t get caught inserting unsupported marketing claims in between. For example this is pure nonsense, and very much unneeded:

“The codebase extensively uses C++ idioms that promote code stability, such as smart pointers and RAII. Because of this, the possibility that the watchdog contains crashing bugs is extremely low.”

How extremely low? Once in a million years? How do you know that? Low compared to what? Aren’t you essentially saying “well we use a programming language, and some libraries, we followed other smart people’s guidelines and since it’s only a small codebase, and because we tested it quite a lot, we HOPE there are fewer bugs in that component, than in the average work WE deliver.” That’s about what you have, anything more, is plain self-important.

BECAUSE
- You write to educate others, to show by example how Phusion strives to take Rails to higher levels. And probably to inspire others to do the same, with similar openness. If you start ‘selling’ yourself or your product, inspiration is lost. Whom did you learn most from at University; the all-knowing arrogant professor, or the modest and forgiving Ph.D. assistant?
- You loose your credibility whenever people find out you are presenting guesses as facts.

HOW
- We know you’re talking about Phusion, I like your honesty, please leave out the self-indulgant “we’re sooooo much better than everyone else out there” statements. I would not be reading your post, if I was not contemplating testing Phusion 3 in beta, or researching its sources to find a solution to my problem.

For example, don’t get caught raising bars that are over your head. Let me LOWER yours a little for you:

” Instead of using words convince them, we would rather convince them with real results. Phusion Passenger 3 raises the bar in the areas of performance, stability, robustness and availability yet higher, but it doesn’t stop here.”

Phusion Passenger 3 is here implicitly compared to other Rails servers, of which – frankly – there aren’t too many. You are not raising any bars in the minds of the people doubting “Ruby in Production”. And it’s counter-productive to blabber like this to them. Their bars are actually a lot higher, they consider your technology ancient, and – like me – wonder why you didn’t put it in in the first place. How can you do without it? What else are you missing, that is required to make seriously heavy production systems stable and robust?

Finally, I find it striking that you even bring up raising bars, instead of just being honest and saying something like:

“Instead of using words to convince people, we prefer to be judged by our results. We challenge any Phusion 2 site, as heavy as can be, to try Phusion 3, and try out the test scenarios we featured in this article. If you get Phusion 3 to drop a single request, we’ll buy you a pizza. Success or failure, in either case, we expect you to blog it. ”

To catch all this in two sentences:
Microsoft tried to repeat that their product “Bob” was so awesome it was beyond awesomeness. This MS Bob technology inside! Apple’s iPhone just makes you quiet the first time you use it, while Jobs says “This is nice huh? It’s running BSD Unix, like our computers. That’s as stable as we can make it for you.” References and technological modesty. There you have it.