Turbocaching security changes

by Hongli Lai

Phusion Passenger version 5.0.0 beta 3 was released today, with a number of changes to the turbocache. The turbocache is a component in Passenger 5 that automatically caches HTTP responses in an effort to speed up the application. The turbocache is not a feature-rich HTTP cache like Varnish, but instead it's more like a "CPU L1 cache for the web" -- small and fast, requires little configuration, but has fewer features by design.

The turbocache was implemented with HTTP caching standards in mind, namely RFC 7234 (HTTP 1.1 Caching) and RFC 2109 (HTTP State Management Mechanism). Our initial mindset in implementing the turbocache was like that of compiler implementors: if something is allowed by the standards, then we'll implement it in order pursue maximum possible performance.

But it turns out that following the standards strictly may raise security concerns. A while ago, we were contacted by Chris Heald, who provided convincing cases on why the turbocache's behavior is problematic from a security standpoint. Chris is a veteran Varnish user and has a lot of experience on the subject of HTTP caching.

The gist is that the turbocache made it too easy to accidentally cache responses which should not be cached. Imagine that a certain response contains sensitive information and is only meant to be served to one user. Such sensitive information may include security credentials or session cookies. If the application sets HTTP caching headers incorrectly, then the turbocache may serve that response to other users, resulting in an information leak.

Such problems are technically bugs in the application. After all, Passenger is just following the standards. But Chris asserted that such mistakes are very easily made. Indeed, he has seen quite a number of applications that send out incorrect HTTP caching headers. For this reason, he believes that the turbocache should be more conservative by default.

Cookies also deserve special attention. RFC 2109 mentions that cookies may only be cached when they are intended to be shared by multiple users. But it is impossible for Passenger to know the intention of the cookies without configuration from the application developer. At the same time, providing such configuration goes against the spirit of the turbocache, namely that it should be easy and automatic.

It was clear that something had to be done.

Turbocache changes

More conservative

After contemplating the issues, we've decided to make the turbocache more conservative in order to avoid the most common security issues.

In beta 1 and beta 2, the turbocache caches all "default cacheable responses", as defined by RFC 7234. These are responses to GET requests; with status code 200, 203, 204, 300, 301, 400, 405, 410, 414 or 501; and for which no headers are set that prevent caching, e.g. "Cache-Control: no-store". This means that a GET request which yields a 200 response with no caching headers, is actually cacheable per the standards, and so it was cached by the turbocache.

In beta 3, responses are no longer cached unless the application explicitly sends caching headers. That means that a GET request which yields a 200 response is only cached if the application sends an "Expires" or a "Cache-Control" header (which must not contain "private", "no-store", etc).

While it is still possible for some application responses to be unintentionally cached, this change solves the majority of the issues. The only way to avoid these issues out of the box is by disabling caching completely, which has the downside of not being able to leverage said caching features. Instead, we believe it is reasonable that we only cache requests in case the application asks for this explicitly. A potential caveat is that even though the application and Passenger respect the specifications, an application developer may not have a full understanding of how caching behaves according to said specs. A developer should choose to disable caching entirely in such cases.

Bugs fixed

While investigating Chris's case, we also uncovered some bugs in the turbocache, such as the incorrect handling of certain headers. These bugs have been fixed. The full details can be found in the Passenger 5 beta 3 release notes.

Using the turbocache and speeding up applications

Now that the turbocache has changed in behavior, here are some practical tips on what you can do to make good use of the turbocache.

Learn about HTTP caching headers

The first thing you should do is to learn how to use HTTP caching headers. It's pretty simple and straightforward. Since the turbocache is just a normal HTTP shared cache, it respects all the HTTP caching rules.

Set an Expires or Cache-Control header

To activate the turbocache, the response must contain either an "Expires" header or a "Cache-Control" header.

The "Expires" header tells the turbocache how long to cache a response. Its value is an HTTP timestamp, e.g. "Thu, 01 Dec 1994 16:00:00 GMT".

The Cache-Control header is a more advanced header that not only allows you to set the caching time, but also how the cache should behave. The easiest way to use it is to set the max-age flag, which has the same effect as setting "Expires". For example, this tells the turbocache that the response is cacheable for at most 60 seconds:

Cache-Control: max-age=60

As you can see, a "Cache-Control" header is much easier to generate than an "Expires" header. Furthermore, "Expires" doesn't work if the visitor's computer's clock is wrongly configured, while "Cache-Control" does. This is why we recommend using "Cache-Control".

Another flag to be aware of is the private flag. This flag tells any shared caches -- caches which are meant to store responses for many users -- not to cache the response. The turbocache is a shared cache. However, the browser's cache is not, so the browser can still cache the response. You should set the "private" flag on responses which are meant for a single user, as you will learn later in this article.

And finally, there is the no-store flag, which tells all caches -- even the browser's -- not to cache the response.

Here is an example of a response which is cacheable for 60 seconds by the browser's cache, but not by the turbocache:

Cache-Control: max-age=60,private

The HTTP specification specifies a bunch of other flags, but they're not relevant for the turbocache.

Only GET requests are cacheable

The turbocache currently only caches GET requests. POST, PUT, DELETE and other requests are never cached. If you want your response to be cacheable by the turbocache, be sure to use GET requests, but also be sure that your request is idempotent.

Avoid using the "Vary" header

The "Vary" header is used to tell caches that the response depends on one or more request headers. But the turbocache does not implement support for the "Vary" header, so if you output a "Vary" header then the turbocache will not cache your response at all. Avoid using the "Vary" header where possible.

Common application caching bugs

Even though the turbocache has become more conservative now, it is still possible for application responses to be unintentionally cached if the application outputs incorrect caching headers. Here are a few tips on preventing common caching bugs.

Varying response by Ajax

A common pattern is to return a different response depending on whether or not it was an Ajax call. For example, consider this Rails app, which returns a JSON response for Ajax calls, HTML response otherwise:

def show
  headers["Cache-Control"] = "max-age: 600"
  if request.xhr?
    render json: "show"
  else
    render
  end
end

This can cause the turbocache to return the JSON response for non-Ajax calls, or to return the HTML response for Ajax calls.

There are two ways to solve this problem:

  1. Set the "Vary: X-Requested-With" header so that the cache knows the response depends on this header. Rails's request.xhr? method checks this header. Note that the turbocache currently disables caching altogether upon encountering a "Vary" header.
  2. Do not vary the response based on whether or not it is an Ajax call. Instead, vary the response based on the URI. For example, you can return JSON responses only if the URI ends with ".json".

Be careful with caching when setting cookies

If your application outputs cookies then you should be careful with your caching headers. You should only allow the turbocache to cache the response if all cookies may be cacheable by multiple users. If any of your cookies contain user-specific information, or if you're not sure, then you should set the "private" flag in "Cache-Control" to prevent the turbocache from caching it.

We realize that lots of responses can output cookies, so we're working on ways to improve the cacheability of responses with cookies.

Set "Cache-Control: private" when working with sessions

If your application works with sessions then you must ensure that all your "Cache-Control" headers set the "private" flag so that the turbocache does not cache it.

Note that your app may work with sessions indirectly. Any Rails page which outputs a form will result in Rails setting a CSRF token in the session. Luckily Rails sets "Cache-Control: private" by default.

Performance considerations

The turbocache plays a major role in the performance of Passenger 5. The performance claims we made are with turbocaching enabled. Now that the turbocache has become more conservative, the performance claims still stand, but may require more application modifications than before.

By strictly following the caching standards, we had hoped that we'd be able to deliver performance for most apps out-of-the-box without modifications. But we also value security, and we value that more so than performance, which is why we made the turbocache more conservative.

But this raises a question: suppose that we can't rely on the turbocache anymore as a major factor in performance, what else can we do to improve Passenger 5's performance, and is it worth it?

There is in fact one more thing we can do. and that is by introducing a new operational mode which bypasses a layer. In the current Passenger 5 architecture, all requests go through a process called the HelperAgent. This extra process introduces some context switching overhead and processing overhead. The overhead is negligible in most real-world scenarios, but the overhead is very apparent in hello world benchmarks. It is possible to bypass this process, allowing clients to directly communicate with application processes. But doing so comes at a cost:

  • Some features -- e.g. load balancing, multitenancy, out-of-band garbage collection, memory checking and statistics collection -- are best implemented in the HelperAgent. Reimplementing them inside application processes is either difficult or less efficient.
  • Rearchitecting Passenger this way requires a non-trivial amount of development time.

So it remains to be seen whether bypassing the HelperAgent is worth it. Investing time in this means that we'll have less time available to pursue other goals on the roadmap. What do you think about this? Just post a comment and let us know.

Conclusion

Passenger 5 beta 3 is the first "more or less usable in production" release of the Passenger 5 series. Previous releases were not ready for production, but with beta 3 we are confident enough that the most important issues are solved. The turbocache issue as described in this article is one of them. If you were on a previous Passenger 5 release, then we strongly recommend you to upgrade.

We would like to thank Chris Heald for his excellent feedback. We couldn't have done this without him.

We expect the Passenger 5 final stable version to be released in February. Beta 3 is supposed to be the last beta. Next up will be Release Candidate 1, followed by the final stable release.

But the story doesn't end there. We will publish a roadmap in the near future, which describes all the ambitious plans we have in store for Passenger. Passenger is constantly improving and evolving, so please stay tuned for updates.