Phusion white papers Phusion overview

Securing the Rails session secret

By Hongli Lai on January 4th, 2013

The recent Rails SQL injection vulnerability sparked very large discussions. One of the discussion threads in particular caught my attention.

In one of the exploitable scenarios, the attacker must know the session secret key. This is not so problematic for proprietary apps where the session secret is kept hidden, but is problematic for many open source Rails apps because the session secret is stored in version control for all to see. Commenter Dan Kaminsky and some other people were of the opinion that this is a bug in Rails. I disagreed because I saw it as the responsibility of the developer to omit secret keys upon public distribution.

However, regardless of whose fault it is, can this problem be prevent? Rails allows storing the session secret key in version control by default for the following reasons:

  1. It is not a problem for the majority of the users, who write proprietary apps. The number of open source Rails apps is quite small in comparison.
  2. It is convenient. Requiring the user (which in production environments is the system administrator, and in development environments is the developer) to set a secret key adds one more installation step which is not user friendly.

Is there a way to avoid the secret key from leaking out upon public distribution, without making things less convenient? Let’s explore a few possibilities.

  1. Omit the secret key from version control, and require the user to manually generate one.

    This works but is inconvenient. You can make it more convenient by adding a hypothetical rake save_secret task to automate it, but that’s still a manual step. It also raises the learning curve for new users. You can have rails new automatically run it for you, but that doesn’t solve things in production. You will have to generate a secret key on the production server for every app you deploy, and modify your Capistrano script to symlink to that key on every new deploy. As authors of Phusion Passenger we dislike any kind of installation-time sit-ups.

  2. Omit the secret key from version control, but auto-generate a random key if missing.

    The problem with this is that every time you redeploy with Capistrano, the secret key would be regenerated. This would invalidate all previous session cookies, but not in a nice way. Rails would detect invalid signatures on previous cookies and throw an exception. There are two solutions to this:

    1. Make Rails silently discard invalid cookies instead of throwing an exception. Users would still be logged out on every deploy, but I believe this is a minor problem for the people who can’t be bothered to do (2).
    2. Create the secret key on the production server once and symlink to it on every deploy, but you have to know about it and it makes your Capistrano scripts slightly larger.
    3. A reader said that Capistrano can be modified to generate this key and store it in the ‘shared’ directory if it doesn’t exist.
  3. Omit the secret key from version control, but auto-generate a non-random key if missing.

    Instead of generating a random key, the key would depend on something that is unique to the system so that the key changes across different machines but not on the same machine. However secret keys are supposed to have high entropy so you will have to choose your “something unique” very carefully. What options do we have? From the top of my head, this is what I’ve come up with:

    • Host name – low entropy and can be guessed.
    • MAC address – it’s not inconceivable that it can be guessed.
    • IP address – this is public information, so not a good idea.
    • Modification time of the root filesystem – low entropy. There’s a high chance that the server was installed in the past 5 years.
    • SHA-512 of all file contents in /etc – slow, and changes your key every time you modify something in /etc.

    None of these are very good sources. But maybe we can generate a machine-unique property that has high entropy. The property would be stored in /etc/machine-uuid and would be world-readable because it’s only used for deriving secret keys. During startup, Rails would check for this file and tell you to run rake secret | sudo tee /etc/machine-uuid if it doesn’t exist. This only has to be done once per machine. machine-uuid is never supposed to be distributed outside the local machine.

    You don’t want all apps on the same machine to have the same HMAC key, so Rails would derive the HMAC key as follows:

    hash("#{machine_uuid}-#{hostname}-#{app_name}")
    

    where hash is a cryptographically secure hashing function.

    But let’s be paranoid for a bit. If the attacker has gained access to a random local user account then he can very easily derive the HMAC key. But in this case you probably have more things to worry about than whether the attacker can tamper session data.

  4. Allow storing the secret key in version control, but use the secret key and unique machine properties to sign HMACs.

    Like with (3), machine properties must have sufficient entropy. The HMAC key would be derived as follows:

    hash("#{machine_uuid}-#{hostname}-#{secret_key}")
    

    This is more secure than (3) if you’ve set proper permissions on your application files. An attacker that has obtained access to a local user account, but not the one that your app is running on, knows machine_uuid and hostname but not secret_key. Of course he can still do other nasty things so I’m not sure whether one should worry about this scenario.

  5. Store a default secret key in version control, but allow customizing it through an environment variable

    A reader said he prefers the following:

    Rails.application.config.secret_token = ENV['SECRET_TOKEN'] || 'fallback_token_for_development'
    

    Unfortunately this does not solve the convenience problem. It requires the administrator to know about the secret token and it requires the administrator to perform some setup.

I prefer (2) in combination with patching Rails to discard invalid cookies instead of throwing an exception. What do you think? Can you think of any other ways to make it more secure without sacrificing convenience? Are my security analyses correct? Please leave a comment here, on Hacker News or on Reddit.

  • http://twitter.com/barttenbrinke barttenbrinke

    I would treat this as I would handle any other certificate or key. Don’t have them in version control and deploy them through a different means then your app. You could easily roll the key out through Chef or Puppet and place it in the shared directory of the app.

  • http://developerfounder.com/ Michiel Sikkes

    I guess they are good measures and I agree with you that deploying should be easy.

    However, I don’t think having to symlink a on a Cap deploy would be that bad. For instance, if Capistrano detects that there is no shared/session_key.rb it will generate it, and on subsequent deploys it will symlink it because the file already exists.

    For people who don’t have a lot of experience to customize their own Cap recipes, you could hide this in the default Rails deploy hooks.

    I find such a way a better thing than invalidating the key on every deploy. For example, I might do 4-10 deploys in a single day, maybe even within 5-10 minutes and I wouldn’t want my users to suddenly logout (and loose other session data) 5 times a day.

  • Tim

    I prefer to store my secret token as an environment variable and check this into version control:
    Rails.application.config.secret_token = ENV[‘SECRET_TOKEN’] || ‘fallback_token_for_development’
    You have to take care of setting it on your server, though (but this can be taken care of by chef/puppet/capistrano/…). Otherwise you’re very vulnerable.

  • http://twitter.com/chrismwelsh Chris M. Welsh

    I use environment variables on Heroku for secret keys. In a similar way, perhaps Rails core could come up with some standard method of secure environment variable management. Another method I use is to maintain a secret deployment repository responsible for setting up the environment and running capistrano/chef/whatever scripts to deploy the application.

  • http://www.phusion.nl/ Hongli Lai

    I like this method.

  • http://www.phusion.nl/ Hongli Lai

    I’ve written my thoughts about this in point 5.

  • http://blog.mmediasys.com Luis Lavena

    What about securekey?

    https://github.com/will/securekey

  • http://developerfounder.com/ Michiel Sikkes

    Of course this will provide a problem when you are load balancing your app to a new server. But I think, when you’ve got an app that large, you know how to do that stuff with custom deploy recipes/chef/puppet/whatever.

  • Thibaut Barrère

    I definitely agree. Having something generated automatically in development.rb/test.rb for convenience is ok, but for production I’d rather make sure it’s a password, just like what is inside database.yml.

  • sytses

    The secret key is similar to the salt of a user password. Why not store it in the database so that it is persisted between deploys to new servers and the same when you have multiple servers?

  • Schneems

    I think the ENV var should be an option, i’ve been working over at Heroku to be able to auto add a secure rotating key to Rails apps. You can ping me on twitter for more info.

  • http://twitter.com/MattRogish Matt Rogish

    If you’re a company that does continuous deployment #2 is right out. On a busy day, we deploy a few times an hour. Continuously logging out our users is not good usability. :)

    We opt for #5 with a guard that raises an exception if Rails.env.production? && the secret is blank. That way, we keep the keys in Heroku ENV VAR (and the folks that have access to that are small). Whenever someone leaves the company, we generate new tokens for all the environments. The admin overhead is quite small, all things considered.

  • http://twitter.com/MattRogish Matt Rogish

    It seems like it ought to be straightforward to do with the heroku toolbelt, similar to pg:credentials –rotate

  • Daniele Sluijters

    To be perfectly honest I’m a bit shocked this is an issue. Wether you’re an open source or closed source application is in my mind completely irrelevant. The same practices should apply when it comes to the security of your application and hence the user data you are entrusted with.

    I realise it’s convenient to have everything done for you automagically but that convenience comes at a price. I don’t have a real preference to whichever solution people want to use, just find some other way than committing that shared secret to version control.

    If you can’t educate users to generate a secret key themselves or aren’t capable enough to run a rake task to do so we have a completely different problem to solve and I don’t think you can solve that by automating it away. For something as important as the shared secret of your application, people need to understand what’s going on and what the implications are.

    At least when it comes to production systems an administrator should be capable enough to generate a secret key or deploy one through other means. If that’s not the case, you either need to (h|f)ire someone or look at educating that person/yourself.

  • Samuel Kadolph

    6. Store a development and test version in config/app.yml and during your deloy, sym link in the production copy of that into your deploy’s config/app.yml.

  • http://twitter.com/ndm Neil Matatall

    What about passw3rd? Tokenize passwords/keys/secrets, protect a single secret in a separate fashion. No .gitignore. https://github.com/oreoshake/passw3rd disclosure: I am the author

  • http://profiles.google.com/stoune Petro Sasnyk

    I think that generation unique to machine session key is good idea only for less loaded application. If your application is distributed across several hosts using load balancer this approach will be a bad idea.
    I prefer using ENV veraiables, because it is convenient and industry adopted. See Heroku adn AWS.

  • ThomasW

    I think, this touches a rails problem I have since I am using rails: What is the conf-folder all about? On the one hand the developer does configure the rails framework and other gems he uses here. On the other hand, the site admin configures his site specific installation of the app here. In most cases the admin should not touch most of the configuration but still find among so many settings those, which are important for him. This never felt good to me.

    So what about a “app/conf”, where the developer configures the framework etc. and a conf, which is for the site confs, and which does not go to version control at all except for example files provided by the developer.

  • Evan Grantham-Brown

    What about a mix of 5 and 2? Pull the secret key from an environment variable. If the env variable is not found, auto-generate it.

    The key is not in source control. It does not require the administrator to know about it or do anything. And it is not invalidated on redeploy.