Phusion white papers Phusion overview

Bundler and public applications

By Hongli Lai on January 19th, 2012

I think Bundler is a great tool. Its strength lies not in its ability to install all the gems that you’ve specified, but in automatically figuring out a correct dependency graph so that nothing conflicts with each other, and in the fact that it gives you rock-solid guarantees that whatever gems you’re using in development is exactly what you get in production. No more weird gem version conflict errors.

This is awesome for most Ruby web apps that are meant to be used internally, e.g. things like Twitter, Basecamp, Union Station. Unfortunately, this strength also turns in a kind of weakness when it comes to public apps like Redmine and Juvia. These apps typically allow the user to choose their database driver through config/database.yml. However the driver must also be specified inside Gemfile, otherwise the app cannot load it. The result is that the user has to edit both database.yml and Gemfile, which introduces the following problems:

  • The user may not necessarily be a Ruby programmer. The Gemfile will confuse him.
  • The user is not able to use the Gemfile.lock that the developer has provided. This makes installing in deployment mode with the developer-provided Gemfile.lock impossible.

This can be worked around in a very messy form with groups. For example:

group :driver_sqlite do
  gem 'sqlite3'
end

group :driver_mysql do
  gem 'msyql'
end

group :driver_postgresql do
  gem 'pg'
end

And then, if the user chose to use MySQL:

bundle install --without='driver_postgresql driver_sqlite'

This is messy because you have to exclude all the things you don’t want. If the app supports 10 database drivers then the user has to put 9 drivers on the exclusion list.

How can we make this better? I propose supporting conditionals in the Gemfile language. For example:

condition :driver => 'sqlite' do
  gem 'sqlite3'
end

condition :driver => 'mysql' do
  gem 'mysql'
end

condition :driver => 'postgresql' do
  gem 'pg'
end

condition :driver => ['mysql', 'sqlite'] do
  gem 'foobar'
end

The following command would install the mysql and the foobar gems:

bundle install --condition driver=mysql

Bundler should enforce that the driver condition is set: if it’s not set then it should raise an error. To allow for the driver condition to not be set, the developer must explicitly define that the condition may be nil:

condition :driver => nil do
  gem 'null-database-driver'
end

Here, bundle install will install null-database-driver.

With this proposal, user installation instructions can be reduced to these steps:

  1. Edit database.yml and specify a driver.
  2. Run bundle install --condition driver=(driver name)

I’ve opened a ticket for this proposal. What do you think?

  • Matt Jones

    Two thoughts:

    * including database drivers in a shipping Gemfile.lock for a public app is probably wrong, since whatever version is specified may get stale.

    * handing an application install to someone who’s “confused” by Gemfile.lock seems exceedingly risky for other reasons. If *anything* goes wrong, that person will be hopelessly lost. For instance, Passenger has incredibly helpful error screens when the app fails to start – but they’re probably unintelligible to someone who’s totally unfamiliar with Ruby.

  • http://www.phusion.nl/ Hongli Lai

    @Matt:

    Regarding shipping database drivers in Gemfile.lock: that may be the case, but Bundler is built around the notion that it knows your *entire* dependency graph, including the exact versions that you were using in development, and locks down your gem environment to exactly that. If you don’t include your database drivers in your Gemfile.lock then you won’t have any version guarantees anymore. The alternative is to specify exact versions directly in Gemfile (not Gemfile.lock) and allow the user to regenerate Gemfile.lock, but this makes life extremely hard for the developer. I’ve been down this path and it ain’t pretty.

    I’m not sure what you’re getting at with your other statement. Of course the person is lost if something goes wrong, that’s why I’m suggesting a change that makes less things go wrong and less things confusing.

  • http://bibwild.wordpress.com Jonathan Rochkind

    Redmine and Juvia are both Rails applications.

    In Rails 3.x, I think there should no longer BE “shared”/”public” rails apps — all such apps should instead be delivered as engine gems.

    This keeps strictly seperate the ‘shared codebase’ part (in the gem, which can specify dependencies in the usual way through it’s gemspec), and the ‘local app’ part, which includes any configuration (such as database.yml), as well as the Gemfile and Gemfile.lock.

    The ‘local app’ might have nothing _but_ a database.yml, a Gemfile.lock and a skeleton rails app including the ‘shared’ gem. Or it could have some configuration too. Or it could have some over-rides and local functionality too.

    But anything that can be done in an app, in rails 3.x, can be done instead as a skeleton app + engine gem. I think that’s clearly the right way to distribute shared rails apps in 3.x, you should never actually distribute a shared app itself, it should be an engine gem. That eliminates the sort of problems talked about here, has other ancillary benefits, and is what I’ve done with all such things I develop.

    There’s no reason you should ever be shipping a Gemfile.lock at all, in theory. You should be shipping a gem with dependencies, the Gemfile.lock should be the LOCAL record of _exactly_ which gems were used in last invocation, as opposed to your gem’s gemspec, which is a listing of ranges of versions acceptable to it. You certainly could lock down to very specific gem versions in the gemspec if you wanted to, though, to get the same effect.

  • http://www.phusion.nl/ Hongli Lai

    @Jonathan: How’s shipping apps as an engine going to make things easier to install for the end user who’s not a Ruby programmer? Where does the ‘local app’ part come from?

  • rbq

    I think we need a standard installer for public Rails apps deployed by users on a single host. It should act similar to the installers popular PHP apps provide. If the app launches without being properly configured it should ask for certain variables the developer specifies, copy -dist files to the according destination and fill in the blanks, check for other dependencies (Ruby version, write permissions, Postgres/Mongo/Redis/Memcached, …), re-run bundler and touch tmp/restart.txt. Maybe this could be integrated on a very basic level, even before Bundler runs, so it could catch most problems, even if something with the configuration is fundamentally wrong (old Ruby, Bundler not available, …).

    Just look at WordPress and others. It’s ridiculously easy these days to install them and upgrade to a new major version, including migrations being executed for you. Would be nice if there was a default way to add this to a public Rails app.

  • http://twitter.com/mrreynolds Robert

    This is actually a problem we have with shipping our open source app. We want to support both the application to be runnable as a standalone app as well as an includable Rails engine. Currently, we ship it with a checked in Gemfile.lock which requires a multitude of database drivers for different groups **and** different platforms (ruby, jruby). Especially if you want to support easy heroku deployments you have to include Postgres.

    This can not be the desired pattern.

    I wonder WWYD (What Would Yehuda Do)? :)

  • http://www.yehudakatz.com Yehuda Katz

    I actually like this proposal. Back in the 1.0 days, we talked about having something like:

    group :sqlite, :optional => true do
      gem "sqlite3"
    end

    You would then do: `bundle install –with sqlite`. However, this didn’t feel entirely satisfying, because it didn’t really hit the nail on the head for the use-cases in question. Your proposal does. I’d want to talk with other Bundler folks before committing to anything, but I think that this actually satisfies an important use-case in an elegant way.

    It’s worth noting that bundler would still need to include all of the conditionals in the Gemfile.lock, and they would not be able to have conflicts. You can find more information about the rationale for that at http://gembundler.com/rationale.html#faq-3. However, in most cases where conditionals exist, like database drivers, the drivers don’t conflict with each other, but you only want one, so it would work fine.