Why Ruby app servers break on macOS High Sierra and what can be done about it

by Hongli Lai

People who have upgraded to macOS High Sierra and who are using a preforking app server such as Puma or Unicorn (with the right settings), may have noticed this error:

objc[81924]: +[__NSPlaceholderDictionary initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.

This cryptic error is triggered under the following conditions:

  1. You are using Unicorn with preload_app, or Puma in cluster mode, or iodine in prefork mode, or Passenger in smart spawning mode.
  2. And you are using MRI.
  3. And your application uses a gem that is either directly or indirectly linked to the macOS Foundation framework.

This error is caused by changes in how fork() behaves in High Sierra. This article covers:

  • What is this and why did Apple change it?
  • How are the Puma, Unicorn and Passenger authors responding to this?
  • What can you do about it, and do you need to do anything at all?
  • What should the wider Ruby ecosystem do?

What is forking?

Puma, Unicorn and Passenger all implement a feature known as preforking. In Puma it's called the cluster mode, in Unicorn it's called preload_app and in Passenger it's called smart spawning. This feature works like this:

  1. The app server loads the application code and gem code inside its own process memory.
  2. The app server tells the OS to make multiple exact, virtual copies of itself. This is called forking, and the copies are called child processes.
  3. The copies, not the original, are used to process requests.

Forking is faster and uses less memory than simply starting multiple instances of the app server. First, forking a process is nearly instant. Compare this to starting another app server instance, which can take multiple seconds.

Second, modern operating systems implement something called virtual memory. They don't really copy the original process's memory immediately. Instead they make a copy-on-write copy: the copy is virtual, and the actual memory is only copied the moment the original process or the child processes tries to modify that memory. What's more, only the modified memory region is copied, not the entire memory space.

Preforking implementation in Passenger: smart spawning. All processes, as well as the preloader, share the same application code and web framework code.

Lots of software use this signature Unix technique. Apache and Nginx for example use forking extensively. And before Unix and Linux had good multithreading support, Unix software tended to implement concurrency by forking instead of spawning threads.

Forking can be dangerous

Forking has one danger. It is fundamentally at odds with multithreading because of the following properties:

  1. When you fork a process, only the thread that called fork() will remain. All other threads disappear.
  2. At the moment a process forks, the other threads may be in arbitrary inconsistent states.

The second property is especially dangerous. In general, you cannot assume anything at all about the application's state after forking a multithreaded application, so almost everything you do is dangerous. Any code that directly or indirectly reads or modifies memory can crash or freeze.

Consider the following typical example, which we have reproduced multiple times during the development of Passenger:

  1. An application spawns a thread X. This thread is allocating a memory buffer. The memory allocator locks a global mutex.
  2. While thread X is allocating a memory buffer and also has a lock on the global mutex, the application's main thread forks.
  3. The child process also allocates some memory. But the memory allocation mutex is locked, and X is gone, so it will never be unlocked. The child process deadlocks forever.

The child cannot simply unlock that mutex either, because the fact that the mutex is still locked means that the heap is in an inconsistent state.

Bitten by Apple's good intentions

Ironically, the Objective-C error mentioned at the beginning of this article is caused by the fact that Apple tried to add support for forking in their Objective-C libraries.

What can you safely do after you've forked a multithreaded application? You can execute async-signal-safe code. This is like thread-safe code, but with stronger guarantees. You can't rely on mutexes to define critical sections: the biggest critical section you can count on is one CPU instruction.

Apple's Objective-C libraries have historically not supported any kind of async-signal-safety. If you try to call Objective-C libraries after forking then it would work most of the time, but when it fails it probably fails catastrophically.

In High Sierra, Apple has defined some rules on what is allowed and not allowed after forking, and they have also added async-signal-safety to a limited number of APIs. They have also started enforcing rules. Here is what happens if you call a disallowed function after forking:

  • Before High Sierra: appearing to work most of the time, occasionally failing catastrophically.

    Most of the time, you're lucky and no other threads are doing important stuff. The application will appear to work fine. There may still be corrupted state somewhere, but you don't notice it — at least not immediately.

    If you're unlucky then the application crashes or freezes in mysterious ways. In theory it can even accidentally wipe your hard drive.

  • Since High Sierra: Apple makes the application crash immediately.

Takeaway: the problem has always been there, even before High Sierra and even on Linux. It's just that High Sierra has chosen to fail fast instead of silently allowing corruption.

How app servers trigger the error

Ruby, Puma, Unicorn and Passenger don't have any direct dependencies on Apple's Objective-C libraries. But one of the gems you use may have such a dependency, either directly or indirectly. Jessica Stokes cites an example with the pg gem:

[The application is] loading pg which has a native extension linked against Postgres’ libpq.5 which in turn is linked against Kerberos.framework and LDAP.framework.

So here's an example trace:

  1. User starts app server in preforking mode.
  2. Ruby loads gem bundle.
  3. App server forks.
  4. App calls a gem function, which (maybe through multiple layers) calls NSPlaceholderDictionary.initialize.
  5. One of the rules that Apple defined is that you may not call Foundation class initializers after forking. This function crashes the process immediately.

Responses from app server authors

This issue has spawned a long discussion on the Puma issue tracker and on Reddit /r/ruby. Jessica Stokes, Misty De Meo and Boaz Segev have done a lot of the initial work involved with figuring out the root cause of the problem, as well as finding workarounds (which we will cover later in this article).

It is clear the app servers are not the root cause of the issue, and that app servers cannot really solve the issue.

A major part of the discussion revolved around where a workaround belongs. Should app servers invoke the workaround, because they are the ones implementing preforking? Or should the workaround be included in Ruby, because it should provide a consistent environment across all operating systems? Or should users activate the workaround manually? As of 2017 October 13, the general consensus in that thread is that the workaround should be included in Ruby. While participants have also identified reasons why the Ruby core team may not want to include the workaround, it wouldn't hurt to try to convince them.

On October 12, Jessica opened an issue report on the Ruby bug tracker in an effort to convince the Ruby core team to include the workaround. I am also actively participating in this issue report because I too think that a workaround belongs in Ruby.

Puma and iodine: no decision so far

The Puma and iodine authors have not made a decision so far on what to do with the workarounds.

Unicorn: chosen to take no action

The Unicorn author, Eric Wong, has chosen not to take action on this issue on the grounds that he has no access to macOS.

Passenger: workaround included

Passenger's philosophy is to put user experience at the forefront. We believe that writing and managing apps should be easy, hassle-free and productive, and that software should serve users. Therefore we have chosen to include the workaround in the next Passenger version (5.1.11*), so that everything Just Works(tm) as much as possible.

How to make your app server work: activating the workaround manually

So if you are not using Passenger then you will have to manually activate a workaround to suppress the error message. Set the following environment variable before starting Puma, Unicorn or iodine:

export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES

If you are using Passenger, upgrade to 5.1.11, which will activate a workaround automatically.

Activating this workaround effectively returns macOS to pre-High Sierra behavior.

  • Note: that 5.1.9 and 5.1.10 will be skipped. We'll announce in the near future why that is so.

Towards a real solution: cooperation from gem authors required

The workaround does not really solve the problem. The fundamental problem -- that forking is at odds with multithreading -- is still there, even before High Sierra, and even on Linux. The workaround merely disables a safety check, and users are just lucky that they don't run into bad situations most of the time.

To guarantee that forking is safe, the application must not be running any threads at the point of fork (or the other threads must only be executing async-signal-safe code). MRI can guarantee this about its own code to some degree, but third party native libraries may spawn their own threads. So any gems that the user pulls in must not spawn threads until the app server has forked.

I propose the following to authors of gems with native extensions: all such gems should ensure that no threads are spawned during require time. Instead, they should ensure that threads are only spawned after calling an explicit initialization method. That way, users can configure the app server in such a way that threads are only initialized after forking, while preserving the advantages of copy-on-write preforking.

Conclusion

Ruby app servers use the preforking technique to improve speed and efficiency, but this technique is not without its dangers. In particular, it is fundamentally at odds with multithreading, no matter which operating system is used (it's not a macOS-specific problem). High Sierra introduced some sanity checks to prevent some of the pitfalls related to forking, which is why you may run into errors when using Ruby app servers on High Sierra.

When using Puma, Unicorn or iodine, you have to activate a workaround manually, but when using Passenger you simply have to upgrade to 5.1.11 and everything will be taken care of for you automatically.

To truly solve the problem, cooperation from the wider Ruby ecosystem is required. But until then, the status quo seems to be good enough in most situations, even though it's only because of luck of the draw.

Checkout these references and related stuff

Please consider reading the references below to deepen your understanding about this issue.

Also consider checking out Passenger. What sets us apart is our hardcore dedication towards performance and scalability, while also focusing on user experience, developer efficiency and happiness. Passenger is constantly being improved, with the latest major version being twice as fast thanks to a brand-new zero-copy HTTP engine, and features for keeping production stable. We take care of the hard stuff so that you don't have to.

Finally: we are hiring, for example for a developer advocate position.

Happy developing!