When spork puts a fork in your cucumber and a spanner in your specs

TL;DR: Getting Rails, RSpec, Cucumber and SimpleCov to play nicely with spork is a pain. However, it is possible to get them all working together. Ensure config.cache_classes = true, that Rails threadsafe mode (config.threadsafe!) is not enabled, and then see my spec_helper.rb file below.

So, I’m hacking again. How I have missed it. It has been a long time since I’ve had a good solid run of a week to write some code, largely, but not completely, uninterrupted by things like grant writing and meetings. It’s been a lot of fun. But I did lose a day this week due to a not so fun problem in my testing setup…

With a colleague, I’m working on a Rails app that will serve as a platform for some research we want to do along with the boss, and will hopefully be useful in its own right, thereby enabling us to make lots of moolah, build an interplanetary spacecraft, and retire to a little ecodome on Mars. But we sort of just plunged in. No specs, no integration tests. Naughty. It was getting to a point where it was a pain in the butt to test things by opening a browser and refreshing web pages. It’s not always easy to track down the source of a problem when loading web pages manually is your only recourse. And it’s a damn slow way of operating. And since we hadn’t written our specs and features up front, it means I wasn’t being very disciplined in what code I was writing:

“Look, I just wrote a couple of hundred lines of code!”

“Great! Was it all necessary? What feature does it implement?”

“Oh.”

This may be a slight exaggeration of a situation that may or may not have actually occurred. The point is, it was time (well past time, actually) to say hello to my friends RSpec, Cucumber, Machinist and SimpleCov. Getting an initial set of specs and features written was easy enough. But then, because the specs and features were taking longer to run than I thought they should, I tried to improve the runtime of the tests with spork. Things went a bit belly up at this point.

The main symptom was getting “NameError: uninitialized constant” all over the place. My tests couldn’t find my models or controllers. I tried moving code from spork’s prefork block to the each_run block. No cigar. I tried explicitly requiring libraries in each_run. I added the app directory of the Rails project to the autoload path since I suspected the eager_load! hacks that spork-rails introduces might have been playing silly buggers with Mongoid. Nothing. And I tried requiring everything under the app directory explicitly in the each_run block, which got me part of the way, but bombed out on Devise-related stuff. I googled. There was lots to read. But mostly, it was about how to stop spork from preloading all your models when you’re using Mongoid in your persistence layer. At this point I’d have been happy if my models were being preloaded! I did find one trick that fixed my specs but made Cucumber complain: set config.cache_classes to false in test.rb.

Then I remembered having switched on config.threadsafe! in application.rb a week or two earlier to solve a problem with EventMachine, Thin and Cramp (yes, the Cramp stuff will be served from a different web server than the Rails stuff shortly, but let’s get back to the problem at hand, shall we?), and I wondered whether this was the source of my current problems. I took a look at what threadsafe mode actually does:

# Enable threaded mode. Allows concurrent requests to controller actions and
# multiple database connections. Also disables automatic dependency loading
# after boot, and disables reloading code on every request, as these are
# fundamentally incompatible with thread safety.
def threadsafe!
  @preload_frameworks = true
  @cache_classes = true
  @dependency_loading = false
  @allow_concurrency = true
  self
end

Hmm… That looked like a possible cause, since spork was probably relying on reloading code on each run. So, I shifted the call to config.threadsafe! from application.rb to production.rb and development.rb, so that threadsafe mode would be on for production and development environments but not for my test environment. Et voilà! RSpec was happy. Cucumber was (almost) happy. Spork was happy. And now that I’d figured out what to Google, it turns out others have been onto this issue, too, though nobody seems to have delved into it very deeply as far as I can tell.

There was one more problem with my Cucumber features, though. It was complaining about a Devise helper that it didn’t seem to know about. But someone had been there before in the RSpec context. To fix the problem, I added the following code to the bottom of the spork prefork block in my Cucumber env.rb:

  class DeviseController
    include DeviseHelper
    include ActionView::Helpers::TagHelper
    helper_method :devise_error_messages!
    helper_method :content_tag
  end

I thought my problems were all solved. But alas, I had forgotten to check whether SimpleCov was still working as expected. Nope. There are well known issues when SimpleCov is used with spork. The most glaring one is that SimpleCov just doesn’t get run under spork. Starting SimpleCov in the each_run block when using spork as suggested by a participant in a discussion about the issue on GitHub at least enabled SimpleCov to run. However, I was getting vastly different coverage reports depending on whether I was running under spork or not. That wasn’t cool. SimpleCov needs to be started prior to loading any code that you want coverage reports for, but config/initializers/*.rb are run in prefork, before SimpleCov was starting (since I was now starting SimpleCov in each_run), and one of my initializers was executing custom code I had put in the lib directory of our Rails project. So, what to do?

The fix wasn’t too difficult to figure out. Instead of requiring environment.rb in the prefork block, I required application.rb. This meant that I could load the app without running the initializers. I could then call the initialize! method on my application explicitly in the each_run block after starting SimpleCov. A problem with this is that after the each_run block exits, spork calls initialize! again, blowing things up. But if I didn’t call initialize! explicitly in each_run, I ran into the original NameError problem. So, I just stubbed out the initialize! method after calling it. Ugly, but works. Here’s the entire spec_helper.rb file complete with some Spork.trap_method tricks I pinched from a blog article to prevent Mongoid from preloading the models:

require 'spork'
#uncomment the following line to use spork with the debugger
#require 'spork/ext/ruby-debug'

Spork.prefork do
  ENV["RAILS_ENV"] ||= 'test'
  unless ENV['DRB']
    require 'simplecov'
    SimpleCov.start 'rails'
  end

  require 'rails/application'

  # Use of https://github.com/sporkrb/spork/wiki/Spork.trap_method-Jujutsu
  Spork.trap_method(Rails::Application, :reload_routes!)
  Spork.trap_method(Rails::Application::RoutesReloader, :reload!)

  require 'rails/mongoid'
  Spork.trap_class_method(Rails::Mongoid, :load_models)

  # Prevent main application to eager_load in the prefork block (do not load files in autoload_paths)
  Spork.trap_method(Rails::Application, :eager_load!)

  require Rails.root.join("config/application")

  # Load all railties files
  Rails.application.railties.all { |r| r.eager_load! }

  require 'rspec/rails'
  require 'email_spec'
  require 'rspec/autorun'

  RSpec.configure do |config|
    config.include(EmailSpec::Helpers)
    config.include(EmailSpec::Matchers)

    config.infer_base_class_for_anonymous_controllers = true

    # Clean up the database
    require 'database_cleaner'
    config.before(:suite) do
      DatabaseCleaner.strategy = :truncation
      DatabaseCleaner.orm = "mongoid"
    end

    config.before(:each) do
      DatabaseCleaner.clean
    end

  end
  ActiveSupport::DescendantsTracker.clear
  ActiveSupport::Dependencies.clear
end

Spork.each_run do
  if ENV['DRB']
    require 'simplecov'
    SimpleCov.start 'rails'
    Statusdash::Application.initialize!
    class Statusdash::Application
      def initialize!; end
    end
  end

  # Requires supporting ruby files with custom matchers and macros, etc,
  # in spec/support/ and its subdirectories.
  Dir[Rails.root.join("spec/support/**/*.rb")].each {|f| require f}
end

Even after this, rspec spec and rspec –drb spec were giving slightly different results, but I was very close. It seemed one of the files from the aforementioned code in the lib directory was still being preloaded. Why? Well, I intend to eventually split out the code in the lib directory into its own repo/gem, and so I had structured it as a gem. In preparation for this eventuality, instead of loading it by adding it to the Rails autoload_paths in application.rb, I added it to Gemfile using the :path option, which meant that the “root” file in the gem was being loaded by bundler in boot.rb. This meant, of course, it was loading in the prefork block. My fix for this was to add :require => false in the Gemfile for that “gem”, and then require it explicitly in the intializer file that used it (which was now not being run until after forking due to the above hacks in spec_helper.rb).

Bingo! SimpleCov was now giving exactly the same coverage results for my specs with and without spork.

Finally, I have RSpec, Cucumber and SimpleCov all running nicely (and speedily) under spork. For what it’s worth, I’m seeing a five to six times speed up in user time for my specs with spork compared to no spork, and I have them automatically run with guard, which along with mongod and some other background processes is in turn managed by foreman. Was it worth all the hassle? It won’t get me my ecodome on Mars, but it was an interesting challenge, my tests run faster, and hopefully this post will help someone else.

Note: this post was updated on 24 July, 2012 to correct the claim of a seven to eight times speed up to a five to six times speed up in terms of user time. Roughly 8.5 seconds compared to 1.5 seconds on a small suite of specs with SimpleCov enabled.