Categories
Innovation

Spork to Spin

In my last post, I documented the trials and tribulations of getting spork working properly with RSpec, Cucumber, SimpleCov and Mongoid, and a solution I devised to the problems. I also posted my solution to the SimpleCov issues list on GitHub.

Then Christoph Olszowka, the maintainer of SimpleCov introduced me to spin, a lightweight alternative to spork for fast Rails testing written by Jesse Storimer. It takes a different approach to DRb-based solution like spork, which use remote method invocations. Essentially, spin loads up Rails (but does not initialize it) in one process (spin serve), and then you submit RSpec or Test::Unit jobs to it over a unix socket using another process (spin push). This causes spin serve to fork a child process (in much the same way as spork), which loads up the relevant test framework (RSpec or Test::Unit), initializing Rails along the way. This splitting of loading and initializing Rails mirrors part of the spork solution in my previous post. Another thing about spin is that, unlike spork, it doesn’t require you to alter your spec_helper.rb or test_helper.rb file. It just works. It’s also tiny: one small executable ruby file. Simple solutions appeal to me greatly.

However (there’s always a however), it didn’t support Cucumber. Also, spin serve would load up only one test environment (either Test::Unit or RSpec, but not both). Also, the guard plugin for spin (guard-spin) is in its early days, and as such is a little immature. For instance, the Guardfile template it ships with doesn’t watch config/**/*.rb for changes so that it knows when to restart the spin serve process. And if you added that watcher in, it still wouldn’t work because the plugin is only expecting changes on application and test/spec code (i.e., so that it runs spin push). I guess you could always force a reload on the guard CLI, but I shouldn’t have to do that. The other thing with guard-spin is that the spin configuration block in Guardfile needs to watch the relevant files for RSpec and Test::Unit. And with me about to add Cucumber support, that was going to get long and messy.

So, yesterday, sick at home with the flu (which hasn’t yet let up, dammit!) and a nice fever, I set about modifying spin so that it would support Cucumber. But I wanted it so that one spin serve process would handle Cucumber and RSpec (and Test::Unit, but I don’t use it) at the same time. This is because managing one spin serve process through guard is much easier than managing several. Take a look at the guard-spork runner.rb, which needs to manage sporks and global sporks (?) and whatnot to enable multiple sporks to run at the same time listening on a priori agreed ports for the various testing frameworks.

Changing spin to enable this was fairly simple, because spin itself was already an elegant piece of code, IMHO. The other thing I needed to change in spin was the client side of things. Instead of issuing spin push <path1> <path2> ... to submit jobs to spin serve, I wanted to be able to issue spin rspec, spin cucumber or spin testunit with the paths as argument (and also with no paths as argument, so that spin serve would look in the default locations for specs/features/tests). This way the client selects which test framework to run. You just need to make sure you invoked the server with the right options to load the required test frameworks (using the --rspec, --cucumber and --testunit options to spin serve). Interestingly, this all seems to work even when Rails is running in threadsafe mode, which I was not able to achieve with my spork solution. My version of spin is over on GitHub, and I’m going to have a chat with Jesse about whether this is something he might want to pull back into his version of spin.

Of course, a simpler solution isn’t worth as much if it doesn’t retain the performance gains of the more complex solution. Recall that I was seeing a five- to six-fold improvement (in terms of user time) with spork over plain rspec. Here is the comparison between plain rspec and spin rspec on the same specs (and with SimpleCov enabled) that I tested my spork solution with:

Plain rspec:

real	0m9.499s
user	0m8.455s
sys 	0m0.990s

Now spin rspec:

real	0m4.915s
user	0m1.528s
sys 	0m0.106s

Awesome! About the same improvement as the DRb solution, but without the various bits of spork jujitsu and spec_helper.rb modification to make it work. Note that to conduct this very simple benchmark I used the --push-results switch with spin serve so that the output was written to STDOUT by the client rather than the server, which is how rspec --drb works.

That done, the next step was making it all work with guard. I modified guard-spin so that it only handles the spin serve side of things (just like guard-spork handles only starting/restarting/stopping spork), and created the rather ugly-named guard-spin_rspec and guard-spin_cucumber (just like guard-rspec and guard-cucumber). I haven’t done guard-spin_testunit. These guys watch the application and test code directories for changes, and then issue spin rspec|cucumber <path1> <path2> ... as necessary.

Here are the relevant bits of my Gemfile for getting this set up:

gem "spin", :github => "rickyrobinson/spin", :branch => "cucumber"
gem "guard", ">= 0.6.2"
gem "guard-spin", :github => "rickyrobinson/guard-spin", :branch => "cucumber"
gem "guard-spin_rspec", :github => "rickyrobinson/guard-spin_rspec"
gem "guard-spin_cucumber", :github => "rickyrobinson/guard-spin_cucumber"

And, this is the relevant Guardfile config, which you can include automatically with guard init spin|spin_rspec|spin_cucumber:

# Start the spin server with RSpec and Cucumber support, and report time for each run
guard 'spin', :cli => "--time --rspec --cucumber" do
  # Spin itself
  watch('config/application.rb')
  watch('config/environment.rb')
  watch(%r{^config/environments/.*\.rb$})
  watch(%r{^config/initializers/.*\.rb$})
  watch('Gemfile')
  watch('Gemfile.lock')
end

guard 'spin_rspec' do
  # RSpec
  # uses the .rspec file
  # --colour --fail-fast --format documentation --tag ~slow
  watch(%r{^spec/(.+)_spec\.rb$})
  watch('spec/spec_helper.rb')                        { "spec" }
  watch(%r{^app/(.+)\.rb$})                           { |m| "spec/#{m[1]}_spec.rb" }
  watch(%r{^app/(.+)\.haml$})                         { |m| "spec/#{m[1]}.haml_spec.rb" }
  watch(%r{^lib/(.+)\.rb$})                           { |m| "spec/lib/#{m[1]}_spec.rb" }
  watch(%r{^app/controllers/(.+)_(controller)\.rb$})  { |m| ["spec/routing/#{m[1]}_routing_spec.rb", "spec/#{m[2]}s/#{m[1]}_#{m[2]}_spec.rb", "spec/requests/#{m[1]}_spec.rb"] }
end

guard 'spin_cucumber' do
  # Cucumber
  watch(%r{^features/(.+)\.feature$})
  watch(%r{^features/support/.+$})          { 'features' }
  watch(%r{^features/step_definitions/(.+)_steps\.rb$}) { |m| Dir[File.join("**/#{m[1]}.feature")][0] || 'features' }
end

There are still a number of things that could be improved, so please go ahead and fork any of this stuff on GitHub. For instance, the client might try to push jobs to the server before the server has started up properly. This might be fixed by enabling the client or the server to create the socket. Also, it would be nice if guard-spin_[rpsec|cucumber] worked a bit more like guard-[rspec|cucumber] which are a little smarter about which specs/features to run when something changes. Ideally, it would be nice to modify cucumber and rspec so that they know about spin. Then we could just pass --spin instead of --drb and we could ditch guard-spin_[rpsec|cucumber]. It might also be better if the spin client determined whether results should be pushed back or not (i.e., the --push-results switch should be given to spin rspec|cucumber|testunit instead of spin serve). Anyhow, please give it a try.

Now to medicate myself.

Categories
Innovation

When spork puts a fork in your cucumber and a spanner in your specs

TL;DR: Getting Rails, RSpec, Cucumber and SimpleCov to play nicely with spork is a pain. However, it is possible to get them all working together. Ensure config.cache_classes = true, that Rails threadsafe mode (config.threadsafe!) is not enabled, and then see my spec_helper.rb file below.

So, I’m hacking again. How I have missed it. It has been a long time since I’ve had a good solid run of a week to write some code, largely, but not completely, uninterrupted by things like grant writing and meetings. It’s been a lot of fun. But I did lose a day this week due to a not so fun problem in my testing setup…

With a colleague, I’m working on a Rails app that will serve as a platform for some research we want to do along with the boss, and will hopefully be useful in its own right, thereby enabling us to make lots of moolah, build an interplanetary spacecraft, and retire to a little ecodome on Mars. But we sort of just plunged in. No specs, no integration tests. Naughty. It was getting to a point where it was a pain in the butt to test things by opening a browser and refreshing web pages. It’s not always easy to track down the source of a problem when loading web pages manually is your only recourse. And it’s a damn slow way of operating. And since we hadn’t written our specs and features up front, it means I wasn’t being very disciplined in what code I was writing:

“Look, I just wrote a couple of hundred lines of code!”

“Great! Was it all necessary? What feature does it implement?”

“Oh.”

This may be a slight exaggeration of a situation that may or may not have actually occurred. The point is, it was time (well past time, actually) to say hello to my friends RSpec, Cucumber, Machinist and SimpleCov. Getting an initial set of specs and features written was easy enough. But then, because the specs and features were taking longer to run than I thought they should, I tried to improve the runtime of the tests with spork. Things went a bit belly up at this point.

The main symptom was getting “NameError: uninitialized constant” all over the place. My tests couldn’t find my models or controllers. I tried moving code from spork’s prefork block to the each_run block. No cigar. I tried explicitly requiring libraries in each_run. I added the app directory of the Rails project to the autoload path since I suspected the eager_load! hacks that spork-rails introduces might have been playing silly buggers with Mongoid. Nothing. And I tried requiring everything under the app directory explicitly in the each_run block, which got me part of the way, but bombed out on Devise-related stuff. I googled. There was lots to read. But mostly, it was about how to stop spork from preloading all your models when you’re using Mongoid in your persistence layer. At this point I’d have been happy if my models were being preloaded! I did find one trick that fixed my specs but made Cucumber complain: set config.cache_classes to false in test.rb.

Then I remembered having switched on config.threadsafe! in application.rb a week or two earlier to solve a problem with EventMachine, Thin and Cramp (yes, the Cramp stuff will be served from a different web server than the Rails stuff shortly, but let’s get back to the problem at hand, shall we?), and I wondered whether this was the source of my current problems. I took a look at what threadsafe mode actually does:

# Enable threaded mode. Allows concurrent requests to controller actions and
# multiple database connections. Also disables automatic dependency loading
# after boot, and disables reloading code on every request, as these are
# fundamentally incompatible with thread safety.
def threadsafe!
  @preload_frameworks = true
  @cache_classes = true
  @dependency_loading = false
  @allow_concurrency = true
  self
end

Hmm… That looked like a possible cause, since spork was probably relying on reloading code on each run. So, I shifted the call to config.threadsafe! from application.rb to production.rb and development.rb, so that threadsafe mode would be on for production and development environments but not for my test environment. Et voilà! RSpec was happy. Cucumber was (almost) happy. Spork was happy. And now that I’d figured out what to Google, it turns out others have been onto this issue, too, though nobody seems to have delved into it very deeply as far as I can tell.

There was one more problem with my Cucumber features, though. It was complaining about a Devise helper that it didn’t seem to know about. But someone had been there before in the RSpec context. To fix the problem, I added the following code to the bottom of the spork prefork block in my Cucumber env.rb:

  class DeviseController
    include DeviseHelper
    include ActionView::Helpers::TagHelper
    helper_method :devise_error_messages!
    helper_method :content_tag
  end

I thought my problems were all solved. But alas, I had forgotten to check whether SimpleCov was still working as expected. Nope. There are well known issues when SimpleCov is used with spork. The most glaring one is that SimpleCov just doesn’t get run under spork. Starting SimpleCov in the each_run block when using spork as suggested by a participant in a discussion about the issue on GitHub at least enabled SimpleCov to run. However, I was getting vastly different coverage reports depending on whether I was running under spork or not. That wasn’t cool. SimpleCov needs to be started prior to loading any code that you want coverage reports for, but config/initializers/*.rb are run in prefork, before SimpleCov was starting (since I was now starting SimpleCov in each_run), and one of my initializers was executing custom code I had put in the lib directory of our Rails project. So, what to do?

The fix wasn’t too difficult to figure out. Instead of requiring environment.rb in the prefork block, I required application.rb. This meant that I could load the app without running the initializers. I could then call the initialize! method on my application explicitly in the each_run block after starting SimpleCov. A problem with this is that after the each_run block exits, spork calls initialize! again, blowing things up. But if I didn’t call initialize! explicitly in each_run, I ran into the original NameError problem. So, I just stubbed out the initialize! method after calling it. Ugly, but works. Here’s the entire spec_helper.rb file complete with some Spork.trap_method tricks I pinched from a blog article to prevent Mongoid from preloading the models:

require 'spork'
#uncomment the following line to use spork with the debugger
#require 'spork/ext/ruby-debug'

Spork.prefork do
  ENV["RAILS_ENV"] ||= 'test'
  unless ENV['DRB']
    require 'simplecov'
    SimpleCov.start 'rails'
  end

  require 'rails/application'

  # Use of https://github.com/sporkrb/spork/wiki/Spork.trap_method-Jujutsu
  Spork.trap_method(Rails::Application, :reload_routes!)
  Spork.trap_method(Rails::Application::RoutesReloader, :reload!)

  require 'rails/mongoid'
  Spork.trap_class_method(Rails::Mongoid, :load_models)

  # Prevent main application to eager_load in the prefork block (do not load files in autoload_paths)
  Spork.trap_method(Rails::Application, :eager_load!)

  require Rails.root.join("config/application")

  # Load all railties files
  Rails.application.railties.all { |r| r.eager_load! }

  require 'rspec/rails'
  require 'email_spec'
  require 'rspec/autorun'

  RSpec.configure do |config|
    config.include(EmailSpec::Helpers)
    config.include(EmailSpec::Matchers)

    config.infer_base_class_for_anonymous_controllers = true

    # Clean up the database
    require 'database_cleaner'
    config.before(:suite) do
      DatabaseCleaner.strategy = :truncation
      DatabaseCleaner.orm = "mongoid"
    end

    config.before(:each) do
      DatabaseCleaner.clean
    end

  end
  ActiveSupport::DescendantsTracker.clear
  ActiveSupport::Dependencies.clear
end

Spork.each_run do
  if ENV['DRB']
    require 'simplecov'
    SimpleCov.start 'rails'
    Statusdash::Application.initialize!
    class Statusdash::Application
      def initialize!; end
    end
  end

  # Requires supporting ruby files with custom matchers and macros, etc,
  # in spec/support/ and its subdirectories.
  Dir[Rails.root.join("spec/support/**/*.rb")].each {|f| require f}
end

Even after this, rspec spec and rspec –drb spec were giving slightly different results, but I was very close. It seemed one of the files from the aforementioned code in the lib directory was still being preloaded. Why? Well, I intend to eventually split out the code in the lib directory into its own repo/gem, and so I had structured it as a gem. In preparation for this eventuality, instead of loading it by adding it to the Rails autoload_paths in application.rb, I added it to Gemfile using the :path option, which meant that the “root” file in the gem was being loaded by bundler in boot.rb. This meant, of course, it was loading in the prefork block. My fix for this was to add :require => false in the Gemfile for that “gem”, and then require it explicitly in the intializer file that used it (which was now not being run until after forking due to the above hacks in spec_helper.rb).

Bingo! SimpleCov was now giving exactly the same coverage results for my specs with and without spork.

Finally, I have RSpec, Cucumber and SimpleCov all running nicely (and speedily) under spork. For what it’s worth, I’m seeing a five to six times speed up in user time for my specs with spork compared to no spork, and I have them automatically run with guard, which along with mongod and some other background processes is in turn managed by foreman. Was it worth all the hassle? It won’t get me my ecodome on Mars, but it was an interesting challenge, my tests run faster, and hopefully this post will help someone else.

Note: this post was updated on 24 July, 2012 to correct the claim of a seven to eight times speed up to a five to six times speed up in terms of user time. Roughly 8.5 seconds compared to 1.5 seconds on a small suite of specs with SimpleCov enabled.