Spork to Spin

In my last post, I documented the trials and tribulations of getting spork working properly with RSpec, Cucumber, SimpleCov and Mongoid, and a solution I devised to the problems. I also posted my solution to the SimpleCov issues list on GitHub.

Then Christoph Olszowka, the maintainer of SimpleCov introduced me to spin, a lightweight alternative to spork for fast Rails testing written by Jesse Storimer. It takes a different approach to DRb-based solution like spork, which use remote method invocations. Essentially, spin loads up Rails (but does not initialize it) in one process (spin serve), and then you submit RSpec or Test::Unit jobs to it over a unix socket using another process (spin push). This causes spin serve to fork a child process (in much the same way as spork), which loads up the relevant test framework (RSpec or Test::Unit), initializing Rails along the way. This splitting of loading and initializing Rails mirrors part of the spork solution in my previous post. Another thing about spin is that, unlike spork, it doesn’t require you to alter your spec_helper.rb or test_helper.rb file. It just works. It’s also tiny: one small executable ruby file. Simple solutions appeal to me greatly.

However (there’s always a however), it didn’t support Cucumber. Also, spin serve would load up only one test environment (either Test::Unit or RSpec, but not both). Also, the guard plugin for spin (guard-spin) is in its early days, and as such is a little immature. For instance, the Guardfile template it ships with doesn’t watch config/**/*.rb for changes so that it knows when to restart the spin serve process. And if you added that watcher in, it still wouldn’t work because the plugin is only expecting changes on application and test/spec code (i.e., so that it runs spin push). I guess you could always force a reload on the guard CLI, but I shouldn’t have to do that. The other thing with guard-spin is that the spin configuration block in Guardfile needs to watch the relevant files for RSpec and Test::Unit. And with me about to add Cucumber support, that was going to get long and messy.

So, yesterday, sick at home with the flu (which hasn’t yet let up, dammit!) and a nice fever, I set about modifying spin so that it would support Cucumber. But I wanted it so that one spin serve process would handle Cucumber and RSpec (and Test::Unit, but I don’t use it) at the same time. This is because managing one spin serve process through guard is much easier than managing several. Take a look at the guard-spork runner.rb, which needs to manage sporks and global sporks (?) and whatnot to enable multiple sporks to run at the same time listening on a priori agreed ports for the various testing frameworks.

Changing spin to enable this was fairly simple, because spin itself was already an elegant piece of code, IMHO. The other thing I needed to change in spin was the client side of things. Instead of issuing spin push <path1> <path2> ... to submit jobs to spin serve, I wanted to be able to issue spin rspec, spin cucumber or spin testunit with the paths as argument (and also with no paths as argument, so that spin serve would look in the default locations for specs/features/tests). This way the client selects which test framework to run. You just need to make sure you invoked the server with the right options to load the required test frameworks (using the --rspec, --cucumber and --testunit options to spin serve). Interestingly, this all seems to work even when Rails is running in threadsafe mode, which I was not able to achieve with my spork solution. My version of spin is over on GitHub, and I’m going to have a chat with Jesse about whether this is something he might want to pull back into his version of spin.

Of course, a simpler solution isn’t worth as much if it doesn’t retain the performance gains of the more complex solution. Recall that I was seeing a five- to six-fold improvement (in terms of user time) with spork over plain rspec. Here is the comparison between plain rspec and spin rspec on the same specs (and with SimpleCov enabled) that I tested my spork solution with:

Plain rspec:

real	0m9.499s
user	0m8.455s
sys 	0m0.990s

Now spin rspec:

real	0m4.915s
user	0m1.528s
sys 	0m0.106s

Awesome! About the same improvement as the DRb solution, but without the various bits of spork jujitsu and spec_helper.rb modification to make it work. Note that to conduct this very simple benchmark I used the --push-results switch with spin serve so that the output was written to STDOUT by the client rather than the server, which is how rspec --drb works.

That done, the next step was making it all work with guard. I modified guard-spin so that it only handles the spin serve side of things (just like guard-spork handles only starting/restarting/stopping spork), and created the rather ugly-named guard-spin_rspec and guard-spin_cucumber (just like guard-rspec and guard-cucumber). I haven’t done guard-spin_testunit. These guys watch the application and test code directories for changes, and then issue spin rspec|cucumber <path1> <path2> ... as necessary.

Here are the relevant bits of my Gemfile for getting this set up:

gem "spin", :github => "rickyrobinson/spin", :branch => "cucumber"
gem "guard", ">= 0.6.2"
gem "guard-spin", :github => "rickyrobinson/guard-spin", :branch => "cucumber"
gem "guard-spin_rspec", :github => "rickyrobinson/guard-spin_rspec"
gem "guard-spin_cucumber", :github => "rickyrobinson/guard-spin_cucumber"

And, this is the relevant Guardfile config, which you can include automatically with guard init spin|spin_rspec|spin_cucumber:

# Start the spin server with RSpec and Cucumber support, and report time for each run
guard 'spin', :cli => "--time --rspec --cucumber" do
  # Spin itself
  watch('config/application.rb')
  watch('config/environment.rb')
  watch(%r{^config/environments/.*\.rb$})
  watch(%r{^config/initializers/.*\.rb$})
  watch('Gemfile')
  watch('Gemfile.lock')
end

guard 'spin_rspec' do
  # RSpec
  # uses the .rspec file
  # --colour --fail-fast --format documentation --tag ~slow
  watch(%r{^spec/(.+)_spec\.rb$})
  watch('spec/spec_helper.rb')                        { "spec" }
  watch(%r{^app/(.+)\.rb$})                           { |m| "spec/#{m[1]}_spec.rb" }
  watch(%r{^app/(.+)\.haml$})                         { |m| "spec/#{m[1]}.haml_spec.rb" }
  watch(%r{^lib/(.+)\.rb$})                           { |m| "spec/lib/#{m[1]}_spec.rb" }
  watch(%r{^app/controllers/(.+)_(controller)\.rb$})  { |m| ["spec/routing/#{m[1]}_routing_spec.rb", "spec/#{m[2]}s/#{m[1]}_#{m[2]}_spec.rb", "spec/requests/#{m[1]}_spec.rb"] }
end

guard 'spin_cucumber' do
  # Cucumber
  watch(%r{^features/(.+)\.feature$})
  watch(%r{^features/support/.+$})          { 'features' }
  watch(%r{^features/step_definitions/(.+)_steps\.rb$}) { |m| Dir[File.join("**/#{m[1]}.feature")][0] || 'features' }
end

There are still a number of things that could be improved, so please go ahead and fork any of this stuff on GitHub. For instance, the client might try to push jobs to the server before the server has started up properly. This might be fixed by enabling the client or the server to create the socket. Also, it would be nice if guard-spin_[rpsec|cucumber] worked a bit more like guard-[rspec|cucumber] which are a little smarter about which specs/features to run when something changes. Ideally, it would be nice to modify cucumber and rspec so that they know about spin. Then we could just pass --spin instead of --drb and we could ditch guard-spin_[rpsec|cucumber]. It might also be better if the spin client determined whether results should be pushed back or not (i.e., the --push-results switch should be given to spin rspec|cucumber|testunit instead of spin serve). Anyhow, please give it a try.

Now to medicate myself.