Automated cross-browser testing with BrowserStack and CircleCI

By now, automated testing of code has hopefully become an industry standard. Ideally, you write your tests first and make them a runnable specification of what your code should do. When done right, test-driven development can improve code design, not mentioning you have a regression test suite to stop you from accidentally breaking things in the future. 

However, unit testing does just what it says on the tin: tests the code units (modules, classes, functions) in isolation. To know the whole application or system works, you need to test the integration of those modules.

That's nothing new either. At least in the web application world, which this post is about, we've had tools like Cucumber (which lets you write user scenarios in an almost human language) for years. You can then run these tests on a continuous integration server (we use the amazing CircleCI) and get a green light for every commit you push.

But when it comes to testing how things work in different web browsers, the situation is not that ideal. Or rather it wasn't. 

Automated testing in a real browser

The golden standard of automated testing against a real browser is Selenium, the browser automation tool that can drive many different browsers using a common API. In the ruby world, there are tools on top of Selenium providing a nice DSL for driving the browsers using domain specific commands like page.click 'Login' and expectations like page.has_content?('something').

Selenium will open a browser and run through your scripted scenario and check that everything you expected to happen did actually happen. This should still be an old story to you. You can improve on the default setup by using a faster headless browser (like PhantomJS), although watching your test complete a payment flow on PayPal is kinda cool. There is still a big limitation though.

When you need to test your application on multiple browsers, versions, operating systems and devices, you first need to have all that hardware and software and second, you need to run your test suite on all of them.

So far, we've mostly solved this by having human testers. But making humans test applications is a human rights violation and a time of a good tester is much better spent creatively trying to break things in an unexpected way. For some projects, there even isn't enough budget for a dedicated tester.

This is where cloud services, once again, come to the rescue. And the one we'll use is called BrowserStack.

BrowserStack

BrowserStack allows you to test your web applications in almost every combination of browser and OS/Device you can think of, all from your web browser. It spins up the right VM for you and gives you a remote screen to play around. That solves the first part of our problem, we no longer need to have all those devices and browsers. You can try it yourself at http://www.browserstack.com/.

Amazingly, BrowserStack solves even the second part of the problem by offering the automate feature: it can act as a Selenium server, to which you can connect your test suite by using Selenium remote driver and automate the testing. It even offers up to ten parallel testing sessions!

Testing an existing website

To begin with, let's configure a Cucumber test suite to run against a staging deployment of your application. That has it's limitations - you can only do things to the application that a real user could, so forget mocking and stubbing for now (but keep on reading).

We'll demonstrate the setup with a rails application, using cucumber and Capybara and assume you already have some scenario to run.

First, you need to tell Capybara what hostname to use instead of localhost

# features/support/capybara.rb

Capybara.app_host = 'http://staging.my-app.com'

Next, loosely following the BrowserStack documentation we'll configure the remote driver. Start with building the browser stack URL using environment variables to set the username and API authorization key.

# features/support/cross_browser.rb

url = "http://#{ENV['BS_USERNAME']}:#{ENV['BS_AUTHKEY']}@hub.browserstack.com/wd/hub"

then we need to set the desired capabilities of the remote browser. Let's ask for Chrome 33 on OS X Mavericks.

capabilities = Selenium::WebDriver::Remote::Capabilities.new
capabilities["os"] = "OS X"
capabilities["os_version"] = "Mavericks"
capabilities["browser"] = "chrome"
capabilities["browser_version"] = "33.0"
capabilities['browserstack.debug'] = true
capabilities['takesScreenshot'] = true

capabilities['project'] = "My project"
if ENV['BUILD_NUM']
  capabilities['build'] = ENV['BUILD_NUM']
end

Next step is to register a driver with these capabilities with Capybara 

# register a driver
driver_name = "chrome_33.0_Mavericks"
Capybara.register_driver(driver_name) do |app|
  Capybara::Selenium::Driver.new(app, :browser => :remote, :url => url, :desired_capabilities => capabilities)
end

and use it

# use the registered driver
Capybara.default_driver = driver_name
Capybara.javascript_driver = driver_name

If you run cucumber now, it should connect to BrowserStack and run your scenario. You can even watch it happen live in the Automate section!

Ok, that was a cool experiment, but we wanted multiple browsers and the ability to run on BrowserStack only when needed would be good as well.

Multiple different browsers

What we want then, is to be able to run a simple command to run cross-browser tests in one browser or a whole set of them. Something like

rake cross_browser

and

rake cross_browser:chrome

In fact, let's do exactly that. First of all, list all the browsers you want in a browsers.json in the root of your project 

Each of those browser configurations is stored under a short key we'll use throughout the configuration to make things simple.

The rake task will look something like the following

{
   "chrome": {
      "human": "Chrome 33 on OS X Mavericks",
      "browser_version":"33.0",
      "browser":"chrome",
      "os":"OS X",
      "os_version":"Mavericks"
   },
   "firefox": {
      "human": "Firefox 27 on OS X Mavericks",
      "browser_version":"27.0",
      "browser":"Firefox",
      "os":"OS X",
      "os_version":"Mavericks"
   },
   "ie_9": {
      "human": "IE 9 on Windows 7",
      "browser_version":"9.0",
      "browser":"IE",
      "os":"Windows",
      "os_version":"7"
   },
   "ie_10": {
      "human": "IE 10 on Windows 7",
      "browser_version":"10.0",
      "browser":"IE",
      "os":"Windows",
      "os_version":"7"
   }
}

First we load the JSON file and store it in a constant. Then we define a task that goes through the list and for each browser executes a browser specific task. The browser tasks are under a cross_browser namespace.

To pass the browser configuration to Capybara when Cucumber gets executed we'll use an environment variable. Instead of passing the whole configuration we can just pass the browser key and load the rest in the configuration itself. To be able to pass the environment variable based on the task name, we need to wrap the actual cucumber task in another task.

The inner task then extends the Cucumber::Rake::Task and provides some configuration for cucumber. Notice especially the --tags option, which means you can specifically tag Cucumber scenarios for cross-browser execution, only running the necessary subset to keep the time down (your daily time running BrowserStack sessions is likely limited after all).

The cross_browser.rb changes to the following:

# features/support/cross_browser.rb

def remote_browser?
  ENV.has_key? 'BROWSER_TASK_NAME'
end

if remote_browser?
  raise 'Please ensure you supply ENV variables BS_USERNAME and BS_AUTHKEY from browser stack' if !ENV['BS_USERNAME'] || !ENV['BS_AUTHKEY']

  url = "http://#{ENV['BS_USERNAME']}:#{ENV['BS_AUTHKEY']}@hub.browserstack.com/wd/hub"

  # load browser configuration
  browser_data = JSON.load(open('.browsers.json'))
  browser_name = ENV['BROWSER_TASK_NAME']

  browser = browser_data[browser_name]

  puts "Testing in #{browser['human']} (#{ENV['BROWSER_TASK_NAME']})..."

  # translate into Selenium Capabilities
  capabilities = Selenium::WebDriver::Remote::Capabilities.new
  capabilities["os"] = browser["os"]
  capabilities["os_version"] = browser["os_version"]
  capabilities["browser"] = browser["browser"]
  capabilities["browser_version"] = browser["browser_version"]
  capabilities['browserstack.debug'] = true
  capabilities['takesScreenshot'] = true

  capabilities['project'] = "My project"
  if ENV['BUILD_NUM']
    capabilities['build'] = ENV['BUILD_NUM']
  end

  # register a driver
  driver_name = "#{browser['browser']}_#{browser['browser_version']}_#{browser['os']}_#{browser['os_version']}"
  Capybara.register_driver(driver_name) do |app|
    Capybara::Selenium::Driver.new(app, :browser => :remote, :url => url, :desired_capabilities => capabilities)
  end

  # use the registered driver
  Capybara.default_driver = driver_name
  Capybara.javascript_driver = driver_name
end

That should now let you run

rake cross_browser

and watch the four browsers fly through your your scenarios one after another.

We've used this setup with a few modifications for a while. It has a serious limitation however. Because the remote browsers is accessing a real site, it can only do as much as a real user can do. The initial state setup and repeatability is difficult. Not mentioning it isn't the fastest solution. We really need to run the application locally.

Local testing

Running your application locally and letting Capybara start your server enables you to do everything you are used to in your automated tests - load fixtures, create data with factories, mock and stub pieces of your infrastructure, etc. But how can a browser running in a cloud access your local machine? You will need to dig a tunnel.

BrowserStack provides a set of binaries able to open a tunnel to the remote VM and connect to any hostname and port from the local one. The remote browser can then connect to that hostname as if it could itself access it. You can read all about it in the documentation.

After you downloaded a BrowserStack tunnel binary for your platform, you'll need to change the configuration again. The app_host is localhost once again and we also need Capybara to start a local server for us.

# features/support/capybara.rb

Capybara.run_server = true
Capybara.server_port = 3001
Capybara.app_host = 'http://127.0.0.1:3001'

We also need to tell BrowserStack we want to use the tunnel. Just add

capabilities['browserstack.local'] = true

to the list of capabilities. Start the tunnel and run the specs again

./BrowserStackLocal -skipCheck $BS_AUTHKEY 127.0.0.1,3001 &
rake cross_browser

This time everything should go a bit faster. You can also test more complex systems that need external APIs or direct access to your data store because you can now mock those.

This is great! I want that to run for every single build before it's deployed like my unit tests. Testing everything as much as possible is what CI servers are for after all.

Running on CircleCI

We really like CircleCI for it's reliability, great UI and especially it's ease of configuration and libraries and services support.

On top of that, their online chat support deserves a praise in a separate paragraph. Someone is in the chat room all the time, responds almost immediately and they are always very helpful. They even fix an occasional bug in near real time.

To run our cross browser tests on CircleCI we will need a circle.yml file and a few changes to the configuration. The circle.yml will contain the following

dependencies:
  cache_directories:
    - browserstack
test:
  override:
    - RAILS_ENV=test bundle exec rake spec
    - RAILS_ENV=test bundle exec cucumber
    - script/ci/browserstack_tunnel.sh
    - browserstack/BrowserStackLocal -skipCheck $BS_AUTHKEY 127.0.0.1,3001
        background: true
    - RAILS_ENV=test bundle exec rake cross_browser
    - script/ci/browserstack_tunnel.sh stop:

We run unit tests, then cucumber specs normally, then open the tunnel and run our rake task. When it's done, we can close the tunnel again. To download and eventually stop the tunnel we wrote a little shell script

#!/bin/bash

if [[ "$1" == "stop" ]]
then
  echo "Stopping browserstack tunnel..."
  killall BrowserStackLocal
  exit
fi

if [[ ! -e browserstack ]]
then
  echo "Downloading browserstack tunnel..."
  mkdir -p browserstack
  curl https://www.browserstack.com/browserstack-local/BrowserStackLocal-linux-x64.zip > browserstack/BrowserStackLocal-linux-x64.zip
  cd browserstack && unzip BrowserStackLocal-linux-x64.zip && chmod a+x BrowserStackLocal
  cd ..
fi

echo "Browserstack tunnel ready..."

It downloads the 64-bit linux browserstack binary and unpacks it into a browserstack directory (which is cached by CircleCI). When passed a stop parameter, it will kill all the browserstack tunnels running. (We will eventually make the script start the tunnel as well, but we had problems with backgrounding the process so it's done as an explicit step for now).

Finally, we can update the configuration to use the project name and build number supplied by Circle to name the builds for BrowserStack

capabilities['project'] = ENV['CIRCLE_PROJECT_REPONAME'] if ENV['CIRCLE_PROJECT_REPONAME']
capabilities['build'] = ENV['CIRCLE_BUILD_NUM'] if ENV['CIRCLE_BUILD_NUM']

That setup should work, but it will take a while going through all the browsers. That is a problem when you work in multiple branches in parallel, because the testing becomes a race for resources. We can use another brilliant feature of CircleCI to limit the impact of this issue: we can run the tests in parallel.

The holy grail

Marking any task in circle.yml with parallel: true will make it run in multiple containers at the same time. You can than scale your build up to as many containers you want (and are willing to pay for). We are limited by the concurrency BrowserStack offers us and on top of that we're using just 4 browsers anyway, so let's start with four, but plan for more devices.

First, we need to spread the individual browser jobs across the containers. We can use the environment variables provided by CircleCI to see which container we're running on. Our final rake task will look like this

desc 'Run all cross browser tests in parallel (with ENV["nodes"] parallel limit)'
task :cross_browser => [:environment] do
  parallel_limit = ENV["nodes"] || 2
  parallel_limit = parallel_limit.to_i

  sliced = BROWSERS.keys.each_slice(parallel_limit).to_a
  browser_groups = sliced.first.zip(*sliced[1..-1])

  browser_groups.each_with_index do |browser_group, group_index|
    browser_group.compact.each do |browser_name|
      if ENV['CIRCLE_NODE_INDEX'].to_i == group_index
        puts "Cross browser testing against #{browser_name}."

        Rake::Task["reset"].execute
        Rake::Task["cross_browser:#{browser_name}"].execute
      end
    end
  end

Reading the nodes environment variable we check the concurrency limit and spread the browsers across the same number of buckets. For each bucket, we'll only run the actual test if the CIRCLE_NODE_INDEX is the same as the order of the bucket.

Because we're now opening multiple tunnels to BrowserStack, we need to name them. Add

capabilities['browserstack.localIdentifier'] = "#{ENV['CIRCLE_PROJECT_REPONAME']}-node-#{ENV['CIRCLE_NODE_INDEX']}"

to the capabilities configuration in cross_browser.rb. The final file looks like this

def remote_browser?
  ENV.has_key? 'BROWSER_TASK_NAME'
end

if remote_browser?
  raise 'Please ensure you supply ENV variables BS_USERNAME and BS_AUTHKEY from browser stack' if !ENV['BS_USERNAME'] || !ENV['BS_AUTHKEY']

  url = "http://#{ENV['BS_USERNAME']}:#{ENV['BS_AUTHKEY']}@hub.browserstack.com/wd/hub"

  # load browser configuration
  browser_data = JSON.load(open('.browsers.json'))
  browser_name = ENV['BROWSER_TASK_NAME']

  browser = browser_data[browser_name]

  puts "Testing in #{browser['human']} (#{ENV['BROWSER_TASK_NAME']})..."

  # translate into Selenium Capabilities
  capabilities = Selenium::WebDriver::Remote::Capabilities.new
  capabilities["os"] = browser["os"]
  capabilities["os_version"] = browser["os_version"]
  capabilities["browser"] = browser["browser"]
  capabilities["browser_version"] = browser["browser_version"]
  capabilities['browserstack.debug'] = true
  capabilities['browserstack.local'] = true
  capabilities['browserstack.localIdentifier'] = "#{ENV['CIRCLE_PROJECT_REPONAME']}-node-#{ENV['CIRCLE_NODE_INDEX']}"
  capabilities['takesScreenshot'] = true

  capabilities['project'] = ENV['CIRCLE_PROJECT_REPONAME'] if ENV['CIRCLE_PROJECT_REPONAME']
  capabilities['build'] = ENV['CIRCLE_BUILD_NUM'] if ENV['CIRCLE_BUILD_NUM']

  # register a driver
  driver_name = "#{browser['browser']}_#{browser['browser_version']}_#{browser['os']}_#{browser['os_version']}"
  Capybara.register_driver(driver_name) do |app|
    Capybara::Selenium::Driver.new(app, :browser => :remote, :url => url, :desired_capabilities => capabilities)
  end

  # use the registered driver
  Capybara.default_driver = driver_name
  Capybara.javascript_driver = driver_name

  # Resize browser window to be consistant when testing cross browser
  Before do
    page.driver.browser.manage.window.resize_to(1280, 800)
  end
end

We need to supply the same identifier when openning the tunnel from circle.yml. We also need to run all the cross-browser related commands in parallel. Final circle.yml will look like the following (notice the added nodes=4 when running the tests)

dependencies:
  cache_directories:
    - browserstack
test:
  override:
    - RAILS_ENV=test bundle exec rake spec
    - RAILS_ENV=test bundle exec cucumber
    - script/ci/browserstack_tunnel.sh:
        parallel: true
    - browserstack/BrowserStackLocal -localIdentifier "$CIRCLE_PROJECT_REPONAME-node-$CIRCLE_NODE_INDEX" -skipCheck $BS_AUTHKEY 127.0.0.1,3001:
        parallel: true
        background: true
    - RAILS_ENV=test bundle exec rake cross_browser nodes=4:
        parallel: true
    - script/ci/browserstack_tunnel.sh stop:
        parallel: true

And that's it. You can now scale your build out to four containers and run the tests in paralel. For us this gets the build time down to about 12 minutes on a complex app and 5 minutes on a very simple one.

Conclusions

We are really happy with this setup. It's really stable, fast, individual test runs are completely isolated and we don't need to deploy anything anywhere. It has just one drawback compared to the previous setup which first deployed the application to a staging environment and then ran cross-browsers tests against it. It doesn't test the app in it's real runtime environment (Heroku in our case). Otherwise it's a complete win on all fronts.

We plan to solve that remaining problem by writing a separate test suite testing our whole system (consisting from multiple services consuming each other's APIs) cleanly from the outside. It won't go into as much detail as the normal tests since it is only there to confirm that the different pieces fit together and users can complete the most important journes. Coupled with Heroku's slug promotion feature, we will actually test the exact thing that will end up in production in the exact same environment. And you can look forward to another blogpost about that soon.