Headless Browser Testing on Mac

Functional testing is a fundamental aspect of BDD. It generally takes longer to run than unit testing because you are write tests that mimic the user experience. For web apps, this means writing tests that can use the browser the same way a user would. An excellent way to accomplish this level of functional testing is with Selenium. Selenium enables remote control browser testing via lightweight commands.

Part of what Selenium does is to launch a browser and execute commands using a DSL called Selenese to interact with the browser the way a user would. This could be filling in form fields, clicking on buttons, selecting an option from a dropdown, etc. At each step of the way, results can be confirmed directly from the browser. Often, a suite of tests will cause multiple browsers to be launched and can take minutes to complete. Seeing browser windows opening and closing on your screen can be disruptive to your workflow. The end result is often just sitting idly while the test suite is running.

On Linux, it is quite easy to setup a headless environment in which the browser can run so as to not be disruptive. It is a little more challenging to do this on Mac, but it is doable. When properly setup, you can run an entire test suite without ever seeing the browser on your screen. It is this setup that is the focus of this post.

Firefox-x11 and Xvfb are used to accomplish a headless Selenium testing environment on Mac. Read on for instructions on how to accomplish this setup on your machine.

Pre-requisites

You will need to have the following installed on your system before you can follow the rest of these instructions:

1) Xcode – http://developer.apple.com/xcode/
2) MacPorts – http://www.macports.org/install.php – The firefox-x11 install is the most complex part of the installation. MacPorts automatically downloads, compiles and installs all of the many required supporting libraries as well as firefox-x11 itself. If you already have MacPorts installed, execute the following to make sure you are on the latest version:

sudo port selfupdate
sudo port selfupdate
sudo port upgrade

NOTE: selfupdate is run twice because sometimes port downloads an update but doesn’t install it until the next run.

Understanding What’s going on

This section provides some background on what’s going on behind the scenes with the setup outlined above. Feel free to skip ahead if you just want to get it working.

X11 is the old school UNIX graphic window system. X11 is a reverse client/server system. This means that you run the X11 server locally and the graphic UI client remotely.

In order to get Selenium tests to run in a headless environment, you will run a headless X11 server. Selenium will then launch a browser as a client of this headless X11 environment. So, the browser itself is not headless. Rather, it is the X11 server that provides a virtual headless display.

Although the underbelly of OS X is UNIX, the graphic windowing system has diverged to a proprietary system called Aqua. X11, however, does still run on the Mac. The regular Mac versions of the browsers (Chrome, Firefox, Safari, etc.) do not support running in an X11 environment. As a result, part of the installation involves downloading and compiling an X11 version of Firefox.

Installation

1) firefox-x11

Install firefox-x11 by issuing the following command in the terminal:

sudo port install firefox-x11

NOTE: This step will take over an hour to complete as there are many required supporting libraries that are compiled and installed

2) Mac setup

D-Bus is a message bus system used for application to application communication. This is required in order for firefox-x11 to run properly on the mac.

Execute the following in the terminal:

sudo launchctl load -w /Library/LaunchDaemons/org.freedesktop.dbus-system.plist
sudo launchctl load -w /Library/LaunchAgents/org.freedesktop.dbus-session.plist
launchctl load -w /Library/LaunchDaemons/org.freedesktop.dbus-system.plist
launchctl load -w /Library/LaunchAgents/org.freedesktop.dbus-session.plist

3) Headless X11 Server

The following command runs the headless X11 server:

Xvfb :5 -ac -screen 0 1024x768x8 -extension GLX -kb &

Running Xvfb in this way binds it to display 5, screen 0 on the localhost.

Testing the Test Environment

There are many testing frameworks that make use of Selenium. For the purposes of this post, I wrote a shell script that issues commands directly to Selenium. This keeps the focus on the task at hand: running tests in a headless environment. This setup should work with any supported language that interacts with Selenium. We use Soda in conjunction with Peanut for our functional tests of our node.js code.

First, get Selenium running. If you don’t already have it, it can be downloaded here.

DISPLAY=:5.0 java -jar selenium-server-standalone-2.0.0.jar

Putting the DISPLAY environment variable at the beginning of the command ensures that any X11 programs (such as firefox-x11) launched from Selenium will be bound to that display. The DISPLAY environment variable is ignored for native apps, such as Google Chrome. Remember, Xvfb, the headless X11 Server, is bound top display 5 screen 0.

Here is the shell script to test that you’ve setup the headless environment properly:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
#!/bin/bash
 
URL_BASE="http://localhost:4444/selenium-server/driver/"
 
STATUS=""
SESSION=""
 
function parseSession() {
  OIFS=$IFS
  IFS=','
  set $1
  IFS=$OIFS
  STATUS=$1
  SESSION=$2
}
 
if [ -z "$1" ]
then
  BROWSER="*googlechrome"
else
  BROWSER="*firefox%20%2Fopt%2Flocal%2Flib%2Ffirefox-x11%2Ffirefox-bin"
fi
 
LAUNCH_BROWSER="$URL_BASE?cmd=getNewBrowserSession&1=$BROWSER&2=http%3A%2F%2Fwww.google.com"
RESPONSE=`curl -s $LAUNCH_BROWSER`
parseSession $RESPONSE
 
CMDS=(
  "cmd=open&1=/"
  "cmd=clickAndWait&1=link%3DAdvanced%20search"
  "cmd=type&1=as_q&2=Hello%20World"
  "cmd=clickAndWait&1=%2F%2Finput%5B%40type%3D'submit'%20and%20%40value%3D'Advanced%20Search'%5D"
  "cmd=assertAttribute&1=q%40value&2=Hello%20World"
  "cmd=testComplete"
)
 
for CMD_BASE in "${CMDS[@]}"
do
  CMD="$URL_BASE?$CMD_BASE&sessionId=$SESSION"
  RESPONSE=`curl -s $CMD`
  echo $RESPONSE
done

There are two ways to run this shell script: with and without an argument. If run without an argument, the script will cause Selenium to use the native Google Chrome browser for the tests. If passed an argument (which can be anything), the script will cause Selenium to use the firefox-x11 browser. Because of the DISPLAY environment variable set when you launched Selenium above, firefox-x11 will be run inside the headless environment.

Let’s run the command both ways, and then we will review the shell script line-by-line.

./SeleniumTest.sh
./SeleniumTest.sh headless

In both cases, you should get the same output:

OK
OK
OK
OK
OK
OK

When you run the script without any arguments, you should see the Google Chrome browser launched on your screen.

When you run the script with an argument, you should see the same output, but you should not see any browsers launched on your screen.

If you look at the Selenium output, you should see something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
12:28:12.332 INFO - Command request: getNewBrowserSession[*firefox /opt/local/lib/firefox-x11/firefox-bin, http://www.google.com] on session null
12:28:12.332 INFO - creating new remote session
12:28:12.333 INFO - Allocated session ebc24031a88b4e75b5a4ac2f0ae6b53d for http://www.google.com, launching...
12:28:12.353 INFO - Preparing Firefox profile...
12:28:15.183 INFO - Launching Firefox...
12:28:20.171 INFO - Got result: OK,ebc24031a88b4e75b5a4ac2f0ae6b53d on session ebc24031a88b4e75b5a4ac2f0ae6b53d
12:28:20.376 INFO - Command request: open[/, ] on session ebc24031a88b4e75b5a4ac2f0ae6b53d
12:28:21.738 INFO - Got result: OK on session ebc24031a88b4e75b5a4ac2f0ae6b53d
12:28:21.846 INFO - Command request: clickAndWait[link=Advanced search, ] on session ebc24031a88b4e75b5a4ac2f0ae6b53d
12:28:22.052 INFO - Got result: OK on session ebc24031a88b4e75b5a4ac2f0ae6b53d
12:28:22.065 INFO - Command request: type[as_q, Hello World] on session ebc24031a88b4e75b5a4ac2f0ae6b53d
12:28:23.278 INFO - Got result: OK on session ebc24031a88b4e75b5a4ac2f0ae6b53d
12:28:23.289 INFO - Command request: clickAndWait[//input[@type='submit' and @value='Advanced Search'], ] on session ebc24031a88b4e75b5a4ac2f0ae6b53d
12:28:27.062 INFO - Got result: OK on session ebc24031a88b4e75b5a4ac2f0ae6b53d
12:28:27.240 INFO - Command request: assertAttribute[q@value, Hello World] on session ebc24031a88b4e75b5a4ac2f0ae6b53d
12:28:27.891 INFO - Got result: OK on session ebc24031a88b4e75b5a4ac2f0ae6b53d
12:28:27.902 INFO - Command request: testComplete[, ] on session ebc24031a88b4e75b5a4ac2f0ae6b53d
12:28:27.902 INFO - Killing Firefox...
12:28:27.952 INFO - Got result: OK on session ebc24031a88b4e75b5a4ac2f0ae6b53d

Let’s look at the shell script to understand what it’s doing.

Line 3 set’s up the connection to Selenium on the standard port.

Lines 5 – 15 are used to parse the session value out of the returned value from Selenium. Every time you launch a browser with Selenium, a session is established that should be reused in subsequent commands. When first launching the browser, the response from Selenium will be:

OK,<session id>

You can see this on line 6 of the Selenium output above.

Lines 9 – 10 in the script use a Bash shell trick that changes the default Internal Field Separator (IFS). By default, IFS is set to whitespace (space, tab, and newline). Line 9 saves the current value of IFS. Line 10 sets IFS to comma (,) and line 11 breaks the parameter passed in to the function based on the IFS. Line 12 sets the IFS back to the saved value. Lines 13 and 14 store the values created from line 11.

Line 17 tests to see if any arguments were passed to the script. If not, Line 19 sets the browser to the native Google Chrome browser. If any argument was passed in to the script, then Line 21 sets the browser to be firefox-x11. Selenium has a number of built-in defaults for a variety of browsers. These can be overridden by supplying a path to an executable as a second argument. NOTE: As a security measure, Selenium will NOT run a shell script. The second argument must be a fully qualified path to the binary executable of the browser. Line 21 is URL encoded since the command is passed to Selenium via an HTTP connection. Decoded the line looks like this:

*firefox /opt/local/lib/firefox-x11/firefox-bin

This is the default path to firefox-x11 as set by MacPorts.

Line 24 sets up the command that will be used by curl to have Selenium launch the browser. It too is URL encoded. Decoded, it looks like this:

LAUNCH_BROWSER="$URL_BASE?cmd=getNewBrowserSession&1=$BROWSER&2=http://www.google.com"

Note that line 24 uses the value for $URL_BASE (set on line 3) and $BROWSER (set on lines 17 – 22).

Line 25 executes curl and saves the response from Selenium in a variable called $RESPONSE.

Line 26 calls the parseSession function described above so that the $SESSION variable can be saved.

Lines 28 – 35 sets up an array of commands that will be send to Selenium. Each command is URL encoded. Decoded, this is how this block of code looks:

CMDS=(
  "cmd=open&1=/"
  "cmd=clickAndWait&1=link=Advanced search"
  "cmd=type&1=as_q&2=Hello World"
  "cmd=clickAndWait&1=//input[@type='submit' and @value='Advanced Search']"
  "cmd=assertAttribute&1=q@value&2=Hello World"
  "cmd=testComplete"
)

The Selenium command at line 29 navigates to “/” in the open browser with the url:

http://www.google.com/

Line 30 tells Selenium to click the – Advanced search – link.

Line 31 tells Selenium to put the string – Hello World – in the field identified by – as_q – (you can see this if you view-source on Google’s advanced search page).

Line 32 tells Selenium to click the submit button on the form and wait for the page to load

Line 33 asks Selenium to confirm that the input field identifed by – q – contains – Hello World – as its value.

Line 34 tells Selenium that the test is complete. This causes Selenium to kill the browser it launched.

Line 37 iterates over the $CMDS array.

Line 39 sets up the command to be sent to Selenium. Note that it uses the $URL_BASE (set on line 3), $CMD_BASE which is the current element from the $CMDS array, and $SESSION (set on line 26).

Line 40 sends the command to Selenium using curl

Line 41 echoes the response from Selenium to the terminal. If everything goes smoothly, the response should be – OK – for each command in the $CMDS array.

Summary

Selenium is a powerful way to do browser based integration & functional testing. But, having Selenium launch browsers while running the tests can be disruptive to your workflow.

Having Selenium launch browsers in a headless environment enables long running test suites to be executed behind the scenes without affecting the display you are working in.



One Response to “ “Headless Browser Testing on Mac”

  1. Manoj says:

    Awesome tutorial.

Leave a Reply