Evaluating Gwen with Modern JavaScript Web Apps

Gwen-web was designed to take the development pain out of web automation and make it easier to achieve with all types of web applications regardless of the underlying technologies or frameworks they were built with. Our goal was to make Gwen work consistently with web pages built on any kind of server or client side framework. All web pages are just HTML documents when they hit the browser after all. But this HTML is not always static and with the recent trend of modern JavaScript framework adoption, web pages are getting more and more dynamic. A lot more things happen in web pages nowadays. Lots of JavaScript code gets loaded and all types of browser events trigger functions and Ajax requests at various times resulting in all sorts of dynamic and asynchronous rendering that makes for a very rich user experience. This might make things more pleasant for human users but it is often a hard challenge for automation programs (or robots).

The Evaluation Test

So I thought I’d try Gwen out on some web pages built on the popular and modern JS frameworks of today to see how well it performs. To do this, I wrote a feature suite that mimics these Serenity tests and ran it over the various JS implementations of the publicly available TodoMVC web application (this is a project which offers the same Todo web application implemented on different JavaScript frameworks).

It is important to note that Gwen uses the Java implementation of the Selenium web driver and is not a JS framework itself but rather a Java executable that reads plain text Gherkin features and dynamically interprets them into automated web page interactions. The evaluation performed here provides us with an indication of how well Gwen interacts with web applications built on different types of JS frameworks.

The Evaluation Results

The suite consists of 26 scenarios which I ran over 33 different JS implementations of the Todo app (for a total of 858 scenarios). I ran it in a Chrome browser on a Mac. It took about 50 minutes to run every scenario in sequence. I also ran the tests in parallel (on a dual CPU quad core machine) which took about half the time and produced consistent results. Gwen was executed with the gwen.feature.failfast setting overriden to false to force all scenarios in a feature to execute even if one or more prior ones fail.

94% of Scenarios Passed ūüôā

All tests completed successfully and passed for the following frameworks.

  • JavaScript:
    • Backbone.js
    • AngularJS
    • Ember.js
    • KnockoutJS
    • Dojo
    • Knockback.js
    • CanJS
    • Polymer
    • React
    • Mithril
    • Ampersand
    • Vue.js
    • Marionette.js
    • Vanilla JS
    • Vanilla ES6
    • jQuery
  • Compile-to-JS:
    • Spine
    • Dart
    • GWT
    • TypeScript + Backbone.js
    • TypeScript + AngularJS
    • TypeScript + React
    • Serenade.js
    • Reagent
    • Scala.js + React
    • Scala.js + Binding.scala
    • js_of_ocaml
    • Humble + GopherJS

6% of Scenarios Failed

Some tests failed for the following frameworks. I have not investigated these in detail but have included my initial findings below.

  • JavaScript:
    • Flight
      • 15 scenarios failed, 11 passed
      • Intermittent element hits and misses
      • Entered data leaks or is lost
    • TroopJS + RequireJS
      • 15 scenarios failed, 11 passed
      • Unresponsive to enter key being sent to field
  • Compile-to-JS:
    • Closure
      • 5 scenarios failed, 21 passed
      • Unresponsive to checkbox being clicked
    • Elm
      • 8 scenarios failed, 18 passed
      • Intermittently not sending some characters to field
    • AngularDart
      • 3 scenarios failed, 23 passed
      • Intermittently not rendering conditionally visible buttons


Gwen interacted successfully with a large majority of the popular JS implementations. All tests passed for 28 of the 33 implementations evaluated. More than 94% of all scenarios passed in total.

Try it Yourself

All the todoMVC features used in this evaluation are also bundled in the gwen-web distribution as of release 2.3.3. If you install the latest gwen-web distribution, you can run it by calling the following command in the directory where you install Gwen (to run in parallel mode, just add the --parallel switch):

  • Windows:
    • gwen features/todoMVC
  • Linux:
    • ./gwen features/todoMVC

Gwen 2 released

Gwen 2 has been released with the latest technologies including Java 8 and Selenium WebDriver 3. All binary dependencies have also been updated and the project is now built using Scala 2.12.

There is one impact to Firefox users but nothing else has changed. With the upgrade to Selenium 3, all browser vendors (including Mozilla) are responsible for developing and providing the native drivers for their own browsers. To use Firefox, you will now need to download a native Firefox driver and set the following in your gwen.properties file:


The settings page on our Gwen wiki has also been updated.

Gwen 2.x will only run on Java 8 environments. Gwen 1.x will still run on Java 7 and above as before, but will no longer have new features added to it.

The latest Gwen release is available here. See also install instructions.

Configurable User Properties in gwen-web

One of the really cool features of gwen-web is the ability to load in different parameters at run time.  This allows you to run gwen-web against multiple environments without changing either the feature or the meta files.

Lets take one of the examples with gwen-web and flood.io ¬† Below are two examples of how one could set the value. ¬†The first example sets “the how old are you dropdown” to 21, the second example uses a property, in this case different files for two different environments to set the value.

Lets define how the dropdown can be found in the meta first.


And the heading should be “Step 2”
And the how old are you dropdown can be located by id “challenger_age”


When I select “21” in the how old are you dropdown

Now with the properties approach. Lets create two files, DEV.properties and SIT.properties. The contents of the two files are below:



In addition we need to make a small modification to both the feature and the meta files


And the heading should be “Step 2”
And my age is defined by property “user.age”
And the how old are you dropdown can be located by id “challenger_age”

Now replace the strikethrough line with the new line in the feature


When I select “21” in the how old are you dropdown
When I select my age in the how old are you dropdown

Editting is complete. To run gwen-web with the environment specific age, simply type “gwen-web features/floodio/ -p dev.properties -b -r devReport”, or alternatively you can run “gwen-web features/floodio/ -p sit.properties -b -r sitReport”.

Remote WebDriver feature now available in gwen-web

For my first post, I will be talking about a new feature that has been added to gwen-web.  For those of you not familiar with gwen-web, its a web automation engine that runs inside our gwen interpreter and is used to automate the browser.  gwen-web is unique in that it allows automation of the web browser, using english(dsl) rather than coding or using visual record/playback. We believe this will remove some of the entry level requirements often associated with a coded tool.  For more information please head on over to our gwen github page https://github.com/gwen-interpreter/gwen.

Remote WebDriver

Just recently we added the remote webdriver capability.  This means gwen-web can now connect to selenium grid and drive not just the local browser, but also a browser that is installed on a different os, or a browser that has a different version.

Setting up Remote WebDriver with gwen-web

So how do I setup gwen-web with selenium remote webdriver?  What I am going to walk you through is setting up a selenium grid, and attaching nodes to it.  Firstly download the selenium-server-standalone-2.45.0.jar (or most recent) from http://selenium-release.storage.googleapis.com/2.45/selenium-server-standalone-2.45.0.jar.   Once this has been downloaded, you can start up selenium grid by running the following:

java -jar selenium-server-standalone-2.45.0.jar -role hub

Now lets make sure that its running by navigating to http://localhost:4444/grid/console.  You will notice that there are currently no browsers available.


Running selenium grid in this mode, means we need to register nodes against the grid.

Adding browsers to the current running grid

Lets start by downloading the latest version of chromedriver (specific for the OS) Рfrom http://chromedriver.storage.googleapis.com/index.html?path=2.15/  (at the time of writing the blog chromedriver 2.15 is the latest)

Unzip chromedriver and note the path.

Now lets start up selenium-server-standalone as a node role, and register both the chrome driver and firefox driver.  Remembering firefox driver comes with the selenium-server-standalone.

java -jar selenium-server-standalone-2.45.0.jar 
-role node 
-hub http://localhost:4444/grid/register 
-browser browserName=firefox,version=37.0,maxInstances=5,platform=LINUX 
-browser browserName=chrome,version=42.0,maxInstances=5,platform=LINUX

Let me explain the parts above:

  1. java -jar selenium-server-standalone-2.45.0.jar  (standard selenium startup)
  2. -Dwebdriver.chrome.driver= (path to chromedriver including the executable)
  3. -role node (run the selenium-server-standalone as a node)
  4. -hub http://localhost:4444/grid/register ¬†(this is the location of the grid that you started back under the previous heading “Setting up ¬†Remote WebDriver with gwen-web”
  5. -browser browserName=firefox,version=37.0,maxInstance=5,platform=LINUX  (the browser, version label and the number of maximum instances to run on this node)  Note:  firefox requires the binary to be in the PATH, alternatively it would need to be specified using -Dwebdriver.firefox.bin
  6. -browser browserName=chrome,version=42.0,maxInstances=5,platform=LINUX.  (register that this node is capable of running chrome)

Testing the grid console

So how do we now confirm the currently registered browsers on the grid.   Navigate to http://localhost:4444/grid/console and now the console should show both firefox and chrome registered and waiting for a connection.


When I setup a grid, I typically like to confirm whether or not the browsers do indeed start up.  To do this I navigate to the node webdriver/hub   E.g.   (you can see the IP address of each node that is registered against this grid).


Then I create two sessions.  One for chrome and one for firefox.  If its working, both chrome and firefox will start.  Just delete the sessions once that has been confirmed.  If its not working, then you will need to go back and troubleshoot.  Typically its driver / binary path related.  If either browser pops up and then closes unexpectedly, make sure to keep both the selenium-server-standalone and the browser versions in sync.  The latest version of selenium-standalone may not work with a version of the browser that is 6 versions out of date.

Setting up gwen-web to work with selenium-grid

Once you have a working grid, gwen web only requires a small change to the gwen.properties file. (eg. edit ~/gwen.properties and add the following line.)


This tells the gwen-web engine that it needs to communicate with selenium grid which will effectively use the RemoteWebDriver with different browser capabilities.

Running gwen-web with selenium-grid

To run gwen-web with two different browsers it is suggested that you remove gwen.web.browser property from the gwen.properties and pass it in as a “-D” system property. ¬†The below example uses the floodio challenge sample with gwen-web.


bin/gwen-web -Dgwen.web.browser=chrome -b features/floodio -r reports/chrome
bin/gwen-web -Dgwen.web.browser=firefox -b features/floodio -r reports/firefox

Once gwen-web is running, navigate to the selenium node and you will see something like this

Screen Shot 2015-04-30 at 9.42.28 pm

and navigate to the grid localhost:4444/grid/console and you will now see both firefox and chrome running.

Screen Shot 2015-04-30 at 9.42.47 pm