Gwen Automation with Data Tables

Gwen now supports data tables and in this post I’m going to show you some examples of how they can be used with the web engine to automate some repetitive tasks on the TodoMVC web application.

Data tables are handy for when you want to work with lists of items in Gherkin, such as the list of todo items shown in this example below.

File: features/todo/tables/TodoTables.feature


 Feature: Managing todo items with data tables

Scenario: Complete a todo item

    Given I have the following active items
          | Walk the dog |
          | Get the milk |
          | Feed the cat |

     When I complete the "Get the milk" item

     Then the status of my items should be
          | Item         | Status    |
          | Walk the dog | active    |
          | Get the milk | completed |
          | Feed the cat | active    |

Note that this example is a high level (or declarative) feature specification that describes intended behaviour and not a low level (or imperative) list of statements that describe how automation is to be achieved or on what type of system it will be performed on. Those are automation bindings that can be captured in separate meta files that are also expressed in Gherkin when using Gwen. As it turns out, we already have a meta file that specifies some step definitions that call out to predefined steps in the web engine to interact with the TodoMVC app. One of the capabilities of Gwen is the ability to reuse this meta, so we will reuse it here to help compose some step definitions for our data tables above. You can have a look at that Todo.meta file here if you like.

Data Table Bindings

Looking at the first Given step in our feature, we can see that it contains three todo items in a table. The step implies that we have those three items in our todo list. So what we want to do is load those three items into our list in the todo app when running this step. To do that, we need to create a step definition for it that will launch the app, load the items, and verify that they are all active. We can do that with some new meta as follows:

File: features/todo/tables/TodoTables.meta

 Feature: TodoMVC tables meta

@StepDef
@DataTable(horizontal="Item")
Scenario: I have the following active items
    Given I launch the Todo app
     When I add a "${data[1][Item]}" item
      And I add a "${data[2][Item]}" item
      And I add a "${data[3][Item]}" item
     Then the "${data[Item]}" item should be active for each data record

This meta is also a Gherkin specification with a tagged scenario containing all the steps that will perform the desired operations on the data table.

  • This first @StepDef tag tells Gwen to load this scenario to memory and only execute its steps when a step in a feature references it by name. Note that the name of this scenario is the same expression used in the Given step in the feature. This is how step definitions work in Gwen.
  • The second @DataTable tag contains an attribute that declares the table to contain horizontal data records and that each one of those contains just one data cell that we will call “Item”. We need to do this since the table in the feature does not include a header record at the top and we have to assign each cell a name so we can access its data by that name.
  • The steps from lines 7 to 9 load the items one at a time by explicitly referencing the data in each record
  • The last step on line 10 checks that each item is active by iterating over each record of the table

With this in place and the latest web engine installed, we can now invoke the feature with Gwen as follows.

gwen -b features/todo/tables/TodoTable.feature

Gwen will automatically discover and load the automation bindings in all meta files that exist in the path of the feature file which in our case includes both the Todo.meta we want to reuse and the TodoTables.meta we just created. The Given step in the feature will execute to load the table items and verify that they are all active, and the When step will complete the second item. The Then step in the feature will fail as expected because we haven’t defined a step definition for it yet. The web browser output is shown below:

Similarly, we now add a step definition for the last Then step in our feature to our meta to verify the status of all items and complete all our binding work.

    • The table in this step includes a header record, so we specify header="top" in the @DataTable tag to link the names in that header to the cells in each data record.

    File: features/todo/tables/TodoTables.meta

     Feature: TodoMVC tables meta
    
    @StepDef
    @DataTable(horizontal="Item")
    Scenario: I have the following active items
        Given I launch the Todo app
         When I add a "${data[1][Item]}" item
          And I add a "${data[2][Item]}" item
          And I add a "${data[3][Item]}" item
         Then the "${data[Item]}" item should be active for each data record
    
    @StepDef
    @DataTable(header="top")
    Scenario: the status of my items should be
         Then the "${data[Item]}" item should be ${data[Status]} for each data record
    

    Now the entire feature is executable!

    Simplifying it Further

    For the sake of demonstrating explicit table data access, the steps from lines 7 to 9 in our meta process each record one at a time. Imagine if we had a lot more records? There would be a lot of redundancy wouldn’t there? We can remove this by reducing these steps into a single step that loops through each record in the same way that the steps on line 10 and 15 do.

    File: features/todo/tables/TodoTables.meta

     Feature: TodoMVC tables meta
    
    @StepDef
    @DataTable(horizontal="Item")
    Scenario: I have the following active items
        Given I launch the Todo app
         When I add a "${data[Item]}" item for each data record
         Then the "${data[Item]}" item should be active for each data record
    
    @StepDef
    @DataTable(header="top")
    Scenario: the status of my items should be
         Then the "${data[Item]}" item should be ${data[Status]} for each data record
    

    Now there is no need to explicitly specify the index for each record when referencing each data cell. But for cases where you might need to cross reference different records or really want to be explicit in how you access each data cell, the earlier approach of not iterating will give you the flexibility you need. It really depends on your particular use case and you should always choose the approach that works the best in the given circumstance.

    The feature and meta files created here are available in our samples features/todo/tables folder on Github and are also bundled with the Gwen-web binary distribution. If you download and install it, you can execute the feature in the same way that we did above in your installation directory.

    And, that’s it! Our feature with tables is executable and all redundancy has been removed from the bindings. We have mapped declarative feature steps to imperative steps in meta and reused some existing meta too! ūüôā

Advertisements

Evaluating Gwen with Modern JavaScript Web Apps

Gwen-web was designed to take the development pain out of web automation and make it easier to achieve with all types of web applications regardless of the underlying technologies or frameworks they were built with. Our goal was to make Gwen work consistently with web pages built on any kind of server or client side framework. All web pages are just HTML documents when they hit the browser after all. But this HTML is not always static and with the recent trend of modern JavaScript framework adoption, web pages are getting more and more dynamic. A lot more things happen in web pages nowadays. Lots of JavaScript code gets loaded and all types of browser events trigger functions and Ajax requests at various times resulting in all sorts of dynamic and asynchronous rendering that makes for a very rich user experience. This might make things more pleasant for human users but it is often a hard challenge for automation programs (or robots).

The Evaluation Test

So I thought I’d try Gwen out on some web pages built on the popular and modern JS frameworks of today to see how well it performs. To do this, I wrote a feature suite that mimics these Serenity tests and ran it over the various JS implementations of the publicly available TodoMVC web application (this is a project which offers the same Todo web application implemented on different JavaScript frameworks).

It is important to note that Gwen uses the Java implementation of the Selenium web driver and is not a JS framework itself but rather a Java executable that reads plain text Gherkin features and dynamically interprets them into automated web page interactions. The evaluation performed here provides us with an indication of how well Gwen interacts with web applications built on different types of JS frameworks.

The Evaluation Results

The suite consists of 26 scenarios which I ran over 33 different JS implementations of the Todo app (for a total of 858 scenarios). I ran it in a Chrome browser on a Mac. It took about 50 minutes to run every scenario in sequence. I also ran the tests in parallel (on a dual CPU quad core machine) which took about half the time and produced consistent results. Gwen was executed with the gwen.feature.failfast setting overriden to false to force all scenarios in a feature to execute even if one or more prior ones fail.

94% of Scenarios Passed ūüôā

All tests completed successfully and passed for the following frameworks.

  • JavaScript:
    • Backbone.js
    • AngularJS
    • Ember.js
    • KnockoutJS
    • Dojo
    • Knockback.js
    • CanJS
    • Polymer
    • React
    • Mithril
    • Ampersand
    • Vue.js
    • Marionette.js
    • Vanilla JS
    • Vanilla ES6
    • jQuery
  • Compile-to-JS:
    • Spine
    • Dart
    • GWT
    • TypeScript + Backbone.js
    • TypeScript + AngularJS
    • TypeScript + React
    • Serenade.js
    • Reagent
    • Scala.js + React
    • Scala.js + Binding.scala
    • js_of_ocaml
    • Humble + GopherJS

6% of Scenarios Failed

Some tests failed for the following frameworks. I have not investigated these in detail but have included my initial findings below.

  • JavaScript:
    • Flight
      • 15 scenarios failed, 11 passed
      • Intermittent element hits and misses
      • Entered data leaks or is lost
    • TroopJS + RequireJS
      • 15 scenarios failed, 11 passed
      • Unresponsive to enter key being sent to field
  • Compile-to-JS:
    • Closure
      • 5 scenarios failed, 21 passed
      • Unresponsive to checkbox being clicked
    • Elm
      • 8 scenarios failed, 18 passed
      • Intermittently not sending some characters to field
    • AngularDart
      • 3 scenarios failed, 23 passed
      • Intermittently not rendering conditionally visible buttons

Conclusion


Gwen interacted successfully with a large majority of the popular JS implementations. All tests passed for 28 of the 33 implementations evaluated. More than 94% of all scenarios passed in total.

Try it Yourself

All the todoMVC features used in this evaluation are also bundled in the gwen-web distribution as of release 2.3.3. If you install the latest gwen-web distribution, you can run it by calling the following command in the directory where you install Gwen (to run in parallel mode, just add the --parallel switch):

  • Windows:
    • gwen features/todoMVC
  • Linux:
    • ./gwen features/todoMVC

Gwen 2 released

Gwen 2 has been released with the latest technologies including Java 8 and Selenium WebDriver 3. All binary dependencies have also been updated and the project is now built using Scala 2.12.

There is one impact to Firefox users but nothing else has changed. With the upgrade to Selenium 3, all browser vendors (including Mozilla) are responsible for developing and providing the native drivers for their own browsers. To use Firefox, you will now need to download a native Firefox driver and set the following in your gwen.properties file:

webdriver.gecko.driver=/your-downloaded-driver-location/geckodriver

The settings page on our Gwen wiki has also been updated.

Gwen 2.x will only run on Java 8 environments. Gwen 1.x will still run on Java 7 and above as before, but will no longer have new features added to it.

The latest Gwen release is available here. See also install instructions.

Integrating Gwen with Maven

Although you can download and install Gwen locally to run automated tests, it is sometimes useful to integrate it with a build tool such as Maven and have it download and install it for you and run tests as part of the release verification process. Integrating with Maven can also make it easier for you to run Gwen on a continuous integration build server too.

Gwen does not ship with a Maven plugin, but you can still integrate it with Maven using a POM file like this:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

  <modelVersion>4.0.0</modelVersion>
  <groupId>org.gweninterpreter</groupId>
  <artifactId>gwen-web-maven</artifactId>
  <name>gwen-web-maven</name>
  <version>4.0.0</version>
  <packaging>pom</packaging>

  <profiles>
    <profile>
      <id>gwen</id>
      <activation>
        <property>
          <name>gwen.args</name>
        </property>
      </activation>
      <properties>
        <gwen.web.version>???</gwen.web.version>
        <!--selenium.version>???</selenium.version-->
        <!--selenium.dir>target/selenium-${selenium.version}</selenium.dir-->
        <gwen.dir>target/gwen-web-${gwen.web.version}</gwen.dir>
        <gwen.classpath>${selenium.dir}/*${path.separator}${gwen.dir}/lib/*</gwen.classpath>
      </properties>
      <dependencies>
        <!--dependency>
          <groupId>org.seleniumhq.selenium</groupId>
          <artifactId>selenium-java</artifactId>
          <version>${selenium.version}</version>
        </dependency-->
        <dependency>
          <groupId>org.gweninterpreter</groupId>
          <artifactId>gwen-web</artifactId>
          <version>${gwen.web.version}</version>
          <type>zip</type>
          <scope>provided</scope>
        </dependency>
      </dependencies>
      <build>
        <plugins>
          <plugin>
            <artifactId>maven-dependency-plugin</artifactId>
            <version>2.9</version>
            <executions>
              <execution>
                <id>install-gwen</id>
                <phase>integration-test</phase>
                <goals>
                  <goal>unpack-dependencies</goal>
                </goals>
                <configuration>
                  <includeGroupIds>org.gweninterpreter</includeGroupIds>
                  <includeArtifactIds>gwen-web</includeArtifactIds>
                  <includeTypes>zip</includeTypes>
                  <outputDirectory>target</outputDirectory>
                  <stripClassifier>true</stripClassifier>
                  <stripVersion>true</stripVersion>
                </configuration>
              </execution>
              <!--execution>
                <id>install-selenium</id>
                <phase>integration-test</phase>
                <goals>
                  <goal>copy-dependencies</goal>
                </goals>
                <configuration>
                  <excludeScope>provided</excludeScope>
                  <outputDirectory>${selenium.dir}</outputDirectory>
                </configuration>
              </execution-->
            </executions>
          </plugin>
          <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>exec-maven-plugin</artifactId>
            <version>1.3.2</version>
            <executions>
              <execution>
                <id>launch-gwen</id>
                <phase>verify</phase>
                <goals>
                  <goal>exec</goal>
                </goals>
              </execution>
            </executions>
            <configuration>
              <executable>java</executable>
              <commandlineArgs>-cp ${gwen.classpath} gwen.web.WebInterpreter -b ${gwen.args}</commandlineArgs>
              <successCodes>
                <successCode>0</successCode>
              </successCodes>
            </configuration>
          </plugin>
        </plugins>
      </build>
    </profile>
  </profiles>
  
</project>

Be sure to change the Gwen version by updating the gwen.web.version property at line 20 with the latest version or another version you would like to use.

With this POM and your Gwen settings in place you can have Maven download, install, and launch the latest version of Gwen with the following command (where <args> is a space separated list of arguments that you want to pass to Gwen):

mvn verify -Dgwen.args="<args>"

So if your executable tests are in a folder named ‘features’ relative to your POM file, then you can execute them with a Maven command like this:

mvn verify -Dgwen.args="features"

And to also generate html and junit style reports in the target/reports folder..

mvn verify -Dgwen.args="-r target/reports -f html,junit features"

Similarly, any arguments can be passed to Gwen in this manner through the gwen.args -D command line argument. Note that the -b batch mode switch is implicitly passed to Gwen and therefore you do not need to explicitly specify it as an argument. The gwen profile will only be activated if gwen.args is specified in the call to Maven.

By default, Gwen will use the latest version of Selenium bundled in its distribution but you can change it if you need to by performing the following. You would normally not need to do this unless you are targeting a legacy or beta browser or need to run a specific version of selenium.

  • Uncomment lines 21, 22, 27 to 31, and 61 to 71.
  • Update the selenium.version property (at line 21) with the release number you would like to use