Tag: automation

Visual Testing with Gwen and AppliTools

AppliTools have made visual UI testing extremely simple with their online service. Automating basic visual tests with their Eyes SDK involves just three essential activities:

  1. Starting a new visual test session
  2. Saving one or more checkpoint images
  3. Closing the session and checking the results

Some of our Gwen users that we support have recently enquired about visual testing and we decided to provide it by integrating with AppliTools in gwen-web version 2.32.0.

To perform visual testing with Gwen, you will need:

Set API key in your Environment

Once you have the above, you need to set the APPLITOOLS_API_KEY environment variable on your host (where Gwen will run) to your API key value.

Linux

export APPLITOOLS_API_KEY=your-api-key

Windows

set APPLITOOLS_API_KEY=your-api-key

Start a Visual Test Session

Consider the following executable Gwen feature (and associated meta available here):

Feature:

 Feature: Todo items

Scenario: Add items in my Todo list
    Given I launch the Todo app
     When I add a "Walk the dog" item
      And I add a "Get the milk" item
     Then the number of active items should be "2"

Scenario: Complete one item
     When I tick the "Get the milk" item
     Then the number of active items should be "1"

Scenario: Complete another item
     When I tick the "Walk the dog" item
     Then the number of active items should be "0"

Now let’s say we want to add some visual testing to this.

The first thing we will need to do is instruct Gwen to start a new visual test session and give it a name and viewport size. For example, the following Gwen step will create a new visual test session named “Todo items” and set the viewport size to 600 pixels high and wide:

  • I start visual test as "Todo items" in 600 x 600 viewport (see DSL here)

We can insert this at line 5 in the feature above as follows:

 Feature: Todo items

Scenario: Add items in my Todo list
    Given I launch the Todo app
      And I start visual test as "Todo items" in 600 x 600 viewport
     When ..

Checkpoint Screenshot Images

With the visual test session started, we can now start taking screenshots of the viewport and save them as checkpoints in AppliTools with a name and match level. For example, we can checkpoint the initial state of the Todo UI after the application has loaded and our visual test session has started. At this stage, the Todo UI will have no items in it and we can save the checkpoint of that image under the name “No Todo items” in AppliTools using the STRICT (layout and content) match level as follows:

  • I check viewport visual as "No Todo items" using STRICT match (see DSL here)

We can insert this at line 6 in the feature above as follows:

 Feature: Todo items

Scenario: Add items in my Todo list
    Given I launch the Todo app
      And I start visual test as "Todo items" in 600 x 600 viewport
      And I check viewport visual as "No Todo items" using STRICT match
     When ..

Similarly, we can checkpoint the screenshot after adding two items as follows (line 10 below):

 Feature: Todo items

Scenario: Add items in my Todo list
    Given I launch the Todo app
      And I start visual test as "Todo items" in 600 x 600 viewport
      And I check viewport visual as "No Todo items" using STRICT match
     When I add a "Walk the dog" item
      And I add a "Get the milk" item
     Then the number of active items should be "2"
      And I check viewport visual as "Active Todo items" using STRICT match
      ..

And checkpoint the screenshot after we complete (tick) each item as follows (lines 15 and 20 below):

 Feature: Todo items

Scenario: Add items in my Todo list
    Given I launch the Todo app
      And I start visual test as "Todo items" in 600 x 600 viewport
      And I check viewport visual as "No Todo items" using STRICT match
     When I add a "Walk the dog" item
      And I add a "Get the milk" item
     Then the number of active items should be "2"
      And I check viewport visual as "Active Todo items" using STRICT match

Scenario: Complete one item
     When I tick the "Get the milk" item
     Then the number of active items should be "1"
      And I check viewport visual as "One completed Todo item" using STRICT match

Scenario: Complete another item
     When I tick the "Walk the dog" item
     Then the number of active items should be "0"
      And I check viewport visual as "All completed Todo items" using STRICT match

This gives us a total of four checkpoint images for our visual test.

Close Session and Check Results

The final step is to close the session and check that the visual test has passed. In Gwen we can do this with the following DSL step:

  • the visual test should pass (see DSL here)

We can insert this line at the end of the feature as follows:

 Feature: Todo items

Scenario: Add items in my Todo list
    Given I launch the Todo app
      And I start visual test as "Todo items" in 600 x 600 viewport
      And I check viewport visual as "No Todo items" using STRICT match
     When I add a "Walk the dog" item
      And I add a "Get the milk" item
     Then the number of active items should be "2"
      And I check viewport visual as "Active Todo items" using STRICT match

Scenario: Complete one item
     When I tick the "Get the milk" item
     Then the number of active items should be "1"
      And I check viewport visual as "One completed Todo item" using STRICT match

Scenario: Complete another item
     When I tick the "Walk the dog" item
     Then the number of active items should be "0"
      And I check viewport visual as "All completed Todo items" using STRICT match
      And the visual test should pass

This last step will close the visual test session in AppliTools and return successfully if all checkpoint images pass their respective match level checks. This first time you run a visual test, AppliTools will baseline all checkpoint images and return a successful result. Subsequent runs will compare all respective images with that baseline using the comparison strategy defined by the match level specified at each checkpoint. Any failures will result in an error that Gwen will report.

AppliTools Dasbhoard

Whether all visual checks pass or not, Gwen will provide a link in the console and in the evaluation report to the online AppliTools dashboard containing the detailed results.

Console:

INFO - Evaluating Step: And the visual test should pass
INFO - Visual check status: Passed. Details at: https://applitools/dashboard-url
INFO - [406ms] Passed Step: And the visual test should pass

HTML Report:

Dashboard:

From the AppliTools dashboard, you can inspect and resolve any failures.

Launching the Example

The sample feature described in this post and associated meta are bundled in every Gwen distribution and are available on GitHub here also:

If you install Gwen, you can launch the example in your installation directory on the command line as follows (if you installed a Gwen workspace, you can omit the -r target/reports parameter, don’t forget to set the APPLITOOLS_API_KEY environment variable as described earlier):

Linux

./gwen -b -r target/reports features/visual/todo

Windows

gwen -b -r target/reports features/visual/todo

The Gwen evaluation report will be generated here (relative to your install directory): target/reports/index.html

The example here inlines the visual testing steps in the feature itself, but you could choose to externalise them to Gwen meta to keep your features clean in your own projects if you wish.

More Info

You can also override various AppliTools defaults and runtime aspects in Gwen. For this and more details, see Gwen visual testing on our wiki.

More to Come

Gwen currently supports very basic visual testing with AppliTools (as described above). AppliTools itself has much more features and capabilities which we have not yet fully explored or integrated. We will integrate more capabilities as time goes on and as users start providing us with feedback.

Thanks

We thank the team at AppliTools for helping and supporting us and hope our Gwen users will like this feature and find it very useful.

Advertisements

Integrating Gwen with Maven

Although you can download and install Gwen locally to run automated tests, it is sometimes useful to integrate it with a build tool such as Maven and have it download and install it for you and run tests as part of the release verification process. Integrating with Maven can also make it easier for you to run Gwen on a continuous integration build server too.

Gwen does not ship with a Maven plugin, but you can still integrate it with Maven using a POM file like this:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

  <modelVersion>4.0.0</modelVersion>
  <groupId>org.gweninterpreter</groupId>
  <artifactId>gwen-web-maven</artifactId>
  <name>gwen-web-maven</name>
  <version>4.0.0</version>
  <packaging>pom</packaging>

  <profiles>
    <profile>
      <id>gwen</id>
      <activation>
        <property>
          <name>gwen.args</name>
        </property>
      </activation>
      <properties>
        <gwen.web.version>???</gwen.web.version>
        <!--selenium.version>???</selenium.version-->
        <!--selenium.dir>target/selenium-${selenium.version}</selenium.dir-->
        <gwen.dir>target/gwen-web-${gwen.web.version}</gwen.dir>
        <gwen.classpath>${selenium.dir}/*${path.separator}${gwen.dir}/lib/*</gwen.classpath>
      </properties>
      <dependencies>
        <!--dependency>
          <groupId>org.seleniumhq.selenium</groupId>
          <artifactId>selenium-java</artifactId>
          <version>${selenium.version}</version>
        </dependency-->
        <dependency>
          <groupId>org.gweninterpreter</groupId>
          <artifactId>gwen-web</artifactId>
          <version>${gwen.web.version}</version>
          <type>zip</type>
          <scope>provided</scope>
        </dependency>
      </dependencies>
      <build>
        <plugins>
          <plugin>
            <artifactId>maven-dependency-plugin</artifactId>
            <version>2.9</version>
            <executions>
              <execution>
                <id>install-gwen</id>
                <phase>integration-test</phase>
                <goals>
                  <goal>unpack-dependencies</goal>
                </goals>
                <configuration>
                  <includeGroupIds>org.gweninterpreter</includeGroupIds>
                  <includeArtifactIds>gwen-web</includeArtifactIds>
                  <includeTypes>zip</includeTypes>
                  <outputDirectory>target</outputDirectory>
                  <stripClassifier>true</stripClassifier>
                  <stripVersion>true</stripVersion>
                </configuration>
              </execution>
              <!--execution>
                <id>install-selenium</id>
                <phase>integration-test</phase>
                <goals>
                  <goal>copy-dependencies</goal>
                </goals>
                <configuration>
                  <excludeScope>provided</excludeScope>
                  <outputDirectory>${selenium.dir}</outputDirectory>
                </configuration>
              </execution-->
            </executions>
          </plugin>
          <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>exec-maven-plugin</artifactId>
            <version>1.3.2</version>
            <executions>
              <execution>
                <id>launch-gwen</id>
                <phase>verify</phase>
                <goals>
                  <goal>exec</goal>
                </goals>
              </execution>
            </executions>
            <configuration>
              <executable>java</executable>
              <commandlineArgs>-cp ${gwen.classpath} gwen.web.WebInterpreter -b ${gwen.args}</commandlineArgs>
              <successCodes>
                <successCode>0</successCode>
              </successCodes>
            </configuration>
          </plugin>
        </plugins>
      </build>
    </profile>
  </profiles>
  
</project>

Be sure to change the Gwen version by updating the gwen.web.version property at line 20 with the latest version or another version you would like to use.

With this POM and your Gwen settings in place you can have Maven download, install, and launch the latest version of Gwen with the following command (where <args> is a space separated list of arguments that you want to pass to Gwen):

mvn verify -Dgwen.args="<args>"

So if your executable tests are in a folder named ‘features’ relative to your POM file, then you can execute them with a Maven command like this:

mvn verify -Dgwen.args="features"

And to also generate html and junit style reports in the target/reports folder..

mvn verify -Dgwen.args="-r target/reports -f html,junit features"

Similarly, any arguments can be passed to Gwen in this manner through the gwen.args -D command line argument. Note that the -b batch mode switch is implicitly passed to Gwen and therefore you do not need to explicitly specify it as an argument. The gwen profile will only be activated if gwen.args is specified in the call to Maven.

By default, Gwen will use the latest version of Selenium bundled in its distribution but you can change it if you need to by performing the following. You would normally not need to do this unless you are targeting a legacy or beta browser or need to run a specific version of selenium.

  • Uncomment lines 21, 22, 27 to 31, and 61 to 71.
  • Update the selenium.version property (at line 21) with the release number you would like to use