AppliTools have made visual UI testing extremely simple with their online service. Automating basic visual tests with their Eyes SDK involves just three essential activities:
- Starting a new visual test session
- Saving one or more checkpoint images
- Closing the session and checking the results
Some of our Gwen users that we support have recently enquired about visual testing and we decided to provide it by integrating with AppliTools in gwen-web version 2.32.0.
To perform visual testing with Gwen, you will need:
- gwen-web version 2.32.0 or later
- an AppliTools account and API key (visit applitools.com to acquire these)
Set API key in your Environment
Once you have the above, you need to set the APPLITOOLS_API_KEY environment variable on your host (where Gwen will run) to your API key value.
Linux
export APPLITOOLS_API_KEY=your-api-key
Windows
set APPLITOOLS_API_KEY=your-api-key
Start a Visual Test Session
Consider the following executable Gwen feature (and associated meta available here):
Feature:
Feature: Todo items
Scenario: Add items in my Todo list
Given I launch the Todo app
When I add a "Walk the dog" item
And I add a "Get the milk" item
Then the number of active items should be "2"
Scenario: Complete one item
When I tick the "Get the milk" item
Then the number of active items should be "1"
Scenario: Complete another item
When I tick the "Walk the dog" item
Then the number of active items should be "0"
Now let’s say we want to add some visual testing to this.
The first thing we will need to do is instruct Gwen to start a new visual test session and give it a name and viewport size. For example, the following Gwen step will create a new visual test session named “Todo items” and set the viewport size to 600 pixels high and wide:
I start visual test as "Todo items" in 600 x 600 viewport(see DSL here)
We can insert this at line 5 in the feature above as follows:
Feature: Todo items
Scenario: Add items in my Todo list
Given I launch the Todo app
And I start visual test as "Todo items" in 600 x 600 viewport
When ..
Checkpoint Screenshot Images
With the visual test session started, we can now start taking screenshots of the viewport and save them as checkpoints in AppliTools with a name and match level. For example, we can checkpoint the initial state of the Todo UI after the application has loaded and our visual test session has started. At this stage, the Todo UI will have no items in it and we can save the checkpoint of that image under the name “No Todo items” in AppliTools using the STRICT (layout and content) match level as follows:
I check viewport visual as "No Todo items" using STRICT match(see DSL here)
We can insert this at line 6 in the feature above as follows:
Feature: Todo items
Scenario: Add items in my Todo list
Given I launch the Todo app
And I start visual test as "Todo items" in 600 x 600 viewport
And I check viewport visual as "No Todo items" using STRICT match
When ..
Similarly, we can checkpoint the screenshot after adding two items as follows (line 10 below):
Feature: Todo items
Scenario: Add items in my Todo list
Given I launch the Todo app
And I start visual test as "Todo items" in 600 x 600 viewport
And I check viewport visual as "No Todo items" using STRICT match
When I add a "Walk the dog" item
And I add a "Get the milk" item
Then the number of active items should be "2"
And I check viewport visual as "Active Todo items" using STRICT match
..
And checkpoint the screenshot after we complete (tick) each item as follows (lines 15 and 20 below):
Feature: Todo items
Scenario: Add items in my Todo list
Given I launch the Todo app
And I start visual test as "Todo items" in 600 x 600 viewport
And I check viewport visual as "No Todo items" using STRICT match
When I add a "Walk the dog" item
And I add a "Get the milk" item
Then the number of active items should be "2"
And I check viewport visual as "Active Todo items" using STRICT match
Scenario: Complete one item
When I tick the "Get the milk" item
Then the number of active items should be "1"
And I check viewport visual as "One completed Todo item" using STRICT match
Scenario: Complete another item
When I tick the "Walk the dog" item
Then the number of active items should be "0"
And I check viewport visual as "All completed Todo items" using STRICT match
This gives us a total of four checkpoint images for our visual test.
Close Session and Check Results
The final step is to close the session and check that the visual test has passed. In Gwen we can do this with the following DSL step:
the visual test should pass(see DSL here)
We can insert this line at the end of the feature as follows:
Feature: Todo items
Scenario: Add items in my Todo list
Given I launch the Todo app
And I start visual test as "Todo items" in 600 x 600 viewport
And I check viewport visual as "No Todo items" using STRICT match
When I add a "Walk the dog" item
And I add a "Get the milk" item
Then the number of active items should be "2"
And I check viewport visual as "Active Todo items" using STRICT match
Scenario: Complete one item
When I tick the "Get the milk" item
Then the number of active items should be "1"
And I check viewport visual as "One completed Todo item" using STRICT match
Scenario: Complete another item
When I tick the "Walk the dog" item
Then the number of active items should be "0"
And I check viewport visual as "All completed Todo items" using STRICT match
And the visual test should pass
This last step will close the visual test session in AppliTools and return successfully if all checkpoint images pass their respective match level checks. This first time you run a visual test, AppliTools will baseline all checkpoint images and return a successful result. Subsequent runs will compare all respective images with that baseline using the comparison strategy defined by the match level specified at each checkpoint. Any failures will result in an error that Gwen will report.
AppliTools Dasbhoard
Whether all visual checks pass or not, Gwen will provide a link in the console and in the evaluation report to the online AppliTools dashboard containing the detailed results.
Console:
INFO - Evaluating Step: And the visual test should pass INFO - Visual check status: Passed. Details at: https://applitools/dashboard-url INFO - [406ms] Passed Step: And the visual test should pass
HTML Report:
Dashboard:
From the AppliTools dashboard, you can inspect and resolve any failures.
Launching the Example
The sample feature described in this post and associated meta are bundled in every Gwen distribution and are available on GitHub here also:
If you install Gwen, you can launch the example in your installation directory on the command line as follows (if you installed a Gwen workspace, you can omit the -r target/reports parameter, don’t forget to set the APPLITOOLS_API_KEY environment variable as described earlier):
Linux
./gwen -b -r target/reports features/visual/todo
Windows
gwen -b -r target/reports features/visual/todo
The Gwen evaluation report will be generated here (relative to your install directory): target/reports/index.html
The example here inlines the visual testing steps in the feature itself, but you could choose to externalise them to Gwen meta to keep your features clean in your own projects if you wish.
More Info
You can also override various AppliTools defaults and runtime aspects in Gwen. For this and more details, see Gwen visual testing on our wiki.
More to Come
Gwen currently supports very basic visual testing with AppliTools (as described above). AppliTools itself has much more features and capabilities which we have not yet fully explored or integrated. We will integrate more capabilities as time goes on and as users start providing us with feedback.
Thanks
We thank the team at AppliTools for helping and supporting us and hope our Gwen users will like this feature and find it very useful.
