Gwen now provides seamless integration with report portal for centralised reporting and analytics.
To send all your Gwen evaluation results to a report portal server, just add the following settings to your Gwen properties file and invoke Gwen on your existing feature suite with the -f rp option:
Set [host] and [port] to the host and port of your report portal server
Set [report-portal-api-key] to the unique key of your report portal instance
Set [launch-name] to your desired launch name
Set [project-name] to your desired project name
These are the minimal settings required to achieve integration. Once you’ve configured the above settings, simply include rp in your -f/--format option when you invoke Gwen.
Example: ./gwen -b -f rp features
And if you want to also generate HTML reports:
./gwen -b -r target/reports -f rp,html features
The results of all evaluated Gherkin nodes in your features will then be asynchronously sent to report portal during execution. The complete Gherkin syntax is supported and reported (including Rules, Scenarios, Outlines/Templates, steps, data tables, doc strings, etc, etc..). That’s all there is to it! You will now be able to visualise all your Gwen results in report portal and utilise its analytics features to track and manage results.
Supported options and features
Reruns
Merges
Source references
File attachments
Error traces (inlined or attached)
Screenshots
Optional logging of Gwen meta
Optional logging of StepDefs (inlined or nested)
Optional logging of tags and annotations (as attributes)
Heartbeat for monitoring connection availability
Various test case ID generation strategies (better history association for Gherkin)
Applitools integration has been dropped in Gwen 3.
AppliTools have made visual UI testing extremely simple with their online service. Automating basic visual tests with their Eyes SDK involves just three essential activities:
Starting a new visual test session
Saving one or more checkpoint images
Closing the session and checking the results
Some of our Gwen users that we support have recently enquired about visual testing and we decided to provide it by integrating with AppliTools in gwen-web version 2.32.0.
To perform visual testing with Gwen, you will need:
an AppliTools account and API key (visit applitools.com to acquire these)
Set API key in your Environment
Once you have the above, you need to set the APPLITOOLS_API_KEY environment variable on your host (where Gwen will run) to your API key value.
Linux
export APPLITOOLS_API_KEY=your-api-key
Windows
set APPLITOOLS_API_KEY=your-api-key
Start a Visual Test Session
Consider the following executable Gwen feature (and associated meta available here):
Feature:
Feature: Todo items
Scenario: Add items in my Todo list
Given I launch the Todo app
When I add a "Walk the dog" item
And I add a "Get the milk" item
Then the number of active items should be "2"
Scenario: Complete one item
Given I am on the todo page
When I tick the "Get the milk" item
Then the number of active items should be "1"
Scenario: Complete another item
Given I am on the todo page
When I tick the "Walk the dog" item
Then the number of active items should be "0"
Now let’s say we want to add some visual testing to this.
The first thing we will need to do is instruct Gwen to start a new visual test session and give it a name and viewport size. For example, the following Gwen step will create a new visual test session named “Todo items” and set the viewport size to 600 pixels high and wide:
I start visual test as "Todo items" in 600 x 600 viewport (see DSL here)
We can insert this at line 5 in the feature above as follows:
Feature: Todo items
Scenario: Add items in my Todo list
Given I launch the Todo app
When I start visual test as "Todo items" in 600 x 600 viewport
And I add a "Walk the dog" item
And I add a "Get the milk" item
Then the number of active items should be "2"
..
Checkpoint Screenshot Images
With the visual test session started, we can now start taking screenshots of the viewport and save them as checkpoints in AppliTools with a name and match level. For example, we can checkpoint the initial state of the Todo UI after the application has loaded and our visual test session has started. At this stage, the Todo UI will have no items in it and we can save the checkpoint of that image under the name “No Todo items” in AppliTools using the STRICT (layout and content) match level as follows:
I check viewport visual as "No Todo items" using STRICT match (see DSL here)
We can insert this at after starting the visual test as follows (line 6):
Feature: Todo items
Scenario: Add items in my Todo list
Given I launch the Todo app
When I start visual test as "Todo items" in 600 x 600 viewport
And I check viewport visual as "No items" using STRICT match
And I add a "Walk the dog" item
And I add a "Get the milk" item
And I check viewport visual as "Active items" using STRICT match
Then the number of active items should be "2"
..
And we can checkpoint the screenshot after adding two items as follows (line 9):
Feature: Todo items
Scenario: Add items in my Todo list
Given I launch the Todo app
When I start visual test as "Todo items" in 600 x 600 viewport
And I check viewport visual as "No items" using STRICT match
And I add a "Walk the dog" item
And I add a "Get the milk" item
And I check viewport visual as "Active items" using STRICT match
Then the number of active items should be "2"
..
And checkpoint the screenshot after we completing each item as follows (lines 15 and 21 below):
Feature: Todo items
Scenario: Add items in my Todo list
Given I launch the Todo app
When I start visual test as "Todo items" in 600 x 600 viewport
And I check viewport visual as "No items" using STRICT match
And I add a "Walk the dog" item
And I add a "Get the milk" item
And I check viewport visual as "Active items" using STRICT match
Then the number of active items should be "2"
Scenario: Complete one item
Given I am on the todo page
When I tick the "Get the milk" item
And I check viewport visual as "One completed item" using STRICT match
Then the number of active items should be "1"
Scenario: Complete another item
Given I am on the todo page
When I tick the "Walk the dog" item
And I check viewport visual as "All completed items" using STRICT match
Then the number of active items should be "0"
This gives us a total of four checkpoint images for our visual test.
Close Session and Check Results
The final step is to close the session and check that the visual test has passed. In Gwen we can do this with the following DSL step:
We can insert this line at the end of the feature as follows:
Feature: Todo items
Scenario: Add items in my Todo list
Given I launch the Todo app
When I start visual test as "Todo items" in 600 x 600 viewport
And I check viewport visual as "No items" using STRICT match
And I add a "Walk the dog" item
And I add a "Get the milk" item
And I check viewport visual as "Active items" using STRICT match
Then the number of active items should be "2"
Scenario: Complete one item
Given I am on the todo page
When I tick the "Get the milk" item
And I check viewport visual as "One completed item" using STRICT match
Then the number of active items should be "1"
Scenario: Complete another item
Given I am on the todo page
When I tick the "Walk the dog" item
And I check viewport visual as "All completed items" using STRICT match
Then the number of active items should be "0"
And the visual test should pass
This last step will close the visual test session in AppliTools and return successfully if all checkpoint images pass their respective match level checks. This first time you run a visual test, AppliTools will baseline all checkpoint images and return a successful result. Subsequent runs will compare all respective images with that baseline using the comparison strategy defined by the match level specified at each checkpoint. Any failures will result in an error that Gwen will report.
AppliTools Dasbhoard
Whether all visual checks pass or not, Gwen will provide a link in the console and in the evaluation report to the online AppliTools dashboard containing the detailed results.
Console:
INFO - Evaluating Step: And the visual test should pass
INFO - Visual check status: Passed. Details at: https://applitools/dashboard-url
INFO - [406ms] Passed Step: And the visual test should pass
HTML Report:
Dashboard:
From the AppliTools dashboard, you can inspect and resolve any failures.
Launching the Example
The sample feature described in this post and associated meta are bundled in every Gwen distribution and are available on GitHub here also:
If you install Gwen, you can launch the example in your installation directory on the command line as follows (if you installed a Gwen workspace, you can omit the -r target/reports parameter, don’t forget to set the APPLITOOLS_API_KEY environment variable as described earlier):
Linux
./gwen -b -r target/reports features/visual/todo
Windows
gwen -b -r target/reports features/visual/todo
The Gwen evaluation report will be generated here (relative to your install directory): target/reports/index.html
The example here inlines the visual testing steps in the feature itself, but you could choose to externalise them to Gwen meta to keep your features clean in your own projects if you wish.
More Info
You can also override various AppliTools defaults and runtime aspects in Gwen. For this and more details, see Gwen visual testing on our wiki.
More to Come
Gwen currently supports very basic visual testing with AppliTools (as described above). AppliTools itself has much more features and capabilities which we have not yet fully explored or integrated. We will integrate more capabilities as time goes on and as users start providing us with feedback.
Thanks
We thank the team at AppliTools for helping and supporting us and hope our Gwen users will like this feature and find it very useful.
Although you can download and install Gwen locally to run automated tests, it is sometimes useful to integrate it with a build tool such as Maven and have it download and install it for you and run tests as part of the release verification process. Integrating with Maven can also make it easier for you to run Gwen on a continuous integration build server too.
Gwen does not ship with a Maven plugin, but you can still integrate it with Maven using a POM file like this:
Be sure to change the Gwen version by updating the gwen.web.version property at line 20 with the latest version or another version you would like to use.
With this POM and your Gwen settings in place you can have Maven download, install, and launch the latest version of Gwen with the following command (where <args> is a space separated list of arguments that you want to pass to Gwen):
mvn verify -Dgwen.args="<args>"
So if your executable tests are in a folder named ‘features’ relative to your POM file, then you can execute them with a Maven command like this:
mvn verify -Dgwen.args="features"
And to also generate html and junit style reports in the target/reports folder..
Similarly, any arguments can be passed to Gwen in this manner through the gwen.args -D command line argument. Note that the -b batch mode switch is implicitly passed to Gwen and therefore you do not need to explicitly specify it as an argument. The gwen profile will only be activated if gwen.args is specified in the call to Maven.
By default, Gwen will use the latest version of Selenium bundled in its distribution but you can change it if you need to by performing the following. You would normally not need to do this unless you are targeting a legacy or beta browser or need to run a specific version of selenium.
Uncomment lines 21, 22, 27 to 31, and 61 to 71.
Update the selenium.version property (at line 21) with the release number you would like to use
