Gwen now provides seamless integration with report portal for centralised reporting and analytics.
To send all your Gwen evaluation results to a report portal server, just add the following settings to your Gwen properties file and invoke Gwen on your existing feature suite with the -f rp option:
Set [host] and [port] to the host and port of your report portal server
Set [report-portal-api-key] to the unique key of your report portal instance
Set [launch-name] to your desired launch name
Set [project-name] to your desired project name
These are the minimal settings required to achieve integration. Once you’ve configured the above settings, simply include rp in your -f/--format option when you invoke Gwen.
Example: ./gwen -b -f rp features
And if you want to also generate HTML reports:
./gwen -b -r target/reports -f rp,html features
The results of all evaluated Gherkin nodes in your features will then be asynchronously sent to report portal during execution. The complete Gherkin syntax is supported and reported (including Rules, Scenarios, Outlines/Templates, steps, data tables, doc strings, etc, etc..). That’s all there is to it! You will now be able to visualise all your Gwen results in report portal and utilise its analytics features to track and manage results.
Supported options and features
Reruns
Merges
Source references
File attachments
Error traces (inlined or attached)
Screenshots
Optional logging of Gwen meta
Optional logging of StepDefs (inlined or nested)
Optional logging of tags and annotations (as attributes)
Heartbeat for monitoring connection availability
Various test case ID generation strategies (better history association for Gherkin)
A common secure coding practice is to never log passwords. Automated test cases often use passwords to log into applications under test in various environments. In such cases care should be taken to protect and hide passwords from logs, reports, error messages and console outputs.
Gwen masked settings
Gwen masked settings address this concern by making your sensitive data appear as ●●●●● in all outputs and reports.
To mask the value of a setting named user.password for example:
Define it as user.password\:masked=secret in your Gwen settings/properties file
Or as user.password:masked=secret through the JVM -D option
Then just reference the setting where you need as user.password .
Example
When I enter "${user.password}" in the password field
When evaluated, the above will be logged as follows:
When I enter "●●●●●" in the password field
Note that Gwen will mask sensitive data in all outputs but it is still your responsibility to only enter sensitive data into protected inputs, such as fields that themselves mask the raw value so that they are not displayed as clear text on web pages or in captured screenshots.
You can also change the default masking character '●' by assigning the gwen.mask.char setting to a different character if desired.
Some general rules for writing good Gherkin include:
All steps in Scenarios and Backgrounds should satisfy Given-When-Then order
Given steps should set context
When steps should perform actions
Then steps should assert expectations
When it comes to using Gherkin features to drive automated tests, these rules are often broken. Why? Put simply, they are too easy to break when not enforced.
To help, we have introduced a gwen.behavior.rules setting in Gwen, that when set to strict, will report all such violations as errors.
The Bad and the Imperative
To demonstrate, consider the following bad feature specification that breaks most of the above rules and other good practices:
# file: gwen-workspace/features/Todo.feature
Feature: Bad todo
Scenario: Add and complete items
Given I navigate to "http://todomvc.com/examples/angularjs"
Then I should be on the todo page
When I add a "Walk the dog" item
And I add a "Get the milk" item
Then the active item count should be "2"
When I tick the "Get the milk" item
Then the active item count should be "1"
When I tick the "Walk the dog" item
Then the active item count should be "0"
The automation glue for this feature can be written in a Gwen meta file as follows (custom StepDefs highlighted):
# file: gwen-workspace/features/Todo.meta
Feature: Todo meta
@StepDef
Scenario: I should be on the todo page
Given the heading can be located by tag name "h1"
Then the heading should be "todos"
And the todo field can be located by class name "new-todo"
And the active item count can be located by css selector ".todo-count > strong"
@StepDef
Scenario: I add a "<todo>" item
When I enter "$<todo>" in the todo field
Then the "$<todo>" item can be located by xpath
"""
//label[contains(.,'$<todo>')]/preceding-sibling::input
"""
And the "$<todo>" item should not be ticked
Launching Gwen with the default gwen.behavior.rules=lenient setting will successfully execute the Todo.feature file.
Running the feature again will result in the following error:
ERROR - Given-When-Then order not satisfied by steps in Scenario at features/Todo.feature:5
This tells us that the steps in our Scenario do not follow Given-When-Then order. A brute force fix for this could be to change and order the keywords in each step and remove duplicates as follows:
# file: gwen-workspace/features/Todo.feature
Feature: Bad todo
Scenario: Add and complete items
Given I navigate to "http://todomvc.com/examples/angularjs"
And I should be on the todo page
When I add a "Walk the dog" item
Then I add a "Get the milk" item
And the active item count should be "2"
And I tick the "Get the milk" item
And the active item count should be "1"
And I tick the "Walk the dog" item
And the active item count should be "0"
Running this feature will now result in an new error:
ERROR - Failed step [at line 6]: Given I navigate to "http://todomvc.com/examples/angularjs": Action behavior not permitted where context is expected
You can also see this error in the report generated at target/reports/index.html in your workspace folder.
This error is telling us that we cannot perform an action (the predefined Gwen navigation action in this case) in a Given step which is meant for setting context. When steps are meant for actions but if we change the first step to use When, we will get the previous Given-When-Then order error again.
No matter how we rearrange these steps, the strict validation will always give us an error. So continuing in this manner is futile. We are forced to COMPLETELY rethink what we’re doing if we want to have Good Gherkin!
If we step back a little, it’s easy to see that our scenario has too many steps doing many things. It’s is a mish-mash of imperative operations with no clear behavior.
Let’s do this better!
The Good and the Declarative
The first step is to forget about automation for a moment and rewrite our feature in terms of behavior. Let’s make it about adding and completing items in a todo list with a feature that describes the intended behavior declaratively and conforms to good Gherkin rules like this:
# gwen-workspace/features/Todo.feature
Feature: Good todo
Scenario: Add and complete items
Given a new todo list
When the following items are added
| ITEM |
| Get the milk |
| Walk the dog |
And all items are completed
Then no active items should remain
Note that Gwen also has a feature mode setting that when set to ‘declarative‘ will prevent you from calling predefined and imperative DSL steps in feature files and force you to move all I click this and I enter that clutter into meta files where they belong.
All the imperative automation glue for the declarative feature steps can now be confined to our meta file as follows:
# file: features/Todo.meta
Feature: Todo meta
@StepDef
Scenario: a new todo list
Given the heading can be located by tag name "h1"
When I navigate to "http://todomvc.com/examples/angularjs"
Then the heading should be "todos"
And the todo field can be located by class name "new-todo"
And the active item count can be located by css selector ".todo-count > strong"
@StepDef
@DataTable(header="top")
Scenario: the following items are added
And I add a todo item for each data record
@StepDef
Scenario: I add a todo item
Given the todo item is "${data[ITEM]}"
When I enter the todo item in the todo field
Then the added item can be located by xpath
"""
//label[contains(.,'${the todo item}')]/preceding-sibling::input
"""
And the added item should be unticked
And the active item count should be "${record.number}"
@StepDef
Scenario: all items are completed
When I tick item for each item located by class name "toggle"
@StepDef
Scenario: no active items should remain
Then the active item count should be "0"
The second and third StepDef‘s work together to add each item declared in the data table of the second step in our feature to the list of todo items in the app. For more on data tables, see Gwen data tables.
Custom StepDef Behavior
We’re almost there, but running this feature will result in the following error because the strict validation mechanism in Gwen does not know what type of behavior our step definitions contain.
ERROR - Failed step [at line 6]: Given a new todo list: Missing @Context, @Action, or @Assertion behavior tag on StepDef at features/Todo.meta:6
So we have to annotate each StepDef called by steps in our feature with one of the following tags:
@Context : for context setting StepDefs meant for Given steps
@Action : for action performing StepDefs meant for When steps
@Assertion : for assertion checking StepDefs meant for Then steps
Gwen will then know which StepDef can or cannot be called by Givens, Whens, or Thens and report violations accordingly. This is how behavior types are bound to step definitions.
Note that checking for the presence of behavior tags on StepDefs will only happen in strict behavior mode and Gwen will not perform all the same rules checks on meta files as it does on feature files (since meta files are necessarily imperative automation glue).
Adding behavior tags to all StepDefs called by steps in our feature yields the final and complete meta:
# file: features/Todo.meta
Feature: Todo meta
@StepDef
@Context
Scenario: a new todo list
Given the heading can be located by tag name "h1"
When I navigate to "http://todomvc.com/examples/angularjs"
Then the heading should be "todos"
And the todo field can be located by class name "new-todo"
And the active item count can be located by css selector ".todo-count > strong"
@StepDef
@Action
@DataTable(header="top")
Scenario: the following items are added
And I add a todo item for each data record
@StepDef
Scenario: I add a todo item
Given the todo item is "${data[ITEM]}"
When I enter the todo item in the todo field
Then the added item can be located by xpath
"""
//label[contains(.,'${the todo item}')]/preceding-sibling::input
"""
And the added item should be unticked
And the active item count should be "${record.number}"
@StepDef
@Action
Scenario: all items are completed
When I tick item for each item located by class name "toggle"
@StepDef
@Assertion
Scenario: no active items should remain
Then the active item count should be "0"
We’re all Good Now!
Good Gherkin rules will now be satisfied and running the feature again will succeed.
Adding behavior checks to Gwen was inspired by various articles and discussions about good Gherkin and current challenges in this space. Some of these include:
