Gwen Report Portal Integration

Gwen now provides seamless integration with report portal for centralised reporting and analytics.

To send all your Gwen evaluation results to a report portal server, just add the following settings to your Gwen properties file and invoke Gwen on your existing feature suite with the -f rp option:

# gwen.properties

rp.endpoint = http://[host]:[port]
rp.uuid = [report-portal-api-key]
rp.launch = [launch-name]
rp.project = [project-name]
  • Set [host] and [port] to the host and port of your report portal server
  • Set [report-portal-api-key] to the unique key of your report portal instance
  • Set [launch-name] to your desired launch name
  • Set [project-name] to your desired project name

These are the minimal settings required to achieve integration. Once you’ve configured the above settings, simply include rp in your -f/--format option when you invoke Gwen.

Example: ./gwen -b -f rp features

And if you want to also generate HTML reports:

./gwen -b -r target/reports -f rp,html features

The results of all evaluated Gherkin nodes in your features will then be asynchronously sent to report portal during execution. The complete Gherkin syntax is supported and reported (including Rules, Scenarios, Outlines/Templates, steps, data tables, doc strings, etc, etc..). That’s all there is to it! You will now be able to visualise all your Gwen results in report portal and utilise its analytics features to track and manage results.

Supported options and features

  • Reruns
  • Merges
  • Source references
  • File attachments
  • Error traces (inlined or attached)
  • Screenshots
  • Optional logging of Gwen meta
  • Optional logging of StepDefs (inlined or nested)
  • Optional logging of tags and annotations (as attributes)
  • Heartbeat for monitoring connection availability
  • Various test case ID generation strategies (better history association for Gherkin)
  • And more

References

Protecting Sensitive Data with Gwen

A common secure coding practice is to never log passwords. Automated test cases often use passwords to log into applications under test in various environments. In such cases care should be taken to protect and hide passwords from logs, reports, error messages and console outputs.

Gwen masked settings

Gwen masked settings address this concern by making your sensitive data appear as ●●●●● in all outputs and reports.

To mask the value of a setting named user.password for example:

  • Define it as user.password\:masked=secret in your Gwen settings/properties file
  • Or as user.password:masked=secret through the JVM -D option

Then just reference the setting where you need as user.password .

Example

When I enter "${user.password}" in the password field

When evaluated, the above will be logged as follows:

When I enter "●●●●●" in the password field

Note that Gwen will mask sensitive data in all outputs but it is still your responsibility to only enter sensitive data into protected inputs, such as fields that themselves mask the raw value so that they are not displayed as clear text on web pages or in captured screenshots.

You can also change the default masking character '●' by assigning the gwen.mask.char setting to a different character if desired.

Strictly Good Gherkin with Gwen

Some general rules for writing good Gherkin include:

  • All steps in Scenarios and Backgrounds should satisfy Given-When-Then order
  • Given steps should set context
  • When steps should perform actions
  • Then steps should assert expectations

When it comes to using Gherkin features to drive automated tests, these rules are often broken. Why? Put simply, they are too easy to break when not enforced.

To help, we have introduced a gwen.behavior.rules setting in Gwen, that when set to strict, will report all such violations as errors.

The Bad and the Imperative

To demonstrate, consider the following bad feature specification that breaks most of the above rules and other good practices:

# file: gwen-workspace/features/Todo.feature

Feature: Bad todo

  Scenario: Add and complete items
      Given I navigate to "http://todomvc.com/examples/angularjs"
       Then I should be on the todo page
       When I add a "Walk the dog" item
        And I add a "Get the milk" item
       Then the active item count should be "2"
       When I tick the "Get the milk" item
       Then the active item count should be "1"
       When I tick the "Walk the dog" item
       Then the active item count should be "0"

The automation glue for this feature can be written in a Gwen meta file as follows (custom StepDefs highlighted):

# file: gwen-workspace/features/Todo.meta

Feature: Todo meta

@StepDef
Scenario: I should be on the todo page
    Given the heading can be located by tag name "h1"
     Then the heading should be "todos"
      And the todo field can be located by class name "new-todo"
      And the active item count can be located by css selector ".todo-count > strong"

@StepDef
Scenario: I add a "<todo>" item
     When I enter "$<todo>" in the todo field
     Then the "$<todo>" item can be located by xpath
          """
          //label[contains(.,'$<todo>')]/preceding-sibling::input
          """
      And the "$<todo>" item should not be ticked

Launching Gwen with the default gwen.behavior.rules=lenient setting will successfully execute the Todo.feature file.

Executing the Feature

To install Gwen and run the feature file:

  • Get started with Gwen
  • Copy the above Todo.feature and Todo.meta files into a features folder
  • Open a command prompt in the parent of your features folder:
    • Linux/Mac/PowerShell:
      •  ./gwen features/Todo.feature -b
    • Windows DOS:
      •  gwen features/Todo.feature -b

Enabling Strict Behavior Checks

Now try running it with strict behavior rules enabled. Set the gwen.behavior.rules property in your gwen.properties file to strict.

# behavior rules checking (strict|lenient)
gwen.behavior.rules=strict

Running the feature again will result in the following error:

ERROR - Given-When-Then order not satisfied by steps in Scenario at features/Todo.feature:5

This tells us that the steps in our Scenario do not follow Given-When-Then order. A brute force fix for this could be to change and order the keywords in each step and remove duplicates as follows:

# file: gwen-workspace/features/Todo.feature

Feature: Bad todo

  Scenario: Add and complete items
      Given I navigate to "http://todomvc.com/examples/angularjs"
        And I should be on the todo page
       When I add a "Walk the dog" item
       Then I add a "Get the milk" item
        And the active item count should be "2"
        And I tick the "Get the milk" item
        And the active item count should be "1"
        And I tick the "Walk the dog" item
        And the active item count should be "0"

Running this feature will now result in an new error:

ERROR - Failed step [at line 6]: Given I navigate to "http://todomvc.com/examples/angularjs": Action behavior not permitted where context is expected

You can also see this error in the report generated at target/reports/index.html in your workspace folder.

This error is telling us that we cannot perform an action (the predefined Gwen navigation action in this case) in a Given step which is meant for setting context. When steps are meant for actions but if we change the first step to use When, we will get the previous Given-When-Then order error again.

No matter how we rearrange these steps, the strict validation will always give us an error. So continuing in this manner is futile. We are forced to COMPLETELY rethink what we’re doing if we want to have Good Gherkin!

If we step back a little, it’s easy to see that our scenario has too many steps doing many things. It’s is a mish-mash of imperative operations with no clear behavior.

Let’s do this better!

The Good and the Declarative

The first step is to forget about automation for a moment and rewrite our feature in terms of behavior. Let’s make it about adding and completing items in a todo list with a feature that describes the intended behavior declaratively and conforms to good Gherkin rules like this:

# gwen-workspace/features/Todo.feature

Feature: Good todo

Scenario: Add and complete items
    Given a new todo list
     When the following items are added
          | ITEM          |
          | Get the milk  |
          | Walk the dog  |
      And all items are completed
     Then no active items should remain

Note that Gwen also has a feature mode setting that when set to ‘declarative‘ will prevent you from calling predefined and imperative DSL steps in feature files and force you to move all I click this and I enter that clutter into meta files where they belong.


All the imperative automation glue for the declarative feature steps can now be confined to our meta file as follows:

# file: features/Todo.meta

Feature: Todo meta

  @StepDef
  Scenario: a new todo list
      Given the heading can be located by tag name "h1"
       When I navigate to "http://todomvc.com/examples/angularjs"
       Then the heading should be "todos"
        And the todo field can be located by class name "new-todo"
        And the active item count can be located by css selector ".todo-count > strong"

  @StepDef
  @DataTable(header="top")
  Scenario: the following items are added
        And I add a todo item for each data record

  @StepDef
  Scenario: I add a todo item
      Given the todo item is "${data[ITEM]}"
       When I enter the todo item in the todo field
       Then the added item can be located by xpath
             """
             //label[contains(.,'${the todo item}')]/preceding-sibling::input
             """
        And the added item should be unticked
        And the active item count should be "${record.number}"

  @StepDef
  Scenario: all items are completed
       When I tick item for each item located by class name "toggle"

  @StepDef
  Scenario: no active items should remain
       Then the active item count should be "0"

The second and third StepDef‘s work together to add each item declared in the data table of the second step in our feature to the list of todo items in the app. For more on data tables, see Gwen data tables.


Custom StepDef Behavior

We’re almost there, but running this feature will result in the following error because the strict validation mechanism in Gwen does not know what type of behavior our step definitions contain.

ERROR - Failed step [at line 6]: Given a new todo list: Missing @Context, @Action, or @Assertion behavior tag on StepDef at features/Todo.meta:6

So we have to annotate each StepDef called by steps in our feature with one of the following tags:

  • @Context : for context setting StepDefs meant for Given steps
  • @Action : for action performing StepDefs meant for When steps
  • @Assertion : for assertion checking StepDefs meant for Then steps

Gwen will then know which StepDef can or cannot be called by Givens, Whens, or Thens and report violations accordingly. This is how behavior types are bound to step definitions.


Note that checking for the presence of behavior tags on StepDefs will only happen in strict behavior mode and Gwen will not perform all the same rules checks on meta files as it does on feature files (since meta files are necessarily imperative automation glue).


Adding behavior tags to all StepDefs called by steps in our feature yields the final and complete meta:

# file: features/Todo.meta

Feature: Todo meta

  @StepDef
  @Context
  Scenario: a new todo list
      Given the heading can be located by tag name "h1"
       When I navigate to "http://todomvc.com/examples/angularjs"
       Then the heading should be "todos"
        And the todo field can be located by class name "new-todo"
        And the active item count can be located by css selector ".todo-count > strong"

  @StepDef
  @Action
  @DataTable(header="top")
  Scenario: the following items are added
        And I add a todo item for each data record

  @StepDef
  Scenario: I add a todo item
      Given the todo item is "${data[ITEM]}"
       When I enter the todo item in the todo field
       Then the added item can be located by xpath
             """
             //label[contains(.,'${the todo item}')]/preceding-sibling::input
             """
        And the added item should be unticked
        And the active item count should be "${record.number}"

  @StepDef
  @Action
  Scenario: all items are completed
       When I tick item for each item located by class name "toggle"

  @StepDef
  @Assertion
  Scenario: no active items should remain
       Then the active item count should be "0"

We’re all Good Now!

Good Gherkin rules will now be satisfied and running the feature again will succeed.

gwen-workspace/target/reports/html/features/Todo/Todo.feature.html

Motivation

Adding behavior checks to Gwen was inspired by various articles and discussions about good Gherkin and current challenges in this space. Some of these include: