Gwen Workspaces

Setting up Gwen for teams just got a whole lot easier!

Manually setting up and installing Gwen on multiple machines or build servers in a team environment can be tedious and can also result in inconsistent configurations across workstations. One of the reasons why we created gwen-gpm was to provide consistent installation across machines and platforms. But a team needs more than that. A team needs a consistent and seamless way of getting Gwen configured and running on any user workstation or build server too.

Introducing Gwen workspaces

Gwen workspaces solve this problem by defining a standard project structure on the file system complete with settings files and wrapper scripts that can easily be committed to Git and checked out on any machine. Any team member can then just checkout the workspace and start using Gwen straight away. Gwen and the native web drivers will be automatically downloaded and installed on any user workstation or build server if they are not already present (through an embedded Gwen package manager in the workspace).

It is assumed that the target browser is already installed on the system. If not, you will need to manually install it. The wrapper script installs Gwen and native drivers but not the browser itself.

The structure of this workspace is defined as follows:

gwen-workspace
|--/env                   : Put environment properties here
|--/features              : Put your Gherkin feature files here
|--/meta                  : Put your Gwen meta files here
|  gwen                   : Gwen launcher/wrapper script for linux
|  gwen.bat               : Gwen launcher/wrapper script for windows
|  gwen.properties        : Common Gwen properties
|  log4j.properties       : Gwen log settings
|  gwen-gpm.properties    : Downloadable package checksums
|  gwen-gpm.jar           : Gwen package manager
|  .gitignore             : Git ignore file

Create and Commit a Workspace to Git

To create a Gwen workspace for your team, perform the following (only one person in the team needs to do this):

  • Download and extract the latest gwen-workspace.zip to a location on your file system. (Also have a read of the README.txt file that is extracted in the root).
  • Tweak or add team wide settings to the gwen.properties file in the workspace root. You can also tailor the wrapper scripts if necessary (for example, if you want to change the default Gwen launch options or add some new ones).
  • Verify that it all works by launching gwen (on windows) or ./gwen (on linux). Type exit when done to quit the REPL session.
  • Commit and push the workspace to your remote Git repository (see online Git help if you are not sure how to do this)

Checkout Workspace from Git and Go..

To use the workspace on any machine, perform the following (all team members need to do this):

  • Ensure that the target browser is installed on the system
  • Checkout the workspace from Git
  • Open a command prompt to the root workspace location
  • Type..
    • gwen (on windows) or ./gwen (on linux) followed by the options you require
    • Gwen will execute using the settings in the workspace
      Note: this same command can be used on any build server that checks out the workspace too
    • Gwen and native web drivers will self install on the first call
    • Type exit when done to quit the REPL session (if you started Gwen in REPL mode).

The team can now manage all their Gwen settings, environment configurations, Gherkin feature files, and Gwen meta files in a single workspace that can easily be pulled down and executed on any machine.

Help

For help, open a command prompt in the workspace root and type:

  • gwen help (on windows platforms)
  • ./gwen help (on linux platforms)

Each folder in the workspace also includes a README.txt file that can help to guide you.

Advertisements

Gwen and Selenoid reduced test execution from 50 minutes to 5

Thats 1000% (10 times) quicker than executing features in sequence!!

Gwen has always had the capability to make use of selenium grid which is particularly useful when running in parallel, and after discovering selenoid by the guys over at Aerokube I had to give it a shot.

Installing selenoid was a breeze with the configuration manager docker setup.  Once installed, selenoid downloads the last two driver versions of chrome, firefox and opera, and starts a grid ready to run tests.

Gwen comes with 235 features out of the box, ready for anyone to try.  These features cover a number of different scenarios including a flood.io challenge, navigating etsy, completing a todo, navigating blogs, and searching in google.    The features are also written using both imperative and declarative styles declarative-vs-imperative-gherkin.

Typically running 235 features takes 49 minutes and 33 seconds when running the features sequentially, however running these in parallel with selenoid took only 4 minutes and 41 seconds.  Both samples were run using a 34 core machine.

 

Gwen running 235 features sequentially
Gwen running 235 features in parallel

I have also included a screenshot of the running docker images, taken at a point in time where parallel was running full swing.

Thats all for now, hope you enjoyed a brief look into Selenoid and Gwen.

Evaluating Gwen with Modern JavaScript Web Apps

Gwen-web was designed to take the development pain out of web automation and make it easier to achieve with all types of web applications regardless of the underlying technologies or frameworks they were built with. Our goal was to make Gwen work consistently with web pages built on any kind of server or client side framework. All web pages are just HTML documents when they hit the browser after all. But this HTML is not always static and with the recent trend of modern JavaScript framework adoption, web pages are getting more and more dynamic. A lot more things happen in web pages nowadays. Lots of JavaScript code gets loaded and all types of browser events trigger functions and Ajax requests at various times resulting in all sorts of dynamic and asynchronous rendering that makes for a very rich user experience. This might make things more pleasant for human users but it is often a hard challenge for automation programs (or robots).

The Evaluation Test

So I thought I’d try Gwen out on some web pages built on the popular and modern JS frameworks of today to see how well it performs. To do this, I wrote a feature suite that mimics these Serenity tests and ran it over the various JS implementations of the publicly available TodoMVC web application (this is a project which offers the same Todo web application implemented on different JavaScript frameworks).

It is important to note that Gwen uses the Java implementation of the Selenium web driver and is not a JS framework itself but rather a Java executable that reads plain text Gherkin features and dynamically interprets them into automated web page interactions. The evaluation performed here provides us with an indication of how well Gwen interacts with web applications built on different types of JS frameworks.

The Evaluation Results

The suite consists of 26 scenarios which I ran over 33 different JS implementations of the Todo app (for a total of 858 scenarios). I ran it in a Chrome browser on a Mac. It took about 50 minutes to run every scenario in sequence. I also ran the tests in parallel (on a dual CPU quad core machine) which took about half the time and produced consistent results. Gwen was executed with the gwen.feature.failfast setting overriden to false to force all scenarios in a feature to execute even if one or more prior ones fail.

94% of Scenarios Passed ūüôā

All tests completed successfully and passed for the following frameworks.

  • JavaScript:
    • Backbone.js
    • AngularJS
    • Ember.js
    • KnockoutJS
    • Dojo
    • Knockback.js
    • CanJS
    • Polymer
    • React
    • Mithril
    • Ampersand
    • Vue.js
    • Marionette.js
    • Vanilla JS
    • Vanilla ES6
    • jQuery
  • Compile-to-JS:
    • Spine
    • Dart
    • GWT
    • TypeScript + Backbone.js
    • TypeScript + AngularJS
    • TypeScript + React
    • Serenade.js
    • Reagent
    • Scala.js + React
    • Scala.js + Binding.scala
    • js_of_ocaml
    • Humble + GopherJS

6% of Scenarios Failed

Some tests failed for the following frameworks. I have not investigated these in detail but have included my initial findings below.

  • JavaScript:
    • Flight
      • 15 scenarios failed, 11 passed
      • Intermittent element hits and misses
      • Entered data leaks or is lost
    • TroopJS + RequireJS
      • 15 scenarios failed, 11 passed
      • Unresponsive to enter key being sent to field
  • Compile-to-JS:
    • Closure
      • 5 scenarios failed, 21 passed
      • Unresponsive to checkbox being clicked
    • Elm
      • 8 scenarios failed, 18 passed
      • Intermittently not sending some characters to field
    • AngularDart
      • 3 scenarios failed, 23 passed
      • Intermittently not rendering conditionally visible buttons

Conclusion


Gwen interacted successfully with a large majority of the popular JS implementations. All tests passed for 28 of the 33 implementations evaluated. More than 94% of all scenarios passed in total.

Try it Yourself

All the todoMVC features used in this evaluation are also bundled in the gwen-web distribution as of release 2.3.3. If you install the latest gwen-web distribution, you can run it by calling the following command in the directory where you install Gwen (to run in parallel mode, just add the --parallel switch):

  • Windows:
    • gwen features/todoMVC
  • Linux:
    • ./gwen features/todoMVC

Gwen 2 released

Gwen 2 has been released with the latest technologies including Java 8 and Selenium WebDriver 3. All binary dependencies have also been updated and the project is now built using Scala 2.12.

There is one impact to Firefox users but nothing else has changed. With the upgrade to Selenium 3, all browser vendors (including Mozilla) are responsible for developing and providing the native drivers for their own browsers. To use Firefox, you will now need to download a native Firefox driver and set the following in your gwen.properties file:

webdriver.gecko.driver=/your-downloaded-driver-location/geckodriver

The settings page on our Gwen wiki has also been updated.

Gwen 2.x will only run on Java 8 environments. Gwen 1.x will still run on Java 7 and above as before, but will no longer have new features added to it.

The latest Gwen release is available here. See also install instructions.

Configurable User Properties in gwen-web

One of the really cool features of gwen-web is the ability to load in different parameters at run time.  This allows you to run gwen-web against multiple environments without changing either the feature or the meta files.

Lets take one of the examples with gwen-web and flood.io ¬† Below are two examples of how one could set the value. ¬†The first example sets “the how old are you dropdown” to 21, the second example uses a property, in this case different files for two different environments to set the value.

Lets define how the dropdown can be found in the meta first.

FloodIO.meta

And the heading should be “Step 2”
And the how old are you dropdown can be located by id “challenger_age”

FloodIO.feature

When I select “21” in the how old are you dropdown

Now with the properties approach. Lets create two files, DEV.properties and SIT.properties. The contents of the two files are below:

DEV.properties
user.age=44

SIT.properties
user.age=21

In addition we need to make a small modification to both the feature and the meta files

FloodIO.meta

And the heading should be “Step 2”
And my age is defined by property “user.age”
And the how old are you dropdown can be located by id “challenger_age”

Now replace the strikethrough line with the new line in the feature

FloodIO.feature

When I select “21” in the how old are you dropdown
When I select my age in the how old are you dropdown

Editting is complete. To run gwen-web with the environment specific age, simply type “gwen-web features/floodio/ -p dev.properties -b -r devReport”, or alternatively you can run “gwen-web features/floodio/ -p sit.properties -b -r sitReport”.

Remote WebDriver feature now available in gwen-web

For my first post, I will be talking about a new feature that has been added to gwen-web.  For those of you not familiar with gwen-web, its a web automation engine that runs inside our gwen interpreter and is used to automate the browser.  gwen-web is unique in that it allows automation of the web browser, using english(dsl) rather than coding or using visual record/playback. We believe this will remove some of the entry level requirements often associated with a coded tool.  For more information please head on over to our gwen github page https://github.com/gwen-interpreter/gwen.

Remote WebDriver

Just recently we added the remote webdriver capability.  This means gwen-web can now connect to selenium grid and drive not just the local browser, but also a browser that is installed on a different os, or a browser that has a different version.

Setting up Remote WebDriver with gwen-web

So how do I setup gwen-web with selenium remote webdriver?  What I am going to walk you through is setting up a selenium grid, and attaching nodes to it.  Firstly download the selenium-server-standalone-2.45.0.jar (or most recent) from http://selenium-release.storage.googleapis.com/2.45/selenium-server-standalone-2.45.0.jar.   Once this has been downloaded, you can start up selenium grid by running the following:

java -jar selenium-server-standalone-2.45.0.jar -role hub

Now lets make sure that its running by navigating to http://localhost:4444/grid/console.  You will notice that there are currently no browsers available.

PastedGraphic-1

Running selenium grid in this mode, means we need to register nodes against the grid.

Adding browsers to the current running grid

Lets start by downloading the latest version of chromedriver (specific for the OS) Рfrom http://chromedriver.storage.googleapis.com/index.html?path=2.15/  (at the time of writing the blog chromedriver 2.15 is the latest)

Unzip chromedriver and note the path.

Now lets start up selenium-server-standalone as a node role, and register both the chrome driver and firefox driver.  Remembering firefox driver comes with the selenium-server-standalone.

java -jar selenium-server-standalone-2.45.0.jar 
-Dwebdriver.chrome.driver=chromedriver_215/chromedriver 
-role node 
-hub http://localhost:4444/grid/register 
-browser browserName=firefox,version=37.0,maxInstances=5,platform=LINUX 
-browser browserName=chrome,version=42.0,maxInstances=5,platform=LINUX

Let me explain the parts above:

  1. java -jar selenium-server-standalone-2.45.0.jar  (standard selenium startup)
  2. -Dwebdriver.chrome.driver= (path to chromedriver including the executable)
  3. -role node (run the selenium-server-standalone as a node)
  4. -hub http://localhost:4444/grid/register ¬†(this is the location of the grid that you started back under the previous heading “Setting up ¬†Remote WebDriver with gwen-web”
  5. -browser browserName=firefox,version=37.0,maxInstance=5,platform=LINUX  (the browser, version label and the number of maximum instances to run on this node)  Note:  firefox requires the binary to be in the PATH, alternatively it would need to be specified using -Dwebdriver.firefox.bin
  6. -browser browserName=chrome,version=42.0,maxInstances=5,platform=LINUX.  (register that this node is capable of running chrome)

Testing the grid console

So how do we now confirm the currently registered browsers on the grid.   Navigate to http://localhost:4444/grid/console and now the console should show both firefox and chrome registered and waiting for a connection.

PastedGraphic-2

When I setup a grid, I typically like to confirm whether or not the browsers do indeed start up.  To do this I navigate to the node webdriver/hub   E.g.  http://10.1.1.9:5555/wd/hub   (you can see the IP address of each node that is registered against this grid).

PastedGraphic-3

Then I create two sessions.  One for chrome and one for firefox.  If its working, both chrome and firefox will start.  Just delete the sessions once that has been confirmed.  If its not working, then you will need to go back and troubleshoot.  Typically its driver / binary path related.  If either browser pops up and then closes unexpectedly, make sure to keep both the selenium-server-standalone and the browser versions in sync.  The latest version of selenium-standalone may not work with a version of the browser that is 6 versions out of date.

Setting up gwen-web to work with selenium-grid

Once you have a working grid, gwen web only requires a small change to the gwen.properties file. (eg. edit ~/gwen.properties and add the following line.)

gwen.web.remote.url=http://localhost:4444/wd/hub

This tells the gwen-web engine that it needs to communicate with selenium grid which will effectively use the RemoteWebDriver with different browser capabilities.

Running gwen-web with selenium-grid

To run gwen-web with two different browsers it is suggested that you remove gwen.web.browser property from the gwen.properties and pass it in as a “-D” system property. ¬†The below example uses the floodio challenge sample with gwen-web.

eg.

bin/gwen-web -Dgwen.web.browser=chrome -b features/floodio -r reports/chrome
bin/gwen-web -Dgwen.web.browser=firefox -b features/floodio -r reports/firefox

Once gwen-web is running, navigate to the selenium node and you will see something like this

Screen Shot 2015-04-30 at 9.42.28 pm

and navigate to the grid localhost:4444/grid/console and you will now see both firefox and chrome running.

Screen Shot 2015-04-30 at 9.42.47 pm