Visual Testing Plugin
The plugin for visual testing.
Installation
-
Copy the below line to
dependencies
section of the projectbuild.gradle
filePlease make sure to use the same version for all VIVIDUS dependencies. Example 1. build.gradleimplementation(group: 'org.vividus', name: 'vividus-plugin-visual', version: '0.6.12')
-
If the project was imported to the IDE before adding new dependency, re-generate the configuration files for the used IDE and then refresh the project in the used IDE.
Properties
Property Name | Acceptable values | Default | Description |
---|---|---|---|
|
|
The comma-separated list of locators of elements to ignore element areas |
|
|
|
The comma-separated list of locators of elements to ignore page areas |
|
|
scenario |
|
Used to append scenario based index to a baseline name |
|
|
|
Used for overriding existing ones or creating not existing baselines during compare action |
|
percentage numbers |
|
Add ability to configure sensitivity for visual checks |
|
percentage numbers |
|
Defines required difference baseline vs checkpoint to consider them inequal |
|
|
|
The path to folder for saving debug screenshots. For example, |
|
Any available baseline storage name. |
|
Allows to override default baseline storage. |
How to use
The plugin has the following actions: `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`
ESTABLISH
Creates a baseline image.
Steps to establish baseline:
-
Run step
When I ESTABLISH baseline with name `name`
-
Open report
![Allure report with visual check on establish](_images/report_establish.png)
-
Save image using
contextual mouse click
→ Save as into folder for baselines
![Baseline view](_images/establish_baseline.png)
![Saving baseline](_images/save_baseline.png)
COMPARE_AGAINST
Compares actual appearance of the page or element against the baseline. Make sure that the baseline you compare against actually exists.
Steps to compare baseline:
-
Run step
When I COMPARE_AGAINST baseline with name `name`
-
Open report
![Allure report with visual check on compare](_images/report_compare.png)
-
Review comparison results
![Comparison result view](_images/comparison_result.png)
-
If you need to accept the new baseline:
a) Switch to checkpoint view
![Checkpoint view](_images/checkpoint.png)
b) Contextual click on the checkpoint image → save as to baseline folder
CHECK_INEQUALITY_AGAINST
Work the same way as COMPARE_AGAINST
except it checks if actual appearance differs from the baseline.
Visual testing with custom shooting strategies
In order to support screenshot taking configuration per-project/per-step basis framework provides following approach.
Web Applications
Properties-based screenshot configuration can be specified using web.screenshot.strategy.<YOUR_STRATEGY_NAME>.<PROPERTY_NAME>=<PROPERTY_VALUE>
pattern,
where:
-
YOUR_STRATEGY_NAME
- name of the custom strategy; -
PROPERTY_NAME
- name of the property you want to specify; -
PROPERTY_VALUE
- value of the property in the required format.
Properties-based configuration:
Keep in mind that properties-based configuration can be overridden by the Step based configuration.
Property Name | Acceptable values | Description |
---|---|---|
|
size in pixels |
Web header to cut (could be useful if your site has sticky header) |
|
size in pixels |
Web footer to cut (could be useful if your site has sticky footer) |
|
size in pixels |
Native header to cut |
|
size in pixels |
Native footer to cut |
|
|
Adds margins to coordinates, default one |
|
The duration in ISO-8601 Durations format. |
Timeout for scrolling during performing visual check |
|
locator |
Locator of element for performing scroll action during visual check, steps fails if an element by the locator does not exist |
|
|
Screenshot shooting strategy name ( |
|
size in pixels |
The size of top part of the final screenshot to cut. |
|
size in pixels |
The size of bottom part of the final screenshot to cut. |
|
size in pixels |
The size of left part of the final screenshot to cut. |
|
size in pixels |
The size of right part of the final screenshot to cut. |
|
string |
After configuration necessary properties use this one to enable your custom strategy |
Mind that cut-* modifications applied to the final screenshot, for example final stitched screenshot will be cut for the viewport-pasting strategy.
|
Step based configuration
To use custom configuration per step, two new steps were implemented.
When I $visualAction baseline with `$baselineName` using screenshot configuration:$screenshotConfiguration
When I $visualAction baseline with `$baselineName` ignoring:$ignoringElement using screenshot configuration:$screenshotConfiguration
Examples of usage property based web configuration:
web.screenshot.strategy.custom.web-header-to-cut=80
web.screenshot.strategy.custom.web-footer-to-cut=0
web.screenshot.strategy.custom.scrollable-element=By.cssSelector(.page__inner)
web.screenshot.strategy.custom.scroll-timeout=PT1S
web.screenshot.strategy.custom.shooting-strategy=SIMPLE
web.screenshot.strategy=custom
Examples of usage step based configuration:
When I <action> baseline with `scrollable-element-context` using screenshot configuration:
|scrollableElement |webHeaderToCut|webFooterToCut|scrollTimeout|shootingStrategy|
|By.xpath(//div[@class="page__inner"])|80 |0 |PT1S |SIMPLE |
Please see the image to get a clue about difference between native/web footer/header to cut.
![Difference between native/web footer/header image](_images/example.png)
Modifiers to cut web header or web footer have the default value 100 px (cutting headers and footers up to 100 pixels high).
|
Mobile Native Applications
Properties
Property Name | Acceptable values | Default | Description | ||
---|---|---|---|---|---|
|
|
|
Whether to append Android bottom navigation bar (black bar with system buttons on the screenshots below table) or not.
|
Property based configuration could be specified using mobile.screenshot.strategy.<YOUR_STRATEGY_NAME>.<PROPERTY_NAME>= pattern
,
where:
YOUR_STRATEGY_NAME
- name of the custom strategy;
PROPERTY_NAME
- name of the property you want to specify.
Properties based configuration:
Property Name | Acceptable values | Description |
---|---|---|
|
size in pixels |
The size of top part of the final screenshot to cut. |
|
size in pixels |
The size of bottom part of the final screenshot to cut. |
|
size in pixels |
The size of left part of the final screenshot to cut. |
|
size in pixels |
The size of right part of the final screenshot to cut. |
|
|
Screenshot shooting strategy name ( |
|
string |
After configuration necessary properties use this one to enable your custom strategy |
Examples of usage property based configuration:
mobile.screenshot.strategy.custom.cut-top=100
mobile.screenshot.strategy.custom.cut-left=100
mobile.screenshot.strategy.custom.cut-right=100
mobile.screenshot.strategy.custom.cut-bottom=100
mobile.screenshot.strategy.custom.shooting-strategy=VIEWPORT
mobile.screenshot.strategy=custom
Steps
Run simple visual test
Establishes baseline or compares against existing one.
When I $actionType baseline with name `$name`
-
$actionType
- The `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`. -
$name
- Then name of baseline.
When I COMPARE_AGAINST baseline with name `test`
Run visual test using image
Establishes baseline or compares against existing one using image.
When I $actionType baseline with name `$name` from image `$image`
-
$actionType
- The `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`. -
$name
- Then name of baseline. -
$image
- The data of image.
When I execute HTTP GET request for resource with URL `https://raw.githubusercontent.com/vividus-framework/vividus/master/vividus-tests/src/main/resources/baselines/context.png`
When I compare_against baseline with name `context` from image `${response-as-bytes}`
Run simple visual test with custom baseline storage
Establishes baseline or compares against existing one using baseline storage.
When I $actionType baseline with name `$name` using storage `$storage`
-
$actionType
- The `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`. -
$name
- The name of baseline. -
$storage
- The name of storage. Onyfilesystem
available by the default.
When I COMPARE_AGAINST baseline with name `test` using storage `azure`
Run simple visual test with custom baseline storage using image
Establishes baseline or compares against existing one using baseline storage.
When I $actionType baseline with name `$name` from image `$image` using storage `$storage`
-
$actionType
- The `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`. -
$name
- The name of baseline. -
$image
- The data of image. -
$storage
- The name of storage. Onyfilesystem
available by the default.
When I execute HTTP GET request for resource with URL `https://raw.githubusercontent.com/vividus-framework/vividus/master/vividus-tests/src/main/resources/baselines/context.png`
When I compare_against baseline with name `context` from image `${response-as-bytes}` using storage `filesystem`
Run visual test with specified configuration
Establishes baseline or compares against existing one using the specified configuration.
When I $actionType baseline with name `$name` using screenshot configuration:$screenshotConfiguration
-
$actionType
- The `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`. -
$name
- The name of baseline. -
$screenshotConfiguration
- The screenshot configuration:-
shootingStrategy
- The shooting strategy to use. -
webHeaderToCut
- The header to cut on each screenshot segment during scroll e.g. sticky header. (Web only) -
webFooterToCut
- The footer to cut on each screenshot segment during scroll e.g. sticky footer. (Web only) -
nativeHeaderToCut
- The native header to cut (e.g. system/browser bars). (Web on mobile only) -
nativeFooterToCut
- The native footer to cut (e.g. system/browser bars). (Web on mobile only) -
cutTop
- The header to cut on final screenshot. -
cutLeft
- The left bar to cut on final screenshot. -
cutRight
- The right bar to cut on final screenshot. -
cutBottom
- The footer to cut on final screenshot. -
coordsProvider
- The coords provider used to get ignoring elements position during scroll. (Web only)-
CEILING
- Based on JS API. (Web only) -
WEB_DRIVER
- Based on WebDriver API. (Web only)
-
-
scrollableElement
- The element to scroll. (Web only) -
scrollTimeout
- The scroll timeout. (Web only)
-
When I ESTABLISH baseline with name `test` using screenshot configuration:
|scrollableElement |webFooterToCut|webHeaderToCut|coordsProvider|
|By.xpath(.//header)|100 |100 |CEILING |
Run visual test with specified configuration and baseline storage
Establishes baseline or compares against existing one using the specified configuration and baseline storage.
When I $actionType baseline with name `$name` using storage `$storage` and screenshot configuration:$screenshotConfiguration
-
$actionType
- The `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`. -
$name
- The name of baseline. -
$storage
- The name of storage. Onyfilesystem
available by the default. -
$screenshotConfiguration
- The screenshot configuration:-
shootingStrategy
- The shooting strategy to use. -
webHeaderToCut
- The header to cut on each screenshot segment during scroll e.g. sticky header. (Web only) -
webFooterToCut
- The footer to cut on each screenshot segment during scroll e.g. sticky footer. (Web only) -
nativeHeaderToCut
- The native header to cut (e.g. system/browser bars). (Web on mobile only) -
nativeFooterToCut
- The native footer to cut (e.g. system/browser bars). (Web on mobile only) -
cutTop
- The header to cut on final screenshot. -
cutLeft
- The left bar to cut on final screenshot. -
cutRight
- The right bar to cut on final screenshot. -
cutBottom
- The footer to cut on final screenshot. -
coordsProvider
- The coords provider used to get ignoring elements position during scroll. (Web only)-
CEILING
- Based on JS API. (Web only) -
WEB_DRIVER
- Based on WebDriver API. (Web only)
-
-
scrollableElement
- The element to scroll. (Web only) -
scrollTimeout
- The scroll timeout. (Web only)
-
When I ESTABLISH baseline with name `test` using storage `azure` and screenshot configuration:
|scrollableElement |webFooterToCut|webHeaderToCut|coordsProvider|
|By.xpath(.//header)|100 |100 |CEILING |
Run visual test with ignoring option
Establishes baseline or compares against existing one using the ignoring option.
When I $actionType baseline with name `$name` ignoring:$checkSettings
-
$actionType
- `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`. -
$name
- Then name of baseline. -
$checkSettings
- The examples table ofELEMENT
,AREA
,ACCEPTABLE_DIFF_PERCENTAGE
orREQUIRED_DIFF_PERCENTAGE
.
When I COMPARE_AGAINST baseline with `test` ignoring:
|ELEMENT |AREA |ACCEPTABLE_DIFF_PERCENTAGE|
|By.xpath(.//header)|By.cssSelector(footer)|5 |
Run visual test with ignoring option using image
Establishes baseline or compares against existing one using the ignoring option.
When I $actionType baseline with name `$name` from `$image` ignoring:$checkSettings
-
$actionType
- `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`. -
$name
- Then name of baseline. -
$image
- The data of image. -
$checkSettings
- The examples table ofACCEPTABLE_DIFF_PERCENTAGE
orREQUIRED_DIFF_PERCENTAGE
.
ELEMENT and AREA ignores not supported. |
When I execute HTTP GET request for resource with URL `https://raw.githubusercontent.com/vividus-framework/vividus/master/vividus-tests/src/main/resources/baselines/context.png`
When I compare_against baseline with name `context-element-with-acceptable-diff-percentage` from image `${response-as-bytes}` ignoring:
|ACCEPTABLE_DIFF_PERCENTAGE|
|20 |
Run visual test with ignoring option and baseline storage
Establishes baseline or compares against existing one using the ignoring option and baseline storage.
When I $actionType baseline with name `$name` using storage `$storage` and ignoring:$checkSettings
-
$actionType
- The `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`. -
$name
- The name of baseline. -
$storage
- The name of storage. Onyfilesystem
available by the default. -
$checkSettings
- The examples table ofELEMENT
,AREA
,ACCEPTABLE_DIFF_PERCENTAGE
orREQUIRED_DIFF_PERCENTAGE
.
When I COMPARE_AGAINST baseline with `test` using storage `azure` and ignoring:
|ELEMENT |AREA |ACCEPTABLE_DIFF_PERCENTAGE|
|By.xpath(.//header)|By.cssSelector(footer)|5 |
Run visual test using image and ignoring option and baseline storage
Establishes baseline or compares against existing one using the ignoring option and baseline storage.
When I $actionType baseline with name `$name` from image `$image` using storage `$storage` and ignoring:$checkSettings
-
$actionType
- The `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`. -
$name
- The name of baseline. -
$image
- The data of image. -
$storage
- The name of storage. Onyfilesystem
available by the default. -
$checkSettings
- The examples table ofACCEPTABLE_DIFF_PERCENTAGE
orREQUIRED_DIFF_PERCENTAGE
.
ELEMENT and AREA ignores not supported. |
When I execute HTTP GET request for resource with URL `https://raw.githubusercontent.com/vividus-framework/vividus/master/vividus-tests/src/main/resources/baselines/context.png`
When I compare_against baseline with name `context-element-with-acceptable-diff-percentage` from image `${response-as-bytes}` using storage `filesystem` and ignoring:
|ACCEPTABLE_DIFF_PERCENTAGE|
|20 |
Run visual test with ignoring option and specified configuration
Establishes baseline or compares against existing one using the ignoring option and the specified configuration.
When I $actionType baseline with name `$name` ignoring:$checkSettings using screenshot configuration:$screenshotConfiguration
-
$actionType
- The `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`. -
$name
- The name of baseline. -
$checkSettings
- The examples table ofELEMENT
,AREA
,ACCEPTABLE_DIFF_PERCENTAGE
orREQUIRED_DIFF_PERCENTAGE
. -
$screenshotConfiguration
- The screenshot configuration:-
shootingStrategy
- The shooting strategy to use. -
webHeaderToCut
- The header to cut on each screenshot segment during scroll e.g. sticky header. (Web only) -
webFooterToCut
- The footer to cut on each screenshot segment during scroll e.g. sticky footer. (Web only) -
nativeHeaderToCut
- The native header to cut (e.g. system/browser bars). (Web on mobile only) -
nativeFooterToCut
- The native footer to cut (e.g. system/browser bars). (Web on mobile only) -
cutTop
- The header to cut on final screenshot. -
cutLeft
- The left bar to cut on final screenshot. -
cutRight
- The right bar to cut on final screenshot. -
cutBottom
- The footer to cut on final screenshot. -
coordsProvider
- The coords provider used to get ignoring elements position during scroll. (Web only)-
CEILING
- Based on JS API. (Web only) -
WEB_DRIVER
- Based on WebDriver API. (Web only)
-
-
scrollableElement
- The element to scroll. (Web only) -
scrollTimeout
- The scroll timeout. (Web only)
-
When I ESTABLISH baseline with name `test` ignoring:
|ELEMENT |AREA |ACCEPTABLE_DIFF_PERCENTAGE|
|By.xpath(.//header)|By.cssSelector(footer)|5 |
using screenshot configuration:
|scrollableElement |webFooterToCut|webHeaderToCut|coordsProvider|
|By.xpath(.//header)|100 |100 |CEILING |
Run visual test with ignoring option and specified configuration and baseline storage
Establishes baseline or compares against existing one using the ignoring option, the specified configuration, and baseline storage.
When I $actionType baseline with name `$name` using storage `$storage` and ignoring:$checkSettings and screenshot configuration:$screenshotConfiguration
-
$actionType
- The `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`. -
$name
- The name of baseline. -
$storage
- The name of storage. Onyfilesystem
available by the default. -
$checkSettings
- The examples table ofELEMENT
,AREA
,ACCEPTABLE_DIFF_PERCENTAGE
orREQUIRED_DIFF_PERCENTAGE
. -
$screenshotConfiguration
- The screenshot configurations.-
shootingStrategy
- The shooting strategy to use. -
webHeaderToCut
- The header to cut on each screenshot segment during scroll e.g. sticky header. (Web only) -
webFooterToCut
- The footer to cut on each screenshot segment during scroll e.g. sticky footer. (Web only) -
nativeHeaderToCut
- The native header to cut (e.g. system/browser bars). (Web on mobile only) -
nativeFooterToCut
- The native footer to cut (e.g. system/browser bars). (Web on mobile only) -
cutTop
- The header to cut on final screenshot. -
cutLeft
- The left bar to cut on final screenshot. -
cutRight
- The right bar to cut on final screenshot. -
cutBottom
- The footer to cut on final screenshot. -
coordsProvider
- The coords provider used to get ignoring elements position during scroll. (Web only)-
CEILING
- Based on JS API. (Web only) -
WEB_DRIVER
- Based on WebDriver API. (Web only)
-
-
scrollableElement
- The element to scroll. (Web only) -
scrollTimeout
- The scroll timeout. (Web only)
-
When I ESTABLISH baseline with name `test` using storage `azure` and ignoring:
|ELEMENT |AREA |ACCEPTABLE_DIFF_PERCENTAGE|
|By.xpath(.//header)|By.cssSelector(footer)|5 |
and screenshot configuration:
|scrollableElement |webFooterToCut|webHeaderToCut|coordsProvider|
|By.xpath(.//header)|100 |100 |CEILING |
Baseline storages
Used to store baseline images. User is allowed to implement own baseline storages.
filesystem
The baseline storage is used by default. User is able to specify either embedded into repository baseline folder or any folder on the local filesystem.
Properties
Property Name | Acceptable values | Default | Description |
---|---|---|---|
|
|
|
The path to the folder for saving baselines (root folder is |
In order to store new baselines in the source folder use absolute path to the baselines folder. |
azure-blob-storage
Uses Azure Blob Storage container as a baseline source. In order to use this storage you need to:
-
Install Azure Storage Account plugin.
-
Configure storage account connection Azure Storage Account plugin.
-
Configure baseline storage using properties:
Property Name Acceptable values Default Description ui.visual.baseline-storage.azure-blob-storage.account-key
string
Empty
The logical key used to refer to the storage account.
ui.visual.baseline-storage.azure-blob-storage.container
string
Empty
Describes the name of container with the baselines.