Visual Testing Plugin

The plugin for visual testing.

Installation

  1. Copy the below line to dependencies section of the project build.gradle file

    Please make sure to use the same version for all VIVIDUS dependencies.
    Example 1. build.gradle
    implementation(group: 'org.vividus', name: 'vividus-plugin-visual', version: '0.6.7')

  2. If the project was imported to the IDE before adding new dependency, re-generate the configuration files for the used IDE and then refresh the project in the used IDE.

Properties

Property Name Acceptable values Default Description

ui.visual.ignored-elements

locators

<empty>

The comma-separated list of locators of elements to ignore element areas

ui.visual.ignored-areas

locators

<empty>

The comma-separated list of locators of elements to ignore page areas

ui.visual.indexer

scenario

<empty>

Used to append scenario based index to a baseline name

ui.visual.override-baselines

true false

false

Used for overriding existing ones or creating not existing baselines during compare action

ui.visual.acceptable-diff-percentage

percentage numbers

0

Add ability to configure sensitivity for visual checks

ui.visual.required-diff-percentage

percentage numbers

70

Defines required difference baseline vs checkpoint to consider them inequal

ui.screenshot.debug-directory

string

<empty>

The path to folder for saving debug screenshots. For example, ui.screenshot.debug-directory=C:/temp/debug

ui.visual.baseline-storage

Any available baseline storage name.

filesystem

Allows to override default baseline storage.

How to use

The plugin has the following actions: `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`

ESTABLISH

Creates a baseline image.

Steps to establish baseline:

  1. Run step When I ESTABLISH baseline with name `name`

  2. Open report

Allure report with visual check on establish

  1. Save image using contextual mouse click → Save as into folder for baselines

Baseline view
Saving baseline

COMPARE_AGAINST

Compares actual appearance of the page or element against the baseline. Make sure that the baseline you compare against actually exists.

Steps to compare baseline:

  1. Run step When I COMPARE_AGAINST baseline with name `name`

  2. Open report

Allure report with visual check on compare

  1. Review comparison results

Comparison result view

  1. If you need to accept the new baseline:

a) Switch to checkpoint view

Checkpoint view

b) Contextual click on the checkpoint image → save as to baseline folder

CHECK_INEQUALITY_AGAINST

Work the same way as COMPARE_AGAINST except it checks if actual appearance differs from the baseline.

Visual testing with custom shooting strategies

In order to support screenshot taking configuration per-project/per-step basis framework provides following approach.

Web Applications

Property based configuration could be specified using web.screenshot.strategy.<YOUR_STRATEGY_NAME>.<PROPERTY_NAME>= pattern, where: YOUR_STRATEGY_NAME - name of the custom strategy; PROPERTY_NAME - name of the property you want to specify.

Properties based configuration:

Property Name Acceptable values Description

web.screenshot.strategy.<YOUR_STRATEGY_NAME>.web-header-to-cut

size in pixels

Web header to cut (could be useful if your site has sticky header)

web.screenshot.strategy.<YOUR_STRATEGY_NAME>.web-footer-to-cut

size in pixels

Web footer to cut (could be useful if your site has sticky footer)

web.screenshot.strategy.<YOUR_STRATEGY_NAME>.native-header-to-cut

size in pixels

Native header to cut

web.screenshot.strategy.<YOUR_STRATEGY_NAME>.native-footer-to-cut

size in pixels

Native footer to cut

web.screenshot.strategy.<YOUR_STRATEGY_NAME>.coords-provider

WEB_DRIVER CEILING

Adds margins to coordinates, default one

web.screenshot.strategy.<YOUR_STRATEGY_NAME>.scroll-timeout

The duration in ISO-8601 Durations format.

Timeout for scrolling during performing visual check

web.screenshot.strategy.<YOUR_STRATEGY_NAME>.scrollable-element

locator

Locator of element for performing scroll action during visual check, steps fails if an element by the locator does not exist

web.screenshot.strategy.<YOUR_STRATEGY_NAME>.shooting-strategy

SIMPLE VIEWPORT_PASTING

Screenshot shooting strategy name (VIEWPORT_PASTING by default)

web.screenshot.strategy.<YOUR_STRATEGY_NAME>.cut-top

size in pixels

The size of top part of the final screenshot to cut.

web.screenshot.strategy.<YOUR_STRATEGY_NAME>.cut-bottom

size in pixels

The size of bottom part of the final screenshot to cut.

web.screenshot.strategy.<YOUR_STRATEGY_NAME>.cut-left

size in pixels

The size of left part of the final screenshot to cut.

web.screenshot.strategy.<YOUR_STRATEGY_NAME>.cut-right

size in pixels

The size of right part of the final screenshot to cut.

web.screenshot.strategy=YOUR_STRATEGY_NAME

string

After configuration necessary properties use this one to enable your custom strategy
Mind that cut-* modifications applied to the final screenshot, for example final stitched screenshot will be cut for the viewport-pasting strategy.

Step based configuration

To use custom configuration per step, two new steps were implemented.

When I $visualAction baseline with `$baselineName` using screenshot configuration:$screenshotConfiguration
When I $visualAction baseline with `$baselineName` ignoring:$ignoringElement using screenshot configuration:$screenshotConfiguration

Examples of usage property based web configuration:

web.screenshot.strategy.custom.web-header-to-cut=80
web.screenshot.strategy.custom.web-footer-to-cut=0
web.screenshot.strategy.custom.scrollable-element=By.cssSelector(.page__inner)
web.screenshot.strategy.custom.scroll-timeout=PT1S
web.screenshot.strategy.custom.shooting-strategy=SIMPLE
web.screenshot.strategy=custom

Examples of usage step based configuration:

When I <action> baseline with `scrollable-element-context` using screenshot configuration:
|scrollableElement                    |webHeaderToCut|webFooterToCut|scrollTimeout|shootingStrategy|
|By.xpath(//div[@class="page__inner"])|80            |0             |PT1S         |SIMPLE          |

Please see the image to get a clue about difference between native/web footer/header to cut.

Difference between native/web footer/header image

Mobile Native Applications

Properties

Property Name Acceptable values Default Description

mobile.screenshot.android.append-bottom-navigation-bar

boolean

false

Whether to append Android bottom navigation bar (black bar with system buttons on the screenshots below table) or not.

The property available only for VIEWPORT shooting strategy

Property based configuration could be specified using mobile.screenshot.strategy.<YOUR_STRATEGY_NAME>.<PROPERTY_NAME>= pattern, where: YOUR_STRATEGY_NAME - name of the custom strategy; PROPERTY_NAME - name of the property you want to specify.

Properties based configuration:

Property Name Acceptable values Description

mobile.screenshot.strategy.<YOUR_STRATEGY_NAME>.cut-top

size in pixels

The size of top part of the final screenshot to cut.

mobile.screenshot.strategy.<YOUR_STRATEGY_NAME>.cut-bottom

size in pixels

The size of bottom part of the final screenshot to cut.

mobile.screenshot.strategy.<YOUR_STRATEGY_NAME>.cut-left

size in pixels

The size of left part of the final screenshot to cut.

mobile.screenshot.strategy.<YOUR_STRATEGY_NAME>.cut-right

size in pixels

The size of right part of the final screenshot to cut.

mobile.screenshot.strategy.<YOUR_STRATEGY_NAME>.shooting-strategy

VIEWPORT, FULL_SCREEN

Screenshot shooting strategy name (VIEWPORT by default)

mobile.screenshot.strategy=YOUR_STRATEGY_NAME

string

After configuration necessary properties use this one to enable your custom strategy

Examples of usage property based configuration:

mobile.screenshot.strategy.custom.cut-top=100
mobile.screenshot.strategy.custom.cut-left=100
mobile.screenshot.strategy.custom.cut-right=100
mobile.screenshot.strategy.custom.cut-bottom=100
mobile.screenshot.strategy.custom.shooting-strategy=VIEWPORT
mobile.screenshot.strategy=custom

Examples of usage step based configuration:

When I <action> baseline with `scrollable-element-context` using screenshot configuration:
|cutTop           |shootingStrategy|
|51               |VIEWPORT        |

Steps

Run simple visual test

Establishes baseline or compares against existing one.

When I $actionType baseline with name `$name`

  • $actionType - The `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`.

  • $name - Then name of baseline.

Example 2. Perform simple visual check on compare
When I COMPARE_AGAINST baseline with name `test`

Run visual test using image

Establishes baseline or compares against existing one using image.

When I $actionType baseline with name `$name` from image `$image`

  • $actionType - The `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`.

  • $name - Then name of baseline.

  • $image - The data of image.

Example 3. Perform simple visual check on compare using image
When I execute HTTP GET request for resource with URL `https://raw.githubusercontent.com/vividus-framework/vividus/master/vividus-tests/src/main/resources/baselines/context.png`
When I compare_against baseline with name `context` from image `${response-as-bytes}`

Run simple visual test with custom baseline storage

Establishes baseline or compares against existing one using baseline storage.

When I $actionType baseline with name `$name` using storage `$storage`

  • $actionType - The `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`.

  • $name - The name of baseline.

  • $storage - The name of storage. Ony filesystem available by the default.

Example 4. Perform simple visual check using baseline storage
When I COMPARE_AGAINST baseline with name `test` using storage `azure`

Run simple visual test with custom baseline storage using image

Establishes baseline or compares against existing one using baseline storage.

When I $actionType baseline with name `$name` from image `$image` using storage `$storage`

  • $actionType - The `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`.

  • $name - The name of baseline.

  • $image - The data of image.

  • $storage - The name of storage. Ony filesystem available by the default.

Example 5. Perform simple visual check using baseline storage
When I execute HTTP GET request for resource with URL `https://raw.githubusercontent.com/vividus-framework/vividus/master/vividus-tests/src/main/resources/baselines/context.png`
When I compare_against baseline with name `context` from image `${response-as-bytes}` using storage `filesystem`

Run visual test with specified configuration

Establishes baseline or compares against existing one using the specified configuration.

When I $actionType baseline with name `$name` using screenshot configuration:$screenshotConfiguration

  • $actionType - The `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`.

  • $name - The name of baseline.

  • $screenshotConfiguration - The screenshot configuration:

    • shootingStrategy - The shooting strategy to use.

    • webHeaderToCut - The header to cut on each screenshot segment during scroll e.g. sticky header. (Web only)

    • webFooterToCut - The footer to cut on each screenshot segment during scroll e.g. sticky footer. (Web only)

    • nativeHeaderToCut - The native header to cut (e.g. system/browser bars). (Web on mobile only)

    • nativeFooterToCut - The native footer to cut (e.g. system/browser bars). (Web on mobile only)

    • cutTop - The header to cut on final screenshot.

    • cutLeft - The left bar to cut on final screenshot.

    • cutRight - The right bar to cut on final screenshot.

    • cutBottom - The footer to cut on final screenshot.

    • coordsProvider - The coords provider used to get ignoring elements position during scroll. (Web only)

      • CEILING - Based on JS API. (Web only)

      • WEB_DRIVER - Based on WebDriver API. (Web only)

    • scrollableElement - The element to scroll. (Web only)

    • scrollTimeout - The scroll timeout. (Web only)

Example 6. Perform visual check on establish with specified configuration
When I ESTABLISH baseline with name `test` using screenshot configuration:
|scrollableElement  |webFooterToCut|webHeaderToCut|coordsProvider|
|By.xpath(.//header)|100           |100           |CEILING       |

Run visual test with specified configuration and baseline storage

Establishes baseline or compares against existing one using the specified configuration and baseline storage.

When I $actionType baseline with name `$name` using storage `$storage` and screenshot configuration:$screenshotConfiguration

  • $actionType - The `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`.

  • $name - The name of baseline.

  • $storage - The name of storage. Ony filesystem available by the default.

  • $screenshotConfiguration - The screenshot configuration:

    • shootingStrategy - The shooting strategy to use.

    • webHeaderToCut - The header to cut on each screenshot segment during scroll e.g. sticky header. (Web only)

    • webFooterToCut - The footer to cut on each screenshot segment during scroll e.g. sticky footer. (Web only)

    • nativeHeaderToCut - The native header to cut (e.g. system/browser bars). (Web on mobile only)

    • nativeFooterToCut - The native footer to cut (e.g. system/browser bars). (Web on mobile only)

    • cutTop - The header to cut on final screenshot.

    • cutLeft - The left bar to cut on final screenshot.

    • cutRight - The right bar to cut on final screenshot.

    • cutBottom - The footer to cut on final screenshot.

    • coordsProvider - The coords provider used to get ignoring elements position during scroll. (Web only)

      • CEILING - Based on JS API. (Web only)

      • WEB_DRIVER - Based on WebDriver API. (Web only)

    • scrollableElement - The element to scroll. (Web only)

    • scrollTimeout - The scroll timeout. (Web only)

Example 7. Perform visual check on establish with specified configuration and baseline storage
When I ESTABLISH baseline with name `test` using storage `azure` and screenshot configuration:
|scrollableElement  |webFooterToCut|webHeaderToCut|coordsProvider|
|By.xpath(.//header)|100           |100           |CEILING       |

Run visual test with ignoring option

Establishes baseline or compares against existing one using the ignoring option.

When I $actionType baseline with name `$name` ignoring:$checkSettings

  • $actionType - `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`.

  • $name - Then name of baseline.

  • $checkSettings - The examples table of ELEMENT, AREA, ACCEPTABLE_DIFF_PERCENTAGE or REQUIRED_DIFF_PERCENTAGE.

Example 8. Perform visual check on compare with ignoring options
When I COMPARE_AGAINST baseline with `test` ignoring:
|ELEMENT            |AREA                  |ACCEPTABLE_DIFF_PERCENTAGE|
|By.xpath(.//header)|By.cssSelector(footer)|5                         |

Run visual test with ignoring option using image

Establishes baseline or compares against existing one using the ignoring option.

When I $actionType baseline with name `$name` from `$image` ignoring:$checkSettings

  • $actionType - `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`.

  • $name - Then name of baseline.

  • $image - The data of image.

  • $checkSettings - The examples table of ACCEPTABLE_DIFF_PERCENTAGE or REQUIRED_DIFF_PERCENTAGE.

ELEMENT and AREA ignores not supported.
Example 9. Perform visual check on compare with ignoring options
When I execute HTTP GET request for resource with URL `https://raw.githubusercontent.com/vividus-framework/vividus/master/vividus-tests/src/main/resources/baselines/context.png`
When I compare_against baseline with name `context-element-with-acceptable-diff-percentage` from image `${response-as-bytes}` ignoring:
|ACCEPTABLE_DIFF_PERCENTAGE|
|20                       |

Run visual test with ignoring option and baseline storage

Establishes baseline or compares against existing one using the ignoring option and baseline storage.

When I $actionType baseline with name `$name` using storage `$storage` and ignoring:$checkSettings

  • $actionType - The `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`.

  • $name - The name of baseline.

  • $storage - The name of storage. Ony filesystem available by the default.

  • $checkSettings - The examples table of ELEMENT, AREA, ACCEPTABLE_DIFF_PERCENTAGE or REQUIRED_DIFF_PERCENTAGE.

Example 10. Perform visual check on compare with ignoring options and baseline storage
When I COMPARE_AGAINST baseline with `test` using storage `azure` and ignoring:
|ELEMENT            |AREA                  |ACCEPTABLE_DIFF_PERCENTAGE|
|By.xpath(.//header)|By.cssSelector(footer)|5                         |

Run visual test using image and ignoring option and baseline storage

Establishes baseline or compares against existing one using the ignoring option and baseline storage.

When I $actionType baseline with name `$name` from image `$image` using storage `$storage` and ignoring:$checkSettings

  • $actionType - The `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`.

  • $name - The name of baseline.

  • $image - The data of image.

  • $storage - The name of storage. Ony filesystem available by the default.

  • $checkSettings - The examples table of ACCEPTABLE_DIFF_PERCENTAGE or REQUIRED_DIFF_PERCENTAGE.

ELEMENT and AREA ignores not supported.
Example 11. Perform visual check on compare with ignoring options and baseline storage
When I execute HTTP GET request for resource with URL `https://raw.githubusercontent.com/vividus-framework/vividus/master/vividus-tests/src/main/resources/baselines/context.png`
When I compare_against baseline with name `context-element-with-acceptable-diff-percentage` from image `${response-as-bytes}` using storage `filesystem` and ignoring:
|ACCEPTABLE_DIFF_PERCENTAGE|
|20                        |

Run visual test with ignoring option and specified configuration

Establishes baseline or compares against existing one using the ignoring option and the specified configuration.

When I $actionType baseline with name `$name` ignoring:$checkSettings using screenshot configuration:$screenshotConfiguration

  • $actionType - The `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`.

  • $name - The name of baseline.

  • $checkSettings - The examples table of ELEMENT, AREA, ACCEPTABLE_DIFF_PERCENTAGE or REQUIRED_DIFF_PERCENTAGE.

  • $screenshotConfiguration - The screenshot configuration:

    • shootingStrategy - The shooting strategy to use.

    • webHeaderToCut - The header to cut on each screenshot segment during scroll e.g. sticky header. (Web only)

    • webFooterToCut - The footer to cut on each screenshot segment during scroll e.g. sticky footer. (Web only)

    • nativeHeaderToCut - The native header to cut (e.g. system/browser bars). (Web on mobile only)

    • nativeFooterToCut - The native footer to cut (e.g. system/browser bars). (Web on mobile only)

    • cutTop - The header to cut on final screenshot.

    • cutLeft - The left bar to cut on final screenshot.

    • cutRight - The right bar to cut on final screenshot.

    • cutBottom - The footer to cut on final screenshot.

    • coordsProvider - The coords provider used to get ignoring elements position during scroll. (Web only)

      • CEILING - Based on JS API. (Web only)

      • WEB_DRIVER - Based on WebDriver API. (Web only)

    • scrollableElement - The element to scroll. (Web only)

    • scrollTimeout - The scroll timeout. (Web only)

Example 12. Perform visual check on establish with ignoring options and specified configuration
When I ESTABLISH baseline with name `test` ignoring:
|ELEMENT            |AREA                  |ACCEPTABLE_DIFF_PERCENTAGE|
|By.xpath(.//header)|By.cssSelector(footer)|5                         |
using screenshot configuration:
|scrollableElement  |webFooterToCut|webHeaderToCut|coordsProvider|
|By.xpath(.//header)|100           |100           |CEILING       |

Run visual test with ignoring option and specified configuration and baseline storage

Establishes baseline or compares against existing one using the ignoring option, the specified configuration, and baseline storage.

When I $actionType baseline with name `$name` using storage `$storage` and ignoring:$checkSettings and screenshot configuration:$screenshotConfiguration

  • $actionType - The `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`.

  • $name - The name of baseline.

  • $storage - The name of storage. Ony filesystem available by the default.

  • $checkSettings - The examples table of ELEMENT, AREA, ACCEPTABLE_DIFF_PERCENTAGE or REQUIRED_DIFF_PERCENTAGE.

  • $screenshotConfiguration - The screenshot configurations.

    • shootingStrategy - The shooting strategy to use.

    • webHeaderToCut - The header to cut on each screenshot segment during scroll e.g. sticky header. (Web only)

    • webFooterToCut - The footer to cut on each screenshot segment during scroll e.g. sticky footer. (Web only)

    • nativeHeaderToCut - The native header to cut (e.g. system/browser bars). (Web on mobile only)

    • nativeFooterToCut - The native footer to cut (e.g. system/browser bars). (Web on mobile only)

    • cutTop - The header to cut on final screenshot.

    • cutLeft - The left bar to cut on final screenshot.

    • cutRight - The right bar to cut on final screenshot.

    • cutBottom - The footer to cut on final screenshot.

    • coordsProvider - The coords provider used to get ignoring elements position during scroll. (Web only)

      • CEILING - Based on JS API. (Web only)

      • WEB_DRIVER - Based on WebDriver API. (Web only)

    • scrollableElement - The element to scroll. (Web only)

    • scrollTimeout - The scroll timeout. (Web only)

Example 13. Perform visual check on establish with ignoring options and specified configuration and baseline storage
When I ESTABLISH baseline with name `test` using storage `azure` and ignoring:
|ELEMENT            |AREA                  |ACCEPTABLE_DIFF_PERCENTAGE|
|By.xpath(.//header)|By.cssSelector(footer)|5                         |
and screenshot configuration:
|scrollableElement  |webFooterToCut|webHeaderToCut|coordsProvider|
|By.xpath(.//header)|100           |100           |CEILING       |

Baseline storages

Used to store baseline images. User is allowed to implement own baseline storages.

filesystem

The baseline storage is used by default. User is able to specify either embedded into repository baseline folder or any folder on the local filesystem.

Properties

Property Name Acceptable values Default Description

ui.visual.baseline-storage.filesystem.folder

string

./baseline

The path to the folder for saving baselines (root folder is src/main/resources). For example, ui.visual.baseline-storage.filesystem.folder=C:/Workspace/vividus-tests/src/main/resources/baselines
In order to store new baselines in the source folder use absolute path to the baselines folder.

azure-blob-storage

Uses Azure Blob Storage container as a baseline source. In order to use this storage you need to:

  1. Install Azure Storage Account plugin.

  2. Configure storage account connection Azure Storage Account plugin.

  3. Configure baseline storage using properties:

    Property Name Acceptable values Default Description

    ui.visual.baseline-storage.azure-blob-storage.account-key

    string

    Empty

    The logical key used to refer to the storage account.

    ui.visual.baseline-storage.azure-blob-storage.container

    string

    Empty

    Describes the name of container with the baselines.