Visual Testing Plugin

The plugin for visual testing.

Installation

Example 1. build.gradle
implementation(group: 'org.vividus', name: 'vividus-plugin-visual', version: '0.4.8')

Properties

Property Name Acceptable values Default Description

ui.visual.ignored-elements

locators

<empty>

The comma-separated list of locators of elements to ignore element areas

ui.visual.ignored-areas

locators

<empty>

The comma-separated list of locators of elements to ignore page areas

ui.visual.baselines-folder

string

./baseline

The path to folder for saving baselines (root folder is src/main/resources). For example, ui.visual.baselines-folder=C:/Workspace/vividus-tests/src/main/resources/baselines

ui.visual.indexer

scenario

<empty>

Used to append scenario based index to a baseline name

ui.visual.override-baselines

true false

false

Used for overriding existing ones or creating not existing baselines during compare action

ui.visual.acceptable-diff-percentage

percentage numbers

0

Add ability to configure sensitivity for visual checks

ui.visual.required-diff-percentage

percentage numbers

70

Defines required difference baseline vs checkpoint to consider them inequal

ui.screenshot.debug-directory

string

<empty>

The path to folder for saving debug screenshots. For example, ui.screenshot.debug-directory=C:/temp/debug

How to use

The plugin has the following actions: `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`

ESTABLISH

Creates a baseline image.

Steps to establish baseline:

  1. Run step When I ESTABLISH baseline with `name`

  2. Open report

Allure report with visual check on establish

  1. Save image using contextual mouse click → Save as into folder for baselines

Baseline view
Saving baseline

COMPARE_AGAINST

Compares actual appearance of the page or element against the baseline. Make sure that the baseline you compare against actually exists.

Steps to compare baseline:

  1. Run step When I COMPARE_AGAINST baseline with `name`

  2. Open report

Allure report with visual check on compare

  1. Review comparison results

Comparison result view

  1. If you need to accept the new baseline:

a) Switch to checkpoint view

Checkpoint view

b) Contextual click on the checkpoint image → save as to baseline folder

CHECK_INEQUALITY_AGAINST

Work the same way as COMPARE_AGAINST except it checks if actual appearance differs from the baseline.

Visual testing with custom shooting strategies

In order to support screenshot taking configuration per-project/per-step basis framework provides following approach.

Web Applications

Property based configuration could be specified using web.screenshot.strategy.<YOUR_STRATEGY_NAME>.<PROPERTY_NAME>= pattern, where: YOUR_STRATEGY_NAME - name of the custom strategy; PROPERTY_NAME - name of the property you want specify.

Properties based configuration:

Property Name Acceptable values Description

web.screenshot.strategy.<YOUR_STRATEGY_NAME>.web-header-to-cut

size in pixels

Web header to cut (could be useful if your site has sticky header)

web.screenshot.strategy.<YOUR_STRATEGY_NAME>.web-footer-to-cut

size in pixels

Web footer to cut (could be useful if your site has sticky footer)

web.screenshot.strategy.<YOUR_STRATEGY_NAME>.native-header-to-cut

size in pixels

Native header to cut

web.screenshot.strategy.<YOUR_STRATEGY_NAME>.native-footer-to-cut

size in pixels

Native footer to cut

web.screenshot.strategy.<YOUR_STRATEGY_NAME>.coords-provider

WEB_DRIVER CEILING

Adds margins to coordinates, default one

web.screenshot.strategy.<YOUR_STRATEGY_NAME>.scroll-timeout

time in millis

Timeout for scrolling during performing visual check

web.screenshot.strategy.<YOUR_STRATEGY_NAME>.scrollable-element

locator

Locator of element for performing scroll action during visual check

web.screenshot.strategy.<YOUR_STRATEGY_NAME>.shooting-strategy

SIMPLE VIEWPORT_PASTING

Screenshot shooting strategy name (VIEWPORT_PASTING by default)

web.screenshot.strategy=YOUR_STRATEGY_NAME

string

After configuration necessary properties use this one to enable your custom strategy

Step based configuration

To use custom configuration per step, two new steps were implemented.

When I $visualAction baseline with `$baselineName` using screenshot configuration:$screenshotConfiguration
When I $visualAction baseline with `$baselineName` ignoring:$ignoringElement using screenshot configuration:$screenshotConfiguration

Examples of usage property based configuration:

web.screenshot.strategy.bombaysapphire.web-header-to-cut=80
web.screenshot.strategy.bombaysapphire.web-footer-to-cut=0
web.screenshot.strategy.bombaysapphire.scrollable-element=By.cssSelector(.page__inner)
web.screenshot.strategy.bombaysapphire.scroll-timeout=PT1S
web.screenshot.strategy.bombaysapphire.shooting-strategy=SIMPLE
web.screenshot.strategy=bombaysapphire

Examples of usage step based configuration:

When I <action> baseline with `scrollable-element-context` using screenshot configuration:
|scrollableElement                    |webHeaderToCut|webFooterToCut|scrollTimeout|shootingStrategy|
|By.xpath(//div[@class="page__inner"])|80            |0             |PT1S         |SIMPLE          |

Please see the image to get a clue about difference between native/web footer/header to cut.

Difference between native/web footer/header image

Mobile Native Applications

Properties

Property Name Acceptable values Default Description

mobile.screenshot.downscale

boolean

true

Downscale checkpoint image according to a device DPR value

Please see the image to get a clue about difference between original and downscaled images.

Difference between original and dpr downscaled image

Property based configuration could be specified using mobile.screenshot.strategy.<YOUR_STRATEGY_NAME>.<PROPERTY_NAME>= pattern, where: YOUR_STRATEGY_NAME - name of the custom strategy; PROPERTY_NAME - name of the property you want specify.

Properties based configuration:

Property Name Acceptable values Description

mobile.screenshot.strategy.<YOUR_STRATEGY_NAME>.native-footer-to-cut

size in pixels

Native footer to cut

mobile.screenshot.strategy.<YOUR_STRATEGY_NAME>.shooting-strategy

SIMPLE

Screenshot shooting strategy name (SIMPLE by default)

mobile.screenshot.strategy=YOUR_STRATEGY_NAME

string

After configuration necessary properties use this one to enable your custom strategy

Examples of usage property based configuration:

mobile.screenshot.strategy.bombaysapphire.native-footer-to-cut=100
mobile.screenshot.strategy.bombaysapphire.shooting-strategy=SIMPLE
mobile.screenshot.strategy=bombaysapphire

Examples of usage step based configuration:

When I <action> baseline with `scrollable-element-context` using screenshot configuration:
|nativeFooterToCut|shootingStrategy|
|51               |SIMPLE          |

Steps

Run simple visual test

Establishes baseline or compares against existing one.

When I $actionType baseline with `$name`

  • actionType - `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`

  • name - name of baseline

Example 2. Perform simple visual check on compare
When I COMPARE_AGAINST baseline with `test`

Run visual test with specified configuration

Establishes baseline or compares against existing one using the specified configuration.

When I $actionType baseline with `$name` using screenshot configuration:$screenshotConfiguration

  • actionType - `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`

  • name - name of baseline

  • screenshotConfiguration - configuration to make screenshot

Example 3. Perform visual check on establish with specified configuration
When I ESTABLISH baseline with `test` using screenshot configuration:
|scrollableElement  |webFooterToCut|webHeaderToCut|coordsProvider|
|By.xpath(.//header)|100           |100           |CEILING       |

Run visual test with ignoring option

Establishes baseline or compares against existing one using the ignoring option.

When I $actionType baseline with `$name` ignoring:$ignoredElements

  • actionType - `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`

  • name - name of baseline

  • checkSettings - examples table of ELEMENT, AREA, ACCEPTABLE_DIFF_PERCENTAGE or REQUIRED_DIFF_PERCENTAGE

Example 4. Perform visual check on compare with ignoring options
When I COMPARE_AGAINST baseline with `test` ignoring:
|ELEMENT            |AREA                  |ACCEPTABLE_DIFF_PERCENTAGE|
|By.xpath(.//header)|By.cssSelector(footer)|5                         |

Run visual test with ignoring option and specified configuration

Establishes baseline or compares against existing one using the ignoring option and the specified configuration.

When I $actionType baseline with `$name` ignoring:$ignoredElements using screenshot configuration:$screenshotConfiguration

  • actionType - `ESTABLISH`, `COMPARE_AGAINST` and `CHECK_INEQUALITY_AGAINST`

  • name - name of baseline

  • checkSettings - examples table of ELEMENT, AREA, ACCEPTABLE_DIFF_PERCENTAGE or REQUIRED_DIFF_PERCENTAGE

  • screenshotConfiguration - configuration to make screenshot

Example 5. Perform visual check on establish with ignoring options and specified configuration
When I ESTABLISH baseline with `test` ignoring:
|ELEMENT            |AREA                  |ACCEPTABLE_DIFF_PERCENTAGE|
|By.xpath(.//header)|By.cssSelector(footer)|5                         |
using screenshot configuration:
|scrollableElement  |webFooterToCut|webHeaderToCut|coordsProvider|
|By.xpath(.//header)|100           |100           |CEILING       |