Tests Configuration

configuration.properties

The Vividus configuration includes the following parts: profiles, environments and suites.

Suites

The property configuration.suites defines the suites set. It is a comma separated set of suite file addresses:

Example 1. configuration.properties
configuration.suites=testSuite,anotherDir/anotherTestSuite

The Vividus suite consists of the unlimited number of batches. In its turn the batch consists of any number of stories represented by the files with *.story extension. The batches are run sequentially, while the stories in the batch are run in parallel.

The suites are loaded one by one starting from the last one in the sequence, in other words, the values of the properties defined in the first suite will take precedence over the values of the same properties defined in all subsequent suites.

The following properties are used to configure batches (Vividus doesn’t provide any default batch configuration out of the box). batch-number is one-based batch index.

The properties marked with bold are mandatory.
Property Default value Description

bdd.story-loader.batch-<batch-number>.resource-location

The test project path to the folder with story-files

bdd.story-loader.batch-<batch-number>.resource-include-patterns

<empty>

The comma-separated set of ant-patterns to match the stories to be included in the batch

bdd.story-loader.batch-<batch-number>.resource-exclude-patterns

<empty>

The comma-separated set of ant-patterns to match the stories to be excluded from the batch

bdd.batch-<batch-number>.name

batch-<batch-number>

The name of the batch to display in the report

bdd.batch-<batch-number>.threads

1

The number of parallel threads used to run stories

bdd.batch-<batch-number>.meta-filters

suite meta-filter

The meta-filter used to filter the batch stories and scenarios

bdd.batch-<batch-number>.fail-fast

false

If true and any failure is occurred during the batch execution, the subsequent batches will not be executed.

bdd.batch-<batch-number>.story-execution-timeout

PT3H

The max duration of the single story in the batch.

bdd.batch.fail-fast

false

If set to true the subsequent batches execution will be stopped after first failed assertion

bdd.story.fail-fast

false

If set to true the story execution will be stopped after first failed assertion

bdd.scenario.fail-fast

false

If set to true the scenario execution will be stopped after first failed assertion

statistics.print-failures

false

If set to true prints table of failures including: story, scenario, step and error message

Known Issues

Known issues mechanism allows to distinguish failures from known issues presented in the system under test.

JSON file should contain a valid JSON. If JSON is not valid known issues will be ignored and warning about invalid file will be printed.

Properties

Property Default value Description

known-issue-provider.fileName

known-issues.json

Defines the location of the known issues file

known-issue-provider.detect-potentially-known-issues

true

Defines are potentially known issues detected

Known Issues file

The file represents a collection of JSON elements describing known issues.

Example 2. known-issues.json
 {
    "VVD-1": { (1)
        "type": "Internal",(2)
        "assertionPattern": "Expected: a value equal to 200, Actual: \\[404\\]" (3)
        "storyPattern": "Validate the Math", (4)
        "scenarioPattern": "Validate equality", (5)
        "stepPattern": "Then `1` is = `2`.+", (6)
        "variablePatterns": { (7)
            "var":  "value-.*"
        },
        "failScenarioFast": true, (8)
        "failStoryFast": true (9)
    }
}
1 [Mandatory] Identifier (This could be an ID of the issue in your bug tracking system)
2 [Mandatory] Issue type. One of: INTERNAL (application issue), EXTERNAL (3rd party issue), AUTOMATION (test automation issue)
3 [Mandatory] Assertion pattern describes which failed assertions should be considered as known
4 Story pattern to match failed story name
5 Scenario pattern to match failed scenario name
6 Step pattern to match failed step.
7 Variable patterns. Patterns to match variables.
8 Skip the rest of scenario in case of failure identified as this known issue
9 Skip the rest of story in case of failure identified as this known issue
Use \\ for escaping of special characters in regular expressions

Examples

  • To consider failed assertion known or potentially-known assertionPattern should match assertion failure, if defined variablePatterns should match also.

  • If assertionPattern/variablePatterns matched and you have story/scenario/step/Pattern defined, to consider issue known all of them should match.

  • If assertionPattern/variablePatterns matched and you have story/scenario/step/Pattern defined, when at least one not matched issue is considered potentially known.

Meta tags

Meta tags can be used to reach several goals:

  • Grouping of stories and scenarios in the report.

  • Dynamic selection of tests to execute by meta tags.

  • Management of capabilities for the particular tests.

Levels

Meta tags may be used at both story and scenario level. Meta tags specified at the story level are propagated to every scenario, if the same meta tag is used at the scenario level, then the scenario level value takes precedence.

Example 3. Levels of meta tags
!-- Story level
Meta:
     @group Login

Scenario: Open Epam main page
!-- Scenario level
Meta:
  @severity 1
  @layout desktop
Given I am on a page with the URL 'https://www.epam.com/'
Then the page title is equal to 'EPAM | Enterprise Software Development, Design & Consulting'

Scenario: Open Google main page
Meta:
  @severity 2
  @layout desktop
  @skip
Given I am on a page with the URL 'https://www.google.com/'
Then the page title is equal to 'Google'

Properties

The prorerty bdd.all-meta-filters is used to override values of the default meta filters based on the tags: @skip, @layout, @browserWindowSize and etc.

The property bdd.meta-filters relies on groovy synax can be used to filters tests to be executed.

Example 4. Execute stoies and scenarios marked with '@group Login' tag and without '@skip' tag
bdd.meta-filters=groovy: (group == 'Login' && !skip)

The following meta tags are available out of the box.

1D tags (@key)

Tag Description

@skip

Used to exclude particular scenarios from the test execution

2D tags (@key  value)

Tag Key Allowed values Description Example

@severity

Numeric values (range 1-5)

Used to mark importance of the test where, most usually, the smallest number is the most important Test, the highest number is the least important.

@severity 1

@layout

desktop, phone, phone_portrait, phone_landscape, chrome_phone, chrome_responsive, chrome_tablet, tablet, tablet_landscape, tablet_portrait

Used to specify layout, using which marked story or scenario should be executed. In case if this meta tag is not specified, default (desktop) value is used for web applications.

@layout desktop

@browserWindowSize

Any browser window size in pixels, e.g. 800x600. Note: browser window size should be smaller than the current screen resolution.

Used to specify browser window size. It can be applied on both story and scenario levels. In case, if browser size differs between adjacent scenarios, browser window will be simply resized to desired values, browser session will not be restarted. If the tag is not set for story or scenario, the browser window will be maximized to the screen size during local test execution.

@browserWindowSize 1024x768

Reporting

User has possibility to link stories/scenarios to an external systems like TMS or bug-tracking systems.

There is a few steps to achieve this:

  1. Configure link pattern via properties

  2. Add metatag to the story/sceanrio meta

Example 5. environment.properties
system.allure.link.issue.pattern=https://vividus.dev/issues/{}
Example 6. Test.story
Meta:
    @issueId VVD-1

Scenario: Should check variables equals
Then `1` is = `1`
Link in allure report

Configuration

Property Meta tag Description Example

system.allure.link.issue.pattern

@issueId

Defines issue link pattern

https://github.com/vividus-framework/vividus/issues/{}

system.allure.link.requirement.pattern

@requirementId

Defines requirement link pattern

https://github.com/vividus-framework/vividus/issues/{}

system.allure.link.tms.pattern

@testCaseId

Defines test case link pattern

https://vividus.jira.com/issues/{}

Mutiple endpoints

It is possible to have multiple endpoints. To achieve this user should define system suffix for the properties and meta tags.

Example 7. environment.properties
# Default property
system.allure.link.issue.pattern=https://vividus.prod/issues/{}
# Additional property with .dev suffix, that used to create links annotated with @isssueId.dev
system.allure.link.issue.dev.pattern=https://vividus.dev/issues/{}
Example 8. Test.story
Meta:
    @issueId VVD-1
    @issueId.dev DEV-1

Scenario: Should check variables equal
Then `1` is = `1`
Multiple links

Story Prioritization

In order to start stories in particular order it is possible to assign the stories a numeric priority, the stories with the higher priority start first.

To enable the feature create a file spring.xml in the src/main/resources directory and add the following XML data into that file.

Example 9. Configuration
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans https://www.springframework.org/schema/beans/spring-beans.xsd"
       default-lazy-init="false">

    <bean class="org.vividus.bdd.priority.MetaBasedStoryExecutionPriority" factory-method="byNumericMetaValue">
        <constructor-arg index="0" value="story_priority" />
    </bean>
</beans>

The story_priority value that corresponds to a meta name in stories can be changed to arbitrary value without spaces.

Taking story_priority as an example the priority meta should be placed at the story level as the following example shows:

Example 10. Prioritized.story
Meta: @story_priority 15

Scenario: Time consuming test
When I perform highly time consuming task