Tests Configuration

configuration.properties

VIVIDUS configuration includes the following parts: suites, profiles and environments.

Properties prefixed with configuration. should be specified in the following locations only:

  • The system properties.

  • The configuration.properties file.

  • The overriding.properties file.

Suites

The property configuration.suites defines the suites set. It is a comma separated set of suite file addresses:

Example 1. configuration.properties
configuration.suites=testSuite,anotherDir/anotherTestSuite

VIVIDUS suite consists of the unlimited number of batches. In its turn the batch consists of any number of stories represented by the files with *.story extension. The batches are run sequentially, while the stories in the batch are run in parallel.

The suites are loaded one by one starting from the last one in the sequence, in other words, the values of the properties defined in the first suite will take precedence over the values of the same properties defined in all subsequent suites. If any of the suites in the sequence is not found (i.e. no *.properties file is present at the given path), the error will be thrown and the execution will be stopped. However if suites property is empty (configuration.suites=) and no suite is configured in the root (i.e no files can be found by /properties/suite/*.properties pattern), no error will be thrown and the execution will be continued.

The following properties are used to configure batches (VIVIDUS doesn’t provide any default batch configuration out of the box). batch-number is one-based batch index.

The properties marked with bold are mandatory.
Property Default value Description

batch-<batch-number>.resource-location

The test project path to the folder with story-files

batch-<batch-number>.resource-include-patterns

<empty>

The comma-separated set of ant-patterns to match the stories to be included in the batch. Leading and trailing whitespaces in patterns are omitted.

batch-<batch-number>.resource-exclude-patterns

<empty>

The comma-separated set of ant-patterns to match the stories to be excluded from the batch. Leading and trailing whitespaces in patterns are omitted.

batch-<batch-number>.name

batch-<batch-number>

The name of the batch to display in the report

batch-<batch-number>.threads

1

The number of parallel threads used to run stories

batch-<batch-number>.meta-filters

suite meta-filter

The meta-filter used to filter the batch stories and scenarios

batch-<batch-number>.fail-fast

false

If true and any failure is occurred during the batch execution, the subsequent batches will not be executed.

batch-<batch-number>.story.fail-fast

<empty>

If set the value overrides global setting story.fail-fast.

batch-<batch-number>.scenario.fail-fast

<empty>

If set the value overrides global setting scenario.fail-fast.

batch-<batch-number>.story.execution-timeout

PT3H

The max duration of the single story in the batch in ISO-8601 Durations format. Overrides value specified via story.execution-timeout.

batch.fail-fast

false

If set to true the subsequent batches execution will be stopped after first failed assertion

story.fail-fast

false

If set to true the story execution will be stopped after first failed assertion

scenario.fail-fast

false

If set to true the scenario execution will be stopped after first failed assertion

story.execution-timeout

PT3H

The max duration of the single story in ISO-8601 Durations format. Could be overriden via corresponging batch setting.

bdd.configuration.dry-run

false

Enables dry-run execution mode (no actual steps will be executed, dynamic variables and expressions won’t be resolved). For example dry-run could be useful to debug what stroies will be executed with provided config.

bdd.configuration.skip-story-if-given-story-failed

false

Skips the story in case of failure in the given story

bdd.configuration.formats

JSON,XML

Available formats of the output files with test execution results

statistics.print-failures

false

If set to true prints table of failures including: story, scenario, step and error message

report.text-length-diff-threshold

100

Defines text threshold length to enable unified diff report publishing on failure, for variable comparisons. If you want to have this report despite the size of the compared data set 0.

engine.composite-paths

<empty>

Paths to the composite steps e.g. steps/*.steps

Profiles

A profile defines the subject of the test, i.e. it answers the question "What are specifics of the browser/device I want to run tests on?".

These specifics (properties) should be written in the profile.properties file. Path to this file is set in configuration.properties file:

Example 2. configuration.properties
configuration.profiles=web

It’s allowed to create multiple profiles. This can be useful for cross-browsing or cross-device testing: there is no need of changing all the properties, just switch the profile.
As well as the suites, profiles are loaded one by one starting from the last one in the sequence. This allows to reuse common properties and separate specific ones.

Profiles setup

Define the number of profiles you need to set up. In case of multiple profiles, distinguish common and specific properties and put them into profile.properties files in appropriate places.

Please keep in mind that running multiple profiles (e.g. for Chrome and Firefox simultaneously) is not supported. All the properties would be merged, but not iterated as separate profiles for scenario execution.

Let’s create an example. Imagine that our project contains automated scenarios for a couple of platforms: web and mobile. We will need to run autotests on different browsers and different devices. Therefore some of the properties would be common, but many of them should only belong to a certain subject. Let’s see profiles structure for this case:

└─ src/main/resources/properties
                            └─  profile (1)
                                    ├─ web (2)
                                        ├─ chrome (4)
                                                └─ profile.properties (6)
                                        ├─ firefox (4)
                                                └─ profile.properties
                                        └─ profile.properties (5)
                                    ├─ mobile_app (2)
                                            ├─ iOS (7)
                                                └─ profile.properties
                                            ├─ android (7)
                                                └─ profile.properties (9)
                                            └─ profile.properties (8)
                                    └─ profile.properties (3)
1 Profile directory which contains all the profiles properties;
2 General type of the profiles, either a browser or a mobile device;
3 The highest level of profile properties. Values written in this file will be applied to any browser or device;
4 Web browsers' profiles (directories);
5 Web-oriented properties. Values from this file will be applied for any web browser, but not for mobiles;
6 Chrome-specific profile. Any properties written here will not affect any other browsers or mobile devices;
7 Mobile profiles (directories);
8 Mobile-oriented properties. Values from this file will be applied for iOS and Android devices only;
9 Android-specific profile. Any properties written here will not affect iOS devices or any browser setup;

By default, profile property is pointed to the /src/main/resources/properties/profile directory, so use the relative path starting from profile when you have nested directories:

Example 3. configuration.properties
configuration.profiles=mobile_app/android

Built-in profiles

Some of the plugins, like web-app, have built-in profiles. You can find them in project explorer after installation by the following path:

Referenced Libraries -> vividus-plugin-name -> properties -> profile

Environments

Depending on the stage of the project, you will likely need to run automated scenarios on different environments, such as development, testing, and production servers.

Rather than using specific variables in each profile or suite, always updating them according to the currently selected server, and eventually making mistakes, we instead use the environment configuration.

Environment configuration allows putting environment-related variables (e.g. host, API endpoint) into environment.properties file. The path to this file is set in configuration.properties file. Create as many property files as a number of environments you have and easily switch between them:

Example 4. configuration.properties
configuration.environments=dev
#configuration.environments=test
#configuration.environments=prod

Environment variables can also be declared in multiple files. Define common and specific properties for your environments and take the same approach of reusing and separating values.

Known Issues

Known issues mechanism allows to distinguish failures from known issues presented in the system under test.

JSON file should contain a valid JSON. If JSON is not valid known issues will be ignored and warning about invalid file will be printed.

Properties

Property Default value Description

known-issue-provider.fileName

known-issues.json

Defines the location of the known issues file

known-issue-provider.detect-potentially-known-issues

true

Defines are potentially known issues detected

Known Issues file

The file represents a collection of JSON elements describing known issues.

Example 5. known-issues.json
 {
    "VVD-1": { (1)
        "type": "Internal",(2)
        "assertionPattern": "Expected: a value equal to 200, Actual: \\[404\\]", (3)
        "storyPattern": "Validate the Math", (4)
        "scenarioPattern": "Validate equality", (5)
        "stepPattern": "Then `1` is = `2`.+", (6)
        "variablePatterns": { (7)
            "var":  "value-.*"
        },
        "failScenarioFast": true, (8)
        "failStoryFast": true, (9)
        "description": "Missing endpoint configuration on server side" (10)
    }
}
1 [Mandatory] Identifier (This could be an ID of the issue in your bug tracking system)
2 [Mandatory] Issue type. One of: INTERNAL (application issue), EXTERNAL (3rd party issue), AUTOMATION (test automation issue)
3 [Mandatory] Assertion pattern describes which failed assertions should be considered as known
4 Story pattern to match the file name of the failed story. Pay attention, the file name extension (.story) should not be included in the pattern, only base file name is matched.
5 Scenario pattern to match failed scenario name
6 Step pattern to match failed step.
7 Variable patterns. Patterns to match variables.
8 Skip the rest of scenario in case of failure identified as this known issue
9 Skip the rest of story in case of failure identified as this known issue
10 Known issue description, it’s used to give the meaningful explanation of the issue if it’s not tracked in bug tracking system or if the issue is too complicated and to improve overall transparency
Use \\ for escaping of special characters in regular expressions

Examples

  • To consider failed assertion known or potentially-known assertionPattern should match assertion failure, if defined variablePatterns should match also.

  • If assertionPattern/variablePatterns matched and you have story/scenario/step/Pattern defined, to consider issue known all of them should match.

  • If assertionPattern/variablePatterns matched and you have story/scenario/step/Pattern defined, when at least one not matched issue is considered potentially known.

Meta tags

Meta tags can be used to reach several goals:

  • Grouping of stories and scenarios in the report.

  • Dynamic selection of tests to execute by meta tags.

  • Management of capabilities for the particular tests.

Levels

Meta tags may be used at both story and scenario level. Meta tags specified at the story level are propagated to every scenario, if the same meta tag is used at the scenario level, then the scenario level value takes precedence.

Example 6. Levels of meta tags
!-- Story level
Meta:
     @group Login

Scenario: Open Epam main page
!-- Scenario level
Meta:
  @severity 1
  @layout desktop
Given I am on page with URL `https://www.epam.com/`
Then the page title is equal to 'EPAM | Enterprise Software Development, Design & Consulting'

Scenario: Open Google main page
Meta:
  @severity 2
  @layout desktop
  @skip
Given I am on page with URL `https://www.google.com/`
Then the page title is equal to 'Google'

Properties

The property bdd.meta-filters is used to filter tests to be executed and relies on Groovy syntax.

Example 7. Execute stoies and scenarios marked with '@group Login' tag and matching default filters
bdd.meta-filters=group == 'Login'

The prorerty bdd.all-meta-filters is used to override values of the default meta filters based on the tags: @skip, @layout etc.

Example 8. Execute stoies and scenarios marked with '@group Login' tag and without '@skip' tag
bdd.all-meta-filters=groovy: (group == 'Login' && !skip)

Limitations

Since meta filters properties are based on Groovy, meta tags names support only normal Groovy identifiers by default. If it’s required to use arbitrary characters in meta tags names (e.g.follow kebab-case notation which actively relies on the hyphen char), then quoted identifiers or subscript operator must be used in the properties.

Example 9. Execute stoies and scenarios not marked with '@test-stand' tag (quoted identifiers approach)
bdd.all-meta-filters=groovy: !this.'test-stand'
Example 10. Execute stoies and scenarios not marked with '@test-stand' tag (subscript operator approach)
bdd.all-meta-filters=groovy: !this['test-stand']

1D tags (@key)

Tag Description

@skip

Used to exclude particular scenarios from the test execution

2D tags (@key  value)

Tag Key Allowed values Description Example

@severity

Numeric values (range 1-5)

Used to mark importance of the test where, most usually, the smallest number is the most important Test, the highest number is the least important.

@severity 1

@layout

desktop, phone, phone_portrait, phone_landscape, chrome_phone, chrome_responsive, chrome_tablet, tablet, tablet_landscape, tablet_portrait

Used to specify layout, using which marked story or scenario should be executed. In case if this meta tag is not specified, default (desktop) value is used for web applications.

@layout desktop

Story Prioritization

In order to start stories in particular order it is possible to assign the stories a numeric priority, the stories with the higher priority start first.

To enable the feature create a file spring.xml in the src/main/resources directory and add the following XML data into that file.

Example 11. Configuration
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans https://www.springframework.org/schema/beans/spring-beans.xsd"
       default-lazy-init="false">

    <bean class="org.vividus.priority.MetaBasedStoryExecutionPriority" factory-method="byNumericMetaValue">
        <constructor-arg index="0" value="story_priority" />
    </bean>
</beans>

The story_priority value that corresponds to a meta name in stories can be changed to arbitrary value without spaces.

Taking story_priority as an example the priority meta should be placed at the story level as the following example shows:

Example 12. Prioritized.story
Meta: @story_priority 15

Scenario: Time consuming test
When I perform highly time consuming task

Exit codes

The tests execution returns one of the following exit codes.

Code Description Status

0

Test report is generated, contains only passed tests.

GREEN

1

Test report is generated, contains only passed and tests with known issues.

AMBER

2

Test report is generated, contains any tests except passed and with known issues or doesn’t contain tests at all.

RED

3

Test report isn’t generated.

Common errors:

  • "InvalidConfigurationException: No batches with tests to execute are configured" - the configured suite has 0 batches, i.e. no tests to execute are configured.

RED

HTTP Configuration

Here one could find description of general HTTP properties applied across all the HTTP interactions in VIVIDUS.

Property Name Acceptable values Default Description

General

http.circular-redirects-allowed

boolean

false

Allow circular redirects (redirects to the same location)

http.connection-request-timeout

integer

30000

The timeout in milliseconds used when requesting a connection from the client connections pool. A timeout value of zero is interpreted as an infinite timeout. A negative value is interpreted as undefined (system default if applicable)

http.connect-timeout

integer

30000

The timeout in milliseconds until a connection is established. A timeout value of zero is interpreted as an infinite timeout. A negative value is interpreted as undefined (system default if applicable)

http.max-total-connections

integer

80

The number of maximum total connections for the client connections pool

http.max-connections-per-route

integer

60

The number of maximum connections per route for the client connections pool

http.socket-timeout

integer

3000

The timeout in milliseconds used for waiting for data or, put differently, a maximum period inactivity between two consecutive data packets. A timeout value of zero is interpreted as an infinite timeout. A negative value is interpreted as undefined (system default if applicable)

http.cookie-spec

default, ignoreCookies, standard-strict, standard, netscape

default

The name of the cookie specification to be used for HTTP state management

Authentication

http.auth.username

character sequence

<empty>

The username, presence of this property requires the http.auth.password to be also set

http.auth.password

character sequence

<empty>

The password, presence of this property requires the http.auth.username to be also set

http.auth.preemptive-auth-enabled

boolean

false

If enabled the HTTP client will send the basic authentication data even before the service gives an unauthorized response (401), useful when the service doesn’t support unauthorized response (401) or to reduce the overhead of making the connection

SSL handshake

http.ssl.check-certificate

boolean

true

Verify certificates sent by the server during SSL handshake, the most common errors occured during verfication are Expired SSL Certificate Error, SSL Certificate Not Trusted Error, SSL Certificate Revoked Error etc.

http.ssl.verify-hostname

boolean

true

Verify if the domain name in the SSL certificate matches the domain name in URL where the request is sent to

Two-Way SSL authentication

http.key-store.type

Keystore type

JKS

The keystore type

http.key-store.path

Resource

<empty>

The keystore to load

http.key-store.password

string

<empty>

The password used to unlock the keystore and for the integrity checking

http.ssl.private-key-password

string

<empty>

The password for the private key

DNS resolver

http.local-dns-storage.

key-value mapping

<empty>

The property prefix used to override the OS DNS lookup by specifying hostname to IP address mappings e.g. http.local-dns-storage.coursera.org=52.84.197.20

Service unavailability retry handler

http.service-unavailable-retry.max-retries

integer

1

The maximum number of allowed retries if the server responds with a HTTP code specified in http.service-unavailable-retry.status-codes property

http.service-unavailable-retry.status-codes

set of integers

<empty>

The set of HTTP status codes to be retried. If it’s empty, no retries will be performed

http.service-unavailable-retry.retry-interval

ISO-8601 Durations format

PT1S

The retry interval between subsequent requests

Idempotent retry handler

http.idempotent-methods-sending-request-body

set of integers

<empty>

The list of HTTP methods containing request body which should be treated as idempotent for retries

Externalized Configuration

Most common use cases for externalized configuration are tests parametrization with properties passed from CI/CD pipeline, passing of secrets into test run etc. The available approaches are listed below.

  1. Declare an environment variable containing the desired property value and then point the property to the declared environment variable:

    export EXTERNAL_CONFIG_VALUE=value
    variables.some-key=${EXTERNAL_CONFIG_VALUE}

  2. Pass the property with its value into a test run using command line arguments prepending it with the -Pvividus. prefix. The prefix only plays role during test start-up phase and cut off during the actual test run:

    ./gradlew runStories -Pvividus.variables.some-key-1=value1 -Pvividus.variables.some-key-2=value2

    This approach can be used only when tests are run via Gradle.

  3. Generate the overriding.properties file on-the-fly and put it into corresponding folder in a test project.

    The following script assumes that we are in the root of the test project folder.

    cat > src/main/resources/overriding.properties << EOD
variables.some-key-1=value1
variables.some-key-2=value2
EOD

    This approach can be used only when tests are run via Gradle.

Sensitive Data Encryption

VIVIDUS allows to store values of the properties in the encrypted form. VIVIDUS uses Jasypt (Java Simplified Encryption) which provides two-way encryption mechanism. While performing two-way encryption, apart from feeding plain-text it’s required to feed the secret text (i.e. password) and this secret text is used to decrypt the encrypted text. The default encryption algorithm is PBEWithMD5AndDES.

How to encrypt a string

Option 1 (Recommended): using Jasypt CLI

  1. Download the latest Jasypt release

  2. Unpack the downloaded zip-archive

  3. Go to bin folder

  4. Run the command performing encryption:

    Example 13. Windows:
    encrypt.bat input="This is my message to be encrypted" password=MYPAS_WORD
    Example 14. Linux/UNIX/macOS:
    ./encrypt.sh input="This is my message to be encrypted" password=MYPAS_WORD

    where

    • input - Any string to be encrypted.

    • password - Your encryption password.

      MYPAS_WORD is a sample password and should be never used, own strong password is required for the encryption.

  5. Find the encrypted data in OUTPUT section

    Example 15. Encryption command output
    ----ENVIRONMENT-----------------

Runtime: Eclipse Adoptium OpenJDK 64-Bit Server VM 17+35



----ARGUMENTS-------------------

input: This is my message to be encrypted
password: MYPAS_WORD



----OUTPUT----------------------

WVnXhzG704KRZZFYL7hfpA4YoSm8V2F5eNXyr7CRYCichteHvNS3nB23zaI8CuLy
Find more details in Jasypt CLI Tools documentation

Option 2: using Jasypt Online

Use this tool at your own risk, since there is a chance of potential leakage of sensitive data

  1. Paste plain text string to encrypt to the corresponding text field.

  2. Set Type of Encryption to "Two Way Encryption (With Secret Text)".

  3. Enter secret key/text in the corresponding text field.

  4. Click "Encrypt".

  5. The resulting encrypted string can be copied from the corresponding field.

How to use an encrypted string in VIVIDUS

  1. Set the password in one of the following ways:

    1. Pass the key with its password-value into a test run using command line:

      ./gradlew runStories -Pvividus.encryptor.password=MYPAS_WORD

    2. Add the password to the system project properties as the value of:

      system.vividus.encryptor.password=MYPAS_WORD

    3. Set via OS environment variable VIVIDUS_ENCRYPTOR_PASSWORD

      Keep in mind that a password specified in the command line takes priority over the project system property, and the project system property takes priority over the environment variable.

      This password must be kept secret and must not be committed to version control system.
      MYPAS_WORD is a sample password and should be never used, own strong password is required for the encryption.

  2. Use the wrapping ENC(…​) for any encrypted property value. e.g.

    http.auth.password=ENC(WVnXhzG704KRZZFYL7hfpA4YoSm8V2F5eNXyr7CRYCichteHvNS3nB23zaI8CuLy)

Please see Externalized Configuration to get more information about how encryptor password can be passed to the tests.

Composite steps

Composite steps allow to reuse set of actions via custom steps with own wordings.

How to use composite steps

  1. Define composite step in *.steps file, e.g. steps/ui/my-app.steps:

    Example 16. Composite step definition
    Composite: When I login with username `$username` and password `$password`
When I enter `<username>` in field located by `id(username)`
When I enter `<password>` in field located by `id(password)`
When I click on element located by `id(signInButton)`

  2. Configure paths to files with custom steps:

    Example 17. Composite steps paths configuration
    engine.composite-paths=steps/ui/*.steps

  3. Use your step in the story:

    Example 18. Composite step usage example
    Scenario: Verify login
Given I am on main application page
When I login with username `admin` and password `pa$$w0rd`
Users are allowed to use composite steps in composite step definition.

Steps aliases

VIVIDUS allows to define set of aliases for steps in JSON format and then use them in stories.

How to use steps aliases

  1. Select a step you want to create alias for, as an example let’s take the following step:

    Example 19. VIVIDUS step
    When I hover mouse over element located `$locator`

  2. Create a new JSON file (for example, aliases/aliases.json) and put the following content with one or more aliases:

    Example 20. aliases/aliases.json
    [
    {
        "name": "When I hover mouse over element located `$locator`",
        "aliases":
        [
            {
                "name": "When I place mouse pointer over web element located by `$locator`"
            },
            {
                "name": "When I hover mouse pointer over web element located `$locator`"
            }
        ]
    }
]

  3. Point the alias configuration property to an exact location or location pattern with the JSON files containing aliases:

    engine.alias-paths=aliases/aliases.json

  4. Use the defined alias in place of actual step in your story

    When I place mouse pointer over web element located by `id(loginButton)`