Tests Configuration
configuration.properties
VIVIDUS configuration includes the following parts: suites, profiles and environments.
Properties prefixed with
|
Suites
The property configuration.suites
defines the suites set. It is a comma separated set of suite file addresses:
configuration.suites=testSuite,anotherDir/anotherTestSuite
VIVIDUS suite consists of the unlimited number of batches. In its turn the batch consists of any number of stories represented by the files with *.story
extension. The batches are run sequentially, while the stories in the batch are run in parallel.
The suites are loaded one by one starting from the last one in the sequence, in other words, the values of the properties defined in the first suite will take precedence over the values of the same properties defined in all subsequent suites. If any of the suites in the sequence is not found (i.e. no *.properties
file is present at the given path), the error will be thrown and the execution will be stopped. However if suites property is empty (configuration.suites=
) and no suite is configured in the root (i.e no files can be found by /properties/suite/*.properties
pattern), no error will be thrown and the execution will be continued.
The following properties are used to configure batches (VIVIDUS doesn’t provide any default batch configuration out of the box). batch-number
is one-based batch index.
The properties marked with bold are mandatory. |
Property | Default value | Description |
---|---|---|
|
The test project path to the folder with story-files |
|
|
|
The comma-separated set of ant-patterns to match the stories to be included in the batch. Leading and trailing whitespaces in patterns are omitted. |
|
|
The comma-separated set of ant-patterns to match the stories to be excluded from the batch. Leading and trailing whitespaces in patterns are omitted. |
|
|
The name of the batch to display in the report |
|
|
The number of parallel threads used to run stories |
|
suite meta-filter |
The meta-filter used to filter the batch stories and scenarios |
|
|
If |
|
|
If set the value overrides global setting |
|
|
If set the value overrides global setting |
|
|
The max duration of the single story in the batch in ISO-8601 format. Overrides value specified via |
|
|
If set to |
|
|
If set to |
|
|
If set to |
|
|
The max duration of the single story in ISO-8601 format. Could be overriden via corresponging batch setting. |
|
|
Enables dry-run execution mode (no actual steps will be executed, dynamic variables and expressions won’t be resolved). For example dry-run could be useful to debug what stroies will be executed with provided config. |
|
|
Skips the story in case of failure in the given story |
|
|
Available formats of the output files with test execution results |
|
|
If set to |
|
|
Defines text threshold length to enable unified diff report publishing on failure, for variable comparisons. If you want to have this report despite the size of the compared data set 0. |
|
<empty> |
Paths to the composite steps e.g. |
Profiles
A profile defines the subject of the test, i.e. it answers the question "What are specifics of the browser/device I want to run tests on?".
These specifics (properties) should be written in the profile.properties
file. Path to this file is set in configuration.properties
file:
configuration.profiles=web
It’s allowed to create multiple profiles. This can be useful for cross-browsing or cross-device testing: there is no need of changing all the properties, just switch the profile.
As well as the suites, profiles are loaded one by one starting from the last one in the sequence. This allows to reuse common properties and separate specific ones.
Profiles setup
Define the number of profiles you need to set up. In case of multiple profiles, distinguish common and specific properties and put them into profile.properties
files in appropriate places.
Please keep in mind that running multiple profiles (e.g. for Chrome and Firefox simultaneously) is not supported. All the properties would be merged, but not iterated as separate profiles for scenario execution. |
Let’s create an example. Imagine that our project contains automated scenarios for a couple of platforms: web and mobile. We will need to run autotests on different browsers and different devices. Therefore some of the properties would be common, but many of them should only belong to a certain subject. Let’s see profiles structure for this case:
└─ src/main/resources/properties
└─ profile (1)
├─ web (2)
├─ chrome (4)
└─ profile.properties (6)
├─ firefox (4)
└─ profile.properties
└─ profile.properties (5)
├─ mobile_app (2)
├─ iOS (7)
└─ profile.properties
├─ android (7)
└─ profile.properties (9)
└─ profile.properties (8)
└─ profile.properties (3)
1 | Profile directory which contains all the profiles properties; |
2 | General type of the profiles, either a browser or a mobile device; |
3 | The highest level of profile properties. Values written in this file will be applied to any browser or device; |
4 | Web browsers' profiles (directories); |
5 | Web-oriented properties. Values from this file will be applied for any web browser, but not for mobiles; |
6 | Chrome-specific profile. Any properties written here will not affect any other browsers or mobile devices; |
7 | Mobile profiles (directories); |
8 | Mobile-oriented properties. Values from this file will be applied for iOS and Android devices only; |
9 | Android-specific profile. Any properties written here will not affect iOS devices or any browser setup; |
By default, profile property is pointed to the /src/main/resources/properties/profile
directory, so use the relative path starting from profile
when you have nested directories:
configuration.profiles=mobile_app/android
Built-in profiles
Some of the plugins, like web-app, have built-in profiles. You can find them in project explorer after installation by the following path:
Referenced Libraries -> vividus-plugin-name -> properties -> profile
Environments
Depending on the stage of the project, you will likely need to run automated scenarios on different environments, such as development, testing, and production servers.
Rather than using specific variables in each profile or suite, always updating them according to the currently selected server, and eventually making mistakes, we instead use the environment configuration.
Environment configuration allows putting environment-related variables (e.g. host, API endpoint) into environment.properties
file. The path to this file is set in configuration.properties
file. Create as many property files as a number of environments you have and easily switch between them:
configuration.environments=dev
#configuration.environments=test
#configuration.environments=prod
Environment variables can also be declared in multiple files. Define common and specific properties for your environments and take the same approach of reusing and separating values.
Known Issues
Known issues mechanism allows to distinguish failures from known issues presented in the system under test.
JSON file should contain a valid JSON. If JSON is not valid known issues will be ignored and warning about invalid file will be printed. |
Properties
Property | Default value | Description |
---|---|---|
|
|
Defines the location of the known issues file |
|
|
Defines are potentially known issues detected |
Known Issues file
The file represents a collection of JSON elements describing known issues.
{
"VVD-1": { (1)
"type": "Internal",(2)
"assertionPattern": "Expected: a value equal to 200, Actual: \\[404\\]", (3)
"storyPattern": "Validate the Math", (4)
"scenarioPattern": "Validate equality", (5)
"stepPattern": "Then `1` is = `2`.+", (6)
"variablePatterns": { (7)
"var": "value-.*"
},
"failScenarioFast": true, (8)
"failStoryFast": true, (9)
"description": "Missing endpoint configuration on server side" (10)
}
}
1 | [Mandatory] Identifier (This could be an ID of the issue in your bug tracking system) |
2 | [Mandatory] Issue type. One of: INTERNAL (application issue), EXTERNAL (3rd party issue), AUTOMATION (test automation issue) |
3 | [Mandatory] Assertion pattern describes which failed assertions should be considered as known |
4 | Story pattern to match the file name of the failed story. Pay attention, the file name extension (.story ) should not be included in the pattern, only base file name is matched. |
5 | Scenario pattern to match failed scenario name |
6 | Step pattern to match failed step. |
7 | Variable patterns. Patterns to match variables. |
8 | Skip the rest of scenario in case of failure identified as this known issue |
9 | Skip the rest of story in case of failure identified as this known issue |
10 | Known issue description, it’s used to give the meaningful explanation of the issue if it’s not tracked in bug tracking system or if the issue is too complicated and to improve overall transparency |
Use \\ for escaping of special characters in regular expressions |
Examples
-
To consider failed assertion known or potentially-known assertionPattern should match assertion failure, if defined variablePatterns should match also.
-
If assertionPattern/variablePatterns matched and you have story/scenario/step/Pattern defined, to consider issue known all of them should match.
-
If assertionPattern/variablePatterns matched and you have story/scenario/step/Pattern defined, when at least one not matched issue is considered potentially known.
Meta tags
Meta tags can be used to reach several goals:
-
Grouping of stories and scenarios in the report.
-
Dynamic selection of tests to execute by meta tags.
-
Management of capabilities for the particular tests.
Levels
Meta tags may be used at both story and scenario level. Meta tags specified at the story level are propagated to every scenario, if the same meta tag is used at the scenario level, then the scenario level value takes precedence.
!-- Story level
Meta:
@group Login
Scenario: Open Epam main page
!-- Scenario level
Meta:
@severity 1
@layout desktop
Given I am on a page with the URL 'https://www.epam.com/'
Then the page title is equal to 'EPAM | Enterprise Software Development, Design & Consulting'
Scenario: Open Google main page
Meta:
@severity 2
@layout desktop
@skip
Given I am on a page with the URL 'https://www.google.com/'
Then the page title is equal to 'Google'
Properties
The property bdd.meta-filters
is used to filter tests to be executed and
relies on Groovy syntax.
bdd.meta-filters=group == 'Login'
The prorerty bdd.all-meta-filters
is used to override values of the default
meta filters based on the tags: @skip
, @layout
etc.
bdd.all-meta-filters=groovy: (group == 'Login' && !skip)
Limitations
Since meta filters properties are based on Groovy, meta tags names support only normal Groovy identifiers by default. If it’s required to use arbitrary characters in meta tags names (e.g.follow kebab-case notation which actively relies on the hyphen char), then quoted identifiers or subscript operator must be used in the properties.
bdd.all-meta-filters=groovy: !this.'test-stand'
bdd.all-meta-filters=groovy: !this['test-stand']
2D tags (@key value)
Tag Key | Allowed values | Description | Example |
---|---|---|---|
@severity |
Numeric values (range 1-5) |
Used to mark importance of the test where, most usually, the smallest number is the most important Test, the highest number is the least important. |
|
@layout |
|
Used to specify layout, using which marked story or scenario should be executed. In case if this meta tag is not specified, default ( |
|
Story Prioritization
In order to start stories in particular order it is possible to assign the stories a numeric priority, the stories with the higher priority start first.
To enable the feature create a file spring.xml
in the src/main/resources
directory and add the following XML data into that file.
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans https://www.springframework.org/schema/beans/spring-beans.xsd"
default-lazy-init="false">
<bean class="org.vividus.priority.MetaBasedStoryExecutionPriority" factory-method="byNumericMetaValue">
<constructor-arg index="0" value="story_priority" />
</bean>
</beans>
The story_priority
value that corresponds to a meta name in stories can be changed to arbitrary value without spaces.
Taking story_priority
as an example the priority meta should be placed at the story level as the following example shows:
Meta: @story_priority 15
Scenario: Time consuming test
When I perform highly time consuming task
Exit codes
The tests execution returns one of the following exit codes.
Code | Description | Status |
---|---|---|
0 |
Test report is generated, contains only passed tests. |
|
1 |
Test report is generated, contains only passed and tests with known issues. |
|
2 |
Test report is generated, contains any tests except passed and with known issues or doesn’t contain tests at all. |
|
3 |
Test report isn’t generated. Common errors:
|
|
HTTP Configuration
Here one could find description of general HTTP properties applied across all the HTTP interactions in VIVIDUS.
Property Name | Acceptable values | Default | Description |
---|---|---|---|
General |
|||
|
|
|
Allow circular redirects (redirects to the same location) |
|
|
|
The timeout in milliseconds used when requesting a connection from the client connections pool. A timeout value of zero is interpreted as an infinite timeout. A negative value is interpreted as undefined (system default if applicable) |
|
|
|
The timeout in milliseconds until a connection is established. A timeout value of zero is interpreted as an infinite timeout. A negative value is interpreted as undefined (system default if applicable) |
|
|
|
The number of maximum total connections for the client connections pool |
|
|
|
The number of maximum connections per route for the client connections pool |
|
|
|
The timeout in milliseconds used for waiting for data or, put differently, a maximum period inactivity between two consecutive data packets. A timeout value of zero is interpreted as an infinite timeout. A negative value is interpreted as undefined (system default if applicable) |
|
|
|
The name of the cookie specification to be used for HTTP state management |
Authentication |
|||
|
character sequence |
|
The username, presence of this property requires the |
|
character sequence |
|
The password, presence of this property requires the |
|
|
|
If enabled the HTTP client will send the basic authentication data even before the service gives an unauthorized response (401), useful when the service doesn’t support unauthorized response (401) or to reduce the overhead of making the connection |
SSL handshake |
|||
|
|
|
Verify certificates sent by the server during SSL handshake, the most common errors occured during verfication are Expired SSL Certificate Error, SSL Certificate Not Trusted Error, SSL Certificate Revoked Error etc. |
|
|
|
Verify if the domain name in the SSL certificate matches the domain name in URL where the request is sent to |
Two-Way SSL authentication |
|||
|
|
The keystore type |
|
|
|
The keystore to load |
|
|
|
|
The password used to unlock the keystore and for the integrity checking |
|
|
|
The password for the private key |
DNS resolver |
|||
|
|
|
The property prefix used to override the OS DNS lookup by specifying hostname to IP address mappings e.g. |
Service unavailability retry handler |
|||
|
|
|
The maximum number of allowed retries if the server responds with a HTTP code specified in |
|
set of integers |
|
The set of HTTP status codes to be retried. If it’s empty, no retries will be performed |
|
ISO-8601 Durations format |
|
The retry interval between subsequent requests |
Idempotent retry handler |
|||
|
set of integers |
|
The list of HTTP methods containing request body which should be treated as idempotent for retries |
Externalized Configuration
Most common use cases for externalized configuration are tests parametrization with properties passed from CI/CD pipeline, passing of secrets into test run etc. The available approaches are listed below.
-
Declare an environment variable containing the desired property value and then point the property to the declared environment variable:
export EXTERNAL_CONFIG_VALUE=value
variables.some-key=${EXTERNAL_CONFIG_VALUE}
-
Pass the property with its value into a test run using command line arguments prepending it with the
-Pvividus.
prefix. The prefix only plays role during test start-up phase and cut off during the actual test run:./gradlew runStories -Pvividus.variables.some-key-1=value1 -Pvividus.variables.some-key-2=value2
This approach can be used only when tests are run via Gradle.
-
Generate the
overriding.properties
file on-the-fly and put it into corresponding folder in a test project.The following script assumes that we are in the root of the test project folder.
cat > src/main/resources/overriding.properties << EOD variables.some-key-1=value1 variables.some-key-2=value2 EOD
This approach can be used only when tests are run via Gradle.
Sensitive Data Encryption
VIVIDUS allows to store values of the properties in the encrypted form.
VIVIDUS uses Jasypt (Java Simplified Encryption) which provides two-way
encryption mechanism. While performing two-way encryption, apart from feeding
plain-text it’s required to feed the secret text (i.e. password) and this secret
text is used to decrypt the encrypted text. The default encryption algorithm is
PBEWithMD5AndDES
.
How to encrypt a string
Option 1 (Recommended): using Jasypt CLI
-
Download the latest Jasypt release
-
Unpack the downloaded zip-archive
-
Go to
bin
folder -
Run the command performing encryption:
Example 13. Windows:encrypt.bat input="This is my message to be encrypted" password=MYPAS_WORD
Example 14. Linux/UNIX/macOS:./encrypt.sh input="This is my message to be encrypted" password=MYPAS_WORD
where
-
input
- Any string to be encrypted. -
password
- Your encryption password.MYPAS_WORD
is a sample password and should be never used, own strong password is required for the encryption.
-
-
Find the encrypted data in
OUTPUT
sectionExample 15. Encryption command output----ENVIRONMENT----------------- Runtime: Eclipse Adoptium OpenJDK 64-Bit Server VM 17+35 ----ARGUMENTS------------------- input: This is my message to be encrypted password: MYPAS_WORD ----OUTPUT---------------------- WVnXhzG704KRZZFYL7hfpA4YoSm8V2F5eNXyr7CRYCichteHvNS3nB23zaI8CuLy
Find more details in Jasypt CLI Tools documentation |
Option 2: using Jasypt Online
Use this tool at your own risk, since there is a chance of potential leakage of sensitive data |
-
Paste plain text string to encrypt to the corresponding text field.
-
Set Type of Encryption to "Two Way Encryption (With Secret Text)".
-
Enter secret key/text in the corresponding text field.
-
Click "Encrypt".
-
The resulting encrypted string can be copied from the corresponding field.
How to use an encrypted string in VIVIDUS
-
Set the password in one of the following ways:
-
Pass the key with its password-value into a test run using command line:
./gradlew runStories -Pvividus.encryptor.password=MYPAS_WORD
-
Add the password to the system project properties as the value of:
system.vividus.encryptor.password=MYPAS_WORD
-
Set via OS environment variable
VIVIDUS_ENCRYPTOR_PASSWORD
Keep in mind that a password specified in the command line takes priority over the project system property, and the project system property takes priority over the environment variable.
This password must be kept secret and must not be committed to version control system. MYPAS_WORD
is a sample password and should be never used, own strong password is required for the encryption.
-
-
Use the wrapping
ENC(…)
for any encrypted property value. e.g.http.auth.password=ENC(WVnXhzG704KRZZFYL7hfpA4YoSm8V2F5eNXyr7CRYCichteHvNS3nB23zaI8CuLy)
Please see Externalized Configuration to get more information about how encryptor password can be passed to the tests.
Composite steps
Composite steps allow to reuse set of actions via custom steps with own wordings.
How to use composite steps
-
Define composite step in
*.steps
file, e.g.steps/ui/my-app.steps
:Example 16. Composite step definitionComposite: When I login with username `$username` and password `$password` When I enter `<username>` in field located by `id(username)` When I enter `<password>` in field located by `id(password)` When I click on element located by `id(signInButton)`
-
Configure paths to files with custom steps:
Example 17. Composite steps paths configurationengine.composite-paths=steps/ui/*.steps
-
Use your step in the story:
Example 18. Composite step usage exampleScenario: Verify login Given I am on the main application page When I login with username `admin` and password `pa$$w0rd`
Users are allowed to use composite steps in composite step definition. |
Steps aliases
VIVIDUS allows to define set of aliases for steps in JSON format and then use them in stories.
How to use steps aliases
-
Select a step you want to create alias for, as an example let’s take the following step:
Example 19. VIVIDUS stepWhen I hover mouse over element located `$locator`
-
Create a new JSON file (for example,
aliases/aliases.json
) and put the following content with one or more aliases:Example 20. aliases/aliases.json[ { "name": "When I hover mouse over element located `$locator`", "aliases": [ { "name": "When I place mouse pointer over web element located by `$locator`" }, { "name": "When I hover mouse pointer over web element located `$locator`" } ] } ]
-
Point the alias configuration property to an exact location or location pattern with the JSON files containing aliases:
engine.alias-paths=aliases/aliases.json
-
Use the defined alias in place of actual step in your story
When I place mouse pointer over web element located by `id(loginButton)`