Table Transformers

VIVIDUS provides a set of pre-registered table transformers. Also plugins may register own table transformers (they are documented in the corresponding plugins articles).

FROM_LANDSCAPE

FROM_LANDSCAPE transformer is designed to transform the table from a landscape form.

Example 1. Usage of FROM_LANDSCAPE transformer
{transformer=FROM_LANDSCAPE}
|header 1|value 11| ... | value m1|
...
|header n|value 1n| .... |value mn|

RESOLVING_EXPRESSIONS_EAGERLY

RESOLVING_EXPRESSIONS_EAGERLY transformer processes expressions which are used inside ExamplesTable eagerly, i.e. at the table loading stage. If expressions generating random data are used in the table with RESOLVING_EXPRESSIONS_EAGERLY transfomer, the random data will be generated once and each placeholder will be replaced with this value. While in regular flow placeholders are replaced with the expression and data is generated at step execution stage, in other words the same placeholders will be replaced by different randomly generated datasets.

Example 2. Usage of RESOLVING_EXPRESSIONS_EAGERLY transformer
{transformer=RESOLVING_EXPRESSIONS_EAGERLY}
|firstName                     |lastName                     |
|#{generate(Address.firstName)}|#{generate(Address.lastName)}|

RESOLVING_SELF_REFERENCES_EAGERLY

RESOLVING_SELF_REFERENCES_EAGERLY transformer enables eagerly resolution of references to one column from another column within single ExamplesTable ("eagerly" means at the table loading stage). Circular references will result in error.

Example 3. Usage of RESOLVING_SELF_REFERENCES_EAGERLY transformer
{transformer=RESOLVING_SELF_REFERENCES_EAGERLY}
|column1|column2        |column3             |
|A      |foo<column3>bar|before<column1>after|

FORMATTING

FORMATTING transformer formats the table according to the max width of each column. Also this transformer removes the commented lines.

Example 4. Usage of FORMATTING transformer
{transformer=FORMATTING}
|key|
|value|
|-- |value|
|longValue|

REPLACING

REPLACING transformer replaces all substrings inside table values with replacement according to the given parameters.

Parameter Description

replacing

the sequence of chars to be replaced in the table cells

replacement

the replacing sequence of chars
Example 5. Usage of REPLACING transformer
{transformer=REPLACING, replacing=k, replacement=l}
|key  |
|vakue|

SORTING

SORTING transformer is designed to sort the table rows by the values in the columns.

Parameter Description

byColumns

column names ordered according to the priority of applying sorting (the column names are joined using |)

order

the optional sorting order: ASCENDING or DESCENDING. By default - ASCENDING.

sortingTypes

the optional sorting types ordered according to the column names from the byColumns parameter: STRING or NUMBER (only for number values). By default - STRING. The sorting types are joined using | (if one sorting type is defined, it applies to all column names from the byColumns parameter)
Example 6. Usage of SORTING transformer with DESCENDING sorting and sorting type by default (STRING). The following transformer:
{transformer=SORTING, byColumns=key1|key2, order=DESCENDING}
|key1|key2|key3|
|bb  |d   |1   |
|ba  |c   |2   |
|bb  |b   |3   |
|aa  |a   |4   |

will result in the following ExamplesTable:

|key1|key2|key3|
|bb  |d   |1   |
|bb  |b   |3   |
|ba  |c   |2   |
|aa  |a   |4   |
Example 7. Usage of SORTING transformer with DESCENDING sorting and NUMBER sorting type. The following transformer:
{transformer=SORTING, byColumns=key2, order=DESCENDING, sortingTypes=NUMBER}
|key1|key2|
|a   |10  |
|b   |2.3 |
|c   |10.2|

will result in the following ExamplesTable:

|key1|key2|
|c   |10.2|
|a   |10  |
|b   |2.3 |
Example 8. Usage of SORTING transformer with DESCENDING sorting and STRING sorting type. The following transformer:
{transformer=SORTING, byColumns=key2, order=DESCENDING, sortingTypes=STRING}
|key1|key2|
|a   |10  |
|b   |2.3 |
|c   |10.2|

will result in the following ExamplesTable:

|key1|key2|
|b   |2.3 |
|c   |10.2|
|a   |10  |
Example 9. Usage of SORTING transformer with sorting by default (ASCENDING) and STRING sorting type for column name key1 and NUMBER sorting type for column name key2. The following transformer:
{transformer=SORTING, byColumns=key1|key2, sortingTypes=STRING|NUMBER}
|key1|key2|index|
|a1  |10  |1    |
|a1  |2.3 |2    |
|a2  |10.2|3    |

will result in the following ExamplesTable:

|key1|key2 |index|
|a1   |2.3 |2    |
|a1   |10  |1    |
|a2   |10.2|3    |
Example 10. Usage of SORTING transformer with sorting by default (ASCENDING) and sorting type by default (STRING)
{transformer=SORTING, byColumns=key|key1}
|key   |key1  |
|value3|value4|
|value1|value2|

ITERATING

ITERATING transformer generates simple one-column table with header iterator and numbers as values: 0, 1, 2…​ .

Parameter Description

startInclusive

value specifying which number to start generating values from, it should be less than or equal to endInclusive value

endInclusive

threshold value specifying at which number to stop generating values, it should be greater than or equal to startInclusive value
Example 11. Usage of ITERATING transformer
{transformer=ITERATING, startInclusive=0, endInclusive=4}

REPEATING

REPEATING transformer generates ExamplesTable by copy-pasting the initial table the desired number of times.

Parameter Description

times

the number of times to copy-paste the initial table

The following REPEATING transformer:

{transformer=REPEATING, times=3}
|column1|column2|
|A      |C      |
|B      |D      |

will result in the following ExamplesTable:

|column1|column2|
|A      |C      |
|B      |D      |
|A      |C      |
|B      |D      |
|A      |C      |
|B      |D      |

FILTERING

FILTERING transformer filters the table using the specified parameters.

Parameter Description

byMaxColumns

The maximum number of columns to keep. If the number of columns in the original table is less than the value specified in this parameter, then all the original table columns are kept.

byMaxRows

The maximum number of rows to keep. If the number of rows in the original table is less than the value specified in this parameter, then all the original table rows are kept.

byRandomRows

The number of random rows to keep. If the number of rows in the original table is less than the value specified in this parameter, then all the original table rows are kept.

byColumnNames

The names of the columns to keep separated by semicolon.

byRowIndexes

The zero-based indexes of the rows to keep, allowing individual indexes and ranges of indexes separated by semicolons. The range of indexes is specified using the hyphen (-) symbol between two indexes (e.g., 0-5).

column.<column name>=<regex>

Select rows where the value in the specified column matches the given regex.

filtering by regex is not allowed to be used alongside with the following properties: byMaxColumns, byColumnNames, byMaxRows, byRowIndexes, byRandomRows.
byMaxColumns and byColumnNames are competing properties and only one can be specified at the same time. The same restriction is applied to the properties byMaxRows, byRandomRows and byRowIndexes.
In case if byColumnNames contains nonexistent table columns the exception with list of the invalid columns will be thrown.
Example 12. Usage of FILTERING transformer with filtering by column names and by max number of rows
{transformer=FILTERING, byColumnNames=key1;key3, byMaxRows=1}
|key1  |key2  |key3  |
|value1|value2|value3|
|value4|value5|value6|
Example 13. Usage of FILTERING transformer with filtering by max number of columns and by row indexes
{transformer=FILTERING, byMaxColumns=2, byRowIndexes=0;2-4;6}
|key1   |key2   |key3   |
|value1 |value2 |value3 |
|value4 |value5 |value6 |
|value7 |value8 |value9 |
|value10|value11|value12|
|value13|value14|value15|
|value16|value17|value18|
|value19|value20|value21|

will result in the following ExamplesTable:

|key1   |key2   |
|value1 |value2 |
|value7 |value8 |
|value10|value11|
|value13|value14|
|value19|value20|
Example 14. Usage of FILTERING transformer with filtering by column names and the number of random rows to keep
{transformer=FILTERING, byColumnNames=key2;key3, byRandomRows=2}
|key1  |key2  |key3  |
|value1|value2|value3|
|value4|value5|value6|
|value7|value8|value9|
Example 15. Usage of FILTERING transformer with filtering by column value matching regex
{transformer=FILTERING, column.sequence=^((?!bbb).)*$}
|sequence |
|aaabbbccc|
|qqqwwwlll|
|bbbfffttt|
|iiioooddd|
|uuuhhhbbb|

in the exampe above we select only those rows that do not contain bbb string in the sequence column, as a result we get the following ExamplesTable:

|sequence |
|qqqwwwlll|
|iiioooddd|

DISTINCTING

DISTINCTING transformer generates ExamplesTable from the specified columns, deleting the rows with repeating values.

Parameter Default value Description

byColumnNames

The names of the columns to keep (separated by semicolon).

keepAllColumns

false

If false, only the columns specified in byColumnNames will appear in the resulting table. Otherwise, all columns from the initial table will be included in the resulting table. If the initial table has rows with repeated values, the value from the first row will be retained in the resulting table.

This DISTINCTING transformer:

{transformer=DISTINCTING, byColumnNames=column1;column2}
|column1|column2|column3|
|A      |C      |0      |
|B      |D      |1      |
|A      |C      |1      |
|A      |C      |0      |
|B      |D      |1      |

will result in the following ExamplesTable:

|column1|column2|
|A      |C      |
|B      |D      |

This DISTINCTING transformer keeping all columns:

{transformer=DISTINCTING, byColumnNames=column1;column2, keepAllColumns=true}
|column1|column2|column3|
|A      |C      |0      |
|B      |D      |1      |
|A      |C      |1      |
|A      |C      |0      |
|B      |D      |0      |

will result in the following ExamplesTable:

|column1|column2|column3|
|A      |C      |0      |
|B      |D      |1      |

MERGING

MERGING transformer merges several tables into one.

Parameter Description

mergeMode

rows or columns

tables

paths to examples tables or transformers to merge

fillerValue

value to fill new cells while merging tables with different number of rows (in columns mode) or columns (in rows mode)

  • to merge tables with different number of rows or columns use fillerValue parameter to fill new cells

  • to merge tables in rows mode they must have the same numbers of columns + the same headers(keys)

  • to merge tables in columns mode they must have the same number of rows + unique headers(keys)

  • number of unique tables paths must be more than 1

  • when using transformers in tables parameter it is necessary to escape brackets, commas and semicolon in them: for unary nested- "", for double - "\" etc.

  • empty tables are allowed
Example 16. Usage of MERGING transformer with 'rows' merge mode
{transformer=MERGING, mergeMode=rows, tables=story/tables/test1.table;story/tables/test2.table}
Example 17. Usage of MERGING transformer with 'columns' merge mode
{transformer=MERGING, mergeMode=columns, tables=story/tables/test1.table;story/tables/test2.table;story/tables/test3.table}
Example 18. Usage of MERGING transformer with nested tables
{transformer=MERGING, mergeMode=columns, fillerValue=null, tables=\{transformer=FROM_EXCEL\, path=TestTemplate.xlsx\, sheet=Data\, addresses=A2\;A3\, column=test1\};\{transformer=FROM_EXCEL\, path=TestTemplate.xlsx\, sheet=Data\, range=B2:B4\, column=test2\}}
Example 19. Usage of nested MERGING transformers
{
 transformer=MERGING,
 mergeMode=rows,
 fillerValue=null,
 tables=
 \{
   transformer=MERGING\,
   mergeMode=columns\,
   tables=
        \\{
           transformer=FROM_EXCEL\\,
           path=TestTemplate.xlsx\\,
           sheet=Data\\,
           addresses=E6\\,
           column=text1
        \\}
        \;
        \\{
           transformer=FROM_EXCEL\\,
           path=TestTemplate.xlsx\\,
           sheet=Data\\,
           addresses=E7\\,
           column=text2
        \\}
 \}
 ;
 \{
   transformer=FROM_EXCEL\,
   path=TestTemplate.xlsx\,
   sheet=Data\,
   addresses=E8\,
   column=text1
 \}
}
Example 20. Usage of MERGING transformer with table body
{transformer=MERGING, mergeMode=columns, tables=/data/some-table.table}
|column1|
|value1 |

JOINING

JOINING transformer joins values from rows/columns of the table.

Parameter Description

joinMode

rows or columns

joinedColumn

[columns mode only] name of the new column with joined values

columnsToJoin

[columns mode only] colon-separated list of the columns to join
JOINING transformer should be the last while joining columns/rows of the table which created using another transformer.
Example 21. Usage of JOINING transformer with columns join mode
Scenario: Verify JOINING transformer in default columns mode
Then `<joinedColumn>` is equal to `A B`
Examples:
{transformer=JOINING, joinMode=columns, joinedColumn=joinedColumn}
|column1|column2|
|A      |B      |
Example 22. Usage of JOINING transformer with columns join mode and definition of columns to join
Scenario: Verify JOINING transformer in configured columns mode
Then `<joinedColumn>` is equal to `B D`
Examples:
{transformer=JOINING, joinMode=columns, joinedColumn=joinedColumn, columnsToJoin=column2;column4}
|column1|column2|column3|column4|
|A      |B      |C      |D      |
Example 23. Usage of JOINING transformer with rows join mode
Scenario: Verify JOINING transformer in default rows mode
Then `<column1>` is equal to `A B`
Then `<column2>` is equal to `C D`
Examples:
{transformer=JOINING, joinMode=rows}
|column1|column2|
|A      |C      |
|B      |D      |
Example 24. Usage of JOINING transformer in the chain of transformers
{transformer=MERGING, mergeMode=columns, tables=
  \{transformer=FROM_EXCEL\, path=/TestTemplate.xlsx\, sheet=Mapping\, range=A4:A5\, column=header1\, joinValues=true\};
  \{transformer=FROM_EXCEL\, path=/TestTemplate.xlsx\, sheet=Mapping\, range=B4:B5\, column=header2\, joinValues=true\}
}
{transformer=JOINING, joinMode=columns, joinedColumn=header}
Example 25. Usage of JOINING transformer with table body containing path to table file
{transformer=JOINING, joinMode=columns, joinedColumn=header}
/test1.table

CARTESIAN_PRODUCT

CARTESIAN_PRODUCT transformer creates a cartesian product from several tables.

Parameter Description

tables

paths to examples tables or transformers to create a cartesian product

  • headers of input tables must be different

  • number of input tables must be more than 1

  • if any of tables is empty - the resulting table will also be empty
Example 26. Usage of CARTESIAN_PRODUCT transformer
{transformer=CARTESIAN_PRODUCT, tables=story/tables/test1.table;story/tables/test2.table}
Example 27. Usage of CARTESIAN_PRODUCT transformer with table body
{transformer=CARTESIAN_PRODUCT, tables=/data/some-table.table}
|column1|
|value1 |
Example 28. Usage of CARTESIAN_PRODUCT transformer with nested transformers
{transformer=CARTESIAN_PRODUCT, tables=\{transformer=REPEATING\, times=100\};\{transformer=FROM_CSV\, csvPath=/data/csv.csv\}}

INDEXING

INDEXING transformer adds a column with the rows indices in the specified order. Indices are zero-based

Parameter Description

order

The mandatory indexing order: ASCENDING or DESCENDING.
Example 29. ASCENDING order example
{transformer=INDEXING, order=ASCENDING}
|key   |
|value |
|value2|
Example 30. ASCENDING order resulting table
|key   |index|
|value |0    |
|value2|1    |
Example 31. DESCENDING order example
{transformer=INDEXING, order=DESCENDING}
|key   |
|value |
|value2|
Example 32. DESCENDING order resulting table
|key   |index|
|value |1    |
|value2|0    |

INNER_JOIN

INNER_JOIN transformer combines rows from two tables whenever there are matching values between the columns.

Parameter Description

leftTableJoinColumn

the column name for matching in the left table

rightTableJoinColumn

the column name for matching in the right table

tables

ExamplesTable-s to join

  • left table is the first ExamplesTable declared in tables parameter,

  • right table is the second ExamplesTable declared in tables parameter or table body put under the transformer definition.

  • The number of used ExamplesTable-s must be equal to 2 (left and right).

  • The column names of input tables must be different (except the column names for matching).

  • If any of tables is empty - the resulting table will also be empty.
Example 33. Usage of INNER_JOIN transformer with table body
{transformer=INNER_JOIN, leftTableJoinColumn=joinID, rightTableJoinColumn=joinID, tables=/tables/some-table.table}
|joinID|column2|
|5     |row25  |
|3     |row23  |
|1     |row21  |

where ExamplesTable from /tables/some-table.table:

|joinID|column1|
|1     |row11  |
|2     |row12  |
|3     |row13  |
|4     |row14  |

will result in the following ExamplesTable:

|column1|joinID|column2|
|row11  |1     |row21  |
|row13  |3     |row23  |
Example 34. Usage of INNER_JOIN transformer with the same column names for matching
{transformer=INNER_JOIN, leftTableJoinColumn=joinID, rightTableJoinColumn=joinID, tables=/tables/test1.table;/tables/test2.table}
Example 35. Usage of INNER_JOIN transformer with the different column names for matching
{transformer=INNER_JOIN, leftTableJoinColumn=joinID1, rightTableJoinColumn=joinID2, tables=/tables/test1.table;/tables/test2.table}
Example 36. Usage of INNER_JOIN transformer with nested tables
{
 transformer=INNER_JOIN,
 leftTableJoinColumn=joinID,
 rightTableJoinColumn=joinID,
 tables=
 \{
   transformer=FROM_EXCEL\,
   path=TestTemplate.xlsx\,
   sheet=Data\, addresses=A2\;A3\,
   column=joinID
 \};
 \{
   transformer=FROM_EXCEL\,
   path=TestTemplate.xlsx\,
   sheet=Data\, range=B2:B4\,
   column=joinID
 \}
}