HTML Plugin

The plugin provides the ability to work with HTML documents: data extraction and validation.

Installation

Copy the below line to dependencies section of the project build.gradle file

Please make sure to use the same version for all VIVIDUS dependencies.
Example 1. build.gradle
```
implementation(group: 'org.vividus', name: 'vividus-plugin-html')
```
If the project was imported to the IDE before adding new dependency, re-generate the configuration files for the used IDE and then refresh the project in the used IDE.

Table Transformers

FROM_HTML

FROM_HTML transformer generates a table based on the text content, HTML content or attributes of HTML elements found in the requested HTML page.

Parameter Name Description

pageUrl

The URL of the page to build the table upon.

The pageUrl parameter is deprecated and will be removed in VIVIDUS 0.7.0, please use variableName instead.

The pageUrl parameter can not be used together with the variableName or path parameter.

variableName

The name of the variable containing source HTML, only variables of scopes global and next_batches are allowed. Exceptions are cases when the transformer using in step which initializes a variable with a table.

The variableName parameter can not be used together with the pageUrl or path parameter.

path

The HTML resource name or the HTML file path.

The path parameter can not be used together with the variableName or pageUrl parameter.

column

The column name in the generated table.

xpathSelector

The XPath selector to select HTML elements in the HTML page.

By using XPath selector we can extract element’s HTML content, attributes and text content like its shown in the following example:

//a - extract the link HTML content, e.g. <a href="/hello">Say Hello</a>
//a/text() - extract the link text, e.g. Say Hello
//a/@href - extract the link href attribute, e.g. /hello

baseUri

The base URI can be used to get an absolute URL from an attribute that may be a relative URL: make sure to add the prefix abs: before the attribute name to xPath selector, e.g. //a/@abs:href. If defined pageUrl parameter as the base URI implicitly using value of this parameter.

Property Name Acceptable values Default Description

Property Name	Acceptable values	Default	Description
`transformer.from-html.headers.<header name>=<header value>`			Set of headers to set when requesting the page. `transformer.from-html.headers.x-vercel-protection-bypass=1fac2b25014d35e5103b`

transformer.from-html.headers.<header name>=<header value>

Set of headers to set when requesting the page.

transformer.from-html.headers.x-vercel-protection-bypass=1fac2b25014d35e5103b

Example 2. Given the following HTML page

<!DOCTYPE html>
<html>
    <body>
        <a href="/r">R</a>
        <a href="/g">G</a>
        <a href="/b">B</a>
    </body>
</html>

Example 3. Applyng FROM_HTML to the page

Examples:
{transformer=FROM_HTML, column=relative-url, pageUrl=https://mypage.com, xpathSelector=//a/@href}

Example 4. Output table

|relative-url|
|/r          |
|/g          |
|/b          |

Example 5. The transformer configuration based on local HTML file

transformer=FROM_HTML, column=relative-url, path=data/index.html, xpathSelector=//a/@href

Steps

Validate elements

Validates the number of elements found by the locator matches the specified rules.

Then number of elements found by $htmlLocatorType `$htmlLocator` in HTML `$html` is $comparisonRule `$number`

$htmlLocatorType - The HTML locator type, either CSS selector or XPath.
$htmlLocator - The actual locator.
$html - The HTML document to validate.
$comparisonRule - The comparison rule.
$quantity - The expected number of elements.

Example 6. Validate starfield release date

When I execute HTTP GET request for resource with URL `https://bethesda.net/en/game/starfield`
Then number of elements found by XPath `//h3[contains(., '111122')]` in HTML `${response}` is = `1`

Validate element text

Validates whether the element found by the locator contains the expected text.

Then element found by $htmlLocatorType `$htmlLocator` in HTML `$html` contains text `$expectedText`

$htmlLocatorType - The HTML locator type, either CSS selector or XPath.
$htmlLocator - The actual locator.
$html - The HTML document to validate.
$expectedText - The expected element text.

Example 7. Validate starfield release date

When I execute HTTP GET request for resource with URL `https://bethesda.net/en/game/starfield`
Then element found by XPath `//h3` in HTML `${response}` contains text `111122`

Save attribute value

Saves the attribute value into the variable.

When I save `$attributeName` attribute value of element found by $htmlLocatorType `$htmlLocator` in HTML `$html` to $scopes variable `$variableName`

$attributeName - The name of the attribute.
$htmlLocatorType - The HTML locator type, either CSS selector or XPath.
$htmlLocator - The actual locator.
$html - The HTML document to find element.
$scopes - The comma-separated set of the variables scopes.
$variableName - The name of the variable to save the attribute value.

Example 8. Validate paragraph title

Given I initialize story variable `html` with value
`
<html>
  <head>
    <title>Page Title</title>
    <script>//<![CDATA[Here comes the data//]]></script>
  </head>
  <body>
    <h1>This is a Heading</h1>
    <p title="paragraph">This is a paragraph.</p>
  </body>
</html>
`
When I save `title` attribute value of element found by CSS selector `p` in HTML `${html}` to scenario variable `title`
Then `${title}` is = `paragraph`

Save data/text of element

Saves the data or the text of element. Where the data is characters between the start-tag and end-tag of the element and the text is the textual content of the element without any inner element.

When I save $dataType of element found by $htmlLocatorType `$htmlLocator` in HTML `$html` to $scopes variable `variableName`

$dataType - Either data or text
$html - The HTML document to find element.
$htmlLocatorType - The HTML locator type, either CSS selector or XPath.
$htmlLocator - The actual locator.
- $scopes - The comma-separated set of the variables scopes.
- $variableName - The name of the variable to save the data or text.

Example 9. Validate data/text

Given I initialize story variable `html` with value
`
<html>
  <head>
    <title>Page Title</title>
    <script>//<![CDATA[Here comes the data//]]></script>
  </head>
  <body>
    <h1>This is a Heading</h1>
    <p title="paragraph">This is a paragraph.</p>
  </body>
</html>
`
When I save data of element found by <locatorType> `<script>` in HTML `${html}` to scenario variable `data`
When I save text of element found by <locatorType> `<header>` in HTML `${html}` to scenario variable `text`
Then `${data}` is equal to `//<![CDATA[Here comes the data//]]>`
Then `${text}` is equal to `This is a Heading`
Examples:
|locatorType |script  |header|
|CSS selector|script  |h1    |
|XPath       |//script|//h1  |

Save number of elements

Saves the number of elements contained in HTML document.

When I save number of elements found by $htmlLocatorType `$htmlLocator` in HTML `$html` to $scopes variable `$variableName`

$htmlLocatorType - The HTML locator type, either CSS selector or XPath.
$htmlLocator - The actual locator.
$html - The HTML document.
$scopes - The comma-separated set of the variables scopes.
$variableName - The name of the variable to save the number of elements.

Example 10. Validate number of Heading 1 elements

Given I initialize story variable `html` with value
`
<html>
  <body>
    <h1>This is a Heading</h1>
    <p title="paragraph">This is a paragraph.</p>
  </body>
</html>
`
When I save number of elements found by XPath `//h1` in HTML `${html}` to scenario variable `heading1`
Then `${heading1}` is equal to `1`