Parquet Plugin

The plugin provides the ability to create Parquet files.

Installation

  1. Copy the below line to dependencies section of the project build.gradle file

    Please make sure to use the same version for all VIVIDUS dependencies.
    Example 1. build.gradle
    implementation(group: 'org.vividus', name: 'vividus-plugin-parquet', version: '0.6.4')
  2. If the project was imported to the IDE before adding new dependency, re-generate the configuration files for the used IDE and then refresh the project in the used IDE.

Properties

Property Name Acceptable values Default Description

expression.csv-to-parquet.csv-format

DEFAULT

Specifies the format of a CSV file to create a Parquet file from.

Expressions

The expression parameters marked with bold are mandatory.

convertCsvToParquetFile

Creates temporary Parquet file, writes CSV data records to the file using Avro schema and returns its path.

#{convertCsvToParquetFile($csv-resource-path, $avro-schema-resource-path)}
  • $csv-resource-path - The path to a resource containing CSV data.

  • $avro-schema-resource-path - The path to a resource containing Avro schema.

Example 2. Initialize a variable with a path to a Parquet file
Given I initialize scenario variable `parquet-path` with value `#{convertCsvToParquetFile(data/pets.csv, data/pets-schema.avsc)}`