Azure Data Factory Plugin

The plugin provides functionality to interact with Azure Data Factory

Installation

Example 1. build.gradle
implementation(group: 'org.vividus', name: 'vividus-plugin-azure-data-factory', version: '0.3.11')

Configuration

Authentication

The authentication process relies on the configuration of the environment variables.

See the official "Azure identity" guide to get more details on what types of authentication could be used.

Azure environment selection

Azure environment could be optionally specified using either global property azure.environment (sets the environment for all Azure plugins) or plugin-specific property azure.data-factory.environment. The plugin-specific property takes precedence over the global one. The default value is AZURE.

The supported environments are only:

  • AZURE

  • AZURE_CHINA

  • AZURE_GERMANY

  • AZURE_US_GOVERNMENT

Azure subscription selection

Azure subscription must be configured via AZURE_SUBSCRIPTION_ID environment variable.

Steps

Run Pipeline with expected status

Creates a run of a pipeline in Data Factory, waits for its completion or until the timeout is reached and validates the run status is equal to the expected one.

When I run pipeline `$pipelineName` in Data Factory `$factoryName` from resource group `$resourceGroupName` with wait timeout `$waitTimeout` and expect run status to be equal to `$expectedPipelineRunStatus`

  • $pipelineName - The name of the pipeline to run.

  • $factoryName - The name of the factory.

  • $resourceGroupName - The name of the resource group of the factory.

  • $waitTimeout - The maximum duration of time to wait for the pipeline completion in ISO-8601 format.

  • $expectedPipelineRunStatus - The expected pipeline run status, e.g. Succeeded.

Example 2. Run pipeline
When I run pipeline `vividus-pipeline` in Data Factory `vividus-data-factory` from resource group `vividus-resource-group-ingestion` with wait timeout `PT30S` and expect run status to be equal to `Succeeded`

Run Pipeline

This step is deprecated and will be removed in VIVIDUS 0.4.0. The replacement is the step accepting an expected status as input parameter.

Creates a run of a pipeline in Data Factory, waits for its completion or until the timeout is reached and validates the run status is Succeeded.

When I run pipeline `$pipelineName` in Data Factory `$factoryName` from resource group `$resourceGroupName` with wait timeout `$waitTimeout`

  • $pipelineName - The name of the pipeline to run.

  • $factoryName - The name of the factory.

  • $resourceGroupName - The name of the resource group of the factory.

  • $waitTimeout - The maximum duration of time to wait for the pipeline completion in ISO-8601 format.

Example 3. Run pipeline
When I run pipeline `vividus-pipeline` in Data Factory `vividus-data-factory` from resource group `vividus-resource-group-ingestion` with wait timeout `PT30S`