Ot-Cad: Data Pipelines

Mon, 01 Jan 0001 00:00:00 +0000

Data pipeline

A data pipeline also known as a data loader is a declarative, automated way to process data continuously from external sources like Amazon S3. They allow you to define where, when, and how you need to load data into OTCAD with minimal manual intervention.

Data pipelines UI tour

You can view and manage data pipelines. After logging in to the OTCAD application, you land on the home page. Select the More options button and Data Pipelines. The Data pipelines page appears as shown in this image:

This page displays the following information in the Overview card:

Total pipelines - The Total Pipelines card displays the total number of data pipelines configured in the system.
Failed execution - The Failed Execution card displays the total number of data pipelines that failed to execute.
Active pipelines - The Active Pipelines card displays the total number of data pipelines that are in the Active status. The Pipelines area displays the following information:
Pipeline name - The name of the data pipeline.
Created by - The user ID of the person who created the data pipeline.
Data source - The source location of the files that contain the data to be loaded.
Schema - The tables, columns (fields), data types, and relationships among different tables in the database.
Destination table - The database table where data is written after it has been processed or transformed from a source table or other data source.
Last runtime - The timestamp at which the data pipeline was last run.
Last run status - Indicates the status of the data pipeline when it was last run.
Pipeline status - Indicates the present status of the data pipeline.

Option	Description
a	Duration: From this list, select the duration for which you need to view the data pipelines. You can choose to view data pipelines for any one of these durations: Last 24 hours Last week Last month Last 6 months Last year All time
b	Filter by: Select this option to filter data by different criteria.
c	Search pipeline name or schema: Search for a data pipeline based on the pipeline name or schema.
d	Actions: Select the options in this column to perform one of these operations on the data pipelines: View details of the data pipeline Edit the data pipeline Clone a data pipeline Pause schedule Delete

Create a data pipeline

You can create a data pipeline using either the AWS object store or Kafka data source.

Note

When creating data pipelines, ensure that mulitple database objects do not have the same name.

Create a data pipeline using AWS object store

AWS object store, primarily Amazon Simple Storage Service (S3) is a highly scalable, durable, and cost-effective cloud storage service for unstructured data. It stores data as objects in flat containers called buckets.

To create a data pipeline using AWS object store, do the following:

In the Data pipelines page, select +Create a pipeline. The Create a pipeline page is displayed.
In the Pipeline name field, enter the name of the pipeline.
Choose AWS object store.
In the Access key ID field, enter your AWS account access key id.
In the Secret access key field, enter your AWS account secret access key.

Note
Provide valid AWS credentials for the Access key ID and Secret access key. Invalid AWS credentials do not allow you to create a data pipeline.
From the Region list, select the region (geography) of the S3 bucket where the files are present.
In the S3 Bucket/File/Folder path field, enter the name or the folder path where the files are present.
Select the Data is encrypted option to specify the following parameters:
- Select either AWS Key Management Service Key or Customer managed keys if you wish to encrypt and load data into the S3 bucket.
- Select Encryption key ID.
Select Next.
In the Retry limit field, specify the number of times the system should attempt to retry a failed file load.
In the Parameters field, specify the copy parameters. For more information, see Parameters.
Select Next.
From the Destination table list, select the destination table to which you need to load the data.
Select Next.
Specify the schedule at which the data pipeline needs to run. Do one of the following:
- Select Schedule.
  - From the date pickers, select the Start date and End date.
  - In the Repeat every field, specify the duration at which the data pipeline needs to run.
  - From the Unit list, select the minute, hour, day, week, or month at which the data pipeline needs to run.
  - Select the On day option and specify the day on which the data pipeline needs to run.
  - Select the On option and specify the exact day and month on which the data pipeline needs to run.
  - Select the option Trigger when something is added to run the data pipeline when a file is added to the S3 bucket.
  - Enter the SQS credentials in the Access key ID, Secret access key, and Resource URL fields.
  <--Or-->
- Select Execute once.
Select Finish.

The data pipeline is created and displayed in the Data pipelines page.

Create a data pipeline using Kafka data source

Define data pipelines to ingest real-time data through Apache Kafka, an open-source distributed real-time streaming platform by leveraging Kafka topics for efficient streaming and processing. A Kafka topic is a category or feed name for streams of data, similar to a table in a database. Consumers read data from Kafka topics, with topics being organized into partitions for parallel processing. Each message in a topic is a record with a key, value, and timestamp. Kafka topics are logs where messages are ordered by an offset within each partition.

To create a data pipeline using Kafka data source, do the following:

In the Data pipelines page, select +Create a pipeline. The Create a pipeline page is displayed.
In the Pipeline name field, enter the name of the pipeline.

Choose Kafka.

Note

  Create a destination table from the SQL editor before creating the data pipeline.

In the Bootstrap servers field, enter the initial list of Kafka broker addresses that a Kafka client uses to connect to the Kafka cluster.

Note

  You can use only those bootstrapped servers that are whitelisted by OpenText. For assistance, contact the technical support team.

Note

  For multi-cluster configurations, provide the addresses as a comma-separated list. For example, 10.20.41.15:9092,10.20.41.16:9092.

Do one of the following:
- Select SSL if your Kafka broker is configured with SSL.
  
  SSL is short for Secure Sockets Layer. It is a protocol that creates an encrypted link between a web server and a web browser to ensure that all data transmitted between them is confidential.
- In the Client key field, enter the encrypted client key generated for the SSL certificate.
- In the Password field, enter the password associated with the encrypted client key.
- In the Client certificate field, enter the certificate used to authenticate the client with the Kafka broker.
- In the CA certificate field, enter the certificate authority (CA) certificate used to validate the Kafka broker.
<--Or-->
- Select SASL if your Kafka broker is authenticated with Simple Authentication and Security Layer (SASL).
  
  SASL is a Kafka framework for authenticating clients and brokers, which can be used with or without TLS/SSL encryption. It allows Kafka to support different authentication mechanisms.
- In the SASL mechanism list, select one of the following:
  - Plain - A simple username/password authentication mechanism used with TLS for encryption to implement secure authentication.
  - SCRAM - SHA - 256 - A secure password-based authentication method that uses a challenge-response protocol and the SHA-256 hashing algorithm to verify user credentials without sending the password in plain text over the network.
  - SCRAM - SHA - 512 - A secure authentication mechanism that uses the SHA-512 cryptographic hash function to verify a user's credentials in a "challenge-response" format, which prevents the password from being sent directly over the network.
  - In the Username field, enter a valid username.
  - In the Password field, enter the password for SASL authentication.
Select Proceed.
In the Define configuration area, specify the configuration settings for the data source.

In the Topic field, enter the Kafka topic. A Kafka topic is a logical grouping of messages, split into multiple partitions for parallelism.
In the Partition box, type or select the number of partitions for the Kafka topic. A partition is a sequence of records within a topic, stored on brokers and consumed independently.
In the Start offset box, type or select the incremental ID for each partition. The offset is a unique identifier for each record within a partition, used to track consumer progress.
Select +Add topic to add more topics and partitions.
Select one of the available parser options depending on the message type in the topic:
- AVRO - An Avro schema registry is a crucial component in systems that utilize Apache Kafka and Avro for data serialization. Its primary purpose is to centralize the management and evolution of Avro schemas, providing a robust mechanism for ensuring data compatibility and governance in streaming data pipelines.
- In the URL field, enter the schema registry URL.
- In the Subject field, enter the subject information from the schema registry.
- In the Version fieled, enter the version of the schema registry.
  
  <--Or-->
- In the External Schema field, enter the schema of the AVRO message. Ensure that the schema is in JSON format.
- JSON - JSON schema registry for Kafka provides a mechanism to define, manage, and enforce the structure of JSON data being produced to and consumed from Kafka topics. This ensures data consistency and compatibility, especially in distributed systems where multiple applications interact with the same data streams.
- Kafka - A Kafka schema registry is an external service that acts as a central repository for managing and validating schemas for data in a Kafka cluster. It allows producers to register schemas and consumers to retrieve them, ensuring data consistency and compatibility as schemas evolve over time.

Select Proceed.
From the Destination table list, select the destination table to which you need to load the data.
Note
```
    New data is added to the end of the selected table without changing existing records.
```
Select Proceed.
Specify the schedule at which the data pipeline needs to run. Do one of the following:

Select Schedule.
- From the date pickers, select the Start date and End date.
- In the Repeat every field, specify the duration at which the data pipeline needs to run.
- From the Unit list, select the minute, hour, day, week, or month at which the data pipeline needs to run.
- Select the On day option and specify the day on which the data pipeline needs to run.
- Select the On option and specify the exact day of the week on which the data pipeline needs to run.
<--Or-->
Select Execute once.

Select Finish.

The data pipeline is created and displayed in the Data pipelines page.

Note

     The data pipeline is created successfully only if the SQL query execution is successful. If the SQL query execution fails, the reason for the failure appears in a message. Resolve the SQL query to ensure successful query execution and data pipeline creation.

Create a data pipeline by uploading a file

You can create a data pipeline by uploading data from a file in your system.

In the Data pipelines page, select +Create a pipeline. The Create a pipeline page is displayed.
In the Pipeline name field, enter the name of the pipeline.
Choose Upload file. Ensure that a destination table is created and available before you proceed. If no destination table exists, create a new one from the SQL editor and continue to create the data pipeline.

Click Browse and choose the file that you need to upload. You can upload files that are of type .csv, .tsv, or .parquet only. Ensure that the file size is less than 1 GB.

Note

   All uploaded code or content is the sole responsibility of the uploading party. OpenText does not review, endorse, or accept any liability for uploaded content. Uploads are made at the uploading party's own risk.

From the File type list, select the type of file you are uploading. Files can be of type .csv, .tsv, or .parquet. For information about these file types, see:
In the Is header present area, choose Yes only if the file you upload has a header. Else, choose No.

If the file has a header row, it is displayed in the Preview area along with the first 5 rows of data. If no header row is present in the file, the columns are displayed with default header names such as ucol0, ucol1, and so on.
From the Delimiter list, select the character or sequence of characters to define boundaries between separate, independent data elements in the file. Select one of the following values:
- , - comma delimiter
- /t - tab delimiter
- | - pipe delimiter
- custom - Enter a single ASCII character in the range E'\000' to E'\177'.
  
  Note
  The fields Is header present and Delimiter are displayed only for files of type .csv or .tsv. These fields are not displayed if the file is of type .parquet.
Click Proceed.
From the Destination table list, select a destination table to which the data needs to be loaded.

Note
If the destination table has fewer columns than the source table, a message "There is a mismatch in source and destination columns" is displayed. However, if the destination table has more columns than the source table, the destination and source tables are mapped successfully. The extra columns in the destination table are filled with NULL values.
In the Map columns area, the source columns and destination columns are displayed.

In the Destination columns area, expand the list and select the required destination column. You can either choose to retain the default destination column or select a different one based on your requirement.

Note

     If the destination column has a different data type than the source and the system cannot coerce the data, the system inserts NULL into the column or returns an error. For example, when a source value of "ABC" (string) is mapped to a destination column of type INTEGER, coercion fails. The system inserts NULL or returns an error.

Click Create a pipeline.
The data pipeline is created and displayed in the Data pipelines page.

Filter a data pipeline

You can filter data pipelines based on certain criteria. Filter data pipelines in one of these ways:

A date range on which the data pipelines last ran.
- Select the Filter icon in the Data Pipelines page.
- Expand the Last run range drop-down list.
- In the date picker, select a date range (start date and end date) for which you need to view the data pipelines. For example, select a range from 1 May to 31 May to view data pipelines that were created in the month of May.
Schema
- Select the Filter icon in the Data Pipelines page.
- Expand the Schema drop-down list.
- Select the required schema.
Data source
- Select the Filter icon in the Data Pipelines page.
- Expand the Data source drop-down list.
- Select the data source or data sources for which you wish to view the data pipeline.
Pipeline status
- Select the Filter icon in the Data Pipelines page.
- Expand the Pipeline status drop-down list.
- Select Active to view the data pipelines that are active. A data pipeline is in the Active status when the schedule end date is a future date and there are one or more ingestions that are yet to complete.
- Select Inactive to view the data pipelines that are inactive. A data pipeline is in the Inactive status either when the end date is past or there are no ingestions that are yet to complete.
Id of the person who created the pipeline
- Select the Filter icon in the Data Pipelines page.
- Expand the Created by drop-down list.
- Select the user Id of the person who created the data pipeline.

Search a data pipeline

All data pipelines are displayed by default. You can search data pipelines using specific search criteria. To search data pipelines, do the following:

In the Data pipelines page, select +Search pipeline name or schema.
Enter one of the following search criteria:
- Pipeline name
- Owner of the data pipeline (Created by)
- Schema
- Destination table
You can sort the data pipeline by the pipeline name or the date on which it was last run. To sort the data pipelines in the ascending or descending order, select the Sort icon for the Pipeline name or Last run on column in the Data Pipelines page.

View pipeline details

You can view pipeline details in the Data pipelines page. To view the details of a data pipeline, do the following:

In the Data pipeline page, mouse over the Pipeline name column and select the +View details icon for a data pipeline. You can view the following details of the data pipeline in this page.
- Overview - Displays the owner (user ID) of the data pipeline.
- Configurations - Displays the source and destination paths of the data pipeline. In the Configurations card, click the Edit icon to edit the data source and destination paths of the data pipeline. For more information, see Creating a data pipeline.
- Total instances - Displays the total number of jobs. Select the drop-down list and view information about the instances for the last 24 hours, last week, last month, last 6 months, last one year, or all time.
- To view the reasons for the failure of a job, mouse over an instance with job status Failed and select the View error logs icon that appears for a file name. The error log provides information about the row that failed execution, reason for failure, and rejected data. You can troubleshoot the job with this information and ensure that this job is successfully executed.
- Incident Overview - Displays information about the data pipeline job in a pie chart. Select the drop-down list and view information about the jobs for the last 24 hours, last week, last month, last 6 months, last one year, or all time.
Select Execute pipeline to execute the selected data pipeline. Executing a data pipeline loads all files that have not already been loaded and that have not reached the retry limit. Executing the data pipeline commits the transaction.
Select Edit pipeline to edit the data pipeline.
In the Access key ID field, enter your AWS account access key id.
In the Secret access key field, enter your AWS account secret access key. All the details about the data pipeline are populated, except the Access key ID and Secret access key.
Note
```
    Provide valid AWS credentials for the **Access key ID** and **Secret access key**. Invalid AWS credentials do not allow you to edit a data pipeline.  
```
For more information about editing a data pipeline, see Creating a data pipeline.

Edit a data pipeline

After creating a data pipeline, you can edit the details to suit your requirements. To edit a data pipeline, do the following:

In the Data Pipelines page, mouse over the Pipeline name column and select the +Edit pipeline icon for a data pipeline.
In the Access key ID field, enter your AWS account access key id.
In the Secret access key field, enter your AWS account secret access key. All the details about the data pipeline are populated, except the Access key ID and Secret access key.
Note
```
     Provide valid AWS credentials for the **Access key ID** and **Secret access key**. Invalid AWS credentials do not allow you to edit a data pipeline.  
```
For more information about editing a data pipeline, see Creating a data pipeline.

Clone a data pipeline

You can create a clone or replica of an existing data pipeline. The configurations of the existing data pipeline are copied to the cloned data pipeline. You can edit these configuration settings in the cloned data pipeline.

In the Data Pipelines page, mouse over the Pipeline name column and select the Clone pipeline icon for a data pipeline.
In the Confirmation dialog, select Confirm. The Create a pipeline page displays.
In the Pipeline name field, enter the name of the pipeline.
In the Access key ID field, enter your AWS account access key id.

In the Secret access key field, enter your AWS account secret access key.

Note

     Provide valid AWS credentials for the **Access key ID** and **Secret access key**. Invalid AWS credentials do not allow you to edit a data pipeline.

Information in all other fields is pre-populated. You can edit this information.

Pause data ingestion

You can pause the ingestion of data into a pipeline.

In the Data pipelines page, click ⋮ in the Actions column.
Click +Pause schedule.
In the Confirmation dialog, click Pause. The Pipeline status changes to Paused.

Resume data ingestion

You can resume the ingestion of data into a pipeline.

In the Data pipelines page, select the pipeline for which data ingestion is paused.
Click ⋮ in the Actions column.
Click Resume schedule.
In the Confirmation dialog, click Resume. A message "Data ingestion resumed for data pipeline" is displayed.

Delete a data pipeline

You can delete a data pipeline that is no longer in use or required.

In the Data pipelines page, click ⋮ in the Actions column.
Click +Delete.
In the Confirmation dialog, click Delete.

OpenText Analytics Database 26.2.x – Overview of Data Pipelines

Ot-Cad: Data Pipelines

Data pipeline

Data pipelines UI tour

Create a data pipeline

Note

Create a data pipeline using AWS object store

Note

Create a data pipeline using Kafka data source

Note

Note

Note

Note

Note

Create a data pipeline by uploading a file

Note

Note

Note

Note

Filter a data pipeline

Search a data pipeline

View pipeline details

Note

Edit a data pipeline

Note

Clone a data pipeline

Note

Pause data ingestion

Resume data ingestion

Delete a data pipeline