Mc: About configuring a data load from S3

Mon, 01 Jan 0001 00:00:00 +0000

When you create an S3 Data Load using MC, you have the option of further configuring the load operation. You can optionally specify the following:

Add COPY parameters

MC performs the load operation with COPY. You can use the COPY Parameters field to further configure the COPY operation. This field accepts parameters that are specified after the COPY statement's FROM clause. For details on these parameters and special requirements, see Parameters.

Note

The FILTER and PARSER parameters must appear in that order and precede all other parameters.

For example, you can specify the DELIMITER and SKIP parameters to separate columns with a comma, and skip one record of input data, respectively:

DELIMITER ',' SKIP 1

You can also add comments in this field with standard C comment notation.

Note

This field does not support SQL comment notation (double hyphen --).

Capture rejected data in a table

Set Capture rejected data in a table to Yes to create a table that contains rejected row data. You can view this data in the Load History tab.

This table uses the following naming convention:

schema.s3_load_rejections_target-table-name

You must have CREATE privilege on the schema if the table doesn't already exist. When you invoke multiple load processes for the same target table, MC appends all rejections data to the same table. For details, see Saving rejected data to a table.

Set a rejected records maximum

Set Reject Max to the maximum number of rows that can be rejected before the load operation fails. If COPY rejects the specified maximum rows, OpenText™ Analytics Database rolls back the entire load operation.

Mc: Viewing load history

Mon, 01 Jan 0001 00:00:00 +0000

You can view a history of all your continuous and instance loading jobs in OpenText™ Analytics Database on the Data Load Activity page.

Continuous jobs: Loading jobs that continuously monitor a source and stream data from the source.
Instance jobs: Loading jobs that batch load from a source. Instance jobs are of a fixed length and shorter-term than continuous loads.

View continuous loads

The Continuous tab on the Data Load Activity page displays history of your database’s continuous loading jobs. For example, you can see loading jobs you create using the database integration with Kafka (see Apache Kafka integration). Additionally, if you enable the MC extended monitoring feature, the Continuous tab displays the continuous jobs that stream data from your monitored database to a storage database. (See Extended monitoring for more on how MC can use Kafka to monitor databases externally.)

Use the Continuous tab to view details about continuous jobs, such as their source, target tables, and other microbatch configuration details.

If extended monitoring is enabled, jobs streaming to the MC storage database show mc_dc_kafka_config as the scheduler name. Deselect Show MC data collector monitoring streams at the top of the tab to remove these jobs from the display.

In the Continuous tab, click the labels in the Scheduler, Microbatch, and Errors Last Hour to view additional details about those loading jobs.

For more on continuous data streaming terminology, see Data streaming integration terms.

View load instances

In the Instance tab, you can see a history of your database's one-time loading jobs. For example, you can view instance jobs you created using the COPY command in vsql (see COPY), or instance jobs you created in MC to copy data from an Amazon S3 bucket. (For more about initiating loading jobs in MC, see Loading data from Amazon S3 using MC.)

In the Instance tab, click the labels in the Status column and Rejected Rows column to view more details about completed jobs. For more about rejected rows, see Handling messy data.

The number of load history results on the Instance tab depends on the Data collector retention policy for Requests Issued and Requests Completed. To change the retention policy, see Configuring data retention policies.

OpenText Analytics Database 26.2.x – Loading data from Amazon S3 using MC

Mc: About configuring a data load from S3

Add COPY parameters

Note

Note

Capture rejected data in a table

Set a rejected records maximum

See also

Mc: Viewing load history

View continuous loads

View load instances

See also