Vertica Documentation – Handling messy data

Data-Load: Saving load rejections (REJECTED DATA)

Mon, 01 Jan 0001 00:00:00 +0000

COPY load rejections are data rows that did not load due to a parser exception or, optionally, transformation error. By default, if you do not specify a rejected data file, COPY saves rejected data files to this location:

catalog_dir/CopyErrorLogs/target_table-source-copy-from-rejected-data.`*`n`*

catalog_dir/	The database catalog files directory, for example: `/home/dbadmin/VMart/v_vmart_node0001_catalog`
target_table	The table into which data was loaded (target_table).
source	The source of the load data, which can be STDIN, or a file name, such as `baseball.csv`.
`copy-from-rejected-data.`n	The default name for a rejected data file, followed by n suffix, indicating the number of files, such as `.1`, `.2, .3`. For example, this default file name indicates file 3 after loading from STDIN: `fw-STDIN-copy-from-rejected-data.3`.

Saving rejected data to the default location, or to a location of your choice, lets you review the file contents, resolve problems, and reload the data from the rejected data files. Saving rejected data to a table, lets you query the table to see rejected data rows and the reasons (exceptions) why the rows could not be parsed. Vertica recommends saving rejected data to a table.

Multiple rejected data files

Unless a load is very small (< 10MB), COPY creates more than one file to hold rejected rows. Several factors determine how many files COPY creates for rejected data. Here are some of the factors:

Number of sources being loaded
Total number of rejected rows
Size of the source file (or files)
Cooperative parsing and number of threads being used
UDLs that support apportioned loads
For your own COPY parser, the number of objects returned from prepareUDSources()

Naming conventions for rejected files

You can specify one or more rejected data files with the files you are loading. Use the REJECTED DATA parameter to specify a file location and name, and separate consecutive rejected data file names with a comma (,). Do not use the ON ANY NODE option because it is applicable only to load files.

If you specify one or more files, and COPY requires multiple files for rejected data, COPY uses the rejected data file names you supply as a prefix, and appends a numeric suffix to each rejected data file. For example, if you specify the name my_rejects for the REJECTED_DATA parameter, and the file you are loading is large enough (> 10MB), several files such as the following will exist:

my_rejects-1
my_rejects-2
my_rejects-3

COPY uses cooperative parsing by default, having the nodes parse a specific part of the file contents. Depending on the file or portion size, each thread generates at least one rejected data file per source file or portion, and returns load results to the initiator node. The file suffix is a thread index when COPY uses multiple threads (.1, .2, .3, and so on).

The maximum number of rejected data files cannot be greater than the number of sources being loaded, per thread to parse any portion. The resource pool determines the maximum number of threads. For cooperative parse, use all available threads.

If you use COPY with a UDL that supports apportioned load, the file suffix is an offset value. UDL's that support apportioned loading render cooperative parsing unnecessary. For apportioned loads, COPY creates at least one rejected file per data portion, and more files depending on the size of the load and number of rejected rows.

For all data loads except COPY LOCAL, COPY behaves as follows:

No rejected data file specified... Rejected data file specified...

For a single data file (pathToData or STDIN), COPY stores one or more rejected data files in the default location. For one data file, COPY interprets the rejected data path as a file, and stores all rejected data at the location. If more than one files is required from parallel processing, COPY appends a numeric suffix. If the path is not a file, COPY returns an error.

For multiple source files, COPY stores all rejected data in separate files in the default directory, using the source file as a file name prefix, as noted.

No rejected data file specified...	Rejected data file specified...
For a single data file (`pathToData` or `STDIN`), COPY stores one or more rejected data files in the default location.	For one data file, `COPY` interprets the rejected data path as a file, and stores all rejected data at the location. If more than one files is required from parallel processing, COPY appends a numeric suffix. If the path is not a file, `COPY` returns an error.
For multiple source files, COPY stores all rejected data in separate files in the default directory, using the source file as a file name prefix, as noted.	For multiple source files, `COPY` interprets the rejected path as a directory. `COPY` stores all information in separate files, one for each source. If path is not a directory, `COPY` returns an error. `COPY` accepts only one path per node. For example, if you specify the rejected data path as my_rejected_data, `COPY` creates a directory of that name on each node. If you provide more than one rejected data path, `COPY` returns an error.
Rejected data files are returned to the initiator node.	Rejected data files are not shipped to the initiator node.

For multiple source files, COPY interprets the rejected path as a directory. COPY stores all information in separate files, one for each source. If path is not a directory, COPY returns an error.

COPY accepts only one path per node. For example, if you specify the rejected data path as my_rejected_data, COPY creates a directory of that name on each node. If you provide more than one rejected data path, COPY returns an error.

Rejected data files are returned to the initiator node. Rejected data files are not shipped to the initiator node.

Maximum length of file names

Loading multiple input files in one statement requires specifying full path names for each file. Keep in mind that long input file names, combined with rejected data file names, can exceed the operating system's maximum length (typically 255 characters). To work around file names that exceed the maximum length, use a path for the rejected data file that differs from the default path—for example, \tmp\<shorter-file-name>.

Data-Load: Saving rejected data to a table

Mon, 01 Jan 0001 00:00:00 +0000

Use the REJECTED DATA parameter with the AS TABLE clause to specify a table in which to save rejected data. Saving rejected data to a file is mutually exclusive with using the AS TABLE clause.

When you use the AS TABLE clause, Vertica creates a new table if one does not exist, or appends to an existing table. If no parsing rejections occur during a load, the table exists but is empty. The next time you load data, Vertica inserts any rejected rows to the existing table.

The load rejection tables are a special type of table with the following capabilities and limitations:

Support SELECT statements
Can use DROP TABLE
Cannot be created outside of a COPY statement
Do not support DML and DDL activities
Are not K-safe

To make the data in a rejected table K-safe, you can do one of the following:

Write a CREATE TABLE..AS statement, such as this example:

=> CREATE TABLE new_table AS SELECT * FROM rejected_table;

Create a table to store rejected records, and run INSERT..SELECT operations into the new table

Using COPY NO COMMIT

If the COPY statement includes options NO COMMIT and REJECTED DATA AS TABLE, and the reject-table does not already exist, Vertica Analytic Database saves the rejected data table as a LOCAL TEMP table and returns a message that a LOCAL TEMP table is being created.

Rejected-data tables are useful for Extract-Load-Transform workflows, where you will likely use temporary tables more frequently. The rejected-data tables let you quickly load data and identify which records failed to load. If you load data into a temporary table that you created using the ON COMMIT DELETE clause, the COPY operation will not commit.

Location of rejected data table records

When you save rejected records to a table, using the REJECTED DATA AS TABLE table_name option, the data for the table is saved in a database data subdirectory, RejectionTableData. For example, for a VMart database, table data files reside here:

/home/dbadmin/VMart/v_vmart_node0001_data/RejectionTableData

Rejected data tables include both rejected data and the reason for the rejection (exceptions), along with other data columns, described next. Vertica suggests that you periodically drop any rejected data tables that you no longer require.

Querying a rejected data table

When you specify a rejected data table when loading data with COPY, you can query that table for information about rejected data after the load operation is complete. For example:

Create the loader table:

=> CREATE TABLE loader(a INT)
CREATE TABLE

Use COPY to load values, saving rejected data to a table, loader_rejects:

=> COPY loader FROM STDIN REJECTED DATA AS TABLE loader_rejects;
Enter data to be copied followed by a newline.
End with a backslash and a period on a line by itself.
>> 1
>> 2
>> 3
>> a
>> \.

Query the loader table after loading data:

=> SELECT * FROM loader;
 x
---
 1
 2
 3
(3 rows)

Query the loader_rejects table to see its column rows:

=> SELECT * FROM loader_rejects;
-[ RECORD 1 ]-------------+--------------------------------------------
node_name                 | v_vmart_node0001
file_name                 | STDIN
session_id                | v_vmart_node0001.example.-24016:0x3439
transaction_id            | 45035996274080923
statement_id              | 1
batch_number              | 0
row_number                | 4
rejected_data             | a
rejected_data_orig_length | 1
rejected_reason           | Invalid integer format 'a' for column 1 (x)

The rejected data table has the following columns:

Column	Data Type	Description
`node_name`	`VARCHAR`	The name of the Vertica node on which the input load file was located.
`file_name`	`VARCHAR`	The name of the file being loaded, which applies if you loaded a file (as opposed to using STDIN).
`session_id`	`VARCHAR`	The session ID number in which the COPY statement occurred.
`transaction_id`	`INTEGER`	Identifier for the transaction within the session, if any; otherwise NULL.
`statement_id`	`INTEGER`	The unique identification number of the statement within the transaction that included the rejected data. Tip You can use the `session_id`, `transaction_id`, and `statement_id` columns to create joins with many system tables. For example, if you join against the `QUERY_REQUESTS` table using those three columns, the `QUERY_REQUESTS.REQUEST` column contains the actual `COPY` statement (as a string) used to load this data.
`batch_number`	`INTEGER`	INTERNAL USE. Represents which batch (chunk) the data comes from.
`row_number`	`INTEGER`	The rejected row number from the input file, or -1 if it could not be determined. The value can be -1 when using cooperative parse. Each parse operation resets the row number, so in an apportioned load, there can be several entries with the same row number but different rows.
`rejected_data`	`LONG VARCHAR`	The data that was not loaded.
`rejected_data_orig_length`	`INTEGER`	The length of the rejected data.
`rejected_reason`	`VARCHAR`	The error that caused the rejected row. This column returns the same message that exists in a load exceptions file when you do not save to a table.

Exporting the rejected records table

You can export the contents of the column rejected_data to a file to capture only the data rejected during the first COPY statement. Then, correct the data in the file, save it, and load the updated file.

To export rejected records:

Create a sample table:

=> CREATE TABLE t (i int);
CREATE TABLE

Copy data directly into the table, using a table to store rejected data:

=> COPY t FROM STDIN REJECTED DATA AS TABLE t_rejects;
Enter data to be copied followed by a newline.
End with a backslash and a period on a line by itself.
>> 1
>> 2
>> 3
>> 4
>> a
>> b
>> c
>> \.

Show only tuples and set the output format:

=> \t
Showing only tuples.
=> \a
Output format is unaligned.

Output to a file:

=> \o rejected.txt
=> select rejected_data from t_rejects;
=> \o

Use the catcommand on the saved file:
```
=> \! cat rejected.txt
a
b
c
```

After a file exists, you can fix load errors and use the corrected file as load input to the COPY statement.

Data-Load: Saving load exceptions (EXCEPTIONS)

Mon, 01 Jan 0001 00:00:00 +0000

COPY exceptions consist of informational messages describing why a row of data could not be parsed. The EXCEPTIONS option lets you specify a file to which COPY writes exceptions. If you omit this option, COPY saves exception files to the following path: catalog-dir/CopyErrorLogs/tablename-sourcefilename-copy-from-exceptions, where:

catalog-dir is the directory holding the database catalog files
table is the name of the table being loaded into
sourcefile is the name of the file being loaded

Note

REJECTED DATA AS TABLE is mutually exclusive with EXCEPTIONS.

The file produced by the EXCEPTIONS option indicates the line number and the reason for each exception.

If copying from STDIN, the source file name is STDIN.

You can specify rejected data and exceptions files for individual files in a data load. Separate rejected data and exception file names with commas in the COPY statement.

You must specify a filename in the path to load multiple input files. Keep in mind that long table names combined with long data file names can exceed the operating system's maximum length (typically 255 characters). To work around file names exceeding the maximum length, use a path for the exceptions file that differs from the default path; for example, /tmp/<shorter-file-name>.

If you specify an EXCEPTIONS path:

For one data file, the path must be a file, and COPY stores all information in this file.
For multiple data files, the path must be a directory. COPY creates one file in this directory for each data file.
Exceptions files are not stored on the initiator node.
You can specify only one path per node.

If you do not specify the EXCEPTIONS path, COPY stores exception files in the default directory.

Data-Load: COPY rejected data and exception files

Mon, 01 Jan 0001 00:00:00 +0000

When executing a COPY statement, and parallel processing is ON (the default setting), COPY creates separate threads to process load files. Typically, the number of threads depends on the number of node cores in the system. Each node processes part of the load data. If the load succeeds overall, any parser rejections that occur during load processing are written to that node's specific rejected data and exceptions files. If the load fails, the rejected data file contents can be incomplete, or empty. If you do not specify a file name explicitly, COPY uses a default name and location for rejected data files. See the next topic for specifying your own rejected data and exception files.

Both rejected data and exceptions files are saved and stored on a per-node basis. This example uses multiple files as COPY inputs. Since the statement does not include either the REJECTED DATA or EXCEPTIONS parameters, rejected data and exceptions files are written to the default location, the database catalog subdirectory, CopyErrorLogs, on each node:

\set dir `pwd`/data/ \set remote_dir /vertica/test_dev/tmp_ms/
\set file1 '''':dir'C1_large_tbl.dat'''
\set file2 '''':dir'C2_large_tbl.dat'''
\set file3 '''':remote_dir'C3_large_tbl.dat'''
\set file4 '''':remote_dir'C4_large_tbl.dat'''
=>COPY large_tbl FROM :file1 ON site01,:file2 ON site01,
               :file3 ON site02,
               :file4 ON site02
               DELIMITER '|';

Specifying rejected data and exceptions files

The optional COPY REJECTED DATA and EXCEPTIONS parameters 'path' element lets you specify a non-default path in which to store the files.

If path resolves to a storage location, and the user invoking COPY is not a superuser, these are the required permissions:

The storage location must have been created (or altered) with the USER option (see CREATE LOCATION and ALTER_LOCATION_USE)
The user must already have been granted READ access to the storage location where the file(s) exist, as described in GRANT (storage location)

Both parameters also have an optional ON nodename clause that uses the specified path:

...[ EXCEPTIONS 'path' [ ON nodename ] [, ...] ]...[ REJECTED DATA 'path' [ ON nodename ] [, ...] ]

While 'path' specifies the location of the rejected data and exceptions files (with their corresponding parameters), the optional ON nodename clause moves any existing rejected data and exception files on the node to the specified path on the same node.

Saving rejected data and exceptions files to a single server

The COPY statement does not have a facility to merge exception and rejected data files after COPY processing is complete. To see the contents of exception and rejected data files requires accessing each node's specific files.

Note

To save all exceptions and rejected data files on a network host, be sure to give each node's files unique names, so that different cluster nodes do not overwrite other nodes' files. For instance, if you set up a server with two directories (/vertica/exceptions and /vertica/rejections), specify file names for each Vertica cluster node to identify each node, such as node01_exceptions.txt and node02_exceptions.txt. This way, each cluster node's files are easily distinguishable in the exceptions and rejections directories.

Using VSQL variables for rejected data and exceptions files

This example uses vsql variables to specify the path and file names to use with the exceptions and rejected data parameters (except_s1 and reject_s1). The COPY statement specifies a single input file (large_tbl) on the initiator node:

\set dir `pwd`/data/ \set file1 '''':dir'C1_large_tbl.dat'''
\set except_s1 '''':dir'exceptions'''
\set reject_s1 '''':dir'rejections'''

COPY large_tbl FROM :file1 ON site01 DELIMITER '|'
REJECTED DATA :reject_s1 ON site01
EXCEPTIONS :except_s1 ON site01;

This example uses variables to specify exception and rejected date files (except_s2 and reject_s2) on a remote node. The COPY statement consists of a single input file on a remote node (site02):

\set remote_dir /vertica/test_dev/tmp_ms/\set except_s2 '''':remote_dir'exceptions'''
\set reject_s2 '''':remote_dir'rejections'''

COPY large_tbl FROM :file1 ON site02 DELIMITER '|'
REJECTED DATA :reject_s2 ON site02
EXCEPTIONS :except_s2 ON site02;

This example uses variables to specify that the exception and rejected data files are on a remote node (indicated by :remote_dir). The inputs to the COPY statement consist of multiple data files on two nodes (site01 and site02). The exceptions and rejected data options use the ON nodename clause with the variables to indicate where the files reside (site01 and site02):

\set dir `pwd`/data/ \set remote_dir /vertica/test_dev/tmp_ms/
\set except_s1 '''':dir''''
\set reject_s1 '''':dir''''
\set except_s2 '''':remote_dir''''
\set reject_s2 '''':remote_dir''''
COPY large_tbl FROM :file1 ON site01,
               :file2 ON site01,
               :file3 ON site02,
               :file4 ON site02
               DELIMITER '|'
               REJECTED DATA :reject_s1 ON site01, :reject_s2 ON site02
               EXCEPTIONS :except_s1 ON site01, :except_s2 ON site02;

Data-Load: COPY LOCAL rejection and exception files

Mon, 01 Jan 0001 00:00:00 +0000

Invoking COPY LOCAL (or COPY LOCAL FROM STDIN) does not automatically create rejected data and exceptions files. This behavior differs from using COPY, which saves both files automatically, regardless of whether you use the optional REJECTED DATA and EXCEPTIONS parameters to specify either file explicitly.

Use the REJECTED DATA and EXCEPTIONS parameters with COPY LOCAL and COPY LOCAL FROM STDIN to save the corresponding output files on the client. If you do not use these options, rejected data parsing events (and the exceptions that describe them) are not retained, even if they occur.

You can load multiple input files using COPY LOCAL (or COPY LOCAL FROM STDIN). If you also use the REJECTED DATA and EXCEPTIONS options, the statement writes rejected rows and exceptions and to separate files. The respective files contain all rejected rows and corresponding exceptions, respectively, regardless of how many input files were loaded.

If COPY LOCAL does not reject any rows, it does not create either file.

Note

Because COPY LOCAL (and COPY LOCAL FROM STDIN) must write any rejected rows and exceptions to the client, you cannot use the [ON nodename ] clause with either the rejected data or exceptions options.

Specifying rejected data and exceptions files

To save any rejected data and their exceptions to files:

In the COPY LOCAL (and COPY LOCAL FROM STDIN) statement, use the REJECTED DATA 'path' and the EXCEPTIONS 'path' parameters, respectively.
Specify two different file names for the two options. You cannot use one file for both the REJECTED DATA and the EXCEPTIONS.
When you invoke COPY LOCAL or COPY LOCAL FROM STDIN, the files you specify need not pre-exist. If they do, COPY LOCAL must be able to overwrite them.

You can specify the path and file names with vsql variables:

\set rejected ../except_reject/copyLocal.rejected
\set exceptions ../except_reject/copyLocal.exceptions

Note

Using COPY LOCAL does not support storing rejected data in a table, as you can when using the COPY statement.

When you use the COPY LOCAL or COPY LOCAL FROM STDIN statement, specify the variable names for the files with their corresponding parameters:

=> COPY large_tbl FROM LOCAL rejected data :rejected exceptions :exceptions;
=> COPY large_tbl FROM LOCAL STDIN rejected data :rejected exceptions :exceptions;

Vertica Documentation – Handling messy data

Data-Load: Saving load rejections (REJECTED DATA)

Multiple rejected data files

Naming conventions for rejected files

Maximum length of file names

Data-Load: Saving rejected data to a table

Using COPY NO COMMIT

Location of rejected data table records

Querying a rejected data table

Tip

Exporting the rejected records table

Data-Load: Saving load exceptions (EXCEPTIONS)

Note

Data-Load: COPY rejected data and exception files

Specifying rejected data and exceptions files

Saving rejected data and exceptions files to a single server

Note

Using VSQL variables for rejected data and exceptions files

Data-Load: COPY LOCAL rejection and exception files

Note

Specifying rejected data and exceptions files

Note