OpenText Analytics Database 26.2.x – Database export and import

Data-Export: Configuring connection security between clusters

Mon, 01 Jan 0001 00:00:00 +0000

When copying data between clusters, OpenText™ Analytics Database can encrypt both data and plan metadata.

Data is encrypted if you configure internode encryption (see Internode TLS).

For metadata, by default OpenText™ Analytics Database tries TLS first and falls back to plaintext. You can configure the database to require TLS and to fail if the connection cannot be made. You can also have the database verify the certificate and hostname before connecting.

Enabling TLS between clusters

To use TLS between clusters, you must first configure TLS between nodes:

Set the EncryptSpreadComms parameter.
Configure the data_channel TLS Configuration.
Set the ImportExportTLSMode parameter.

To specify the level of strictness when connecting to another cluster, set the ImportExportTLSMode configuration parameter. This parameter applies for both importing and exporting data. The possible values are:

PREFER: Try TLS but fall back to plaintext if TLS fails.
REQUIRE: Use TLS and fail if the server does not support TLS.

VERIFY_CA: Require TLS (as with REQUIRE), and also validate the other server's certificate using the CA specified by the "server" TLS Configuration's CA certificates (in this case, "ca_cert" and "ica_cert"):

=> SELECT name, certificate, ca_certificate, mode FROM tls_configurations WHERE name = 'server';
  name  |   certificate    |   ca_certificate   |   mode
--------+------------------+--------------------+-----------
 server | server_cert      | ca_cert,ica_cert   | VERIFY_CA
(1 row)

VERIFY_FULL: Require TLS and validate the certificate (as with VERIFY_CA), and also validate the server certificate's hostname.
REQUIRE_FORCE, VERIFY_CA_FORCE, and VERIFY_FULL_FORCE: Same behavior as REQUIRE, VERIFY_CA, and VERIFY_FULL, respectively, and cannot be overridden by CONNECT TO VERTICA.

ImportExportTLSMode is a global parameter that applies to all import and export connections you make using CONNECT TO VERTICA. You can override it for an individual connection.

For more information about these and other configuration parameters, see Security parameters.

Data-Export: Exporting data to another database

Mon, 01 Jan 0001 00:00:00 +0000

EXPORT TO VERTICA exports table data from one OpenText™ Analytics database to another. The following requirements apply:

You already opened a connection to the target database with CONNECT TO VERTICA.
The source database is no more than one major release behind the target database.
The table in the target database must exist.
Source and target table columns must have the same or compatible data types.

Each EXPORT TO VERTICA statement exports data from only one table at a time. You can use the same database connection for multiple export operations.

Export process

Exporting is a three-step process:

Connect to the target database with CONNECT TO VERTICA.

For example:

=> CONNECT TO VERTICA testdb USER dbadmin PASSWORD '' ON 'VertTest01', 5433;
CONNECT

Export the desired data with EXPORT TO VERTICA. For example, the following statement exports all table data in customer_dimension to a table of the same name in target database testdb:
```
=> EXPORT TO VERTICA testdb.customer_dimension FROM customer_dimension;
Rows Exported
---------------
         23416
(1 row)
```
DISCONNECT disconnects from the target database when all export and import operations are complete:
```
=> DISCONNECT testdb;
DISCONNECT
```
Note
Closing your session also closes the database connection. However, it is a good practice to explicitly close the connection to the other database, both to free up resources and to prevent issues with other SQL scripts that might be running in your session. Always closing the connection prevents potential errors if you run a script in the same session that attempts to open a connection to the same database, since each session can only have one connection to a given database at a time.

Mapping between source and target columns

If you export all table data from one database to another as in the previous example, EXPORT TO VERTICA can omit specifying column lists. This is possible only if column definitions in both tables comply with the following conditions:

Same number of columns
Identical column names
Same sequence of columns
Matching or compatible column data types

If any of these conditions is not true, the EXPORT TO VERTICA statement must include column lists that explicitly map source and target columns to each other, as follows:

Contain the same number of columns.
List source and target columns in the same order.
Pair columns with the same (or compatible) data types.

For example:

=> EXPORT TO VERTICA testdb.people (name, gender, age)
   FROM customer_dimension (customer_name, customer_gender, customer_age);

Exporting subsets of table data

In general, you can export a subset of table data in two ways:

Export data of specific source table columns.
Export the result set of a query (including historical queries) on the source table.

In both cases, the EXPORT TO VERTICA statement typically must specify column lists for the source and target tables.

The following example exports data from three columns in the source table to three columns in the target table. Accordingly, the EXPORT TO VERTICA statement specifies a column list for each table. The order of columns in each list determines how OpenText™ Analytics Database maps target columns to source columns. In this case, target columns name, gender, and age map to source columns customer_name, customer_gender, and customer_age, respectively:

=> EXPORT TO VERTICA testdb.people (name, gender, age) FROM customer_dimension
(customer_name, customer_gender, customer_age);
Rows Exported
---------------
         23416
(1 row)

The next example queries source table customer_dimension, and exports the result set to table ma_customers in target database testdb:

=> EXPORT TO VERTICA testdb.ma_customers(customer_key, customer_name, annual_income)
   AS SELECT customer_key, customer_name, annual_income FROM customer_dimension WHERE customer_state = 'MA';
Rows Exported
---------------
          3429
(1 row)

Note

In this example, the source and target column names are identical, so specifying a columns list for target table ma_customers is optional. If one or more of the queried source columns did not have a match in the target table, the statement would be required to include a columns list for the target table.

Exporting IDENTITY columns

You can export tables (or columns) that contain IDENTITY values, but the sequence values are not incremented automatically at the target table. You must use ALTER SEQUENCE to make updates.

Export IDENTITY columns as follows:

If both source and destination tables have an IDENTITY column and configuration parameter CopyFromVerticaWithIdentity is set to true (1), you do not need to list them.
If source table has an IDENTITY column, but target table does not, you must explicitly list the source and target columns.

Caution
Failure to list which IDENTITY columns to export can cause an error, because the IDENTITY column will be interpreted as missing in the destination table.

By default, EXPORT TO VERTICA exports all IDENTITY columns . To disable this behavior globally, set the CopyFromVerticaWithIdentity configuration parameter.

Data-Export: Copying data from another OpenText Analytics Database

Mon, 01 Jan 0001 00:00:00 +0000

COPY FROM VERTICA imports table data from one OpenText™ Analytics database to another. The following requirements apply:

You already opened a connection to the target database with CONNECT TO VERTICA.
The source database is no more than one major release behind the target database.
The table in the target database must exist.
Source and target table columns must have the same or compatible data types.

Import process

Importing is a three-step process:

Connect to the source database with CONNECT TO VERTICA. For example:

=> CONNECT TO VERTICA vmart USER dbadmin PASSWORD '' ON 'VertTest01',5433;
CONNECT

Import the desired data with COPY FROM VERTICA. For example, the following statement imports all table data in customer_dimension to a table of the same name:
```
=> COPY customer_dimension FROM  VERTICA vmart.customer_dimension;
 Rows Loaded
-------------
      500000
(1 row)
=> DISCONNECT vmart;
DISCONNECT
```
Note
Successive COPY FROM VERTICA statements in the same session can import data from multiple tables over the same connection.
DISCONNECT disconnects from the source database when all import and export operations are complete:
```
=> DISCONNECT vmart;
DISCONNECT
```
Note
Closing your session also closes the database connection. However, it is a good practice to explicitly close the connection to the other database, both to free up resources and to prevent issues with other SQL scripts that might be running in your session. Always closing the connection prevents potential errors if you run a script in the same session that attempts to open a connection to the same database, since each session can only have one connection to a given database at a time.

Importing IDENTITY columns

You can import IDENTITY columns as follows:

If both source and destination tables have an IDENTITY column and configuration parameter CopyFromVerticaWithIdentity is set to true (1), you do not need to list them.
If source table has an IDENTITY column, but target table does not, you must explicitly list the source and target columns.

Caution
Failure to list which IDENTITY columns to export can cause an error, because the IDENTITY column will be interpreted as missing in the destination table.

After importing the columns, the IDENTITY column values do not increment automatically. Use ALTER SEQUENCE to make updates.

The default behavior for this statement is to import IDENTITY columns by specifying them directly in the source table. To disable this behavior globally, set the CopyFromVerticaWithIdentity configuration parameter.

Data-Export: Copy and export data on AWS

Mon, 01 Jan 0001 00:00:00 +0000

There are common issues that occur when exporting or copying on AWS clusters, as described below. Except for these specific issues as they relate to AWS, copying and exporting data works as documented in Database export and import.

To copy or export data on AWS:

Verify that all nodes in source and destination clusters have their own elastic IPs (or public IPs) assigned.

If your destination cluster is located within the same VPC as your source cluster, proceed to step 3. Each node in one cluster must be able to communicate with each node in the other cluster. Thus, each source and destination node needs an elastic IP (or public IP) assigned.

(For non-CloudFormation Template installs) Create an S3 gateway endpoint.

If you aren't using a CloudFormation Template (CFT) to install Vertica, you must create an S3 gateway endpoint in your VPC. For more information, see the AWS documentation.

For example, the Vertica CFT has the following VPC endpoint:

"S3Enpoint" : {
    "Type" : "AWS::EC2::VPCEndpoint",
    "Properties" : {
    "PolicyDocument" : {
        "Version":"2012-10-17",
        "Statement":[{
        "Effect":"Allow",
        "Principal": "*",
        "Action":["*"],
        "Resource":["*"]
        }]
    },
    "RouteTableIds" : [ {"Ref" : "RouteTable"} ],
    "ServiceName" : { "Fn::Join": [ "", [ "com.amazonaws.", { "Ref": "AWS::Region" }, ".s3" ] ] },
    "VpcId" : {"Ref" : "VPC"}
}

Verify that your security group allows the AWS clusters to communicate.

Check your security groups for both your source and destination AWS clusters. Verify that ports 5433 and 5434 are open. If one of your AWS clusters is on a separate VPC, verify that your network access control list (ACL) allows communication on port 5434.

Note
Note: This communication method exports and copies (imports) data across the Internet. You can alternatively use non-public IPs and gateways, or VPN to connect the source and destination clusters.
If there are one or more elastic load balancers (ELBs) between the clusters, verify that port 5433 is open between the ELBs and clusters.
If you use the OpenText™ Analytics Database client to connect to one or more ELBs, the ELBs only distribute incoming connections. The data transmission path occurs between clusters.

Data-Export: Changing node export addresses

Mon, 01 Jan 0001 00:00:00 +0000

You can change the export address for your OpenText™ Analytics Database cluster. You might need to do so to export data between clusters in different network subnets.

Create a subnet for importing and exporting data between database clusters. The CREATE SUBNET statement identifies the public network IP addresses residing on the same subnet.
```
=> CREATE SUBNET kv_subnet with '10.10.10.0';
```
Alter the database to specify the subnet name of a public network for import/export.
```
=> ALTER DATABASE DEFAULT EXPORT ON kv_subnet;
```
Create network interfaces for importing and exporting data from individual nodes to other database clusters. The CREATE NETWORK INTERFACE statement identifies the public network IP addresses residing on multiple subnets.
```
=> CREATE NETWORK INTERFACE kv_node1 on v_VMartDB_node0001 with '10.10.10.1';
=> CREATE NETWORK INTERFACE kv_node2 on v_VMartDB_node0002 with '10.10.10.2';
=> CREATE NETWORK INTERFACE kv_node3 on v_VMartDB_node0003 with '10.10.10.3';
=> CREATE NETWORK INTERFACE kv_node4 on v_VMartDB_node0004 with '10.10.10.4';
```
For users on Amazon Web Services (AWS) or using Network Address Translation (NAT), refer to OpenText Analytics Database on Amazon Web Services.

Alter the node settings to change the export address. When used with the EXPORT ON clause, the ALTER NODE specifies the network interface of the public network on individual nodes for importing and exporting data.

=> ALTER NODE v_VMartDB_node0001 export on kv_node1;
=> ALTER NODE v_VMartDB_node0002 export on kv_node2;
=> ALTER NODE v_VMartDB_node0003 export on kv_node3;
=> ALTER NODE v_VMartDB_node0004 export on kv_node4;

Verify if the node address and the export address are different on different network subnets of the database cluster.

=> SELECT node_name, node_address, export_address FROM nodes;
     node_name     | node_address    | export_address
-------------------+-----------------+----------------
v_VMartDB_node0001 | 192.168.100.101 | 10.10.10.1
v_VMartDB_node0002 | 192.168.100.102 | 10.10.10.2
v_VMartDB_node0003 | 192.168.100.103 | 10.10.10.3
v_VMartDB_node0004 | 192.168.100.104 | 10.10.10.4

Creating a network interface and altering the node settings to change the export address takes precedence over creating a subnet and altering the database for import/export.

Data-Export: Using public and private IP networks

Mon, 01 Jan 0001 00:00:00 +0000

In many configurations, OpenText™ Analytics Database cluster hosts use two network IP addresses as follows:

A private address for communication between the cluster hosts.
A public IP address for communication with client connections.

By default, importing from and exporting to another database uses the private network.

Note

Ensure port 5433 or the port the database is using is not blocked.

To use the public network address for copy and export activities, as well as moving large amounts of data, configure the system to use the public network to support exporting to or importing from another database cluster:

The database encrypts data during transmission (if you have configured a certificate). The database attempts to also encrypt plan metadata but, by default, falls back to plaintext if needed. You can configure the database to require encryption for metadata too; see Configuring connection security between clusters.

In certain instances, both public and private addresses exceed the demand capacity of a single Local Area Network (LAN). If you encounter this type of scenario, then configure your database cluster to use two LANs: one for public network traffic and one for private network traffic.

Data-Export: Handling node failure during copy/export

Mon, 01 Jan 0001 00:00:00 +0000

When an export (EXPORT TO VERTICA) or import (COPY FROM VERTICA) task is in progress, and a non-initiator node fails, OpenText™ Analytics Database does not complete the task automatically. A non-initiator node is any node that is not the source or target node in your export or import statement. To complete the task, you must run the statement again.

You address the problem of a non-initiator node failing during an import or export as follows:

Note

Both OpenText™ Analytics databases must be running in a safe state.

You export or import from one cluster to another using the EXPORT TO VERTICA or COPY FROM VERTICA statement.

During the export or import, a non-initiating node on the target or source cluster fails. The database displays an error message that indicates possible node failure, one of the following:
- ERROR 4534: Receive on v_tpchdb1_node0002: Message receipt from v_tpchdb2_node0005 failed
- WARNING 4539: Received no response from v_tpchdb1_node0004 in abandon plan
- ERROR 3322: [tpchdb2] Execution canceled by operator
Complete your import or export by running the statement again. The failed node does not need to be up for the database to successfully complete the export or import.

Data-Export: Using EXPORT functions

Mon, 01 Jan 0001 00:00:00 +0000

OpenText™ Analytics Database provides several EXPORT_ functions that let you recreate a database, or specific schemas and tables, in a target database. For example, you can use the EXPORT_ functions to transfer some or all of the designs and objects you create in a development or test environment to a production database.

The EXPORT_ functions create SQL scripts that you can run to generate the exported database designs or objects. These functions serve different purposes to the export statements, COPY FROM VERTICA (pull data) and EXPORT TO VERTICA (push data). These statements transfer data directly from source to target database across a network connection between both. They are dynamic actions and do not generate SQL scripts.

The EXPORT_ functions appear in the following table. Depending on what you need to export, you can use one or more of the functions. EXPORT_CATALOG creates the most comprehensive SQL script, while EXPORT_TABLES and EXPORT_OBJECTS are subsets of that function to narrow the export scope.

Use this function...	To recreate...
EXPORT_CATALOG	These catalog items: An existing schema design, tables, projections, constraints, views, and stored procedures. The Database Designer-created schema design, tables, projections, constraints, and views A design on a different cluster.
EXPORT_TABLES	Non-virtual objects up to, and including, the schema of one or more tables.
EXPORT_OBJECTS	Catalog objects in order dependency for replication.

Use this function...

To recreate...

EXPORT_CATALOG

These catalog items:

An existing schema design, tables, projections, constraints, views, and stored procedures.
The Database Designer-created schema design, tables, projections, constraints, and views
A design on a different cluster.

EXPORT_TABLES

Non-virtual objects up to, and including, the schema of one or more tables.

EXPORT_OBJECTS

Catalog objects in order dependency for replication.

The designs and object definitions that the script creates depend on the EXPORT_ function scope you specify. The following sections give examples of the commands and output for each function and the scopes it supports.

Saving scripts for export functions

All of the examples in this section were generated using the standard VMART database, with some additional test objects and tables. One output directory was created for all SQL scripts that the functions created:

/home/dbadmin/xtest

If you specify the destination argument as an empty string (''), the function writes the export results to STDOUT.

Note

A superuser can export all available database output to a file with the EXPORT_ functions. For a non-superuser, the EXPORT_ functions generate a script containing only the objects to which the user has access.

OpenText Analytics Database 26.2.x – Database export and import

Data-Export: Configuring connection security between clusters

Enabling TLS between clusters

Data-Export: Exporting data to another database

Export process

Note

Mapping between source and target columns

Exporting subsets of table data

Note

Exporting IDENTITY columns

Caution

Data-Export: Copying data from another OpenText Analytics Database

Import process

Note

Note

Importing IDENTITY columns

Caution

Data-Export: Copy and export data on AWS

Note

Data-Export: Changing node export addresses

Data-Export: Using public and private IP networks

Note

Data-Export: Handling node failure during copy/export

Note

Data-Export: Using EXPORT functions

Saving scripts for export functions

Note