Migrating an enterprise database to Eon Mode

Vertica meta-function MIGRATE_ENTERPRISE_TO_EON migrates an Enterprise database to Eon Mode. The migration process includes the following stages:

1. Check migration prerequisites

2. Verify compliance

3. Execute the migration

4. Check migration results

5. Activate the Eon database

Migration prerequisites

The following conditions must be true; otherwise, MIGRATE_ENTERPRISE_TO_EON returns with an error:

• The source Enterprise database version must be ≥10.0.

• All nodes in the source database must be in an UP state and of type PERMANENT or EPHEMERAL. Verify by querying system table NODES:

  => SELECT node_name, node_type, node_state FROM nodes;
      node_name     | node_type | node_state
  ------------------+-----------+------------
   v_vmart_node0001 | PERMANENT | UP
   v_vmart_node0002 | PERMANENT | UP
   v_vmart_node0003 | PERMANENT | UP
  (3 rows)
  

  Tip

  When planning how to provision the target Eon database, consider that MIGRATE_ENTERPRISE_TO_EON creates the same number of nodes as on the original Enterprise database , and an equal number of segmented shards. Choose the appropriate instance type accordingly.

• The source database must be configured as an elastic cluster. By default, any database created since Vertica release 9.2.1 is configured as an elastic cluster. To verify whether an Enterprise database is configured as an elastic cluster, query system table ELASTIC_CLUSTER:

  => SELECT is_enabled FROM elastic_cluster;
   is_enabled
  ------------
   t
  (1 row)
  

  If the query returns false, call meta-function ENABLE_ELASTIC_CLUSTER on the Enterprise database.

• The source Enterprise database must configure Eon parameters as required by the target Eon object store (see Configuration Requirements below).

• The database must not contain projections that are unsupported by Eon.

Unsupported projections

Eon databases do not support four types of projections, as described below. If MIGRATE_ENTERPRISE_TO_EON finds any of these projection types in the Enterprise database, it rolls back the migration and reports the offending projections or their anchor tables in the migration error log. For example:

The following projections are inconsistent with cluster segmentation. Rebalance them with REBALANCE_CLUSTER() or REBALANCE_TABLE():
Projection(Anchor Table): public.incon1_p1_b0(public.incon1)

Configuration requirements

Before migration, set the following configuration parameters on the source database.

KerberosServiceName
KerberosRealm
KerberosKeytabFile

* Migration only supports [Vertica authentication](/en/hadoop-integration/accessing-kerberized-hdfs-data/using-kerberos-with/#VerticaAuthentication). User authentication through [delegation tokens or proxy users](/en/hadoop-integration/accessing-kerberized-hdfs-data/proxy-users-and-delegation-tokens/) is not supported.

Migration does not support authentication of multiple Kerberos realms.

Compliance verification

Before running migration, check whether the Enterprise source database complies with all migration requirements. You do so by setting the last Boolean argument of MIGRATE_ENTERPRISE_TO_EON to true. This optional argument—by default set to false—specifies whether the meta-function only performs a compliance check. For example:

=> SELECT migrate_enterprise_to_eon('s3://verticadbbucket', '/vertica/depot', true);

If the meta-function discovers any compliance issues, it writes these to the migration error log migrate_enterprise_to_eon_error.log in the database directory.

Migration execution

MIGRATE_ENTERPRISE_TO_EON migrates an Enterprise database to an Eon Mode database. For example:

=> SELECT migrate_enterprise_to_eon('s3://verticadbbucket', '/vertica/depot', false);

If the meta-function's last Boolean argument is omitted or set to false, it executes the migration. MIGRATE_ENTERPRISE_TO_EON runs in the foreground, and until it returns—either with success or an error—it blocks all operations in the same session on the source Enterprise database. If successful, MIGRATE_ENTERPRISE_TO_EON returns with a list of nodes in the migrated database. You can then proceed to revive the migrated Eon database.

Handling interrupted migration

If migration is interrupted before the meta-function returns—for example, the client disconnects, or a network outage occurs—the migration errors out. In this case, call MIGRATE_ENTERPRISE_TO_EON to restart migration.

Communal storage of the target database retains data that was already copied before the error occurred. When you call MIGRATE_ENTERPRISE_TO_EON to resume migration, the meta-function first checks the data on communal storage and only copies unprocessed data from the source database.

Repeating migration

You can repeat migration multiple times to the same communal storage location. This can be useful for backfilling changes that occurred in the source database during the previous migration.

The following constraints apply:

• You can migrate from only one database to the same communal storage location.

• After reviving the newly migrated Eon database, you cannot migrate again to its communal storage, unless you first drop the database and then clean up storage.

Monitoring migration

System table DATABASE_MIGRATION_STATUS displays the progress of a migration in real time, and also stores data from previous migrations. The following example shows data of a migration that is in progress:

=> SELECT node_name, phase, status, bytes_to_transfer, bytes_transferred, communal_storage_location FROM database_migration_status ORDER BY node_name, start_time;
     node_name    |       phase        |  status   | bytes_to_transfer | bytes_transferred | communal_storage_location
------------------+--------------------+-----------+-------------------+------------------+---------------------------
 v_vmart_node0001 | Catalog Conversion | COMPLETED |                 0 |                0 | s3://verticadbbucket/
 v_vmart_node0001 | Data Transfer      | COMPLETED |              1134 |             1134 | s3://verticadbbucket/
 v_vmart_node0001 | Catalog Transfer   | COMPLETED |              3765 |             3765 | s3://verticadbbucket/
 v_vmart_node0002 | Catalog Conversion | COMPLETED |                 0 |                0 | s3://verticadbbucket/
 v_vmart_node0002 | Data Transfer      | COMPLETED |              1140 |             1140 | s3://verticadbbucket/
 v_vmart_node0002 | Catalog Transfer   | COMPLETED |              3766 |             3766 | s3://verticadbbucket/
 v_vmart_node0003 | Catalog Conversion | COMPLETED |                 0 |                0 | s3://verticadbbucket/
 v_vmart_node0003 | Data Transfer      | RUNNING   |           5272616 |           183955 | s3://verticadbbucket/

Error logging

MIGRATE_ENTERPRISE_TO_EON logs migration-related warnings, errors, and hints in migrate_enterprise_to_eon_error.log in the database directory. During execution, the meta-function also prints messages to standard output, together with the error log's pathname.

Conversion results

The following table shows how migration handles visible objects in the source Enterprise database:

Eon database activation

After migration is complete and the Eon database is ready for use, perform these steps:

1. If migrating to an Eon database with HDFS communal storage, create a bootstrapping file to use when you revive the new Eon database. The bootstrapping file must be on the same node where the revive operation is launched, and readable by the user who launches the revive operation.

   A bootstrapping file is required only if the new Eon database uses one or both of the following:

   • High Availability (HA) NameNodes
     Set HadoopConfDir to the location of configuration file hdfs-site.xml—typically, /etc/hadoop/conf. This file defines parameter hdfs.nameservices and individual NameNodes, and must be distributed across all cluster nodes. For details, see Configuring HDFS access.

   • Kerberos authentication
     Set the following Kerberos configuration parameters:

     KerberosServiceName
     KerberosRealm
     KerberosKeytabFile
     

   For example, the bootstrapping file for an Eon database with HA and Kerberos authentication must have the following settings:

   HadoopConfDir = config-path
   KerberosServiceName = principal-name
   KerberosRealm = realm-name
   KerberosKeytabFile = keytab-path
   

2. Revive the database using one of the following methods:

   • From communal storage on S3 or GCP with Management Console.
   • From communal storage on Azure, S3, GCP, or HDFS with admintools.

   In the following example, the admintools revive_db command revives a three-node database that uses S3 communal storage:

   admintools -t revive_db
      -x auth_params.conf \
      --communal-storage-location=s3://verticadbbucket \
      -d VMart \
      -s 172.16.116.27,172.16.116.28,172.16.116.29 \
      --force
   

   In the next example, revive_db revives an three-node database that uses HDFS communal storage:

   admintools -t revive_db
      -x bootstrap_params.conf \
     --communal-storage-location=webhdfs://mycluster/verticadb \
     -d verticadb \
     -s vnode01,vnode02,vnode03
   

3. Check the controlmode setting in /opt/vertica/config/admintools.conf. This setting must be compatible with the network messaging requirements of your Eon implementation. For example, S3/AWS (Amazon Cloud) relies on unicast messaging, which is compatible with a controlmode setting of point-to-point (pt2pt). If the source database controlmode setting was broacast and you migrate to S3/AWS communal storage, you must change controlmode with admintools:

   $ admintools -t re_ip -d dbname -T
   

   Important

   If controlmode is set incorrectly, attempts to start the migrated Eon database will fail.

4. Start the Eon Mode database.

5. Call CLEAN_COMMUNAL_STORAGE to remove unneeded data files that might be left over from the migration.

6. If migrating to S3 on-premises communal storage—Pure Storage or MinIO—set configuration parameter AWSStreamingConnectionPercentage to 0 with ALTER DATABASE...SET PARAMETER.

7. Review the depot storage location size and adjust as needed.