OpenText Analytics Database 26.2.x – Failure recovery

Admin: Restarting the database on a host

Mon, 01 Jan 0001 00:00:00 +0000

When one node in a running database cluster fails, or if any files from the catalog or data directories are lost from any one of the nodes, you can check the status of failed nodes using either the Administration Tools or the Management Console.

Restarting the database on a host using the administration tools

Run Administration tools.
From the Main Menu, select Restart Vertica on Host and click OK.
Select the database host you want to recover and click OK.

Note
You might see additional nodes in the list, which are used internally by the Administration Tools. You can safely ignore these nodes.
Verify recovery state by selecting View Database Cluster State from the Main Menu.

After the database is fully recovered, you can check the status at any time by selecting View Database Cluster State from the Administration Tools Main Menu.

Restarting the database on a host using the Management Console

Connect to a cluster node (or the host on which MC is installed).
Open a browser and connect to MC as an MC administrator.
On the MC Home page, double-click the running database under the Recent Databases section.
Within the Overview page, look at the node status under the Database sub-section and see if all nodes are up. The status will indicate how many nodes are up, critical, down, recovering, or other.
If a node is down, click Manage at the bottom of the page and inspect the graph. A failed node will appear in red.
Click the failed node to select it and in the Node List, click the Start node button.

Admin: Restarting the database

Mon, 01 Jan 0001 00:00:00 +0000

If you lose the database process on more than one node (for example, due to power loss), or if the servers are shut down without properly shutting down the database first, the database cluster indicates that it did not shut down gracefully the next time you start it.

The database automatically detects when the cluster was last in a consistent state and then shuts down, at which point an administrator can restart it.

From the Main Menu in the Administration tools:

Verify that the database has been stopped by clicking Stop Database.

A message displays: No databases owned by <dbadmin> are running
Start the database by selecting Start Database from the Main Menu.
Select the database you want to restart and click OK.

If you are starting the database after an unclean shutdown, messages display, which indicate that the startup failed. Press RETURN to continue with the recovery process.

An epoch represents committed changes to the data stored in a database between two specific points in time. When starting the database, OpenText™ Analytics Database searches for last good epoch.
Upon determining the last good epoch, you are prompted to verify that you want to start the database from the good epoch date. Select Yes to continue with the recovery.

Caution
If you do not want to start from the last good epoch, you may instead restore the data from a backup and attempt to restart the database. For this to be useful, the backup must be more current than the last good epoch.

The database continues to initialize and recover all data prior to the last good epoch.

If recovery takes more than a minute, you are prompted to answer <Yes> or <No> to "Do you want to continue waiting?"

When all the nodes' status have changed to RECOVERING or UP, selecting <No> lets you exit this screen and monitor progress via the Administration Tools Main Menu. Selecting <Yes> continues to display the database recovery window.

Note
Be sure to reload any data that was added after the last good epoch date to which you have recovered.

Admin: Recovering the cluster from a backup

Mon, 01 Jan 0001 00:00:00 +0000

To recover a cluster from a backup, refer to the following topics:

Admin: Phases of a recovery

Mon, 01 Jan 0001 00:00:00 +0000

The phases of a OpenText™ Analytics Database recovery are the same regardless of whether you are recovering by table or node. In the case of a recovery by table, tables become individually available as they complete the final phase. In the case of a recovery by node, the database objects only become available after the entire node completes recovery.

When you perform a recovery, each recovered table goes through the following phases:

Order	Phase	Description	Lock Type
1	Historical	The database copies any historical data it may have missed while in a state of DOWN or INITIALIZING.	none
2	Historical Dirty	The database recovers any DML transactions that committed after the node or table began recovery.	none
3	Current Replay Delete	The database replays any delete transactions that took place during the recovery.	T-lock
4	Aggregate Projections	The database recovers any aggregate projections.	T-lock

After a table completes the last phase, the database considers it fully recovered. At this point, the table can participate in DDL and DML operations.

Admin: Epochs

Mon, 01 Jan 0001 00:00:00 +0000

An epoch represents a cutoff point of historical data within the database. The timestamp of all commits within a given epoch are equal to or less than the epoch's timestamp. Understanding epochs is useful when you need to perform the following operations:

Database recovery: OpenText™ Analytics Database uses epochs to determine the last time data was consistent across all nodes in a database cluster.
Execute historical queries: A SELECT statement that includes an AT epoch clause only returns data that was committed on or before the specified epoch.
Purge deleted data: Deleted data is not removed from physical storage until it is purged from the database. You can purge deleted data from the database only if it precedes the ancient history marker (AHM) epoch.

The database has one open epoch and any number of closed epochs, depending on your system configuration. New and updated data is written into the open epoch, and each closed epoch represents a previous commit to your database. When data is committed with a DML operation (INSERT, UPDATE, MERGE, COPY, or DELETE), the database writes the data, closes the open epoch, and opens a new epoch. Each row committed to the database is associated with the epoch in which it was written.

The EPOCHS system table contains information about each available closed epoch. The epoch_close_time column stores the date and time of the commit. The epoch_number column stores the corresponding epoch number:

=> SELECT * FROM EPOCHS;
       epoch_close_time        | epoch_number
-------------------------------+--------------
 2020-07-27 14:29:49.687106-04 |           91
 2020-07-28 12:51:53.291795-04 |           92
(2 rows)

Epoch milestones

As an epoch progresses through its life cycle, it reaches milestones that the database uses to perform a variety of operations and maintain the state of the database. The following image generally depicts these milestones within the epoch life cycle:

The database defines each milestone as follows:

Current epoch (CE): The current, open epoch that you are presently writing data to.
Latest epoch (LE): The most recently closed epoch.
Checkpoint epoch: Enterprise Mode only. A node-level epoch that is the latest epoch in which data is consistent across all projections on that node.
Last good epoch (LGE): The minimum checkpoint epoch in which data is consistent across all nodes.
Ancient history mark (AHM): The oldest epoch that contains data that is accessible by historical queries.

See Epoch life cycle for detailed information about each stage.

Admin: Best practices for disaster recovery

Mon, 01 Jan 0001 00:00:00 +0000

To protect your database from site failures caused by catastrophic disasters, maintain an off-site replica of your database to provide a standby. In case of disaster, you can switch database users over to the standby database. The amount of data loss between a disaster and fail over to the offsite replica depends on how frequently you save a full database backup.

The solution to employ for disaster recover depends upon two factors that you must determine for your application:

Recovery point objective (RPO): How much data loss can your organization tolerate upon a disaster recovery?
Recovery time objective (RTO): How quickly do you need to recover the database following a disaster?

Depending on your RPO and RTO, OpenText recommends choosing from the following solutions:

Dual-load: During each load process for the database, simultaneously load a second database. You can achieve this easily with off-the-shelf ETL software.
Periodic Incremental Backups: Use the procedure described in Copying the database to another cluster to periodically copy the data to the target database. Remember that the script copies only files that have changed.
Replication solutions provided by Storage Vendors: Although some users have had success with SAN storage, the number of vendors and possible configurations prevent OpenText™ Analytics Database from providing support for SANs.

The following table summarizes the RPO, RTO, and the pros and cons of each approach:

	Dual Load	Periodic Incremental	Storage Replication
RPO	Up to the minute data	Up to the last backup	Recover to the minute
RTO	Available at all times	Available except when backup in progress	Available at all times
Pros	Standby database can have different configuration Can use the standby database for queries	Built-in scripts High performance due to compressed file transfers	Transparent to the database
Cons	Possibly incur additional ETL licenses Requires application logic to handle errors	Need identical standby system	More expensive Media corruptions are also replicated

Admin: Recovery by table

Mon, 01 Jan 0001 00:00:00 +0000

OpenText™ Analytics Database supports node recovery on a per-table basis. Unlike node-based recovery, recovering by table makes tables available as they recover, before the node itself is completely restored. You can prioritize your most important tables so they become available as soon as possible. Recovered tables support all DDL and DML operations.

To enhance recovery speed, the database recovers multiple tables in parallel. The maximum number of tables recoverable at one time is set by the MAXCONCURRENCY parameter in the RECOVERY resource pool.

After a node has fully recovered, it enables full database functionality.

OpenText Analytics Database 26.2.x – Failure recovery

Admin: Restarting the database on a host

Restarting the database on a host using the administration tools

Note

Restarting the database on a host using the Management Console

Admin: Restarting the database

Caution

Note

Admin: Recovering the cluster from a backup

Admin: Phases of a recovery

Admin: Epochs

Epoch milestones

Admin: Best practices for disaster recovery

Admin: Recovery by table