OpenText Analytics Database 26.2.x – Managing nodes

Admin: Stop the database on a node

Mon, 01 Jan 0001 00:00:00 +0000

In some cases, you need to take down a node to perform maintenance tasks, or upgrade hardware. You can do this with one of the following:

Administration Tools
Command line admintools stop_node

Important

Before removing a node from a cluster, check that the cluster has the minimum number of nodes required to comply with K-safety. If necessary, temporarily lower the database K-safety level.

Administration tools

Run Administration Tools, select Advanced Menu, and click OK.
Select Stop Vertica on Host and click OK.
Choose the host that you want to stop and click OK.
Return to the Main Menu, select View Database Cluster State, and click OK. The host you previously stopped should appear DOWN.
You can now perform maintenance.

See Restart the database on a node for more information.

Command line

You can use the command line tool stop_node to stop the database on one or more nodes. stop_node takes one or more node IP addresses as arguments. For example, the following command stops the database on two nodes:

$ admintools -t stop_node -s 192.0.2.1,192.0.2.2

Admin: Restart the database on a node

Mon, 01 Jan 0001 00:00:00 +0000

After stopping a node to perform maintenance tasks such as upgrading hardware, you need to restart the node so it can reconnect with the database cluster.

Run Administration Tools. From the Main Menu select Restart Vertica on Host and click OK.
Select the database and click OK.
Select the host that you want to restart and click OK.

Note
This process may take a few moments.
Return to the Main Menu, select View Database Cluster State, and click OK. The host you restarted now appears as UP, as shown.

Admin: Setting node type

Mon, 01 Jan 0001 00:00:00 +0000

When you create a node, OpenText™ Analytics Database automatically sets its type to PERMANENT. This enables the database to use this node to store data. You can change a node's type with ALTER NODE, to one of the following:

PERMANENT: (default): A node that stores data.
EPHEMERAL: A node that is in transition from one type to another—typically, from PERMANENT to either STANDBY or EXECUTE.
STANDBY: A node that is reserved to replace any node when it goes down. A standby node stores no segments or data until it is called to replace a down node. When used as a replacement node, the database changes its type to PERMANENT. For more information, see Active standby nodes.
EXECUTE: A node that is reserved for computation purposes only. An execute node contains no segments or data.

Note

STANDBY and EXECUTE node types are supported only in Enterprise Mode.

Admin: Active standby nodes

Mon, 01 Jan 0001 00:00:00 +0000

An active standby node exists is a node in an Enterprise Mode database that is available to replace any failed node. Unlike permanent OpenText™ Analytics Database nodes, a standby node does not perform computations or contain data. If a permanent node fails, an active standby node can replace the failed node, after the failed node exceeds the failover time limit. After replacing the failed node, the active standby node contains the projections and performs all calculations of the node it replaced.

In this section

Admin: Large cluster

Mon, 01 Jan 0001 00:00:00 +0000

OpenText™ Analytics Database uses the Spread service to broadcast control messages between database nodes. This service can limit the growth of a database cluster. As you increase the number of cluster nodes, the load on the Spread service also increases as more participants exchange messages. This increased load can slow overall cluster performance. Also, network addressing limits the maximum number of participants in the Spread service to 120 (and often far less). In this case, you can use large cluster to overcome these Spread limitations.

When large cluster is enabled, a subset of cluster nodes, called control nodes, exchange messages using the Spread service. Other nodes in the cluster are assigned to one of these control nodes, and depend on them for cluster-wide communication. Each control node passes messages from the Spread service to its dependent nodes. When a dependent node needs to broadcast a message to other nodes in the cluster, it passes the message to its control node, which in turn sends the message out to its other dependent nodes and the Spread service.

By setting up dependencies between control nodes and other nodes, you can grow the total number of database nodes, and remain in compliance with the Spread limit of 120 nodes.

Note

Technically, when large cluster is disabled, all of the nodes in the cluster are control nodes. In this case, all nodes connect to Spread. When large cluster is enabled, some nodes become dependent on control nodes.

A downside of the large cluster feature is that if a control node fails, its dependent nodes are cut off from the rest of the database cluster. These nodes cannot participate in database activities, and the database considers them to be down as well. When the control node recovers, it re-establishes communication between its dependent nodes and the database, so all of the nodes rejoin the cluster.

Note

The Spread service demon runs as an independent process on the control node host. It is not part of the database process. If the database process goes down on the node—for example, you use admintools to stop the database process on the host—Spread continues to run. As long as the Spread demon runs on the control node, the node's dependents can communicate with the database cluster and participate in database activity. Normally, the control node only goes down if the node's host has an issue—or example, you shut it down, it becomes disconnected from the network, or a hardware failure occurs.

Large cluster and database growth

When your database has large cluster enabled, the database decides whether to make a newly added node into a control or a dependent node as follows:

In Enterprise Mode, if the number of control nodes configured for the database cluster is greater than the current number of nodes it contains, the database makes the new node a control node. In Eon Mode, the number of control nodes is set at the subcluster level. If the number of control nodes set for the subcluster containing the new node is less than this setting, the database makes the new node a control node.
If the Enterprise Mode cluster or Eon Mode subcluster has reached its limit on control nodes, a new node becomes a dependent of an existing control node.

When a newly-added node is a dependent node, the database automatically assigns it to a control node. Which control node it chooses is guided by the database mode:

Enterprise Mode database: The database assigns the new node to the control node with the least number of dependents. If you created fault groups in your database, it chooses a control node in the same fault group as the new node. This feature lets you use fault groups to organize control nodes and their dependents to reflect the physical layout of the underlying host hardware. For example, you might want dependent nodes to be in the same rack as their control nodes. Otherwise, a failure that affects the entire rack (such as a power supply failure) will not only cause nodes in the rack to go down, but also nodes in other racks whose control node is in the affected rack. See Fault groups for more information.
Eon Mode database: The database always adds new nodes to a subcluster. The database assigns the new node to the control node with the fewest dependent nodes in that subcluster. Every subcluster in an Eon Mode database with large cluster enabled has at least one control node. Keeping dependent nodes in the same subcluster as their control node maintains subcluster isolation.

Spread's upper limit of 120 participants can cause errors when adding a subcluster to an Eon Mode database. If your database cluster has 120 control nodes, attempting to create a subcluster fails with an error. Every subcluster must have at least one control node. When your cluster has 120 control nodes, the database cannot create a control node for the new subcluster. If this error occurs, you must reduce the number of control nodes in your database cluster before adding a subcluster.

When to enable large cluster

The database automatically enables large cluster in two cases:

The database cluster contains 120 or more nodes. This is true for both Enterprise Mode and Eon Mode.
You create an Eon Mode subcluster (either a primary subcluster or a secondary subcluster) with an initial node count of 16 or more.

The database does not automatically enable large cluster if you expand an existing subcluster to 16 or more nodes by adding nodes to it.

Note
You can prevent the database from automatically enabling large cluster when you create a subcluster with 16 or more nodes by setting the control-set-size parameter to -1. See Creating subclusters for details.

You can choose to manually enable large cluster mode before the database automatically enables it. Your best practice is to enable large cluster when your database cluster size reaches a threshold:

For cloud-based databases, enable large cluster when the cluster contains 16 or more nodes. In a cloud environment, your database uses point-to-point network communications. Spread scales poorly in point-to-point communications mode. Enabling large cluster when the database cluster reaches 16 nodes helps limit the impact caused by Spread being in point-to-point mode.
For on-premises databases, enable large cluster when the cluster reaches 50 to 80 nodes. Spread scales better in an on-premises environment. However, by the time the cluster size reaches 50 to 80 nodes, Spread may begin exhibiting performance issues.

In either cloud or on-premises environments, enable large cluster if you begin to notice Spread-related performance issues. Symptoms of Spread performance issues include:

The load on the Spread service begins to cause performance issues. The database uses Spread for cluster-wide control messages, so Spread performance issues can adversely affect database performance. This is particularly true for cloud-based databases, where Spread performance problems becomes a bottleneck sooner, due to the nature of network broadcasting in the cloud infrastructure. In on-premises databases, broadcast messages are usually less of a concern because messages usually remain within the local subnet. Even so, eventually, Spread usually becomes a bottleneck before the database automatically enables large cluster automatically when the cluster reaches 120 nodes.
The compressed list of addresses in your cluster is too large to fit in a maximum transmission unit (MTU) packet (1478 bytes). The MTU packet has to contain all of the addresses for the nodes participating in the Spread service. Under ideal circumstances (when your nodes have the IP addresses 1.1.1.1, 1.1.1.2 and so on) 120 addresses can fit in this packet. This is why the database automatically enables large cluster if your database cluster reaches 120 nodes. In practice, the compressed list of IP addresses will reach the MTU packet size limit at 50 to 80 nodes.

Admin: Fault groups

Mon, 01 Jan 0001 00:00:00 +0000

Note

You cannot create fault groups for an Eon Mode database. Rather, OpenText™ Analytics Database automatically creates fault groups on a large cluster Eon database; these fault groups are configured around the control nodes and their dependents of each subcluster. These fault groups are managed internally by the database and are not accessible to users.

Fault groups let you configure an Enterprise Mode database for your physical cluster layout. Sharing your cluster topology lets you use terrace routing to reduce the buffer requirements of large queries. It also helps to minimize the risk of correlated failures inherent in your environment, usually caused by shared resources.

The database automatically creates fault groups around control nodes (servers that run spread) in large cluster arrangements, placing nodes that share a control node in the same fault group. Automatic and user-defined fault groups do not include ephemeral nodes because such nodes hold no data.

Consider defining your own fault groups specific to your cluster's physical layout if you want to:

Use terrace routing to reduce the buffer requirements of large queries.
Reduce the risk of correlated failures. For example, by defining your rack layout, the database can better tolerate a rack failure.
Influence the placement of control nodes in the cluster.

OpenText™ Analytics Database supports complex, hierarchical fault groups of different shapes and sizes. The database platform provides a fault group script (DDL generator), SQL statements, system tables, and other monitoring tools.

See High availability with fault groups for an overview of fault groups with a cluster topology example.

Admin: Terrace routing

Mon, 01 Jan 0001 00:00:00 +0000

Important

Before you apply terrace routing to your database, be sure you are familiar with large cluster and fault groups.

Terrace routing can significantly reduce message buffering on a large cluster database. The following sections describe how OpenText™ Analytics Database implements terrace routing on Enterprise Mode and Eon Mode databases.

Terrace routing on Enterprise Mode

Terrace routing on an Enterprise Mode database is implemented through fault groups that define a rack-based topology. In a large cluster with terrace routing disabled, nodes in a database cluster form a fully connected network, where each non-dependent (control) node sends messages across the database cluster through connections with all other non-dependent nodes, both within and outside its own rack/fault group:

In this case, large database clusters can require many connections on each node, where each connection incurs its own network buffering requirements. The total number of buffers required for each node is calculated as follows:

(numRacks * numRackNodes) - 1

In a two-rack cluster with 4 nodes per rack as shown above, this resolves to 7 buffers for each node.

With terrace routing enabled, you can considerably reduce large cluster network buffering. Each nth node in a rack/fault group is paired with the corresponding nth node of all other fault groups. For example, with terrace routing enabled, messaging in the same two-rack cluster is now implemented as follows:

Thus, a message that originates from node 2 on rack A (A2) is sent to all other nodes on rack A; each rack A node then conveys the message to its corresponding node on rack B—A1 to B1, A2 to B2, and so on.

With terrace routing enabled, each node of a given rack avoids the overhead of maintaining message buffers to all other nodes. Instead, each node is only responsible for maintaining connections to:

All other nodes of the same rack (numRackNodes - 1)
One node on each of the other racks (numRacks - 1)

Thus, the total number of message buffers required for each node is calculated as follows:

(numRackNodes-1) + (numRacks-1)

In a two-rack cluster with 4 nodes as shown earlier, this resolves to 4 buffers for each node.

Terrace routing trades time (intra-rack hops) for space (network message buffers). As a cluster expands with additional racks and nodes, the argument favoring this trade off becomes increasingly persuasive:

In this three-rack cluster with 4 nodes per rack, without terrace routing the number of buffers required by each node would be 11. With terrace routing, the number of buffers per node is 5. As a cluster expands with the addition of racks and nodes per rack, the disparity between buffer requirements widens. For example, given a six-rack cluster with 16 nodes per rack, without terrace routing the number of buffers required per node is 95; with terrace routing, 20.

Enabling terrace routing

Terrace routing depends on fault group definitions that describe a cluster network topology organized around racks and their member nodes. As noted earlier, when terrace routing is enabled, the database first distributes data within the rack/fault group; it then uses nth node-to-nth node mappings to forward this data to all other racks in the database cluster.

You enable (or disable) terrace routing for any Enterprise Mode large cluster that implements rack-based fault groups through configuration parameter TerraceRoutingFactor. To enable terrace routing, set this parameter as follows:

where:

numRackNodes: Number of nodes in a rack
numRacks: Number of racks in the cluster

For example:

#Racks	Nodes/rack	#Connections	Terrace routing enabled if TerraceRoutingFactor less than:
Without terrace routing	With terrace routing
2	16	31	16	1.94
4	16	63	18	3.5
6	16	95	20	4.75
8	16	127	22	5.77

By default, TerraceRoutingFactor is set to 2, which generally ensures that terrace routing is enabled for any Enterprise Mode large cluster that implements rack-based fault groups. It is recommended that you enable terrace routing for any cluster that contains 64 or more nodes, or if queries often require excessive buffer space.

To disable terrace routing, set TerraceRoutingFactor to a large integer such as 1000:

=> ALTER DATABASE DEFAULT SET TerraceRoutingFactor = 1000;

Terrace routing on Eon Mode

As in Enterprise Mode mode, terrace routing is enabled by default on an Eon Mode database, and is implemented through fault groups. However, you do not create fault groups for an Eon Mode database. Rather, OpenText™ Analytics Database automatically creates fault groups on a large cluster database; these fault groups are configured around the control nodes and their dependents of each subcluster. These fault groups are managed internally and are not accessible to users.

Admin: Elastic cluster

Mon, 01 Jan 0001 00:00:00 +0000

Note

Elastic Cluster is an Enterprise Mode-only feature. For scaling your database under Eon Mode, see Scaling your Eon Mode database.

You can scale your cluster up or down to meet the needs of your database. The most common case is to add nodes to your database cluster to accommodate more data and provide better query performance. However, you can scale down your cluster if you find that it is over-provisioned, or if you need to divert hardware for other uses.

You scale your cluster by adding or removing nodes. Nodes can be added or removed without shutting down or restarting the database. After adding a node or before removing a node, OpenText™ Analytics Database begins a rebalancing process that moves data around the cluster to populate the new nodes or move data off nodes about to be removed from the database. During this process, nodes can exchange data that are not being added or removed to maintain robust intelligent K-safety. If the database determines that the data cannot be rebalanced in a single iteration due to lack of disk space, then the rebalance operation spans multiple iterations.

To help make data rebalancing due to cluster scaling more efficient, the database locally segments data storage on each node so it can be easily moved to other nodes in the cluster. When a new node is added to the cluster, existing nodes in the cluster give up some of their data segments to populate the new node. They also exchange segments to minimize the number of nodes that any one node depends upon. This strategy minimizes the number of nodes that might become critical when a node fails. When a node is removed from the cluster, its storage containers are moved to other nodes in the cluster (which also relocates data segments to minimize how many nodes might become critical when a node fails). This method of breaking data into portable segments is referred to as elastic cluster, as it facilitates enlarging or shrinking the cluster.

The alternative to elastic cluster is re-segmenting all projection data and redistributing it evenly among all database nodes any time a node is added or removed. This method requires more processing and more disk space, as it requires all data in all projections to be dumped and reloaded.

Elastic cluster scaling factor

In a new installation, each node has a scaling factor that specifies the number of local segments (see Scaling factor). Rebalance efficiently redistributes data by relocating local segments provided that, after nodes are added or removed, there are sufficient local segments in the cluster to redistribute the data evenly (determined by MAXIMUM_SKEW_PERCENT). For example, if the scaling factor = 8, and there are initially 5 nodes, then there are a total of 40 local segments cluster-wide.

If you add two additional nodes (seven nodes) the database relocates five local segments on two nodes, and six such segments on five nodes, resulting in roughly a 16.7 percent skew. Rebalance relocates local segments only if the resulting skew is less than the allowed threshold, as determined by MAXIMUM_SKEW_PERCENT. Otherwise, segmentation space (and hence data, if uniformly distributed over this space) is evenly distributed among the seven nodes, and new local segment boundaries are drawn for each node, such that each node again has eight local segments.

Note

By default, the scaling factor only has an effect while the database rebalances itself. While rebalancing, each node breaks the projection segments it contains into storage containers, which it then moves to other nodes if necessary. After rebalancing, the data is recombined into ROS containers. It is possible to have the database always group data into storage containers. See Local data segmentation for more information.

Enabling elastic cluster

You enable elastic cluster with ENABLE_ELASTIC_CLUSTER. Query the ELASTIC_CLUSTER system table to verify that elastic cluster is enabled:

=> SELECT is_enabled FROM ELASTIC_CLUSTER;
 is_enabled
------------
 t
(1 row)

Admin: Adding nodes

Mon, 01 Jan 0001 00:00:00 +0000

There are many reasons to add one or more nodes to an OpenText™ Analytics Database cluster:

Increase system performance or capacity. Add nodes due to a high query load or load latency, or increase disk space in Enterprise Mode without adding storage locations to existing nodes.

The database response time depends on factors such as type and size of the application query, database design, data size and data types stored, available computational power, and network bandwidth. Adding nodes to a database cluster does not necessarily improve the system response time for every query, especially if the response time is already short or not hardware-bound.
Make the database K-safe (K-safety=1) or increase K-safety to 2. See Failure recovery for details.
Swap or replace hardware. Swap out a node to perform maintenance or hardware upgrades.

Important

If you install the database on a single node without specifying the IP address or host name (or you used localhost), you cannot expand the cluster. You must reinstall the database and specify an IP address or host name that is not localhost/127.0.0.1.

Adding nodes consists of the following general tasks:

Back up the database.

It is strongly recommended that you back up the database before you perform this significant operation because it entails creating new projections, refreshing them, and then deleting the old projections. See Backing up and restoring the database for more information.

The process of migrating the projection design to include the additional nodes could take a while; however during this time, all user activity on the database can proceed normally, using the old projections.
Configure the hosts you want to add to the cluster.

See Before you install the database. You will also need to edit the hosts configuration file on all of the existing nodes in the cluster to ensure they can resolve the new host.
Add one or more hosts to the cluster.
Add the hosts you added to the cluster (in step 3) to the database.

When you add a host to the database, it becomes a node. You can add nodes to your database using either the Administration tools or the Management Console (See Monitoring with MC). Adding nodes using admintools preserves the specific order of the nodes you add.

After you add nodes to the database, it automatically distributes updated configuration files to the rest of the nodes in the cluster and starts the process of rebalancing data in the cluster. See Rebalancing data across nodes for details.

If you have previously created storage locations using CREATE LOCATION...ALL NODES, you must create those locations on the new nodes.

Admin: Removing nodes

Mon, 01 Jan 0001 00:00:00 +0000

Although less common than adding a node, permanently removing a node is useful if the host system is obsolete or over-provisioned.

Important

Before removing a node from a cluster, check that the cluster has the minimum number of nodes required to comply with K-safety. If necessary, temporarily lower the database K-safety level.

Admin: Replacing nodes

Mon, 01 Jan 0001 00:00:00 +0000

If you have a K-Safe database, you can replace nodes, as necessary, without bringing the system down. For example, you might want to replace an existing node if you:

Need to repair an existing host system that no longer functions and restore it to the cluster
Want to exchange an existing host system for another more powerful system

Note

OpenText™ Analytics Database does not support replacing a node on a K-safe=0 database. Use the procedures to add and remove nodes instead.

The process you use to replace a node depends on whether you are replacing the node with:

A host that uses the same name and IP address
A host that uses a different name and IP address
An active standby node

Prerequisites

Configure the replacement hosts for the database. See Before you install the database.
Read the Important Tips sections under Adding hosts to a cluster and Removing hosts from a cluster.
Ensure that the database administrator user exists on the new host and is configured identically to the existing hosts. OpenText™ Analytics Database will setup passwordless ssh as needed.
Ensure that directories for Catalog Path, Data Path, and any storage locations are added to the database when you create it and/or are mounted correctly on the new host and have read and write access permissions for the database administrator user. Also ensure that there is sufficient disk space.
Follow the best practice procedure below for introducing the failed hardware back into the cluster to avoid spurious full-node rebuilds.

Best practice for restoring failed hardware

Following this procedure will prevent OpenText™ Analytics Database from misdiagnosing missing disk or bad mounts as data corruptions, which would result in a time-consuming, full-node recovery.

If a server fails due to hardware issues, for example a bad disk or a failed controller, upon repairing the hardware:

Reboot the machine into runlevel 1, which is a root and console-only mode.

Runlevel 1 prevents network connectivity and keeps the database from attempting to reconnect to the cluster.
In runlevel 1, validate that the hardware has been repaired, the controllers are online, and any RAID recover is able to proceed.

Note
You do not need to initialize RAID recover in runlevel 1; simply validate that it can recover.
Once the hardware is confirmed consistent, only then reboot to runlevel 3 or higher.

At this point, the network activates, and the database rejoins the cluster and automatically recovers any missing data. Note that, on a single-node database, if any files that were associated with a projection have been deleted or corrupted, the database deletes all files associated with that projection, which could result in data loss.

Admin: Rebalancing data across nodes

Mon, 01 Jan 0001 00:00:00 +0000

OpenText™ Analytics Database can rebalance your database when you add or remove nodes. As a superuser, you can manually trigger a rebalance with Administration Tools, SQL functions, or the Management Console.

A rebalance operation can take some time, depending on the cluster size, and the number of projections and the amount of data they contain. You should allow the process to complete uninterrupted. If you must cancel the operation, call CANCEL_REBALANCE_CLUSTER.

Why rebalance?

Rebalancing is useful or even necessary after you perform one of the following operations:

Change the size of the cluster by adding or removing nodes.
Mark one or more nodes as ephemeral in preparation of removing them from the cluster.
Change the scaling factor of an elastic cluster, which determines the number of storage containers used to store a projection across the database.
Set the control node size or realign control nodes on a large cluster layout.
Specify more than 120 nodes in your initial database cluster configuration.
Modify a fault group by adding or removing nodes.

General rebalancing tasks

When you rebalance a database cluster, OpenText™ Analytics Database performs the following tasks for all projections, segmented and unsegmented alike:

Distributes data based on:
- User-defined fault groups, if specified
- Large cluster automatic fault groups
Ignores node-specific distribution specifications in projection definitions. Node rebalancing always distributes data across all nodes.
When rebalancing is complete, sets the Ancient History Mark the greatest allowable epoch (now).

The database rebalances segmented and unsegmented projections differently, as described below.

Rebalancing segmented projections

For each segmented projection, the database performs the following tasks:

Copies and renames projection buddies and distributes them evenly across all nodes. The renamed projections share the same base name.
Refreshes the new projections.
Drops the original projections.

Rebalancing unsegmented projections

For each unsegmented projection, the database performs the following tasks:

If adding nodes:

Creates projection buddies on them.
Maps the new projections to their shared name in the database catalog.

If dropping nodes: drops the projection buddies from them.

K-safety and rebalancing

Until rebalancing completes, the database operates with the existing K-safe value. After rebalancing completes, the database operates with the K-safe value specified during the rebalance operation. The new K-safe value must be equal to or higher than current K-safety. OpenText™ Analytics Database does not support downgrading K-safety and returns a warning if you try to reduce it from its current value. For more information, see Lowering K-Safety to enable node removal.

Rebalancing failure and projections

If a failure occurs while rebalancing the database, you can rebalance again. If the cause of the failure has been resolved, the rebalance operation continues from where it failed. However, a failed data rebalance can result in projections becoming out of date.

To locate out-of-date projections, query the system table PROJECTIONS as follows:

=> SELECT projection_name, anchor_table_name, is_up_to_date FROM projections
   WHERE is_up_to_date = false;

To remove out-of-date projections, use DROP PROJECTION.

Temporary tables

Node rebalancing has no effect on projections of temporary tables.

For Detailed Information About Rebalancing

See the Knowledge Base articles:

Admin: Redistributing configuration files to nodes

Mon, 01 Jan 0001 00:00:00 +0000

The add and remove node processes automatically redistribute the OpenText™ Analytics Database configuration files. You rarely need to redistribute the configuration files to help resolve configuration issues.

To distribute configuration files to a host:

Log on to a host that contains these files and start Administration Tools.
On the Administration Tools Main Menu, select Configuration Menu and click OK.
On the Configuration Menu, select Distribute Config Files and click OK.
Select Database Configuration.
Select the database where you want to distribute the files and click OK.

Database configuration files are distributed to all other database hosts. If the files already existed on a host, they are overwritten.
On the Configuration Menu, select Distribute Config Files and click OK.
Select SSL Keys.

Certifications and keys are distributed to all other database hosts. If they already existed on a host, they are overwritten.
On the Configuration Menu, select Distribute Config Files and click OK.

Select AdminTools Meta-Data.

Administration Tools metadata is distributed to every host in the cluster.
Restart the database.

Note

To distribute the configuration file admintools.conf via the command line or scripts, use the admintools option distribute_config_files:

$ admintools -t distribute_config_files

Admin: Stopping and starting nodes on MC

Mon, 01 Jan 0001 00:00:00 +0000

You can start and stop one or more database nodes through the Manage page by clicking a specific node to select it and then clicking the Start or Stop button in the Node List.

Note

The Stop and Start buttons in the toolbar start and stop the database, not individual nodes.

On the Databases and Clusters page, you must click a database first to select it. To stop or start a node on that database, click the View button. You'll be directed to the Overview page. Click Manage in the applet panel at the bottom of the page and you'll be directed to the database node view.

The Start and Stop database buttons are always active, but the node Start and Stop buttons are active only when one or more nodes of the same status are selected; for example, all nodes are UP or DOWN.

After you click a Start or Stop button, Management Console updates the status and message icons for the nodes or databases you are starting or stopping.

Admin: Upgrading your operating system on nodes in your database cluster

Mon, 01 Jan 0001 00:00:00 +0000

If you need to upgrade the operating system on the nodes in your OpenText™ Analytics Database cluster, check with the documentation for your Linux distribution to make sure they support the particular upgrade you are planning.

For example, the following articles provide information about upgrading Red Hat:

After you confirm that you can perform the upgrade, follow the steps at Best Practices for Upgrading the Operating System on Nodes in a Database Cluster.

Admin: Reconfiguring node messaging

Mon, 01 Jan 0001 00:00:00 +0000

Sometimes, nodes of an existing, operational OpenText™ Analytics Database cluster require new IP addresses. Cluster nodes might also need to change their messaging protocols—for example, from broadcast to point-to-point. The admintools re_ip utility performs both tasks.

Note

You cannot change from one address family—IPv4 or IPv6—to another. For example, if hosts in the database cluster are identified by IPv4 network addresses, you can only change host addresses to another set of IPv4 addresses.

Changing IP addresses

You can use re_ip to perform two tasks:

In both cases, re_ip requires a mapping file that identifies the current node IP addresses, which are stored in admintools.conf. You can get these addresses in two ways:

Use the admintools utility list_allnodes:

$ admintools -t list_allnodes
Node             | Host          | State | Version        | DB
-----------------+---------------+-------+----------------+-----------
v_vmart_node0001 | 192.0.2.254   | UP    | vertica-12.0.1 | VMart
v_vmart_node0002 | 192.0.2.255   | UP    | vertica-12.0.1 | VMart
v_vmart_node0003 | 192.0.2.256   | UP    | vertica-12.0.1 | VMart

Tip

list_allnodes can help you identify issues that you might have to access the database. For example, if hosts are not communicating with each other, the Version column displays Unavailable.

Print the content of admintools.conf:

$ cat /opt/vertica/config/admintools.conf
...
[Cluster]
hosts = 192.0.2.254, 192.0.2.255, 192.0.2.256

[Nodes]
node0001 = 192.0.2.254/home/dbadmin,/home/dbadmin
node0002 = 192.0.2.255/home/dbadmin,/home/dbadmin
node0003 = 192.0.2.256/home/dbadmin,/home/dbadmin
...

Update node IP addresses

You can update IP addresses with re_ip as described below. re_ip automatically backs up admintools.conf so you can recover the original settings if necessary.

Create a mapping file with lines in the following format:

oldIPaddress newIPaddress[, controlAddress, controlBroadcast]
...

For example:

192.0.2.254 198.51.100.255, 198.51.100.255, 203.0.113.255
192.0.2.255 198.51.100.256, 198.51.100.256, 203.0.113.255
192.0.2.256 198.51.100.257, 198.51.100.257, 203.0.113.255

controlAddress and controlBroadcast are optional. If omitted:

controlAddress defaults to newIPaddress.
controlBroadcast defaults to the host of newIPaddress’s broadcast IP address.

Stop the database.

Run re_ip to map old IP addresses to new IP addresses:

$ admintools -t re_ip -f mapfile

re_ip issues warnings for the following mapping file errors:

IP addresses are incorrectly formatted.
Duplicate IP addresses, whether old or new.

If re_ip finds no syntax errors, it performs the following tasks:

Remaps the IP addresses as listed in the mapping file.
If the -i option is omitted, asks to confirm updates to the database.
Updates required local configuration files with the new IP addresses.
Distributes the updated configuration files to the hosts using new IP addresses.

For example:

Parsing mapfile...
New settings for Host 192.0.2.254 are:
address: 198.51.100.255
New settings for Host 192.0.2.255 are:
address: 198.51.100.256
New settings for Host 192.0.2.254 are:
address: 198.51.100.257

The following databases would be affected by this tool: Vmart

Checking DB status ...
Enter "yes" to write new settings or "no" to exit > yes
Backing up local admintools.conf ...
Writing new settings to local admintools.conf ...

Writing new settings to the catalogs of database Vmart ...
The change was applied to all nodes.
Success. Change committed on a quorum of nodes.

Initiating admintools.conf distribution ...
Success. Local admintools.conf sent to all hosts in the cluster.

Restart the database.

re_ip and export IP address

By default, a node's IP address and its export IP address are identical. For example:

=> SELECT node_name, node_address, export_address FROM nodes;
    node_name      | node_address    | export_address
------------------------------------------------------
v_VMartDB_node0001 | 192.168.100.101 | 192.168.100.101
v_VMartDB_node0002 | 192.168.100.102 | 192.168.100.101
v_VMartDB_node0003 | 192.168.100.103 | 192.168.100.101
v_VMartDB_node0004 | 192.168.100.104 | 192.168.100.101
(4 rows)

The export address is the IP address of the node on the network. This address provides access to other DBMS systems, and enables you to import and export data across the network.

If node IP and export IP addresses are the same, then running re_ip changes both to the new address. Conversely, if you manually change the export address, subsequent re_ip operations leave your export address changes untouched.

Change node control and broadcast addresses

You can map IP addresses for the database only, by using the re_ip option -O (or --db-only). Database-only operations are useful for error recovery. The node names and IP addresses that are specified in the mapping file must be the same as the node information in admintools.conf. In this case, admintools.conf is not updated. The database updates only spread.conf and the catalog with the changes.

You can also use re_ip to change the node control and broadcast addresses. In this case the mapping file must contain the control messaging IP address and associated broadcast address. This task allows nodes on the same host to have different data and control addresses.

Create a mapping file with lines in the following format:

nodeName nodeIPaddress, controlAddress, controlBroadcast
    ...

Tip

Query the system table NODES for node names.

For example:

vertica_node001 192.0.2.254, 203.0.113.255, 203.0.113.258
vertica_node002 192.0.2.255, 203.0.113.256, 203.0.113.258
vertica_node003 192.0.2.256, 203.0.113.257, 203.0.113.258

Stop the database.
Run the following command to map the new IP addresses:
```
$ admintools -t re_ip -f mapfile -O -d dbname
```
Restart the database.

Changing node messaging protocols

You can use re_ip to reconfigure spread messaging between nodes. re_ip configures node messaging to broadcast or point-to-point (unicast) messaging with these options:

-U, --broadcast (default)
-T, --point-to-point

Both options support up to 80 spread daemons. You can exceed the 80-node limit by using large cluster mode, which does not install a spread daemon on each node.

For example, to set the database cluster messaging protocol to point-to-point:

$ admintools -t re_ip -d dbname -T

To set the messaging protocol to broadcast:

$ admintools -t re_ip -d dbname -U

Setting re_ip timeout

You can configure how long re_ip executes a given task before it times out, by editing the setting of prepare_timeout_sec in admintools.conf. By default, this parameter is set to 7200 (seconds).

Admin: Adjusting Spread Daemon timeouts for virtual environments

Mon, 01 Jan 0001 00:00:00 +0000

OpenText™ Analytics Database relies on Spread daemons to pass messages between database nodes. Occasionally, nodes fail to respond to messages within the specified Spread timeout. These failures might be caused by spikes in network latency or brief pauses in the node's VM—for example, scheduled Azure maintenance timeouts. In either case, the database assumes that the non-responsive nodes are down and starts to remove them, even though they might still be running. You can address this issue by adjusting the Spread timeout as needed.

Adjusting spread timeout

By default, the Spread timeout depends on the number of configured Spread segments:

Configured Spread segments	Default timeout
1	8 seconds
> 1	25 seconds

Important

If you deploy your database cluster with Azure Marketplace, the default Spread timeout is set to 35 seconds. If you manually create your cluster in Azure, the default Spread timeout is set to 8 or 25 seconds.

If the Spread timeout is likely to elapse before the network or database nodes can respond, increase the timeout to the maximum length of non-responsive time plus five seconds. For example, if Azure memory-preserving maintenance pauses node VMs for up to 30 seconds, set the Spread timeout to 35 seconds.

If you are unsure how long network or node disruptions are liable to last, gradually increase the Spread timeout until fewer instances of UP nodes leave the database.

Important

The database cannot react to a node going down or being shut down improperly before the timeout period elapses. Changing Spread’s timeout to a value too high can result in longer query restarts if a node goes down.

To see the current setting of the Spread timeout, query system table SPREAD_STATE. For example, the following query shows that the current timeout setting (token_timeout) is set to 8000ms:

=> SELECT * FROM V_MONITOR.SPREAD_STATE;
    node_name     | token_timeout
------------------+---------------
 v_vmart_node0003 |          8000
 v_vmart_node0001 |          8000
 v_vmart_node0002 |          8000
(3 rows)

To change the Spread timeout, call the meta-function SET_SPREAD_OPTION and set the token timeout to a new value. The following example sets the timeout to 35000ms (35 seconds):

=> SELECT SET_SPREAD_OPTION( 'TokenTimeout', '35000');
NOTICE 9003:  Spread has been notified about the change
                   SET_SPREAD_OPTION
--------------------------------------------------------
 Spread option 'TokenTimeout' has been set to '35000'.

(1 row)

=> SELECT * FROM V_MONITOR.SPREAD_STATE;
    node_name     | token_timeout
------------------+---------------
 v_vmart_node0001 |         35000
 v_vmart_node0002 |         35000
 v_vmart_node0003 |         35000
(3 rows);

Note

Changing Spread settings with SET_SPREAD_OPTION has minor impact on your cluster as it pauses while the new settings are propagated across the cluster. Because of this delay, changes to the Spread timeout are not immediately visible in system table SPREAD_STATE.

Azure maintenance and spread timeouts

Azure scheduled maintenance on virtual machines might pause nodes longer than the Spread timeout period. If so, OpenText™ Analytics Database is liable to view nodes that do not respond to Spread messages as down and remove them from the database.

The length of Azure maintenance tasks is usually well-defined. For example, memory-preserving updates can pause a VM for up to 30 seconds while performing maintenance on the system hosting the VM. This pause does not disrupt the node, which resumes normal operation after maintenance is complete. To prevent the database from removing nodes while they undergo Azure maintenance, adjust the Spread timeout as needed.

OpenText Analytics Database 26.2.x – Managing nodes

Admin: Stop the database on a node

Important

Administration tools

Command line

Admin: Restart the database on a node

Note

Admin: Setting node type

Note

Admin: Active standby nodes

In this section

Admin: Large cluster

Note

Note

Large cluster and database growth

When to enable large cluster

Note

Admin: Fault groups

Note

Admin: Terrace routing

Important

Terrace routing on Enterprise Mode

Enabling terrace routing

Terrace routing on Eon Mode

Admin: Elastic cluster

Note

Elastic cluster scaling factor

Note

Enabling elastic cluster

Admin: Adding nodes

Important

Admin: Removing nodes

Important

Admin: Replacing nodes

Note

Prerequisites

Best practice for restoring failed hardware

Note

Admin: Rebalancing data across nodes

Why rebalance?

General rebalancing tasks

Rebalancing segmented projections

Rebalancing unsegmented projections

K-safety and rebalancing

Rebalancing failure and projections

Temporary tables

Admin: Redistributing configuration files to nodes

Note

Admin: Stopping and starting nodes on MC

Note

Admin: Upgrading your operating system on nodes in your database cluster

Admin: Reconfiguring node messaging

Note

Changing IP addresses

Tip

Update node IP addresses

re_ip and export IP address

Change node control and broadcast addresses

Tip

Changing node messaging protocols

Setting re_ip timeout

Admin: Adjusting Spread Daemon timeouts for virtual environments

Adjusting spread timeout

Important

Important

Note

Azure maintenance and spread timeouts

See also