This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Removing nodes

Although less common than adding a node, permanently removing a node is useful if the host system is obsolete or over-provisioned.

Although less common than adding a node, permanently removing a node is useful if the host system is obsolete or over-provisioned.

1 - Automatic eviction of unhealthy nodes

To manage the health of the nodes in your cluster, Vertica performs regular health checks by sending and receiving "heartbeats." During a health check, each node in the cluster verifies read-write access to its catalog, catalog disk, and local storage locations ('TEMP, DATA', TEMP, DATA, and DEPOT).

To manage the health of the nodes in your cluster, Vertica performs regular health checks by sending and receiving "heartbeats." During a health check, each node in the cluster verifies read-write access to its catalog, catalog disk, and local storage locations ('TEMP, DATA', TEMP, DATA, and DEPOT). Upon verification, the node sends a heartbeat. If a node fails to send a heartbeat after five intervals (fails five health checks), then the node is evicted from the cluster.

You can control the time between each health check with the DatabaseHeartBeatInterval parameter. By default, DatabaseHeartBeatInterval is set to 120, which allows five 120-second intervals to pass without a heartbeat.

The amount of time allowed before an eviction is:

TOT = DHBI * 5

where TOT is the total time (in seconds) allowed without a heartbeat before eviction, and DHBI is equal to the value of DatabaseHeartBeatInterval.

If you set the DatabaseHeartBeatInterval too low, it can cause evictions in cases of brief node health issues. Sometimes, such premature evictions result in lower availability and performance of the Vertica database.

See also

DatabaseHeartbeatInterval in General parameters

2 - Lowering K-Safety to enable node removal

A database with a K-safety level of 1 requires at least three nodes to operate, and a database with a K-safety level 2 requires at least 5 nodes to operate.

A database with a K-safety level of 1 requires at least three nodes to operate, and a database with a K-safety level 2 requires at least 5 nodes to operate. You can check the cluster's current K-safety level as follows:

=> SELECT current_fault_tolerance FROM system;
 current_fault_tolerance
-------------------------
                       1
(1 row)

To remove a node from a cluster with the minimum number of nodes that it requires for K-safety, first lower the K-safety level with MARK_DESIGN_KSAFE.

  1. Connect to the database with Administration Tools or vsql.

  2. Call the function MARK_DESIGN_KSAFE:

    SELECT MARK_DESIGN_KSAFE(n);

    where n is the new K-safety level for the database.

3 - Removing nodes from a database

In an Eon Mode database, you remove nodes from the subcluster that contains them, rather than from the database.

As long as there are enough nodes remaining to satisfy the K-Safety requirements, you can remove the node from a database. You cannot drop nodes that are critical for K-safety. See Lowering K-Safety to enable node removal.

You can remove nodes from a database using one of the following:

  • Management Console interface

  • Administration Tools

Prerequisites
Before removing a node from the database, verify that the database complies with the following requirements:

  • It is running.

  • It has been backed up.

  • The database has the minimum number of nodes required to comply with K-safety. If necessary, temporarily lower the database K-safety level.

  • All of the nodes in your database must be either up or in active standby. Vertica reports the error "All nodes must be UP or STANDBY before dropping a node" if you attempt to remove a node while a database node is down. You will get this error, even if you are trying to remove the node that is down.

Management Console

Remove nodes with Management Console from its Manage page:

Remove database nodes as follows:

  1. Choose the node to remove.

  2. Click Remove node in the Node List.

The following restrictions apply:

  • You can only remove nodes that belong to the database cluster.

  • You cannot remove DOWN nodes.

When you remove a node, its state changes to STANDBY. You can later add STANDBY nodes back to the database.

Administration tools

To remove unused hosts from the database using Administration Tools:

  1. Open the Administration Tools. See Using the administration tools for information about accessing the Administration Tools.

  2. On the Main Menu, select View Database Cluster State to verify that the database is running. If the database is not running, start it.

  3. From the Main Menu, choose Advanced Menu and choose OK.

  4. In the Advanced menu, choose Cluster Management and choose OK.

  5. In the Cluster Management menu, choose Remove Host(s) from Database and choose OK.

  6. When warned that you must redesign your database and create projections that exclude the hosts you are going to drop, choose Yes.

  7. Select the database from which you want to remove the hosts and choose OK.

    A list of currently active hosts appears.

  8. Select the hosts you want to remove from the database and choose OK.

  9. When prompted, choose OK to confirm that you want to remove the hosts.

  10. When informed that the hosts were successfully removed, choose OK.

  11. If you removed a host from a Large Cluster configuration, open a vsql session and run realign_control_nodes:

    SELECT realign_control_nodes();

    For more details, see REALIGN_CONTROL_NODES.

  12. If this host is not used by any other database in the cluster, you can remove the host from the cluster. See Removing hosts from a cluster.

4 - Removing hosts from a cluster

If a host that you removed from the database is not used by any other database, you can remove it from the cluster with.

If a host that you removed from the database is not used by any other database, you can remove it from the cluster with update_vertica . You can leave the database running during this operation.

When you use update_vertica to reduce the size of the cluster, it also performs these tasks:

  • Modifies the spread to match the smaller cluster.

  • Configures Administration tools to work with the smaller cluster.

From one of the Vertica cluster hosts, run update_vertica with the –-remove-hosts switch. This switch takes an list of comma-separated hosts to remove from the cluster. You can reference hosts by their names or IP addresses. For example, you can remove hosts host01, host02, and host03 as follows:

 # /opt/vertica/sbin/update_vertica --remove-hosts host01,host02,host03 \
            --rpm /tmp/vertica-version.RHEL8.x86_64.rpm \
            --dba-user mydba

If --rpm specifies a new RPM, then Vertica installs it on the existing cluster hosts before proceeding.

update_vertica uses the same options as install_vertica.For all options, see Install Vertica with the installation script.

Requirements

  • If -remove-hosts specifies a list of multiple hosts, the list must not embed any spaces between hosts.

  • Use the same command line options as in the original installation. If you used non-default values for the database administrator username, password, or directory path, provide the same settings when you remove hosts; otherwise; the procedure fails. Consider saving the original installation options in a properties file that you can reuse on subsequent installation and update operations. See Install Vertica silently.

5 - Remove nodes from an AWS cluster

Use the following procedures to remove instances/nodes from an AWS cluster.

Use the following procedures to remove instances/nodes from an AWS cluster.

To avoid data loss, Vertica strongly recommends that you back up your database before removing a node. For details, see Backing up and restoring the database.

Remove hosts from the database

Before you remove hosts from the database, verify that you have:

  • Backed up the database.

  • Lowered the K-safety of the database.

To remove a host from the database:

  1. While logged on as dbadmin, launch Administration Tools.

    $ /opt/vertica/bin/admintools

  2. From the Main Menu, select Advanced Menu.

  3. From Advanced Menu, select Cluster Management. ClickOK.

  4. From Cluster Management, select Remove Host(s). Click OK.

  5. From Select Database, choose the database from which you plan to remove hosts. Click OK.

  6. Select the host(s) to remove. Click OK.

  7. Click Yes to confirm removal of the hosts.

  8. Click OK. The system displays a message telling you that the hosts have been removed. Automatic rebalancing also occurs.

  9. Click OK to confirm. Administration Tools brings you back to the Cluster Management menu.

Remove nodes from the cluster

To remove nodes from a cluster, run the update_vertica script and specify:

  • The option --remove-hosts, followed by the IP addresses of the nodes you are removing.

  • The option --ssh-identity, followed by the location and name of your *pem file.

  • The option --dba-user-password-disabled.

The following example removes one node from the cluster:

$ sudo /opt/vertica/sbin/update_vertica  --remove-hosts 10.0.11.165  --point-to-point  \
  --ssh-identity ~/name-of-pem.pem --dba-user-password-disabled

Stop the AWS instances

After you have removed one or more nodes from your cluster, to save costs associated with running instances, you can choose to stop the AWS instances that were previously part of your cluster.

To stop an instance in AWS:

  1. On AWS, navigate to your Instances page.

  2. Right-click the instance, and choose Stop.

This step is optional because, after you have removed the node from your Vertica cluster, Vertica no longer sees the node as part of the cluster, even though it is still running within AWS.