OpenText Analytics Database 26.2.x – Managing the database

Admin: Managing nodes

Mon, 01 Jan 0001 00:00:00 +0000

OpenText™ Analytics Database provides the ability to add, remove, and replace nodes on a live cluster that is actively processing queries. This ability lets you scale the database without interrupting users.

Admin: Managing disk space

Mon, 01 Jan 0001 00:00:00 +0000

OpenText™ Analytics Database detects and reports low disk space conditions in the log file so you can address the issue before serious problems occur. It also detects and reports low disk space conditions via SNMP traps if enabled.

Critical disk space issues are reported sooner than other issues. For example, running out of catalog space is fatal; therefore, the database reports the condition earlier than less critical conditions. To avoid database corruption when the disk space falls beyond a certain threshold, the database begins to reject transactions that update the catalog or data.

Caution

A low disk space report indicates one or more hosts are running low on disk space or have a failing disk. It is imperative to add more disk space (or replace a failing disk) as soon as possible.

When the database reports a low disk space condition, use the DISK_RESOURCE_REJECTIONS system table to determine the types of disk space requests that are being rejected and the hosts on which they are being rejected.

To add disk space, see Adding disk space to a node. To replace a failed disk, see Replacing failed disks.

Monitoring disk space usage

You can use these system tables to monitor disk space usage on your cluster:

System table	Description
`DISK_STORAGE`	Monitors the amount of disk storage used by the database on each node.
`COLUMN_STORAGE`	Monitors the amount of disk storage used by each column of each projection on each node.
`PROJECTION_STORAGE`	Monitors the amount of disk storage used by each projection on each node.

Admin: Memory usage reporting

Mon, 01 Jan 0001 00:00:00 +0000

OpenText™ Analytics Database periodically polls its own memory usage to determine whether it is below the threshold that is set by configuration parameter MemoryPollerReportThreshold.Polling occurs at regular intervals—by default, every 2 seconds—as set by configuration parameter MemoryPollerIntervalSec.

The memory poller compares MemoryPollerReportThreshold with the following expression:

RSS / available-memory

When this expression evaluates to a value higher than MemoryPollerReportThreshold—by default, set to 0.93, then the memory poller writes a report to MemoryReport.log, in the database working directory. This report includes information about memory pools, how much memory is consumed by individual queries and session, and so on. The memory poller also logs the report as an event in system table MEMORY_EVENTS, where it sets EVENT_TYPE to MEMORY_REPORT.

The memory poller also checks for excessive glibc allocation of free memory (glibc memory bloat). For details, see Memory trimming.

Admin: Memory trimming

Mon, 01 Jan 0001 00:00:00 +0000

Under certain workloads, glibc can accumulate a significant amount of free memory in its allocation arena. This memory consumes physical memory as indicated by its usage of resident set size (RSS), which glibc does not always return to the operating system. High retention of physical memory by glibc—glibc memory bloat—can adversely affect other processes, and, under high workloads, can sometimes cause the database to run out of memory.

OpenText™ Analytics Database provides two configuration parameters that let you control how frequently the database detects and consolidates much of the glibc-allocated free memory, and then returns it to the operating system:

MemoryPollerTrimThreshold: Sets the threshold for the memory poller to start checking whether to trim glibc-allocated memory.
The memory poller compares MemoryPollerTrimThreshold—by default, set to 0.83— with the following expression:
```
RSS / available-memory
```
If this expression evaluates to a value higher than MemoryPollerTrimThreshold, then the memory poller starts checking the next threshold—set in MemoryPollerMallocBloatThreshold—for glibc memory bloat.

Note
On high-memory machines where very large database RSS values are atypical, consider a higher setting for MemoryPollerTrimThreshold. To turn off auto-trimming, set this parameter to 0.
MemoryPollerMallocBloatThreshold: Sets the threshold of glibc memory bloat.
The memory poller calls glibc function malloc_info() to obtain the amount of free memory in malloc. It then compares MemoryPollerMallocBloatThreshold—by default, set to 0.3—with the following expression:
```
free-memory-in-malloc / RSS
```
If this expression evaluates to a value higher than MemoryPollerMallocBloatThreshold, the memory poller calls glibc function malloc_trim(). This function reclaims free memory from malloc and returns it to the operating system. Details on calls to malloc_trim() are written to system table MEMORY_EVENTS.

For example, the memory poller calls malloc_trim() when the following conditions are true:
- MemoryPollerMallocBloatThreshold is set to 0.5.
- malloc_info() returns 15GB memory in malloc free.
- RSS is 30GB.
Note
This parameter is ignored if MemoryPollerTrimThreshold is set to 0 (disabled).

Trimming memory manually

If auto-trimming is disabled, you can manually reduce glibc-allocated memory by calling function MEMORY_TRIM. This function calls malloc_trim().

Admin: Tuple mover

Mon, 01 Jan 0001 00:00:00 +0000

The Tuple Mover manages ROS data storage. On mergeout, it combines small ROS containers into larger ones and purges deleted data. The Tuple Mover automatically performs these tasks in the background.

The database mode affects which nodes perform Tuple Mover operations:

In an Enterprise Mode database, all nodes run the Tuple Mover to perform mergeout operations on the data they store.
In Eon Mode, the primary subscriber to each shard plans Tuple Mover mergeout operations on the ROS containers in the shard. It can delegate the execution of this plan to another node in the cluster.

Tuple Mover operations typically require no intervention. However, Vertica provides various ways to adjust Tuple Mover behavior. For details, see Managing the tuple mover.

The tuple mover in Eon Mode databases

In Eon Mode, the Tuple Mover's operations are broken into two parts: mergeout planning and mergeout execution. Mergeout planning is always carried out by the primary subscribers of the shards involved in the mergeout. These primary subscribers are part of same the primary subcluster. As part of its mergeout planning, the primary subscriber chooses a node to execute the mergeout plan. It uses two criteria to decide which node should execute the mergeout:

Only nodes that have memory allocated to their TM resource pool are eligible to perform a mergeout. The primary subscriber ignores all nodes in subclusters whose TM pool's MEMORYSIZE and MAXMEMORYSIZE settings are 0.
From the group of nodes able to execute a mergeout, the primary subscriber chooses the node that has the most ROS containers in its depot that are involved in the mergeout.

Limiting which subclusters perform mergeout tasks

You can prevent a secondary subcluster from being assigned mergeout tasks by changing the MEMORYSIZE and MAXMEMORYSIZE settings of the its TM pool to 0. These settings prevent the primary subscribers from assigning mergeout tasks to nodes in the subcluster.

Important

Primary subclusters must always be able to execute mergeout tasks. Only change these settings on secondary subclusters.

For example, this statement prevents the subcluster named dashboard from running mergeout tasks.

=> ALTER RESOURCE POOL TM FOR SUBCLUSTER dashboard MEMORYSIZE '0%'
   MAXMEMORYSIZE '0%';

Admin: Managing workloads

Mon, 01 Jan 0001 00:00:00 +0000

OpenText™ Analytics Database's resource management scheme allows diverse, concurrent workloads to run efficiently on the database. For basic operations, the database pre-configures the built-in GENERAL pool based on RAM and machine cores. You can customize the General pool to handle specific concurrency requirements.

You can also define new resource pools that you configure to limit memory usage, concurrency, and query priority. You can then optionally assign each database user to use a specific resource pool, which controls memory resources used by their requests.

User-defined pools are useful if you have competing resource requirements across different classes of workloads. Example scenarios include:

A large batch job takes up all server resources, leaving small jobs that update a web page without enough resources. This can degrade user experience.

In this scenario, create a resource pool to handle web page requests and ensure users get resources they need. Another option is to create a limited resource pool for the batch job, so the job cannot use up all system resources.
An application has lower priority than other applications and you want to limit the amount of memory and number of concurrent users for the low-priority application.

In this scenario, create a resource pool with an upper limit on the query's memory and associate the pool with users of the low-priority application.

You can also use resource pools to manage resources assigned to running queries. You can assign a run-time priority to a resource pool, as well as a threshold to assign different priorities to queries with different durations. See Managing resources at query run time for more information.

Enterprise Mode and Eon Mode

In Enterprise Mode, there is one global set of resource pools for the entire database. In Eon Mode, you can allocate resources globally or per subcluster. See Managing workload resources in an Eon Mode database for more information.

Admin: Node Management Agent

Mon, 01 Jan 0001 00:00:00 +0000

The Node Management Agent (NMA) lets you administer your cluster with a REST API. The NMA listens on port 5554 and runs on all nodes.

Start the NMA

To start the NMA, run the following on any Vertica node. In addition, if you want to use the recommended vcluster utility to interact with NMA and the HTTPS service, you must run it on all nodes:

$ /opt/vertica/bin/manage_node_agent.sh start node_management_agent

To verify that the NMA is running, you can send a GET request to /v1/health, which returns {"healthy":"true"} if the NMA is running.

When you first start the NMA, Vertica recommends that you perform this verification from inside the cluster. While you can and should still verify that the NMA is reachable from outside the cluster, doing it first from inside the cluster removes possible network and environmental interference:

$ curl https://localhost:5554/v1/health -k

{"healthy":"true"}

To send this and other requests from outside the cluster, see Endpoints.

If the request to /v1/health hangs or otherwise fails, perform the following troubleshooting steps:

Verify that port 5554 is not being used by any other process on the target node.
Verify that the host and port 5554 are accessible by the client.
Open /opt/vertica/log/node_management_agent.log and verify that the endpoint can reach the NMA service.

Stop the NMA

To stop the NMA, send a PUT request to /v1/nma/shutdown:

For simplicity, the following command is run from a Vertica node and specifies paths for certificates generated by the install_vertica script. To send this and other requests from outside the cluster, see Endpoints:

$ curl -X PUT https://localhost:5554/v1/nma/shutdown -k \
    --key /opt/vertica/config/https_certs/dbadmin.key \
    --cert /opt/vertica/config/https_certs/dbadmin.pem \
    --cacert /opt/vertica/config/https_certs/rootca.pem

{"shutdown_error":"Null","shutdown_message":"NMA server stopped","shutdown_scheduled":"NMA server shutdown scheduled"}

Admin: HTTPS service

Mon, 01 Jan 0001 00:00:00 +0000

The HTTPS service lets clients securely access and manage an OpenText™ Analytics Database with a REST API. This service listens on port 8443 and runs on all nodes.

Most HTTPS service endpoints require authentication, and only the dbadmin user can authenticate to the HTTPS service. The following endpoints serve documentation on the endpoints and do not require authentication (unless your TLSMODE is VERIFY_CA):

/swagger/ui
/swagger/{RESOURCE}
/api-docs/oas-3.0.0.json

This service encrypts communications with mutual TLS (mTLS). To configure mTLS, you must alter the server TLS configuration with a server certificate and a trusted Certificate Authority (CA). For mTLS authentication, each client request must include a certificate that is signed by the CA in the server TLS configuration and specifies the dbadmin user in the Common Name (CN). For additional details about these TLS components, see TLS protocol.

Important

During installation, the install_vertica script generates self-signed certificates in the /opt/vertica/config/https_certs directory. Vertica uses these certificates to bootstrap the HTTPS service on a new cluster—they are not suitable for production. Certificates in the TLS configuration supersede those in the /opt/vertica/config/https_certs directory.

Password authentication

The following command connects to the HTTPS service from outside the cluster with the username and password:

$ curl --insecure --user dbadmin:db-password https://10.20.30.40:8443/endpoint

Important

Due to security concerns, this request method is not recommended. For example, the command history can save the dbadmin password.

Certificate authentication

Client requests authenticate to the HTTPS service with a private key and certificate:

$ curl https://10.20.30.40:8443/endpoint \
    --key path/to/client_key.key \
    --cert path/to/client_cert.pem \

When the database server receives the request, it verifies that the client certificate is signed by a trusted CA and specifies the dbadmin user. To establish this workflow, you must complete the following:

Alter the server TLS configuration with a server certificate and a CA.
Generate a client certificate that is signed by the CA in the server TLS configuration. The client certificate SUBJECT must specify the dbadmin user.
Grant TLS access to the database.

Note

To demonstrate a comprehensive setup, the following sections use a self-signed CA certificate that signs both the client and server certificates. In a production environment, you should replace the self-signed CA with a trusted CA.

For details about importing a CA certificate, see Generating TLS certificates and keys.

Create a CA certificate

Important

A self-signed CA certificate is convenient for development purposes, but you should always use a proper certificate authority in a production environment.

A CA is a trusted entity that signs and validates other certificates with its own certificate. The following example generates a self-signed root CA certificate:

Generate or import a private key. The following command generates a new private key:
```
      
=> CREATE KEY ca_private_key TYPE 'RSA' LENGTH 4096;
CREATE KEY
```

Generate the certificate with the following format. Sign the certificate with the private key that you generated or imported in the previous step:

      
=> CREATE CA CERTIFICATE ca_certificate
SUBJECT '/C=country_code/ST=state_or_province/L=locality/O=organization/OU=org_unit/CN=Vertica Root CA'
VALID FOR days_valid
EXTENSIONS 'authorityKeyIdentifier' = 'keyid:always,issuer', 'nsComment' = 'Vertica generated root CA cert'
KEY ca_private_key;

Note

The CA certificate SUBJECT must be different from the SUBJECT of any certificate that it signs.

For example:

      
=> CREATE CA CERTIFICATE SSCA_cert
SUBJECT '/C=US/ST=Massachusetts/L=Cambridge/O=OpenText/OU=Vertica/CN=Vertica Root CA'
VALID FOR 3650
EXTENSIONS 'nsComment' = 'Self-signed root CA cert'
KEY SSCA_key;

Create the server certificate

The server private key and certificate verify the database server's identity for clients:

Generate the server private key:

      
=> CREATE KEY server_private_key TYPE 'RSA' LENGTH 2048;
CREATE KEY

Generate the server certificate with the following format. Include the server_private_key, and sign it with the CA certificate:

      
=> CREATE CERTIFICATE server_certificate
SUBJECT '/C=country_code/ST=state_or_province/L=locality/O=organization/OU=org_unit/CN=Vertica server certificate'
SIGNED BY ca_certificate
KEY server_private_key;
CREATE CERTIFICATE

For example:

      
=> CREATE CERTIFICATE server_certificate
SUBJECT '/C=US/ST=Massachusetts/L=Burlington/O=OpenText/OU=Vertica/CN=Vertica server certificate'
SIGNED BY ca_certificate
KEY server_private_key;
CREATE CERTIFICATE

Alter the TLS configuration

After you generate the server certificate, you must alter the server's default TLS configuration with the server certificate and its CA. When you change the server TLS configuration, the HTTPS service restarts, and the new keys and certificates are added to the catalog and distributed to the nodes in the cluster:

Alter the default server configuration. Mutual TLS requires that you set TLSMODE to TRY_VERIFY or VERIFY_CA. If you use VERIFY_CA, all endpoints (including the documentation-related endpoints /swagger/ui, /swagger/{RESOURCE}, and /api-docs/oas-3.0.0.json) require authentication:
```
      
=> ALTER TLS CONFIGURATION server CERTIFICATE server_certificate ADD CA CERTIFICATES ca_certificate TLSMODE 'VERIFY_CA';
ALTER TLS CONFIGURATION
```

Verify the changes on the TLS configuration object:

 => SELECT name, certificate, ca_certificate, mode FROM TLS_CONFIGURATIONS WHERE name='server';
   name  |    certificate     | ca_certificate |    mode
 --------+--------------------+----------------+------------
  server | server_certificate | ca_certificate | VERIFY_CA
 (1 row)

Create the client certificate

The client private key and certificate verify the client's identity for requests. Generate a client private key and a client certificate that specifies the dbadmin user and sign the client certificate with the same CA that signed the server certificate.

The following steps generate a client key and certificate, and then make them available to the client:

Generate the client key:

      
   => CREATE KEY client_private_key TYPE 'RSA' LENGTH 2048;
   CREATE KEY

Generate the client certificate. Mutual TLS requires that the Common Name (CN) in the SUBJECT specifies a database username:

      
   => CREATE CERTIFICATE client_certificate
   SUBJECT '/C=US/ST=Massachusetts/L=Cambridge/O=OpenText/OU=Vertica/CN=dbadmin/emailAddress=example@example.com'
   SIGNED BY ca_certificate
   EXTENSIONS 'nsComment' = 'Vertica client cert', 'extendedKeyUsage' = 'clientAuth'
   KEY client_private_key;
   CREATE CERTIFICATE

On the client machine, export the client key and client certificate to the client filesystem. The following commands use the vsql client:
```
$ vsql -At -c "SELECT key FROM cryptographic_keys WHERE name = 'client_private_key';" -o client_private_key.key
$ vsql -At -c "SELECT certificate_text FROM certificates WHERE name = 'client_certificate';" -o client_cert.pem
```
In the preceding command:
- -A: enables unaligned output.
- -t: prevents the command from outputting metadata, such as column names.
- -c: instructs the shell to run one command and then exit.
- -o: writes the query output to the specified filename.
For details about all vsql command line options, see Command-line options.
Copy or move the client key and certificate to a location that your client recognizes.

The following commands move the client key and certificate to the hidden directory ~/.client-creds, and then grants the file owner read and write permissions with chmod:
```
$ mkdir ~/.client-creds
$ mv client_private_key.key ~/.client-creds/client_key.key
$ mv client_cert.pem ~/.client-creds/client_cert.pem
$ chmod 600 ~/.client-creds/client_key.key ~/.client-creds/client_cert.pem
```

Create an authentication record

Next, you must create an authentication record in the database. An authentication record defines a set of authentication and the access methods for the database. You grant this record to a user or role to control how they authenticate to the database:

Create the authentication record. The tls method requires that clients authenticate with a certificate whose Common Name (CN) specifies a database username:
```
      
   => CREATE AUTHENTICATION auth_record METHOD 'tls' HOST TLS '0.0.0.0/0';
   CREATE AUTHENTICATION
   
```
Grant the authentication record to a user or to a role. The following example grants the authentication record to PUBLIC, the default role for all users:
```
      
   => GRANT AUTHENTICATION auth_record TO PUBLIC;
   GRANT AUTHENTICATION
   
```

After you grant the authentication record, the user or role can access HTTPS service endpoints.

Mutual TLS for cluster operations

When you configure a database for TLS/SSL security in mutual mode, incoming client requests verify the server certificate. Each client needs to present a certificate and private key so that the server can verify the client. For the server, this means validating that the client certificate was signed by a chain of CA certificates terminating in a trusted CA certificate, typically but not necessarily a root (self-signed) CA certificate. For the client, this means performing the same validation as the server for the server certificate and ensuring that the server certificate belongs to the intended server.

Important

Validating the server hostname is optional. It is based on your network topology, client usage, and security requirements.

Generate and set the certificate for the NMA or HTTPS service with hostname validation

When generating a server certificate where the client validates the server hostname, the certificate must be valid for the expected hostnames and IPs.


=> CREATE CERTIFICATE certificate_usable_by_nma_and_https_service
SUBJECT '/C=US/ST=Massachusetts/L=Cambridge/O=OpenText/OU=Vertica/CN=Vertica server alt/emailAddress=doesntmatter@example.com'
SIGNED BY rootca
EXTENSIONS 'nsComment' = 'Vertica/NMA server cert with IP hostname', 'extendedKeyUsage' = 'serverAuth,clientAuth',
'subjectAltName' = 'IP.1:192.168.1.101,IP.2:192.168.1.102,IP.3:192.168.1.103,IP.4:192.168.1.104,DNS.1:mynode1.mydomain.com,DNS.2:mynode2.mydomain.com,DNS.3:mynode3.mydomain.com,DNS.4:mynode4.mydomain.com'

There are 4 IPs in this example, as it is a 4-node cluster. Generate the NMA certificate with 'serverAuth,clientAuth' as the extendedKeyUsage option when the server TLS configuration in the database is in VERIFY_CA mode or higher. Otherwise, use serverAuth.

Enable Mutual TLS for cluster operations

To enable mutual TLS for cluster operations:

Generate the server and client certificates. For more information, see Create the server certificate and Create the client certificate. To generate the server certificate, use the subjectAltName setting mentioned in Generate and set the certificate for the NMA or HTTPS service with hostname validation.
Either create a separate NMA client/server certificate or reuse the database server certificate that is generated. Specify 'extendedKeyUsage' = 'serverAuth,clientAuth' when generating the certificate to allow NMA to present a certificate to the database.
Export the NMA certificate and key and the CA certificate to a location accessible to each node on the cluster.
For each node, point the NMA at the new certificate, key, and CA certificate. For more information, see Custom certificates.
Set the TLS mode environment variable for NMA before starting NMA as shown here.
```
export NMA_CLIENT_TLS_MODE=VERIFY_FULL
```
Restart the NMA on each node.
Export the CA certificate from Step 1 to a location accessible to the client.
Configure the server TLS CONFIGURATION object as mentioned above. Use VERIFY_CA for the TLS mode option.
For clients calling the API, use the exported client certificate, client key, and trusted CA certificate to perform mutual TLS.

TLSMode environment variable for SQL client connections

The environment variable NMA_CLIENT_TLS_MODE controls TLS mode for NMA SQL client connections. You must export the value on NMA startup. The TLS modes are:

enable # the default
verify_ca
verify_full

Each value specifies a different set of requirements that the database server needs to connect with NMA for operations using the NMA as a SQL proxy. The database server has requirements of the NMA certificate, but this is independent of the NMA requirements of the database server certificate.

Enable – This is the current behavior. The system accepts database server certificates with no validation.
verify_ca – This means the database server certificate is signed by a trusted CA.
verify_full – This means the database server certificate is signed by a trusted CA and validated for the expected hostname.