Vertica Documentation – Managing the database

Admin: Managing nodes

Mon, 01 Jan 0001 00:00:00 +0000

Vertica provides the ability to add, remove, and replace nodes on a live cluster that is actively processing queries. This ability lets you scale the database without interrupting users.

In this section

Admin: Managing disk space

Mon, 01 Jan 0001 00:00:00 +0000

Vertica detects and reports low disk space conditions in the log file so you can address the issue before serious problems occur. It also detects and reports low disk space conditions via SNMP traps if enabled.

Critical disk space issues are reported sooner than other issues. For example, running out of catalog space is fatal; therefore, Vertica reports the condition earlier than less critical conditions. To avoid database corruption when the disk space falls beyond a certain threshold, Vertica begins to reject transactions that update the catalog or data.

Caution

A low disk space report indicates one or more hosts are running low on disk space or have a failing disk. It is imperative to add more disk space (or replace a failing disk) as soon as possible.

When Vertica reports a low disk space condition, use the DISK_RESOURCE_REJECTIONS system table to determine the types of disk space requests that are being rejected and the hosts on which they are being rejected.

To add disk space, see Adding disk space to a node. To replace a failed disk, see Replacing failed disks.

Monitoring disk space usage

You can use these system tables to monitor disk space usage on your cluster:

System table	Description
`DISK_STORAGE`	Monitors the amount of disk storage used by the database on each node.
`COLUMN_STORAGE`	Monitors the amount of disk storage used by each column of each projection on each node.
`PROJECTION_STORAGE`	Monitors the amount of disk storage used by each projection on each node.

Admin: Memory usage reporting

Mon, 01 Jan 0001 00:00:00 +0000

Vertica periodically polls its own memory usage to determine whether it is below the threshold that is set by configuration parameter MemoryPollerReportThreshold.Polling occurs at regular intervals—by default, every 2 seconds—as set by configuration parameter MemoryPollerIntervalSec.

The memory poller compares MemoryPollerReportThreshold with the following expression:

RSS / available‑memory

When this expression evaluates to a value higher than MemoryPollerReportThreshold—by default, set to 0.93, then the memory poller writes a report to MemoryReport.log, in the Vertica working directory. This report includes information about Vertica memory pools, how much memory is consumed by individual queries and session, and so on. The memory poller also logs the report as an event in system table MEMORY_EVENTS, where it sets EVENT_TYPE to MEMORY_REPORT.

The memory poller also checks for excessive glibc allocation of free memory (glibc memory bloat). For details, see Memory trimming.

Admin: Memory trimming

Mon, 01 Jan 0001 00:00:00 +0000

Under certain workloads, glibc can accumulate a significant amount of free memory in its allocation arena. This memory consumes physical memory as indicated by its usage of resident set size (RSS), which glibc does not always return to the operating system. High retention of physical memory by glibc—glibc memory bloat—can adversely affect other processes, and, under high workloads, can sometimes cause Vertica to run out of memory.

Vertica provides two configuration parameters that let you control how frequently Vertica detects and consolidates much of the glibc-allocated free memory, and then returns it to the operating system:

MemoryPollerTrimThreshold: Sets the threshold for the memory poller to start checking whether to trim glibc‑allocated memory.
The memory poller compares MemoryPollerTrimThreshold—by default, set to 0.83— with the following expression:
```
RSS / available‑memory
```
If this expression evaluates to a value higher than MemoryPollerTrimThreshold, then the memory poller starts checking the next threshold—set in MemoryPollerMallocBloatThreshold—for glibc memory bloat.

Note
On high-memory machines where very large Vertica RSS values are atypical, consider a higher setting for MemoryPollerTrimThreshold. To turn off auto-trimming, set this parameter to 0.
MemoryPollerMallocBloatThreshold: Sets the threshold of glibc memory bloat.
The memory poller calls glibc function malloc_info() to obtain the amount of free memory in malloc. It then compares MemoryPollerMallocBloatThreshold—by default, set to 0.3—with the following expression:
```
free‑memory‑in‑malloc / RSS
```
If this expression evaluates to a value higher than MemoryPollerMallocBloatThreshold, the memory poller calls glibc function malloc_trim(). This function reclaims free memory from malloc and returns it to the operating system. Details on calls to malloc_trim() are written to system table MEMORY_EVENTS.

For example, the memory poller calls malloc_trim() when the following conditions are true:
- MemoryPollerMallocBloatThreshold is set to 0.5.
- malloc_info() returns 15GB memory in malloc free.
- RSS is 30GB.
Note
This parameter is ignored if MemoryPollerTrimThreshold is set to 0 (disabled).

Trimming memory manually

If auto-trimming is disabled, you can manually reduce glibc‑allocated memory by calling Vertica function MEMORY_TRIM. This function calls malloc_trim().

Admin: Tuple mover

Mon, 01 Jan 0001 00:00:00 +0000

The Tuple Mover manages ROS data storage. On mergeout, it combines small ROS containers into larger ones and purges deleted data. The Tuple Mover automatically performs these tasks in the background.

The database mode affects which nodes perform Tuple Mover operations:

In an Enterprise Mode database, all nodes run the Tuple Mover to perform mergeout operations on the data they store.
In Eon Mode, the primary subscriber to each shard plans Tuple Mover mergeout operations on the ROS containers in the shard. It can delegate the execution of this plan to another node in the cluster.

Tuple Mover operations typically require no intervention. However, Vertica provides various ways to adjust Tuple Mover behavior. For details, see Managing the tuple mover.

The tuple mover in Eon Mode databases

In Eon Mode, the Tuple Mover's operations are broken into two parts: mergeout planning and mergeout execution. Mergeout planning is always carried out by the primary subscribers of the shards involved in the mergeout. These primary subscribers are part of same the primary subcluster. As part of its mergeout planning, the primary subscriber chooses a node to execute the mergeout plan. It uses two criteria to decide which node should execute the mergeout:

Only nodes that have memory allocated to their TM resource pool are eligible to perform a mergeout. The primary subscriber ignores all nodes in subclusters whose TM pool's MEMORYSIZE and MAXMEMORYSIZE settings are 0.
From the group of nodes able to execute a mergeout, the primary subscriber chooses the node that has the most ROS containers in its depot that are involved in the mergeout.

Limiting which subclusters perform mergeout tasks

You can prevent a secondary subcluster from being assigned mergeout tasks by changing the MEMORYSIZE and MAXMEMORYSIZE settings of the its TM pool to 0. These settings prevent the primary subscribers from assigning mergeout tasks to nodes in the subcluster.

Important

Primary subclusters must always be able to execute mergeout tasks. Only change these settings on secondary subclusters.

For example, this statement prevents the subcluster named dashboard from running mergeout tasks.

=> ALTER RESOURCE POOL TM FOR SUBCLUSTER dashboard MEMORYSIZE '0%'
   MAXMEMORYSIZE '0%';

Admin: Managing workloads

Mon, 01 Jan 0001 00:00:00 +0000

Vertica's resource management scheme allows diverse, concurrent workloads to run efficiently on the database. For basic operations, Vertica pre-configures the built-in GENERAL pool based on RAM and machine cores. You can customize the General pool to handle specific concurrency requirements.

You can also define new resource pools that you configure to limit memory usage, concurrency, and query priority. You can then optionally assign each database user to use a specific resource pool, which controls memory resources used by their requests.

User-defined pools are useful if you have competing resource requirements across different classes of workloads. Example scenarios include:

A large batch job takes up all server resources, leaving small jobs that update a web page without enough resources. This can degrade user experience.

In this scenario, create a resource pool to handle web page requests and ensure users get resources they need. Another option is to create a limited resource pool for the batch job, so the job cannot use up all system resources.
An application has lower priority than other applications and you want to limit the amount of memory and number of concurrent users for the low-priority application.

In this scenario, create a resource pool with an upper limit on the query's memory and associate the pool with users of the low-priority application.

You can also use resource pools to manage resources assigned to running queries. You can assign a run-time priority to a resource pool, as well as a threshold to assign different priorities to queries with different durations. See Managing resources at query run time for more information.

Enterprise Mode and Eon Mode

In Enterprise Mode, there is one global set of resource pools for the entire database. In Eon Mode, you can allocate resources globally or per subcluster. See Managing workload resources in an Eon Mode database for more information.

Vertica Documentation – Managing the database

Admin: Managing nodes

In this section

Admin: Managing disk space

Caution

Monitoring disk space usage

Admin: Memory usage reporting

Admin: Memory trimming

Note

Note

Trimming memory manually

Admin: Tuple mover

The tuple mover in Eon Mode databases

Limiting which subclusters perform mergeout tasks

Important

Admin: Managing workloads

Enterprise Mode and Eon Mode