OpenText Analytics Database 26.2.x – Monitoring the database

Admin: Monitoring log files

Mon, 01 Jan 0001 00:00:00 +0000

When a database is running

When an OpenText™ Analytics Database is running, each node in the cluster writes messages into a file named vertica.log. For example, the Tuple Mover and the transaction manager write INFO messages into vertica.log at specific intervals even when there is no mergeout activity.

You configure the location of the vertica.log file. By default, the log file is in:

catalog-path/database-name/node-name_catalog/vertica.log

catalog-path is the path shown in the NODES system table minus the Catalog directory at the end.
database-name is the name of your database.
node-name is the name of the node shown in the NODES system table.

Note

OpenText often changes the format or content of log files in subsequent releases to benefit both customers and customer support.

To monitor one node in a running database in real time:

In a terminal window enter:

$ tail -f catalog-path/database-name/node-name_catalog/vertica.log

Note

To monitor your overall database (rather than an individual node/host), use the Data Collector, which records system activities and performance. See Data Collector utility for more on Data Collector.

`catalog-path`	The catalog pathname specified when you created the database. See Creating a database.
`database-name`	The database name (case sensitive)
`node-name`	The node name, as specified in the database definition. See Viewing a database.

When the database/node is starting up

During system startup, before the log has been initialized to write messages, each node in the cluster writes messages into a file named dbLog. This log is useful to diagnose situations where the database fails to start before it can write messages into vertica.log. The dblog is located at the following path, using catalog-path and database-name as described above:

catalog-path/database-name/dbLog

Admin: Rotating log files

Mon, 01 Jan 0001 00:00:00 +0000

The LogRotate service periodically rotates logs and removes old rotated logs. To view previous LogRotate events, see LOG_ROTATE_EVENTS.

If the following files exceed the specified maximum size, they are rotated:

vertica.log
UDxFencedProcesses.log
MemoryReport.log
editor.log
dbLog

Rotated files are compressed and marked with a timestamp in the same location as the original log file: path/to/logfile.logtimestamp.gz. For example, /scratch_b/qa/VMart/v_vmart_node0001_catalog/vertica.log is rotated to /scratch_b/qa/VMart/v_vmart_node0001_catalog/vertica.log.2023-11-08-14-09-02-381909-05.gz.

If a log file was rotated, the previously rotated logs (.gz) in that directory are checked against the specified maximum age. Rotated logs older than the maximum age are deleted.

Upgrade behavior

The LogRotate service is automatically enabled, but if your database contains the configuration file for the Linux logrotate utility at /opt/vertica/config/logrotate/database_name, the LogRotate service will not run. If you upgraded from version 23.4 or below, you must delete the configuration file on all nodes to use the LogRotate service.

Automatic rotation

To configure and enable the LogRotate service:

Set the maximum allowed size of logs with LogRotateMaxSize. Log files larger than this age are rotated (compressed).
Set the maximum allowed age of rotated logs with LogRotateMaxAge. Rotated (compressed) logs older than this age are deleted.
Set how often the service runs with LogRotateInterval.
Enable the service with EnableLogRotate.

For details on these parameters, see Monitoring parameters.

To view the current values for these parameters, query CONFIGURATION_PARAMETERS.

The following example configures the LogRotate service to automatically run every four hours, rotating logs that are larger than 1 kibibyte and removing rotated logs that are older than 12 days:

=> ALTER DATABASE DEFAULT SET LogRotateInterval = '4h', LogRotateMaxSize = '1k', LogRotateMaxAge = '12 days';

Manual rotation

You can manually rotate the logs on a particular node with the DO_LOGROTATE_LOCAL meta-function. This function takes optional arguments to force the rotation of all logs and override the values of LogRotateMaxSize and LogRotateMaxAge.

To rotate logs that are larger than 1 kilobyte, and then remove rotated logs that are older than 1 day:

=> SELECT do_logrotate_local('max_size=1K;max_age=1 day');

                                             do_logrotate_local
-----------------------------------------------------------------------------------------------------------
 Doing Logrotate
Considering file /scratch_b/qa/VMart/v_vmart_node0001_catalog/vertica.log
File size: 35753 Bytes
Force rotate? no
Renaming to /scratch_b/qa/VMart/v_vmart_node0001_catalog/vertica.log.2023-11-08-13-55-51-651129-05
Opening new log file /scratch_b/qa/VMart/v_vmart_node0001_catalog/vertica.log
Compressing /scratch_b/qa/VMart/v_vmart_node0001_catalog/vertica.log.2023-11-08-13-55-51-651129-05 to /scratch_b/qa/VMart/v_vmart_node0001_catalog/vertica.log.2023-11-08-13-55-51-651129-05.gz
Done with /scratch_b/qa/VMart/v_vmart_node0001_catalog/vertica.log

Considering file /scratch_b/qa/VMart/v_vmart_node0001_catalog/UDxLogs/UDxFencedProcesses.log
File size: 68 Bytes
Force rotate? no
Rotation not required for file /scratch_b/qa/VMart/v_vmart_node0001_catalog/UDxLogs/UDxFencedProcesses.log
Done with /scratch_b/qa/VMart/v_vmart_node0001_catalog/UDxLogs/UDxFencedProcesses.log
(1 row)

To force rotation and then remove all logs older than 4 days (default value of LogRotateMaxAge):

=> SELECT do_logrotate_local('force;max_age=4 days');

                                             do_logrotate_local
-----------------------------------------------------------------------------------------------------------
 Doing Logrotate
Considering file /scratch_b/qa/VMart/v_vmart_node0001_catalog/vertica.log
File size: 4310245 Bytes
Force rotate? yes
Renaming to /scratch_b/qa/VMart/v_vmart_node0001_catalog/vertica.log.2023-11-10-13-45-15-53837-05
Opening new log file /scratch_b/qa/VMart/v_vmart_node0001_catalog/vertica.log
Compressing /scratch_b/qa/VMart/v_vmart_node0001_catalog/vertica.log.2023-11-10-13-45-15-53837-05 to /scratch_b/qa/VMart/v_vmart_node0001_catalog/vertica.log.2023-11-10-13-45-15-53837-05.gz
Done with /scratch_b/qa/VMart/v_vmart_node0001_catalog/vertica.log

Considering file /scratch_b/qa/VMart/v_vmart_node0001_catalog/UDxLogs/UDxFencedProcesses.log
File size: 68 Bytes
Force rotate? yes
Remove old log file /scratch_b/qa/VMart/v_vmart_node0001_catalog/UDxLogs/UDxFencedProcesses.log.2023-11-06-13-18-27-23141-05.gz
Remove old log file /scratch_b/qa/VMart/v_vmart_node0001_catalog/UDxLogs/UDxFencedProcesses.log.2023-11-07-13-18-30-059008-05.gz
Remove old log file /scratch_b/qa/VMart/v_vmart_node0001_catalog/UDxLogs/UDxFencedProcesses.log.2023-11-08-13-47-11-707903-05.gz
Remove old log file /scratch_b/qa/VMart/v_vmart_node0001_catalog/UDxLogs/UDxFencedProcesses.log.2023-11-09-14-09-02-386402-05.gz
Renaming to /scratch_b/qa/VMart/v_vmart_node0001_catalog/UDxLogs/UDxFencedProcesses.log.2023-11-10-13-45-15-647762-05
Opening new log file /scratch_b/qa/VMart/v_vmart_node0001_catalog/UDxLogs/UDxFencedProcesses.log
Compressing /scratch_b/qa/VMart/v_vmart_node0001_catalog/UDxLogs/UDxFencedProcesses.log.2023-11-10-13-45-15-647762-05 to /scratch_b/qa/VMart/v_vmart_node0001_catalog/UDxLogs/UDxFencedProcesses.log.2023-11-10-13-45-15-647762-05.gz
Done with /scratch_b/qa/VMart/v_vmart_node0001_catalog/UDxLogs/UDxFencedProcesses.log


(1 row)

Rotating with the Linux logrotate utility

Deprecated

This functionality is deprecated and will be removed in a future release. You should instead use the native LogRotate service.

Most Linux distributions include the logrotate utility. By setting up a logrotate configuration file, you can use the utility to complete one or more of these tasks automatically:

Compress and rotate log files
Remove log files automatically
Email log files to named recipients

You can configure logrotate to complete these tasks at specific intervals, or when log files reach a particular size.

If logrotate is present when OpenText™ Analytics Database is installed, then it automatically sets this utility to look for configuration files. Thus, logrotate searches for configuration files in the /opt/vertica/config/logrotate directory on each node.

When you create a database, it creates database-specific logrotate configurations on each node in your cluster, which are used by the logrotate utility. It then creates a file with the path /opt/vertica/config/logrotate/dbname for each individual database.

For information about additional settings, use the man logrotate command.

Executing the Python script through the dbadmin logrotate cron job

During the database installation, the installer configures a cron job for the dbadmin user. This cron job is configured to execute a Python script that runs the logrotate utility. You can view the details of this cron job by viewing the dbadmin.cron file, which is located in the /opt/vertica/config directory.

If you want to customize a cron job to configure logrotate for your database, you must create the cron job under the dbadmin user.

Using the administration tools logrotate utility

You can use the admintools logrotate option to help configure logrotate scripts for a database and distribute the scripts across the cluster. The logrotate option allows you to specify:

How often to rotate logs
How large logs can become before being rotated
How long to keep the logs

Example:

The following example shows you how to set up log rotation on a weekly schedule and keeps for three months (12 logs).

$ admintools -t logrotate -d <dbname> -r weekly -k 12

See Writing administration tools scripts for more usage information.

Configure logrotate for MC

The Management Console log file is:

/opt/vconsole/log/mc/mconsole.log

To configure logrotate for MC, configure the following file:

/opt/vconsole/temp/webapp/WEB-INF/classes/log4j.xml

Edit the log4j.xml file and set these parameters as follows:

Restrict the size of the log:

<param name="MaxFileSize" value="1MB"/>

Restrict the number of file backups for the log:
```
<param name="MaxBackupIndex" value="1"/>
```
Restart MC as the root user:
```
# etc/init.d/vertica-consoled restart
```

Rotating logs manually

To implement a custom log rotation process, follow these steps:

Rename or archive the existing vertica.log file. For example:
```
$ mv vertica.log vertica.log.1
```
Send the database process the USR1 signal, using either of the following approaches:
```
$ killall -USR1 vertica
```
or
```
$ ps -ef | grep -i vertica
$ kill -USR1 process-id
```

Admin: Monitoring process status (ps)

Mon, 01 Jan 0001 00:00:00 +0000

You can use ps to monitor the database and Spread processes running on each node in the cluster. For example:

$ ps aux | grep /opt/vertica/bin/vertica
$ ps aux | grep /opt/vertica/spread/sbin/spread

You should see one database process and one Spread process on each node for common configurations. To monitor Administration Tools and connector processes:

$ ps aux | grep vertica

There can be many connection processes but only one Administration Tools process.

Admin: Monitoring Linux resource usage

Mon, 01 Jan 0001 00:00:00 +0000

You should monitor system resource usage on any or all nodes in the cluster. You can use System Activity Reporting (SAR) to monitor resource usage.

Note

OpenText recommends that you install pstack and sysstat to help monitor Linux resources. The SYSSTAT package contains utilities for monitoring system performance and usage activity, such as sar, as well as tools you can schedule via cron to collect performance and activity data. See the SYSSTAT Web page for details.

The pstack utility lets you print a stack trace of a running process. See the PSTACK man page for details.

Run the top utility

$ top

A high CPU percentage in top indicates that the database is CPU-bound. For example:

top - 11:44:28 up 53 days, 23:47,  9 users,  load average: 0.91, 0.97, 0.81
Tasks: 123 total,   1 running, 122 sleeping,   0 stopped,   0 zombie
Cpu(s):  26.9%us,  1.3%sy,  0.0%ni, 71.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4053136 total,  3882020k used,  171116 free,   407688 buffers
Swap:  4192956 total,   176k used,  4192780 free,  1526436 cached
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
13703 dbadmin    1   0 1374m 678m  55m S 99.9 17.1   6:21.70 vertica
 2606 root      16   0 32152  11m 2508 S  1.0  0.3   0:16.97 X
    1 root      16   0  4748  552  456 S  0.0  0.0   0:01.51 init
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:04.92 migration/0
    3 root      34  19     0    0    0 S  0.0  0.0   0:11.75 ksoftirqd/0
...

Some possible reasons for high CPU usage are:

The Tuple Mover runs automatically and thus consumes CPU time even if there are no connections to the database.
The swappiness kernel parameter may not be set to 0. Execute the following command from the Linux command line to see the value of this parameter:
```
$ cat /proc/sys/vm/swappiness
```
If this value is not 0, change it by following the steps in Check for swappiness.
Some information sources:
- Red Hat
- Indiana University Unix Systems Support Group

Run the iostat utility. A high idle time in top at the same time as a high rate of blocks read in iostat indicates that the database is disk-bound. For example:

$ /usr/bin/iostat
Linux 2.6.18-164.el5 (qa01)     02/05/2011
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.77    2.32    0.76    0.68    0.00   95.47
Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
hda               0.37         3.40        10.37    2117723    6464640
sda               0.46         1.94        18.96    1208130   11816472
sdb               0.26         1.79        15.69    1114792    9781840
sdc               0.24         1.80        16.06    1119304   10010328
sdd               0.22         1.79        15.52    1117472    9676200
md0               8.37         7.31        66.23    4554834   41284840

Admin: Monitoring disk space usage

Mon, 01 Jan 0001 00:00:00 +0000

You can use these system tables to monitor disk space usage on your cluster:

System table	Description
`DISK_STORAGE`	Monitors the amount of disk storage used by the database on each node.
`COLUMN_STORAGE`	Monitors the amount of disk storage used by each column of each projection on each node.
`PROJECTION_STORAGE`	Monitors the amount of disk storage used by each projection on each node.

Admin: Monitoring elastic cluster rebalancing

Mon, 01 Jan 0001 00:00:00 +0000

OpenText™ Analytics Database includes system tables that can be used to monitor the rebalance status of an elastic cluster and gain general insight to the status of elastic cluster on your nodes.

REBALANCE_TABLE_STATUS provides general information about a rebalance. It shows, for each table, the amount of data that has been separated, the amount that is currently being separated, and the amount to be separated. It also shows the amount of data transferred, the amount that is currently being transferred, and the remaining amount to be transferred (or an estimate if storage is not separated).

Note
If multiple rebalance methods were used for a single table (for example, the table has unsegmented and segmented projections), the table may appear multiple times - once for each rebalance method.
REBALANCE_PROJECTION_STATUS can be used to gain more insight into the details for a particular projection that is being rebalanced. It provides the same type of information as above, but in terms of a projection instead of a table.

In each table, the columns SEPARATED_PERCENT and TRANSFERRED_PERCENTcan be used to determine overall progress.

Historical rebalance information

Historical information about work completed is retained, so query with the table column IS_LATEST to restrict output to only the most recent or current rebalance activity. Historical data may include information about dropped projections or tables. If a table or projection has been dropped and information about the anchor table is not available, then NULL is displayed for the table ID and <unknown> for table name. Information on dropped tables is still useful, for example, in providing justification for the duration of a task.

Admin: Monitoring events

Mon, 01 Jan 0001 00:00:00 +0000

To help you monitor your database system, OpenText™ Analytics Database traps and logs significant events that affect database performance and functionality if you do not address their root causes. This section describes where events are logged, the types of events that the database logs, how to respond to these events, the information that the database provides for these events, and how to configure event monitoring.

Admin: Using system tables

Mon, 01 Jan 0001 00:00:00 +0000

OpenText™ Analytics Database system tables provide information about system resources, background processes, workload, and performance—for example, load streams, query profiles, and tuple mover operations. The database collects and refreshes this information automatically.

You can query system tables using expressions, predicates, aggregates, analytics, subqueries, and joins. You can also save system table query results into a user table for future analysis. For example, the following query creates a table, mynode, selecting three node-related columns from the NODES system table:

=> CREATE TABLE mynode AS SELECT node_name, node_state, node_address FROM nodes;
CREATE TABLE
=> SELECT * FROM mynode;
    node_name     | node_state |  node_address
------------------+------------+----------------
 v_vmart_node0001 | UP         | 192.168.223.11
(1 row)

Note

You cannot query system tables if the database cluster is in a recovering state. The database refuses connection requests and cannot be monitored. OpenText™ Analytics Database also does not support DDL and DML operations on system tables.

Where system tables reside

System tables are grouped into two schemas:

V_CATALOG schema: Provides information about persistent objects in the catalog
V_MONITOR schema: Provides information about transient system state

These schemas reside in the default search path. Unless you change the search path to exclude V_MONITOR or V_CATALOG or both, queries can specify a system table name that omits its schema.

You can query the SYSTEM_TABLES table for all database system tables and their schemas. For example:

SELECT * FROM system_tables ORDER BY table_schema, table_name;

System table categories

Database system tables can be grouped into the following areas:

System information
System resources
Background processes
Workload and performance

Database reserves some memory to help monitor busy systems. Using simple system table queries makes it easier to troubleshoot issues. See also SYSQUERY.

Note

You can use external monitoring tools or scripts to query the system tables and act upon the information, as necessary. For example, when a host failure causes the K-safety level to fall below the desired level, the tool or script can notify the database administrator and/or appropriate IT personnel of the change, typically in the form of an e-mail.

Privileges

You can GRANT and REVOKE privileges on system tables, with the following restrictions:

You cannot GRANT privileges on system tables to the SYSMONITOR or PSEUDOSUPERUSER roles.
You cannot GRANT on system schemas.

Case-sensitive system table data

Some system table data might be stored in mixed case. For example, the database stores mixed-case identifier names the way you specify them in the CREATE statement, even though case is ignored when you reference them in queries. When these object names appear as data in the system tables, you'll encounter errors if you query them with an equality (=) operator because the case must exactly match the stored identifier. In particular, data in columns TABLE_SCHEMA and TABLE_NAME in system table TABLES are case sensitive.

If you don't know how the identifiers are stored, use the case-insensitive operator ILIKE. For example, given the following schema:

=> CREATE SCHEMA SS;
=> CREATE TABLE SS.TT (c1 int);
=> CREATE PROJECTION SS.TTP1 AS SELECT * FROM ss.tt UNSEGMENTED ALL NODES;
=> INSERT INTO ss.tt VALUES (1);

A query that uses the = operator returns 0 rows:

=> SELECT table_schema, table_name FROM v_catalog.tables WHERE table_schema ='ss';
table_schema | table_name
--------------+------------
(0 rows)

A query that uses case-insensitive ILIKE returns the expected results:

=> SELECT table_schema, table_name FROM v_catalog.tables WHERE table_schema ILIKE 'ss';
table_schema | table_name
--------------+------------
 SS           | TT
(1 row)

Examples

The following examples illustrate simple ways to use system tables in queries.


=> SELECT current_epoch, designed_fault_tolerance, current_fault_tolerance FROM SYSTEM;
 current_epoch | designed_fault_tolerance | current_fault_tolerance
---------------+--------------------------+-------------------------
           492 |                        1 |                       1
(1 row)

=> SELECT node_name, total_user_session_count, executed_query_count FROM query_metrics;
    node_name     | total_user_session_count | executed_query_count
------------------+--------------------------+----------------------
 v_vmart_node0001 |                      115 |                  353
 v_vmart_node0002 |                      114 |                   35
 v_vmart_node0003 |                      116 |                   34
(3 rows)

=> SELECT DISTINCT(schema_name), schema_owner FROM schemata;
 schema_name  | schema_owner
--------------+--------------
 v_catalog    | dbadmin
 v_txtindex   | dbadmin
 v_func       | dbadmin
 TOPSCHEMA    | dbadmin
 online_sales | dbadmin
 v_internal   | dbadmin
 v_monitor    | dbadmin
 structs      | dbadmin
 public       | dbadmin
 store        | dbadmin
(10 rows)

Admin: Data Collector utility

Mon, 01 Jan 0001 00:00:00 +0000

The Data Collector collects and retains history of important system activities and records essential performance and resource utilization counters. The Data Collector propagates information to system tables.

You can use the Data Collector in the following ways:

Query the past state of system tables and extract aggregate information
See what actions users have taken
Locate performance bottlenecks
Identify potential improvements to the database configuration

The Data Collector does not collect data for nodes that are down, so no historical data is available for affected nodes.

The Data Collector works with Workload Analyzer, a tool that intelligently monitors the performance of SQL queries and workloads and recommends tuning actions based on observations of the actual workload history.

Configuring and accessing Data Collector information

The Data Collector retains the data it gathers according to configurable retention policies. The Data Collector is on by default; you can disable it by setting the EnableDataCollector configuration parameter to 0. You can set the parameter at the database level using ALTER DATABASE or the node level using ALTER NODE. You cannot set it at the session or user level.

You can access metadata on collected data of all components using the DATA_COLLECTOR system table. This table includes information for each component about current collection policies and how much data is retained in memory and on disk.

Collected data is logged on disk in the DataCollector directory under the /catalog path. You can query logged data from component-specific Data Collector tables. You can also manage logged data with meta-functions; see Managing data collection logs.

Admin: Monitoring partition reorganization

Mon, 01 Jan 0001 00:00:00 +0000

When you use ALTER TABLE...REORGANIZE, the operation reorganizes the data in the background.

You can monitor details of the reorganization process by polling the following system tables:

V_MONITOR.PARTITION_STATUS displays the fraction of each table that is partitioned correctly.
V_MONITOR.PARTITION_REORGANIZE_ERRORS logs errors issued by the reorganize process.
V_MONITOR.PARTITIONS displays NULL in the partition_key column for any ROS that was not reorganized.

Note

The corresponding foreground process to ALTER TABLE...REORGANIZE is PARTITION_TABLE.

Admin: Monitoring resource pools

Mon, 01 Jan 0001 00:00:00 +0000

You can use the following to find information about resource pools:

RESOURCE_POOL_STATUS returns current data from resource pools—for example, current memory usage, resources requested and acquired by various requests, and state of the queues.
RESOURCE_ACQUISITIONS displays all resources granted to the queries that are currently running.
SHOW SESSION shows, along with other session-level parameters, the current session's resource pool.

You can also use the Management Console to obtain run-time data on resource pool usage.

Note

The Linux top command returns data on overall CPU usage and I/O wait time across the system. Because of file system caching, the resident memory size returned by top is not the best indicator of actual memory use or available reserves.

Viewing resource pool status

The following example queries RESOURCE_POOL_STATUS for memory size data:

=> SELECT pool_name poolName,
    node_name nodeName,
    max_query_memory_size_kb maxQueryMemSizeKb,
    max_memory_size_kb maxMemSizeKb,
    memory_size_actual_kb memSizeActualKb
    FROM resource_pool_status WHERE pool_name='ceo_pool';
 poolName |     nodeName     | maxQueryMemSizeKb | maxMemSizeKb | memSizeActualKb
----------+------------------+-------------------+--------------+-----------------
 ceo_pool | v_vmart_node0001 |          12179388 |     13532654 |         1843200
 ceo_pool | v_vmart_node0002 |          12191191 |     13545768 |         1843200
 ceo_pool | v_vmart_node0003 |          12191170 |     13545745 |         1843200
(3 rows)

Viewing query resource acquisitions

The following example displays all resources granted to the queries that are currently running. The information shown is stored in system table RESOURCE_ACQUISITIONS table. You can see that the query execution used 708504 KB of memory from the GENERAL pool.

=> SELECT pool_name, thread_count, open_file_handle_count, memory_inuse_kb,
   queue_entry_timestamp, acquisition_timestamp
   FROM V_MONITOR.RESOURCE_ACQUISITIONS WHERE node_name ILIKE '%node0001';

-[ RECORD 1 ]----------+------------------------------
pool_name              | sysquery
thread_count           | 4
open_file_handle_count | 0
memory_inuse_kb        | 4103
queue_entry_timestamp  | 2013-12-05 07:07:08.815362-05
acquisition_timestamp  | 2013-12-05 07:07:08.815367-05
-[ RECORD 2 ]----------+------------------------------
...
-[ RECORD 8 ]----------+------------------------------
pool_name              | general
thread_count           | 12
open_file_handle_count | 18
memory_inuse_kb        | 708504
queue_entry_timestamp  | 2013-12-04 12:55:38.566614-05
acquisition_timestamp  | 2013-12-04 12:55:38.566623-05
-[ RECORD 9 ]----------+------------------------------
...

You can determine how long a query waits in the queue before it can run. To do so, you obtain the difference between acquisition_timestamp and queue_entry_timestamp using a query as this example shows:

=> SELECT pool_name, queue_entry_timestamp, acquisition_timestamp,
    (acquisition_timestamp-queue_entry_timestamp) AS 'queue wait'
   FROM V_MONITOR.RESOURCE_ACQUISITIONS WHERE node_name ILIKE '%node0001';

-[ RECORD 1 ]---------+------------------------------
pool_name             | sysquery
queue_entry_timestamp | 2013-12-05 07:07:08.815362-05
acquisition_timestamp | 2013-12-05 07:07:08.815367-05
queue wait            | 00:00:00.000005
-[ RECORD 2 ]---------+------------------------------
pool_name             | sysquery
queue_entry_timestamp | 2013-12-05 07:07:14.714412-05
acquisition_timestamp | 2013-12-05 07:07:14.714417-05
queue wait            | 00:00:00.000005
-[ RECORD 3 ]---------+------------------------------
pool_name             | sysquery
queue_entry_timestamp | 2013-12-05 07:09:57.238521-05
acquisition_timestamp | 2013-12-05 07:09:57.281708-05
queue wait            | 00:00:00.043187
-[ RECORD 4 ]---------+------------------------------
...

Querying user-defined resource pools

The Boolean column IS_INTERNAL in system tables RESOURCE_POOLS and RESOURCE_POOL_STATUS lets you get data on user-defined resource pools only. For example:

SELECT name, subcluster_oid, subcluster_name, memorysize, maxmemorysize, priority, maxconcurrency
dbadmin->     FROM V_CATALOG.RESOURCE_POOLS where is_internal ='f';
   name       |  subcluster_oid   | subcluster_name | memorysize | maxmemorysize | priority | maxconcurrency
--------------+-------------------+-----------------+------------+---------------+----------+----------------
 load_pool    | 72947297254957395 | default         | 0%         |               |       10 |
 ceo_pool     | 63570532589529860 | c_subcluster    | 250M       |               |       10 |
 ad hoc_pool  |                 0 |                 | 200M       | 200M          |        0 |
 billing_pool | 45579723408647896 | ar_subcluster   | 0%         |               |        0 |              3
 web_pool     |                 0 | analytics_1     | 25M        |               |       10 |              5
 batch_pool   | 47479274633682648 | default         | 150M       | 150M          |        0 |             10
 dept1_pool   |                 0 |                 | 0%         |               |        5 |
 dept2_pool   |                 0 |                 | 0%         |               |        8 |
 dashboard    | 45035996273843504 | analytics_1     | 0%         |               |        0 |
(9 rows)

Admin: Monitoring recovery

Mon, 01 Jan 0001 00:00:00 +0000

When your OpenText™ Analytics Database is recovering from a failure, it's important to monitor the recovery process. There are several ways to monitor database recovery:

Admin: Clearing projection refresh history

Mon, 01 Jan 0001 00:00:00 +0000

The PROJECTION_REFRESHES system table records information about successful and unsuccessful refresh operations. This table normally retains data for a projection until replaced by a new refresh of that projection, but you can also purge the table.

To immediately purge data for all completed refreshes, call CLEAR_PROJECTION_REFRESHES:

=> SELECT clear_projection_refreshes();
clear_projection_refreshes
----------------------------
 CLEAR
(1 row)

This function does not clear data for refreshes that are currently in progress.

Admin: Monitoring OpenText Analytics Database using notifiers

Mon, 01 Jan 0001 00:00:00 +0000

An OpenText™ Analytics Database notifier is a push-based mechanism for sending messages from the database to endpoints like Apache Kafka or syslog. For example, you can configure a long-running script to send notifications at various stages and then at the completion of a task.

To use a notifier:

Use CREATE NOTIFIER to create one of the following notifier types:
- Syslog
- SNS
- Kafka
Send a notification to the NOTIFIER endpoint with any of the following:
- NOTIFY: Manually sends a message to the NOTIFIER endpoint.
- SET_DATA_COLLECTOR_NOTIFY_POLICY: Creates a notification policy, which automatically sends a message to the NOTIFIER endpoint when a specified event occurs.

Admin: Health Watchdog

Mon, 01 Jan 0001 00:00:00 +0000

If the database is under a high concurrent load, it leads to a bad health state on the server. The Health Watchdog is designed to mitigate the bad health state by doing the following:

Detecting the bad health state.
Stopping the transactions from adding to this bad state by blocking DDL/DML transactions.
Once the bad health state has been mitagated, allowing all blocked transactions to proceed.

Note

Only non-super user commands can be stopped by the Health Watchdog check.

The metrics Health Watchdog uses to check the server status and enact the mitigation are:

Truncation Version Lag - tracks the catalog sync service and detects bad health conditions in the server when the current commit version is far ahead of the database truncation version. By default, it is set to 500. It can be tuned using TruncationVersionLag.
GCLX Queue Bloat - tracks the GCLX queue size and stops the GCLX requests when the server is bombarded. By default, it is set to 100. It can be tuned using GCLXBlockParameter.
Mergeout Queue Bloat - tracks the TM queue size and stops DML transactions if the TM pool threads cannot keep up with the number of TM requests. By default, it is set to 100. It can be tuned using MergeoutBlockParameter.
Watchdog Timout Interval - the amount of time a transaction is blocked before it is timed out. By default, it is set to 5 minutes. It can be tuned using WatchdogTimeoutInterval.

You can check the status of the server using check_cluster_health and the health_watchdog_blocked_transactions system table.