OpenText Analytics Database 26.2.x – Monitoring events

Admin: Event logging mechanisms

Mon, 01 Jan 0001 00:00:00 +0000

OpenText™ Analytics Database posts events to the following mechanisms:

Mechanism	Description
`vertica.log`	All events are automatically posted to `vertica.log`. See Monitoring the Log Files.
`ACTIVE_EVENTS`	This SQL system table provides information about all open events. See Using system tables and ACTIVE_EVENTS.
`SNMP`	To post traps to SNMP, enable global reporting in addition to each individual event you want trapped. See Configuring event reporting.
`Syslog`	To log events to syslog, enable event reporting for each individual event you want logged. See Configuring event reporting.

Admin: Event codes

Mon, 01 Jan 0001 00:00:00 +0000

The following table lists the event codes that OpenText™ Analytics Database logs to the events system tables.

Event Code	Severity	Event Code Description	Description/Action
0	Warning	Low Disk Space	Warning indicates one of the following issues: Lack of disk space for database Disk failure I/O hardware failure Action: Add more disk space, or replace the failing disk or hardware as soon as possible. Check `dmesg` to see what caused the problem. Also, use the DISK_RESOURCE_REJECTIONS system table to determine the types of disk space requests that are being rejected and the hosts where they are rejected. See Managing disk space for details.
1	Warning	Read Only File System	Database lacks write access to the file system for data or catalog paths. This sometimes occurs if Linux remounts a drive due to a kernel issue. Action: Give the database write access.
2	Emergency	Loss Of K Safety	The database is no longer K-safe because insufficient nodes are functioning within the cluster. Loss of K-safety causes the database to shut down. Action: Recover the system.
3	Critical	Current Fault Tolerance at Critical Level	One or more nodes in the cluster failed. If the database loses one more node, it will no longer be K-safe and shut down. Action: Restore nodes that failed or shut down.
4	Warning	Too Many ROS Containers	Heavy load activity on one or more projections sometimes generates more ROS containers than the Tuple Mover can handle. The database allows up to 1024 ROS containers per projection before it rolls back additional load jobs and returns a ROS pushback error message. Action: The Tuple Mover typically catches up with pending mergeout requests and the Optimizer can resume executing queries on affected tables (see Mergeout). If this problem does not resolve quickly, or if it occurs frequently, it is probably related to insufficient RAM allocated to MAXMEMORY in the TM resource pool.
5	Informational	WOS Over Flow	Deprecated
6	Informational	Node State Change	The node state changed. Action: Check node status.
7	Warning	Recovery Failure	Database was not restored to a functional state after a hardware or software related failure. Action: Reasons for the warning can vary, see the event description for details.
8	Warning	Recovery Error	Database encountered an error while attempting to recover. If the number of recovery errors exceeds Max Tries, the Recovery Failure event is triggered. Action: Reasons for the warning can vary, see the event description for details.
9	n/a	Recovery Lock Error	Unused
10	n/a	Recovery Projection Retrieval Error	Unused
11	Warning	Refresh Error	The database encountered an error while attempting to refresh. Action: Reasons for the warning can vary, see the event description for details.
12	n/a	Refresh Lock Error	Unused
13	n/a	Tuple Mover Error	Deprecated
14	Warning	Timer Service Task Error	Error occurred in an internal scheduled task. Action: None, internal use only
15	Warning	Stale Checkpoint	Deprecated
16	Notice	License Size Compliance	Database size exceeds license size allowance. Action: See Monitoring database size for license compliance.
17	Notice	License Term Compliance	Database is not in compliance with your OpenText™ Analytics Database license. Action: Check compliance status with GET_COMPLIANCE_STATUS.
18	Error	CRC Mismatch	Cyclic Redundancy Check (CRC) returned an error or errors while fetching data. Action: Review the `vertica.log` file or the SNMP trap utility to evaluate CRC errors.
19	Critical/Warning	Catalog Sync Exceeds Durability Threshold	Severity: Critical when exceeding hard limit, Warning when exceeding soft limit.
20	Critical	Cluster Read-only	Eon Mode) Quorum or primary shard coverage loss forced database into read-only mode. Action: Restart down nodes.

Admin: Event data

Mon, 01 Jan 0001 00:00:00 +0000

To help you interpret and solve the issue that triggered an event, each event provides a variety of data, depending upon the event logging mechanism used.

The following table describes the event data and indicates where it is used.

vertica.log	ACTIVE_EVENTS (column names)	SNMP	Syslog	Description
N/A	NODE_NAME	N/A	N/A	The node where the event occurred.
Event Code	EVENT_CODE	Event Type	Event Code	A numeric ID that indicates the type of event. See Event Types in the previous table for a list of event type codes.
Event Id	EVENT_ID	Event OID	Event Id	A unique numeric ID that identifies the specific event.
Event Severity	EVENT_ SEVERITY	Event Severity	Event Severity	The severity of the event from highest to lowest. These events are based on standard syslog severity types: `0 – Emergency` `1 – Alert` `2 – Critical` `3 – Error` `4 – Warning` `5 – Notice` `6 – Info` `7 – Debug`
PostedTimestamp	EVENT_ POSTED_ TIMESTAMP	N/A	PostedTimestamp	The year, month, day, and time the event was reported. Time is provided as military time.
ExpirationTimestamp	EVENT_ EXPIRATION	N/A	ExpirationTimestamp	The time at which this event expires. If the same event is posted again prior to its expiration time, this field gets updated to a new expiration time.
EventCodeDescription	EVENT_CODE_ DESCRIPTION	Description	EventCodeDescription	A brief description of the event and details pertinent to the specific situation.
ProblemDescription	EVENT_PROBLEM_ DESCRIPTION	Event Short Description	ProblemDescription	A generic description of the event.
N/A	REPORTING_ NODE	Node Name	N/A	The name of the node within the cluster that reported the event.
DatabaseName	N/A	Database Name	DatabaseName	The name of the database that is impacted by the event.
N/A	N/A	Host Name	Hostname	The name of the host within the cluster that reported the event.
N/A	N/A	Event Status	N/A	The status of the event. It can be either: `1 – Open` `2 – Clear`

Admin: Configuring event reporting

Mon, 01 Jan 0001 00:00:00 +0000

Event reporting is automatically configured for vertica.log, and current events are automatically posted to the ACTIVE_EVENTS system table. You can also configure OpenText™ Analytics Database to post events to syslog and SNMP.

Admin: Event reporting examples

Mon, 01 Jan 0001 00:00:00 +0000

Vertica.log

The following example illustrates a Too Many ROS Containers event posted and cleared within vertica.log:

08/14/15 15:07:59 thr:nameless:0x45a08940 [INFO] Event Posted: Event Code:4 Event Id:0 Event Severity: Warning [4] PostedTimestamp:
2015-08-14 15:07:59.253729 ExpirationTimestamp: 2015-08-14 15:08:29.253729
EventCodeDescription: Too Many ROS Containers ProblemDescription:
Too many ROS containers exist on this node. DatabaseName: TESTDB
Hostname: fc6-1.example.com
08/14/15 15:08:54 thr:Ageout Events:0x2aaab0015e70 [INFO] Event Cleared:
Event Code:4 Event Id:0 Event Severity: Warning [4] PostedTimestamp:
2015-08-14 15:07:59.253729 ExpirationTimestamp: 2015-08-14 15:08:53.012669
EventCodeDescription: Too Many ROS Containers ProblemDescription:
Too many ROS containers exist on this node. DatabaseName: TESTDB
Hostname: fc6-1.example.com

SNMP

The following example illustrates a Too Many ROS Containers event posted to SNMP:

Version: 1, type: TRAPREQUESTEnterprise OID: .1.3.6.1.4.1.31207.2.0.1
Trap agent: 72.0.0.0
Generic trap: ENTERPRISESPECIFIC (6)
Specific trap: 0
.1.3.6.1.4.1.31207.1.1 ---> 4
.1.3.6.1.4.1.31207.1.2 ---> 0
.1.3.6.1.4.1.31207.1.3 ---> 2008-08-14 11:30:26.121292
.1.3.6.1.4.1.31207.1.4 ---> 4
.1.3.6.1.4.1.31207.1.5 ---> 1
.1.3.6.1.4.1.31207.1.6 ---> site01
.1.3.6.1.4.1.31207.1.7 ---> suse10-1
.1.3.6.1.4.1.31207.1.8 ---> Too many ROS containers exist on this node.
.1.3.6.1.4.1.31207.1.9 ---> QATESTDB
.1.3.6.1.4.1.31207.1.10 ---> Too Many ROS Containers

Syslog

The following example illustrates a Too Many ROS Containers event posted and cleared within syslog:

Aug 14 15:07:59 fc6-1 vertica: Event Posted: Event Code:4 Event Id:0 Event Severity: Warning [4] PostedTimestamp: 2015-08-14 15:07:59.253729 ExpirationTimestamp:
2015-08-14 15:08:29.253729 EventCodeDescription: Too Many ROS Containers ProblemDescription:
Too many ROS containers exist on this node. DatabaseName: TESTDB Hostname: fc6-1.example.com
Aug 14 15:08:54 fc6-1 vertica: Event Cleared: Event Code:4 Event Id:0 Event Severity:
Warning [4] PostedTimestamp: 2015-08-14 15:07:59.253729 ExpirationTimestamp:
2015-08-14 15:08:53.012669 EventCodeDescription: Too Many ROS Containers ProblemDescription:
Too many ROS containers exist on this node. DatabaseName: TESTDB Hostname: fc6-1.example.com