OpenText Analytics Database 26.2.x – Monitoring with MC

Mc: Viewing the overview page

Mon, 01 Jan 0001 00:00:00 +0000

The Overview page displays a dynamic dashboard view of your database.

The page provides three tabs: Status Summary, System Health, and Query Synopsis. Access these tabs by clicking one of the three icons at the top left of the Overview page. Each tab contains charts and filters displaying information about your cluster. The QuickStats widgets on the right of the page display alerts and statistics about the state of your cluster.

Information on this page updates every two minutes, however you can adjust that value in the MC Settings page on the Monitoring tab. You can postpone updates by deselecting Auto Refresh in the toolbar.

Chart viewing options

You can specify time frames for some charts, which display a calendar icon in their title bars. Click the calendar icon to specify the time frame for that module.

On the Status Summary tab, you can select Synchronize charts to simultaneously apply the specified time frame to all charts on that tab.

If you have enabled extended monitoring on your database, MC can display longer ranges of data in certain charts. See Extended monitoring. If a chart is using extended monitoring data, the rocket ship icon appears in the title bar:

You can expand some charts to view them in larger windows. Click the expand icon in the title bar to do so:

Changing what the chart displays

The charts on the Overview page can display information about the nodes in your database, or the activity in all your database subclusters, in a single subcluster, or on the nodes that are not assigned to a subcluster. Use the dropdown in the title bar to select the type of information you want to display in the chart.

Note

The dropdown list in the CPU/Memory/Disk I/O chart below, and all other MC charts, appears only for Eon Mode databases, and only if subclusters are defined.

Zooming to show chart details

There are several steps you can take to show increasing levels of detail in a chart.

You can click the expand icon in the title bar to view the chart in a larger window:

You can use the cursor to outline a small area you want to expand, shown as a gray rectangle below:

When you release the cursor, the detail area expands to full size:

Hover over any line or point on the chart to see details about those specific data points. This works before or after you expand the chart:

What the lines and dots on the chart represent

The legend below the CPU/Memory/Disk I/O chart explains what the lines and dots on the chart represent.

Each line represents the average of the nodes you selected in the dropdown. If you selected Database - Nodes, the line represents the average for all the nodes in the database. If you selected one subcluster, the line represents the average for the nodes in that subcluster.

Each dot represents an individual entity within your dropdown choice. If you chose Database - Nodes, each dot represents one node in the database. If you chose Database - Subclusters, each dot represents one subcluster in the database. If you chose a single subcluster or the unassigned subclusters, each dot represents an individual node within that set.

You can hover over any line or dot to see a summary about it. You can click on a dot to display the Node Details page for that dot.

Quick stats

The Quick Stats sidebar on the right of the page provides instant alerts and information about your cluster's status.

Database Nodes Health displays which nodes are down, critical, recovering, or up. Critical and recovering nodes are included in the total nodes considered "up" by the database. Click a node value to open the Manage page.
Running and Queued Queries displays current queries in the database. Click the query values to open the Query Monitoring charts.
Projections displays the number of total projections, unsegmented projections, and unsafe projections for the database schema with the most projections. Click a value to open the Table Treemap chart.
Disk Space Usage alerts you to the number of nodes that are low on disk space. Click the value to go to the Manage page. On the Manage page, the Storage Used KPI View is displayed.
Workload Analyzeranalyzes system information retained in SQL system tables and provides tuning recommendations, along with the cost (low, medium, or high) of running the command. See Analyzing workloads for more information.
I/O Wait Notices displays the number of nodes that, in the last hour, have recorded Disk I/O waits and Network I/O waits exceeding the wait threshold (1 second for Disk and 0 seconds for Network).
License Consumption displays the number of licenses your database uses, and the percentage of your Premium Edition license being used.
Unread Messages display the number of unread messages and alerts for the database. This count differs from the number of total messages across all your databases. Click the value to open the Message Center.

Status summary

The Status Summary tab displays four modules that provide a general overview of the status of your cluster:

The CPU/Memory/Disk I/O Usage module shows cluster resource usage. The chart displays the number of nodes in the database cluster and plots average and per-node percentages for CPU, memory, and disk I/O usage.
- Select a resource type from the legend to remove or add it from the chart display.
- Click a data point (which represents a node) to open the Manage page. See Monitoring cluster CPU and Memory with MC.
The General Pool Activity module displays GENERAL pool activity. The chart displays average query queue times, average GENERAL pool free memory, and resource rejections. Use this chart to see how much free memory there is in GENERAL pool, or if there have been high queue times.
- Click the dropdown in the title bar to view the GENERAL pool usage for the entire database (the default), for a specific subcluster, or for the nodes not assigned to a subcluster.
- Click the expand icon in the title bar to open the chart in a bigger window.
- Click a data point to open the Resource Pools Monitoring chart. See Managing workloads.
The Thresholds Notifications module displays alerts generated when a threshold has been exceeded in the database. Notifications are categorized by System Health and Performance.
- In the module, you can acknowledge an alert (which marks it as read) or click the X to stop monitoring that threshold (which stops you receiving similar alerts in the future).
- Customize thresholds and alert priorities for these notifications in the Thresholds tab of the database Settings page. See Customizing Message Thresholds.
The Queries module displays query statistics. The first pie chart displays running and queued queries in the last 24 hours. The second chart displays completed and failed queries for the time frame you specify. Click a query count number above the chart to open the Query Monitoring chart. See Monitoring running queries with MC.

System health

The System Health tab provides a summary of your system resource usage and node information, with filters that allow you to view resource usage within the ranges you specify.

Note

Note: Adjusting the filters on the System Health tab does not affect any database or MC settings.

The Memory Usage filter displays the number of nodes with high and low memory usage. Move the sliders to adjust the memory usage range filter.
For example, if you specify a range of 25% to 75% memory usage, the filter displays how many nodes are using less than 25% of memory (Low) and how many are using more than 75% (High). Hover your cursor over the Low and High values to see lists of what nodes fall, respectively, below or above the memory usage range you specified.
Click a node value to go to the Manage page, which displays the Memory Utilization KPI View.
The Spread Retransmission Rate filter displays the number of nodes with high spread retransmission rates. When a node's retransmission rate is too high, it is not communicating properly with other nodes. Move the slider to adjust the retransmission rate filter.
Hover your cursor over the Nodes value to see a list of what nodes exceeded the spread retransmission rate you specified. Click the node value to view spread retransmit rate alerts in the Message Center.
The CPU Usage chart displays the number of nodes with high and low CPU usage. Move the sliders to adjust the CPU usage range filter. Hover your cursor over the Low and High values to see lists of what nodes are below or above range you specified.
Click a node value to go to the Manage page, which displays the CPU Utilization KPI View.
The Reboot Rate filter displays the number of times nodes in the cluster have rebooted within the specified time frame. Use this filter to discover if nodes have gone down recently, or if there have been an unusual number of reboots. Move the slider to adjust the number of days. Hover over the Times value to see a list of the nodes that have rebooted and the times at which they did so.
The Disk Space Usage filter displays the number of nodes with high disk space usage. Move the slider to adjust the disk usage filter. Hover your cursor over the Nodes value to see a list of what nodes exceed the acceptable range.
Click the nodes value to go to the Manage page, which displays the Storage Used KPI View.
The Cluster Clock Skew Rate module displays the number of nodes that exceed a clock skew threshold. Nodes in a cluster whose clocks are not in sync can interfere with time-related database functions, the accuracy of database queries, and Management Console's monitoring of cluster activity.

Query synopsis

The Query Synopsis page provides two modules that report system query activity and resource pool usage:

The Query Statistics module displays four bar charts that provide an overview of running, queued queries, failed, and completed queries in the past 24 hours.
- Select one of the options at the top of the module to group the queries by Resource Pools, Users, Nodes, or Subclusters.
- Click a bar on the chart to view details about those queries the Query Monitoring activity chart.
The User Query Type Distribution chart provides an overview of user and system query activity. The chart reports the types of operations that ran. The default is to display the types of operations that ran on all nodes in the database. Use the dropdown in the title bar to display the types of operations that ran on the nodes in a specific subcluster, or on the nodes not assigned to a subcluster.
- Hover your cursor over chart points for more details.
- Select a type of operation from the legend to remove or add it from the chart display.
- To zoom to a certain time frame, you can adjust the sliders at the bottom of the chart.
- Click a bar in the graph to open the Queries chart.

Mc: Monitoring same-name databases with MC

Mon, 01 Jan 0001 00:00:00 +0000

If you are monitoring two databases with identical names on different clusters, you can determine which database is associated with which cluster by clicking the database icon on MC's Databases and Clusters page to view its dialog box. Information in the dialog displays the cluster on which the selected database is associated.

Mc: Monitoring cluster nodes with MC

Mon, 01 Jan 0001 00:00:00 +0000

For a visual overview of all cluster nodes, click the running database on the Databases and Clusters page and click the Manage tab at the bottom of the page to open the cluster status page.

The cluster status page displays the nodes in your cluster.

The appearance of the nodes indicate the following states:

Healthy: The nodes appear green.
Up: A small arrow to the right of the node points upward.
Critical: The node appears yellow and displays a warning icon to the right.
Down: The node appears red. To the right of the node, a red arrow points downwards.
Unplugged: An orange outlet and plug icon appears to the right. This is displayed when the MC cannot communicate with the agent running on the node.

You can get information about a particular node by clicking it, an action that opens the node details page.

Filtering what you see

If you have a large cluster, where it might be difficult to view dozens to hundreds of nodes on the MC interface, you can filter what you see. The Zoom filter shows more or less detail on the cluster overall, and the Health Filter lets you view specific node activity; for example, you can slide the bar all the way to the right to show only nodes that are down. A message next to the health filter indicates how many nodes in the cluster are hidden from view.

On this page, you can perform the following actions on your database cluster:

Add, remove and replace nodes
Rebalance data across all nodes
Stop or start (or restart) the database
Refresh the view from information MC gathers from the production database
View key performance indicators (KPI) on node state, CPU, memory, and storage utilization (see Monitoring cluster performance with MC for details)

Note

Starting, stopping, adding, and dropping nodes and rebalancing data across nodes works with the same functionality and restrictions as those same tasks performed through the Administration tools.

If you don't see what you expect

If the cluster grid does not accurately reflect the current state of the database (for example if the MC interface shows a node in INITIALIZING state, but when you use the Administration Tools to View Database Cluster State, you see that all nodes are UP), click the Refresh button in the toolbar. Doing so forces MC to immediately synchronize with the agents and update MC with new data.

Don't press the F5 key, which redisplays the page using data from MC and ignores data from the agent. It can take several seconds for MC to enable all database action buttons.

Mc: Monitoring node activity with MC

Mon, 01 Jan 0001 00:00:00 +0000

If a node fails on an MC-managed cluster or you notice one node is using higher resources than other cluster nodes—which you might observe when monitoring the Overview page—open the Manage page and click the node you want to investigate.

The Node Details page opens and provides summary information for the node (state, name, total memory, and so on), as well as resources the selected node has been consuming for the last three hours, such as average CPU, memory, disk I/O percent usage, network consumption in kilobytes, and the percentage of disk storage the running queries have been using. You can also browse and export log-level data from AgentTools and OpenText™ Analytics Database log files. MC retains a maximum of 2000 log records.

For a more detailed view of node activity, use the mouse to drag-select around a problem area in one of the graphs, such as the large spike in network traffic in the above image. Then hover over the high data point for a summary.

Mc: Monitoring cluster performance with MC

Mon, 01 Jan 0001 00:00:00 +0000

Key Performance Indicators (KPIs) are a type of performance measurement that let you quickly view the health of your database cluster through MC's Manage page. These metrics, which determine a node's color, make it easy for you to quickly identify problem nodes.

Metrics on the database are computed and averaged over the latest 30 seconds of activity and dynamically updated on the cluster grid.

How to get metrics on your cluster

To view metrics for a particular state, click the menu next to the KPI View label at the bottom of the Manage page, and select a state.

MC reports KPI scores for:

Node state—(default view) shows node status (up, down, k-safety critical) by color; you can filter which nodes appear on the page by sliding the health filter from left to right
CPU Utilization—average CPU utilization
Memory Utilization—average percent RAM used
Storage Utilization—average percent storage used

After you make a selection, there is a brief delay while MC transmits information back to the requesting client. You can also click Sync in the toolbar to force synchronization between MC and the client.

Node colors and what they mean

Nodes in the database cluster appear in color. Green is the most healthy and red is the least healthy, with varying color values in between.

Each node has an attached information dialog box that summarizes its score. It is the score's position within a range of 0 (healthiest) to 100 (least healthy) that determines the node's color bias. Color bias means that, depending on the value of the health score, the final color could be slightly biased; for example, a node with score 0 will be more green than than a node with a score of 32, which is still within the green range but influenced by the next base color, which is yellow. Similarly, a node with a score of 80 appears as a dull shade of red, because it is influenced by orange.

MC computes scores for each node's color bias as follows:

0-33: green and shades of green
34-66: yellow and shades of yellow
67-100: red and shades of red shades

If the unhealthy node were to consume additional resources, its color would change from a dull orange-red to a brighter red.

Filtering nodes from the view

The health filter is the slider in the lower left area of page. You can slide it left to right to show or hide nodes; for example, you might want to hide nodes with a score smaller that a certain value so the UI displays only the unhealthy nodes that require immediate attention. Wherever you land on the health filter, an informational message appears to the right of the filter, indicating how many nodes are hidden from view.

Filtering is useful if you have many nodes and want to see only the ones that need attention, so you can quickly resolve issues on them.

Mc: Monitoring cluster CPU and Memory with MC

Mon, 01 Jan 0001 00:00:00 +0000

On the MC Overview page, the CPU/Memory subsection provides a graph-based overview of cluster resources during the last hour, which lets you quickly monitor resource distribution across nodes.

This chart plots average and per-node percentages for both CPU and memory with updates every minute—unless you clear Auto Refresh Charts in the toolbar. You can also filter what the chart displays by clicking components in the legend at the bottom of the subsection to show/hide those components. Yellow data points represent individual nodes in the cluster at that point in time.

Investigating areas of concern

While viewing cluster resources, you might wonder why resources among nodes become skewed. To zoom in, use your mouse to drag around the problem area surrounding the time block of interest.

After you release the mouse, the chart refreshes to display a more detailed view of the selected area. If you hover your cursor over the node that looks like it's consuming the most resources, a dialog box summarizes that node's percent usage.

For more information, click a data point (node) on the graph to open MC's node details page. To return to the previous view, click Reset zoom.

Mc: Monitoring database storage with MC

Mon, 01 Jan 0001 00:00:00 +0000

The Infrastructure page's Storage View provides a summary of the amount of data stored in your database, and the persistent location of that data. Use this view to monitor how much of your storage capacity your databases are using.

For a database running in Eon Mode, MC also displays bar charts in the Storage View that illustrate shard subscription status. Use these charts to determine if your current subscription layout is optimal for querying your Eon Mode database. For information about using subscription status charts, see Monitoring subscription status in Eon Mode.

Monitor storage usage

The storage summary table lists all databases currently monitored by MC and information about their storage:

Database Size. Click Load Size to calculate the total size of the database.
Database Mode. The databases run in Enterprise Mode, or Eon Mode.
Storage Type. Enterprise Mode databases list the OS of the local nodes where data is stored. Eon Mode databases list the type of communal storage location where it stores its data. Eon Mode currently supports only S3-compatible storage locations.
View. The options displayed in this column depend on the database mode and type of data on the database.
- Vertica Tables Storage: For Enterprise Mode databases only. Click for a dialog listing the node and local directories where the database table data is stored.
- Communal/Depot Storage: For Eon Mode databases only. Click for a dialog displaying location paths for your depot and communal storage.
- Communal Storage Subscription: For Eon Mode databases only. Click to view bar charts at the bottom of the Storage View page, illustrating shard subscription status. For more about these charts, Monitoring subscription status in Eon Mode.
- External Tables: Available when there are external tables in your database. Click for a dialog displaying details about all external tables. (Also see Monitoring table utilization and projections with MC.)
- HCatalog Details: Available when your database has access to Hive tables. (See Using the HCatalog Connector.) Click for a dialog displaying details about HCatalog schemas. For any HCatalog schema, click View Tables for details about all tables accessible through that schema. (Also see Monitoring table utilization and projections with MC.)

In front of Eon Mode database names in the list, a plus icon displays. Click the icon to expand more details about the database's depot capacity and usage. The depot is cache-like storage where Eon Mode databases keep local copies of communal storage data for faster query access.

Click Percentage Used to view the Depot Activity chart for that database.
Click View Depot Details by Nodes to see a dialog displaying location paths and depot usage information.

Mc: Monitoring subscription status in Eon Mode

Mon, 01 Jan 0001 00:00:00 +0000

To view subscription charts for any Eon Mode database you monitor, click View Your Infrastructure on the MC Home page. Then click the Storage View tab.

Click the Details action for that database in the storage summary list (highlighted in red in the image below).

When you click Details, two charts become available on the bottom half of the page: The Sharding Subscription chart, and the Node Subscription chart. You can switch between these two charts using the drop down menu to the right of the chart title.

Why monitor shard and node subscriptions?

Shards are segments of the data that is stored persistently in your Eon Mode database's communal storage location, for example Amazon S3 in the cloud or PureStorage if your cluster is on premises. Each node in the database subscribes to a subset of those shards. In this way, the node gets updated on when to populate its depot with new data from communal storage. (See Namespaces and shards.)

For K-safety in an Eon Mode database, shards should have multiple node subscribers to ensure that even if a node goes down or is being used by another query, the data on that shard is still available on other nodes. If a shard has no node subscribers, that could indicate that data loss is occurring.

Subscriptions go through several transitions, which are illustrated by colors in the subscription charts:

Pending (Yellow). The node is ready to subscribe to a certain shard. It cannot yet serve queries because it is not actively subscribed to the shard yet.
Passive (Blue/Teal). The node could potentially serve queries for a shard it is passively subscribed to, but its depot contents for that shard may not yet be up to date, which could negatively impact query performance. The passively subscribed node is waiting for an active node subscriber of the shard to send it the most recent data.
Active (Green). The node is actively subscribed to the shard, can load new data from communal storage, and can serve queries for data in that shard. The actively subscribed node sends data from that shard to other subscribed nodes.
Removing (Dark Red/Maroon). The node is unsubscribing from the shard. It may have the most recent data from that shard, but that state is temporary until data from that shard is cleaned up.
Inactive (Red). The subscribed node is down. It can no longer serve queries for that shard.

Operations such as adding or removing nodes or rebalancing shards can change which nodes subscribe to which shards. Shard subscription changes can prevent object-level restore from backups, though full restore is always possible. If shard subscriptions change, consider making a backup with the new configuration.

Monitor sharding subscription

The Sharding Subscription chart displays how many nodes are subscribed to each shard in your database, and what type of subscription it is.

You can hover over any bar in the chart to see which nodes are subscribed to the shard. Click on a subscription type in the legend to show or hide it in the chart display.

The example below shows the shard subscription status for a running Eon Mode database. The database has three nodes that are up, and one node (Node 4) that has been added to the cluster, but is down.

You can hover over any bar in the chart to see which nodes are subscribed to the shard. In this example, nodes 1 and 3 have active subscriptions to the first shard (green); nodes 1 and 2 to the second shard; and nodes 2 and 3 to the third shard.

The active subscriptions are evenly spread across the shards. This is a k-safe Eon Mode database.

Node 4 was subscribed to two shards; however, because it is down, its subscriptions to the shards are now inactive (red).

Monitor node subscriptions

Use this chart to view how many shards each node in your database is subscribed to, and the state of those subscriptions. The number of shards each node is subscribed to should be about the same to prevent overworking any given node.

Hover over any bar to see the shards it is subscribed to. The color of the bar indicates the state of each subscription. Click on a subscription type in the legend to show or hide it in the chart display.

The example below shows the same database from the Sharding Subscription example above. Nodes 1 through 3 are each actively subscribed to two shards (green). At least two nodes are subscribed to every shard in the database (which you can double check using the Sharding Subscription chart), ensuring that even if one of the nodes is down or being used in a query, another node is still actively subscribed and can access the data of that shard.

Since Node 4 is down, the chart shows that both its shard subscriptions are now inactive.

Mc: Monitoring system resources with MC

Mon, 01 Jan 0001 00:00:00 +0000

MC's Activity page provides immediate visual insight into potential problem areas in your database's health by giving you graph-based views of query and user activity, hardware and memory impact, table and projection usage, system bottlenecks, and resource pool usage.

Select one of the following charts in the toolbar menu:

How up to date is the information?

System-level activity charts automatically update every five minutes, unless you clear Auto Refresh in the toolbar. Depending on your system, it could take several moments for the charts to display when you first access the page or change the kind of resource you want to view.

Chart viewing options

You can specify time frames for some charts, which display a calendar icon in their title bars. Click the calendar icon to specify the time frame for that module.

You can expand some charts to view them in larger windows. Click the expand icon in the title bar to do so:

Mc: Monitoring resource pools with MC

Mon, 01 Jan 0001 00:00:00 +0000

Management Console allows database administrators to monitor and configure resource pools through the Activity and Configuration pages. These pages help you manage workloads by providing visual representations of resource usage as well as resource pool configuration options.

Monitoring resource pools charts

You can monitor your resource pools using the Resource Pools Monitoring charts, accessible through the Management Console Activity page.

Select a resource pool to view using the Resource Pool menu, located in the leftmost sidebar. In the sidebar, Current Usage Activity displays the pool's real-time statistics.

Monitor the selected resource pool using the following charts, which display the pool's historic data:

Resource Usages for Pool: Shows the historically averaged acquired memory usage by each pool across all nodes. The graph uses two y-axes, one that shows memory size, and a second that shows the total number of running queries. Data is collected every hour. Hover over a data point for a summary of the memory usage at that specific point.
Memory Usage in Node: Shows the historically acquired memory usages by all pools across all nodes. Data is collected every hour. Hover over a data point for a summary of the memory usage at that specific point. Use the title bar dropdown to display the memory usage for a specific node. For Eon mode databases, you can also display the memory usage for a specific subcluster, all subclusters, or nodes not assigned to a subcluster. An Eon mode database has one default subcluster, and may have additional user-defined subclusters.
Average Query Execution and Query Time in Pool: Shows the averaged query queue time plotted against the query execution time by each pool across all nodes. Data is collected every minute. Hover over data to get the average query execution and queue time in the specified pool. Click a data point to show detailed individual query information.
Resource Rejections in Pool: Shows the historical total number of resource requests that were rejected in each pool across all nodes. Data is collected every hour. Click a data point to show rejection details and reasons in a pop-up window.

Configuring resource pools in MC

Database administrators can view information about resource pool parameters and make changes to existing parameters through the Management Console Configuration page. You can also create and remove new resource pools, assign resource pool users, and assign cascading pools.

See Configuring Resource Pools in Management Console

Permissions

Only the database administrator can monitor and configure resource pools in Management Console.

Mc: Monitoring database messages and alerts with MC

Mon, 01 Jan 0001 00:00:00 +0000

Management Console periodically checks system health and performance. When an important event occurs or system performance fluctuates beyond user- or system-defined limits, the Management Console generates a message to alert you about the state of your system. View and manage alerts in the Message Center.

Message alert notifications

Management Console uses multiple methods to communicate alert notifications to ensure that you are immediately aware of any changes to your database that might require attention. You receive message notifications by email, and you can view notifications using the following components:

Message envelope icon: This icon is located by the MC Help icon, in the top-right of any database-specific page. Select this icon display the Message Center quick view, and perform archive, read, and delete message actions. For details about message actions and alerts, see Message center.
Unread Messages (This Week) widget: On the database Overview tab, this widget is located in the quick stats sidebar. It displays unread, high-priority messages. Select the number (including 0) in the widget to go to the Message Center.

Pre-configured alerts

Management Console provides pre-configured alerts to provide system monitoring capabilities without manual setup. Each alert has a pre-configured threshold that defines the acceptable performance limit, and MC sends a message notification when the database exceeds that threshold.

By default, pre-configured alerts are not active and require minimal initial setup. For details on how to set pre-configured alert properties, see Alert configuration.

Node health

OpenText™ Analytics Database provides the following pre-configured alerts to monitor node health:

Node CPU
Node Memory
Node Disk Usage
Node Disk I/O
Node CPU I/O Wait
Node Reboot Rate
Node State Change
Node Catalog Memory

Network health

The database provides the Network I/O Error pre-configured alert to monitor network health.

Query

The database provides the following pre-configured alerts to monitor queries:

Queued Query
NumberFailed
Query Number
Spilled Query Number
Retried Query Number
Query Running Time

License status

The database provides the License Usage pre-configured alert to monitor the status of your database license.

Resource pool

MC can send alerts when an individual resource pool reaches a specified state or usage level. For details about resource pool configuration parameters, see Built-in resource pools configuration.

Important

Default settings for resource pool alerts apply to the GENERAL pool only.

You can configure the MC to send the following resource pool alerts:

Queries Reaching the Max Allowed Execution Time: Triggers an alert when the specified number of queries reach the RUNTIMECAP execution threshold for the resource pool. You cannot set this alert if the resource pool does not have a RUNTIMECAP threshold set, or if the resource pool has a secondary resource pool.
Queries With Resource Rejections: Triggers an alert when the specified number of queries exceed a specified number of resource rejections within a set period of time.
Minimum Starting Resource Value: Triggers an alert when the resource pool reaches the minimum amount of resources allocated for the MEMORYSIZE value.

Note
By default, you cannot set MEMORYSIZE for the GENERAL pool. The GENERAL pool must have at least 1GB of memory and it cannot be smaller than 25% of the entire system memory.
Maximum Allowed Resource Value: Triggers an alert when the resource pool reaches the MAXMEMORYSIZE value.
Ended Query With Queue Time Exceeding Limit: Triggers an alert when the specified number of completed queries were queued for a specified length of time within a timeframe.
Ended Query With Run Time Exceeding Limit: Triggers an alert when the specified number of completed queries ran for a specified length of time within a timeframe.

Custom alerts

Create custom alerts to measure system performance metrics that are not monitored by the pre-configured alerts. Create a dynamic SQL query that triggers an alert when it returns any results. You can configure how often an alert is generated, the alert priority, and who receives email alerts. For example, you can create custom alerts that monitor the following:

Failed logins within a configurable time period
Idle database user sessions using a configurable time limit
Database node is DOWN

For details about creating and managing custom alerts, including a tutorial on how to create a custom alert that monitors failed logins, see Custom alerts.

Default notifications

Management Console generates the following messages about the database that appear only in the Message Center:

Low disk space
Read-only file system
Loss of K-safety
Current fault tolerance at critical level
Too many ROS containers
Change in node state
Recovery error
Recovery failure
Recovery lock error
Recovery projection retrieval error
Refresh error
Refresh lock error
Workload analyzer operations
Tuple Mover error
Timer service task error
Last Good Epoch (LGE) lag
License size compliance
License term compliance

Disk space check and cleanup

When the Management Console checks alerts, it generates a result set and saves it to disk. If you use aggressive alert configurations, the result set might use a large amount of disk space. By default, the database reserves 500MB of disk space to save result sets.

The database checks the free disk space 2 times each day and cleans alerts that are older than 7 days. If the available disk space is low, custom alerts are disabled. Notifications and emails are generated when an alert is disabled due to insufficient disk space.

The /opt/console/vconsole/config/console.properties file contains these settings. Edit the following values to configure the how the MC manages your disk space:

Property Description

Property	Description
`console.diskspace.threshold`	The amount of disk space the database reserves to save result sets. Default: 500MB
`customthreshold.alerts.toKeepInDays`	The number of days that alerts are retained on disk. Default: 7

console.diskspace.threshold

The amount of disk space the database reserves to save result sets.

Default: 500MB

customthreshold.alerts.toKeepInDays

The number of days that alerts are retained on disk.

Default: 7

Mc: Monitoring MC user activity using audit log

Mon, 01 Jan 0001 00:00:00 +0000

When an MC user makes changes on the MC interface, whether to an MC-managed database or to the MC itself, their action generates a log entry that records a timestamp, the MC user name, the database and client host (if applicable), and the operation the user performed. You monitor user activity on the Diagnostics > Audit Log page.

MC records the following types of user operations:

User log-on/log-off activities
Database creation
Database connection through the console interface
Start/stop a database
Remove a database from the console view
Drop a database
Database rebalance across the cluster
License activity views on a database, as well as new license uploads
Workload analyzer views on a database
Database password changes
Database settings changes (individual settings are tracked in the audit record)
Syncing the database with the cluster (who clicked Sync on grid view)
Query detail viewings of a database
Closing sessions
Node changes (add, start, stop, replace)
User management (add, edit, enable, disable, delete)
LDAP authentication (enable/disable)
Management Console setting changes (individual settings are tracked in the audit record)
SSL certificate uploads
Message deletion and number deleted
Console restart from the browser interface
Factory reset from the browser interface
Upgrade MC from the browser interface

Background cleanup of audit records

An internal MC job starts every day and, if required, clears audit records that exceed a specified timeframe and size. The default is 90 days and 2K in log size. MC clears whichever limit is first reached.

You can adjust the time and size limits by editing the following lines in the /opt/vconsole/config/console.properties file:

vertica.audit.maxDays=90vertica.audit.maxRecords=2000

Filter and export results

You can manipulate the output of the audit log by sorting column headings, scrolling through the log, refining your search to a specific date/time and you can export audit contents to a file.

If you want to export the log, see Exporting the user audit log.

If you perform a factory reset

If you perform a factory reset on MC's Diagnostics page (restore it to its pre-configured state), MC prompts you to export audit records before the reset occurs.

Mc: Monitoring external data sources with MC

Mon, 01 Jan 0001 00:00:00 +0000

By default, Management Console monitors a database using information from that database's Data Collector (DC) tables. MC can also monitor DC tables you have copied into OpenText™ Analytics Database tables, locally or remotely.

MC administrators provide mappings to local schemas or to an external database containing the corresponding DC data. MC can then render its charts and graphs from the new repository instead of from local DC tables. This offers the benefit of loading larger sets of data faster in MC, and retaining historical data long term.

Note

Note: MC also offers External Monitoring, which allows you to set up a storage database through the MC interface, then use Kafka to stream your data to the storage database. You can use the Data Source mapping process below if you prefer to set up your own alternative data source, or do not plan to use Kafka streaming.

Map an alternative data source

On the MC Settings page, navigate to the Data Source tab.
Select the database for which you are creating the data source mapping.
Choose the database user for which you want to create the mapping.
Set Repository Location to Local or Remote.
If Remote is selected, provide JDBC connection parameters for the remote database repository. Click Validate Connection Properties to confirm a successful connection.
Enter the schema mappings for v_internal and v_catalog. MC does not support mapping the v_monitor schema.
Input your table mappings in one of the following ways:
- Click Auto Discover. MC retrieves the table mappings based on the database and schema mappings you provided.
- Click Manual Entry. Manually input table mappings.
- Click Load Configurations. If you previously saved a data source configuration for the database in a file, import the file to use that configuration for the currently selected user.
Optionally, click Save Configurations to export this configuration file. You can create a mapping for another database user with this configuration file later.
Click Apply to save and apply your configuration settings.

Reports using unmapped schemas

If a report in MC needs to access a locally stored schema or table that is unmapped, MC includes information from the local DC tables for that schema to complete the report.

For remote configurations, if a report depends on an unmapped schema or table, the entire report is run against the local DC tables. If the remote database is down when MC attempts to run a report against it, MC reruns the report against the local database.

When the MC runs a report, it records missing mappings in the MC log under the INFO severity level.

Mc: Monitoring depot activity with MC

Mon, 01 Jan 0001 00:00:00 +0000

The depot is a cache-like component on each node that downloads and stores local copies of table data. Queries that can access the data that they need on the depot, instead of fetching it from communal storage, generally execute much faster. If your database is in Eon Mode, you can use the Depot Activity page to view depot settings and evaluate how efficiently it handles queries and load activity.

To view depot settings and activity, navigate to Database > Activity > Depot Activity Monitoring. The Depot Activity page has the following tabs:

Mc: Monitoring depot storage with MC

Mon, 01 Jan 0001 00:00:00 +0000

To display detailed storage monitoring information for your Eon database:

From the MC home page, select View Your Infrastructure.
On the Infrastructure page, select the Storage View tab.MC displays the Storage View screen, with details about the database storage and links to further detail screens:
To see the loaded size of the database, click Load Size.
To see communal storage details for the database, such as its location and size, and the IP addresses of the nodes, click Communal/Depot Storage.
To view the shard subscriptions for your Eon nodes, click Communal Storage Subscription. MC displays the shard type, how many nodes are subscribed to each shard, and the status of each shard subscription (Active, Inactive, Passive, Pending, Removing).

There are two views:
- Sharding Subscription displays how many nodes store each shard.
- Node Subscription displays how many shards are on each node.
Hover over a bar to display the details.
To display the depot details for all nodes in the database, click View Depot Details by Nodes. MC lists the nodes by node name, and for each node shows the number of bytes the node has in its depot, the total capacity of the depot, the percent used, and the path to the node's depot.

See also
Monitoring depot activity with MC

Mc: Extended monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Enabling extended monitoring allows you to monitor a longer range of data through MC. This can offer insight into long-term trends in your database's health. MC can also continue to display your monitored database's dashboard while it is down.

Extended monitoring uses Kafka to stream monitoring data from your monitored databases to a single MC storage database. MC can query the storage database instead of your monitored database to render some of its charts, reducing impact on your monitored database's performance.

How extended monitoring works

By default, MC monitors your database by querying it directly for monitoring data about system activities, performance, and resource utilization. Typically, the Data collector stores all monitoring data in data collector (DC) tables. However, DC tables have limited retention periods. See Data Collector utility.

Extended monitoring stores your database's monitoring data in a dedicated storage database. OpenText™ Analytics Database streams data from your database's DC tables through Kafka servers to the storage database. To use extended monitoring, you must have access to a running Kafka server. For more information about how the database integrates with Kafka, see Apache Kafka integration.

After you set up and enable extended monitoring for a monitored database, MC renders several of your database's charts and graphs by querying the MC storage database instead of directly querying the database you are monitoring.

You can enable extended monitoring for any, or all, of your monitored databases. The MC storage database provides a single repository for monitoring data from every database that uses enabled extended monitoring.

In the following example, Kafka streams system data from two monitored databases to the storage database. MC uses the storage database to render individual dashboards for each monitored database. Be aware that MC always creates a dashboard that monitors the MC storage database.

Use extended monitoring

Important

Important: To use extended monitoring, OpenText recommends installing Management Console on a host without any other OpenText™ Analytics Database.

When a database has extended monitoring enabled, the MC charts that use the feature display a rocket ship icon in the corner. You can use these charts to access longer-term data about your database's health or performance.

To view historical information in these charts, click the calendar icon to specify the timeframe to display. For example, if your database has been down for several hours, your charts do not display recent activity in your database. You could use the timeframe filter in the System Bottlenecks chart to see unusual resource usage occurred in your database in the hour it went down.

You can view a history of the Kafka streaming jobs loading data into the storage database. MC displays these jobs on the Load tab of your storage database's dashboard. See Viewing load history.

Set up extended monitoring

To set up extended monitoring, see Managing the storage database and Managing extended monitoring on a database.

OpenText Analytics Database 26.2.x – Monitoring with MC

Mc: Viewing the overview page

Chart viewing options

Changing what the chart displays

Note

Zooming to show chart details

What the lines and dots on the chart represent

Quick stats

Status summary

System health

Note

Query synopsis

Mc: Monitoring same-name databases with MC

Mc: Monitoring cluster nodes with MC

Filtering what you see

Note

If you don't see what you expect

Mc: Monitoring node activity with MC

See also

Mc: Monitoring cluster performance with MC

How to get metrics on your cluster

Node colors and what they mean

Filtering nodes from the view

Mc: Monitoring cluster CPU and Memory with MC

Investigating areas of concern

See also

Mc: Monitoring database storage with MC

Monitor storage usage

See also

Mc: Monitoring subscription status in Eon Mode

Why monitor shard and node subscriptions?

Monitor sharding subscription

Monitor node subscriptions

See also

Mc: Monitoring system resources with MC

How up to date is the information?

Chart viewing options

Mc: Monitoring resource pools with MC

Monitoring resource pools charts

Configuring resource pools in MC

Permissions

See also

Mc: Monitoring database messages and alerts with MC

Message alert notifications

Pre-configured alerts

Node health

Network health

Query

License status

Resource pool

Important

Note

Custom alerts

Default notifications

Disk space check and cleanup

Mc: Monitoring MC user activity using audit log

Background cleanup of audit records

Filter and export results

If you perform a factory reset

Mc: Monitoring external data sources with MC

Note

Map an alternative data source

Reports using unmapped schemas

Mc: Monitoring depot activity with MC

Mc: Monitoring depot storage with MC

See also

Mc: Extended monitoring

How extended monitoring works

Use extended monitoring

Important

Set up extended monitoring

See also