Managing streaming services for extended monitoring
When extended monitoring is enabled, Vertica streams data from your database through Kafka servers to the storage database.
For additional parameters that optimize the performance of Kafka with Vertica, see Kafka and Vertica configuration settings.
View streaming details in MC
Click the Load tab on your database's MC dashboard to see the Data Load Activity page. On this page, the Continuous tab displays details about all continuous loading jobs for extended monitoring. You can use this page to monitor whether your extended monitoring data is streaming successfully to the MC storage database.
See Viewing load history for more about the Data Load Activity page.
Tip
Tip: If you do not see loading jobs for extended monitoring, verify that you have selected Show MC data collector monitoring streams at the top of the Continuous tab.Prevent data loss
The Memory Limit buffer allows you to restart the Kafka server without data loss. Vertica queues the streamed data until you restart the Kafka server. When the Kafka server remains down for an extended period of time, data loss occurs when the queue of streamed data exceeds the buffer. You set the buffer size on the Extended Monitoring tab when you enable extended monitoring for a database. See Managing extended monitoring on a database.
The Kafka retention policy determines when data loss occurs during the following scenarios:
-
Restarting the MC storage database (see Managing the storage database)
-
Disabling streaming on the MC storage database (see Managing the storage database)
-
Restart a micro-batch (see Viewing load history)
The Kafka retention policy can allow you to restart these extended monitoring components without data loss. The Kafka server retains the data while the listed components are disabled. Data loss occurs when the streamed data exceeds the Kafka retention policy's log size or retention time limits. See the Apache Kafka documentation for how to configure the retention policy.
Changing the Kafka server
Be aware that when you change Kafka servers for extended monitoring on the MC Storage DB Setup page, you must disable all extended monitoring processes and re-configure the MC storage database. For storage database setup instructions, see Managing the storage database.