Changes to the message.max.bytes setting in Kafka version 0.11 and later

In Kafka version 0.10 and earlier, the message.max.bytes setting configured the maximum allowable size for an individual message. Starting in version 0.11, Kafka began grouping messages into batches. This version changed the meaning of this setting to be the largest allowable size of a message batch.

This change can have a significant impact on how fast your Vertica database loads messages from Kafka 0.11 and later. If you created your streaming job scheduler using Vertica version 9.1.0 or earlier, you might find that its load spec's message.max.bytes value is set too low. Upgrading your scheduler does not adjust this setting automatically.

To prevent slow throughput when using Vertica with Kafka 0.11 and later, manually adjust the message.max.bytes setting in scheduler's load spec. Testing by Vertica suggests the best value for this setting is 25165824 (24 × 1024 × 1024) bytes.

The following example demonstrates updating the message.max.bytes setting in the load spec named weblog_load of an existing scheduler that is defined using the configuration file named weblog.conf:

$ /opt/vertica/packages/kafka/bin/vkconfig load-spec --update \
     --load-spec weblog_load --conf weblog.conf --message-max-bytes 25165824