Authenticating with Kafka using SASL

Kafka supports using Simple Authentication and Security Layer (SASL) to authenticate producers and consumers.

Kafka supports using Simple Authentication and Security Layer (SASL) to authenticate producers and consumers. You can use SASL to authenticate Vertica with Kafka when using most of the Kafka-related functions such as KafkaSource.

Vertica supports using the SASL_PLAINTEXT and SASL_SSL protocols with the following authentication mechanisms:

  • PLAIN

  • SCRAM-SHA-256

  • SCRAM-SHA-512

You must configure your Kafka cluster to enable SASL authentication. See the Kafka documentation for your Kafka version to learn how to configure SASL authentication.

To use SASL authentication between Vertica and Kafka, directly set SASL-related configuration options in the rdkafka library using the kafka_conf parameter. Vertica uses this library to connect to Kafka. See Directly setting Kafka library options for more information on directly setting configuration options in the rdkafka library.

Among the relevant configuration options are:

  • security.protocol sets the security protocol to use to authenticate with Kafka.

  • sasl.mechanism sets the security mechanism.

  • sasl.username sets the SASL user to use for authentication.

  • sasl.password sets the password to use for SASL authentication.

See the rdkafka configuration documentation for a list of all the SASL-related settings.

The following example demonstrates calling KafkaCheckBrokers using the SASL_PLAINTEXT security protocol:

=> SELECT KafkaCheckBrokers(USING PARAMETERS
      brokers='kafka01.example.com:9092',
      kafka_conf='{"sasl.username":"dbadmin", "sasl.password":"pword", "sasl.mechanism":"PLAIN", "security.protocol":"SASL_PLAINTEXT"}'
   ) OVER ();

This example demonstrates using SASL authentication when copying data from Kafka via an SSL connection. This example assumes that Vertica and Kafka have been configures to use TLS/SSL encryption as described in TLS/SSL encryption with Kafka:

=> COPY mytopic_table
      SOURCE KafkaSource(
        stream='mytopic|0|-2',
        brokers='kafka01.example.com:9092',
        stop_on_eof=true,
        kafka_conf='{"sasl.username":"dbadmin", "sasl.password":"pword", "sasl.mechanism":"PLAIN", "security.protocol":"SASL_SSL"}'
      )
      FILTER KafkaInsertDelimiters(delimiter = E'\n')
      DELIMITER ','
      ENCLOSED BY '"';

For more information about using SASL with the rfkafka library, see Using SASL with librdkafka on the rdkafka github site.