Source tool options

Use the vkconfig script's source tool to create, update, or delete a source.

Use the vkconfig script's source tool to create, update, or delete a source.

Syntax

vkconfig source {--create | --read | --update | --delete} --source source_name [other_options...]
--create

Creates a new load spec, cannot be used with --delete, --read, or --update.

--read
Outputs the current setting of the sources defined in the scheduler. The output is in JSON format. Cannot be used with --create, --delete, or --update.

By default this option outputs all of the sources defined in the scheduler. You can limit the output by using the --cluster, --enabled, --partitions, and --source options. The output will only contain sources that match the values in these options. The --enabled option can only have a true or false value. The --source option is case-sensitive.

You can use LIKE wildcards in these options. See LIKE predicate for more information about using wildcards.

--update

Updates an existing Set Snippet Variable Value in Topic. Cannot be used with --create, --delete, or --read.

--delete

Deletes a Set Snippet Variable Value in Topic. Cannot be used with --create, --read, or --update.

--source source_name
Identifies the source to create or alter in the scheduler's configuration. This option is case-sensitive. You can use any name you like for a new source. Most people use the name of the Kafka topic the scheduler loads its data from. This option is required for --create, --update, and --delete.
--cluster cluster_name
Identifies the cluster containing the source that you want to create or edit. You must have already defined this cluster in the scheduler.
--dump

When you use this option along with the --read option, vkconfig outputs the Vertica query it would use to retrieve the data, rather than outputting the data itself. This option is useful if you want to access the data from within Vertica without having to go through vkconfig. This option has no effect if not used with --read.

--enabled TRUE|FALSE
When TRUE, the source is available for use.
--new-cluster cluster_name
Changes the cluster this source belongs to.

All sources referencing the old cluster source now target this cluster.

Requires:--update and --source options

--new-source source_name
Updates the name of an existing source to the name specified by this parameter.

Requires: --update shared utility option

--partitions count
Sets the number of partitions in the source.

Default:

The number of partitions defined in the cluster.

Requires:--create and --source options

You must keep this consistent with the number of partitions in the Kafka topic.

Renamed from --num-partitions.

--validation-type{ERROR|WARN|SKIP}
Controls the validation performed on a created or updated source:

  • ERROR - Cancel configuration or creation if vkconfig cannot validate the source. This is the default setting.

  • WARN - Proceed with task if validation fails, but display a warning.

  • SKIP - Perform no validation.

Renamed from --skip-validation.

See Common vkconfig script options for options that are available in all of the vkconfig tools.

Examples

The following examples show how you can create or update SourceFeed.

Create the source SourceFeed and assign it to the cluster, StreamCluster1 in the scheduler defined by the myscheduler.conf config file:

$ /opt/vertica/packages/kafka/bin/vkconfig source --create --source SourceFeed \
                                           --cluster StreamCluster1 --partitions 3
                                           --conf myscheduler.conf

Update the existing source SourceFeed to use the existing cluster, StreamCluster2 in the scheduler defined by the myscheduler.conf config file:

$ /opt/vertica/packages/kafka/bin/vkconfig source --update --source SourceFeed \
                                           --new-cluster StreamCluster2
                                           --conf myscheduler.conf

The following example reads the sources defined in the scheduler defined by the weblogs.conf file.

$ vkconfig source --read --conf weblog.conf
{"source":"web_hits", "partitions":1, "src_enabled":true,
"cluster":"kafka_weblog",
"hosts":"kafka01.example.com:9092,kafka02.example.com:9092"}