Load spec tool options

The vkconfig script's load spec tool lets you provide parameters for a COPY statement that loads streaming data.

The vkconfig script's load spec tool lets you provide parameters for a COPY statement that loads streaming data.

Syntax

$ vkconfig load-spec {--create | --read | --update | --delete} \
           [--load-spec spec-name] [other-options...]
--create
Creates a new load spec. Cannot be used with --delete, --read, or --update.
--read
Outputs the current settings of the load specs defined in the scheduler. This output is in JSON format. Cannot be used with --create, --delete, or --update.

By default, this option outputs all load specs defined in the scheduler. You can limit the output by supplying a single value or a comma-separated list of values to these options:

  • --load-spec

  • --filters

  • --uds-kv-parameters

  • --parser

  • --message-max-bytes

  • --parser-parameters

The vkconfig script only outputs the configuration of load specs that match the values you supply.

You can use LIKE wildcards in these options. See LIKE for more information about using wildcards.

--update
Updates the settings of spec-name. Cannot be used with --create, --delete, or --read.
--delete
Deletes the load spec named spec-name. Cannot be used with --create, --read, or --update.
--load-spec spec-name
A unique name for copy load spec to operate on. This option is required for --create, --update, and --delete.
--dump

When you use this option along with the --read option, vkconfig outputs the Vertica query it would use to retrieve the data, rather than outputting the data itself. This option is useful if you want to access the data from within Vertica without having to go through vkconfig. This option has no effect if not used with --read.

--filters "filter-name"
A Vertica FILTER chain containing all of the UDFilters to use in the COPY statement. For more information on filters, refer to Parsing custom formats.
--message-max-bytes max-size

Specifies the maximum size, in bytes, of a Kafka protocol batch message.

Default: 25165824

--new-load-spec new-name
A new, unique name for an existing load spec. Requires the --update parameter.
--parser-parameters "key=value[,...]`"`
A list of parameters to provide to the parser specified in the --parser parameter. When you use a Vertica native parser, the scheduler passes these parameters to the COPY statement where they are in turn passed to the parser.
--parser parser-name
Identifies a Vertica UDParser to use with a specified target.This parser is used within the COPY statement that the scheduler runs to load data. If you are using a Vertica native parser, the values supplied to the --parser-parameters option are passed through to the COPY statement.

**Default:**KafkaParser

--uds-kv-parameters key=value[,...]
A comma separated list of key value pairs for the user-defined source.
--validation-type {ERROR|WARN|SKIP}
Specifies the validation performed on a created or updated load spec, to one of the following:

  • ERROR: Cancel configuration or creation if vkconfig cannot validate the load spec. This is the default setting.

  • WARN: Proceed with task if validation fails, but display a warning.

  • SKIP: Perform no validation.

Renamed from --skip-validation.

See Common vkconfig script options for options that are available in all of the vkconfig tools.

Examples

These examples show how you can use the Load Spec utility options.

Create load spec Streamspec1:

$ /opt/vertica/packages/kafka/bin/vkconfig load-spec --create --load-spec Streamspec1 --conf myscheduler.conf

Rename load spec Streamspec1 to Streamspec2:

$ /opt/vertica/packages/kafka/bin/vkconfig load-spec --update --load-spec Streamspec1 \
                                                     --new-load-spec Streamspec2 \
                                                     --conf myscheduler.conf

Update load spec Filterspec to use the KafkaInsertLengths filter and a custom decryption filter:

$ /opt/vertica/packages/kafka/bin/vkconfig load-spec --update --load-spec Filterspec \
                                                     --filters "KafkaInsertLengths() DecryptFilter(parameter=Key)" \
                                                     --conf myscheduler.conf

Read the current settings for load spec streamspec1:

$ vkconfig load-spec --read --load-spec streamspec1 --conf weblog.conf
{"load_spec":"streamspec1", "filters":null, "parser":"KafkaParser",
"parser_parameters":null, "load_method":"TRICKLE", "message_max_bytes":null,
"uds_kv_parameters":null}