Vertica documentation is moving! Use our new documentation site at https://docs.rocketsoftware.com/bundle/vertica_12.0/page/index.html

Vertica tokenizers

A tokenizer does the following:.

A tokenizer does the following:

  • Receives a stream of characters.

  • Breaks the stream into individual tokens that usually correspond to individual words.

  • Returns a stream of tokens.