Vertica tokenizers
A tokenizer does the following:.
A tokenizer does the following:
-
Receives a stream of characters.
-
Breaks the stream into individual tokens that usually correspond to individual words.
-
Returns a stream of tokens.
A tokenizer does the following:
Receives a stream of characters.
Breaks the stream into individual tokens that usually correspond to individual words.
Returns a stream of tokens.