Vertica documentation is moving! Use our new documentation site at https://docs.rocketsoftware.com/bundle/vertica_24.2/page/index.html

Tokenizer base configuration

You can choose among several different tokenizer base configurations:.

You can choose among two tokenizer base configurations:

  • Ngram with position: logNgramTokenizerPositionFactory
  • Ngram without position: logNgramTokenizerFactory

The following example creates an Ngram tokenizer without positional relevance:

=> CREATE TRANSFORM FUNCTION v_txtindex.myNgramTokenizer 
   AS LANGUAGE 'C++' 
   NAME 'logNgramTokenizerFactory' 
   LIBRARY v_txtindex.logSearchLib NOT FENCED;