Tokenizer base configuration
You can choose among several different tokenizer base configurations:.
You can choose among several different tokenizer base configurations:
| Type | Position | Without Position |
|---|---|---|
| Ngram | logNgramTokenizerPositionFactory | logNgramTokenizerFactory |
| Words | logWordITokenizerPositionFactory | logWordITokenizerFactory |
Create a logWord tokenizer without positional relevance:
=> CREATE TRANSFORM FUNCTION v_txtindex.fooTokenizer AS LANGUAGE 'C++' NAME 'logWordITokenizerFactory' LIBRARY v_txtindex.logSearchLib NOT FENCED;