IDOL Category Component 24.4
IDOL Category Component Help

# Character Tokenization

You can tokenize characters into N-grams of a specified size. Set the `NGram` configuration parameter in your language configuration section to the number of characters to use in each N-gram group.

NOTE: You must not use `NGram` with the `SentenceBreaking` configuration parameter.

For example, if you set `NGram` to `2`, Content tokenizes the word Hello as:

    he el ll lo

To tokenize only multibyte strings, set `NGramMultiByteOnly` to `True`.

    [Japanese]
    NGram=2
    NGramMultiByteOnly=True

For this configuration, if you have a document that contains both English and Asian (multibyte) text, Content tokenizes the Asian text according to the `NGram` parameter. It does not tokenize the English text.

To tokenize only multiple-byte strings in Chinese, Japanese, and Korean characters (and ignore multiple-byte strings in other languages), set `NGramSentenceBrokenScriptOnly` to `True`.

    [Japanese]
    NGram=2
    NGramSentenceBrokenScriptOnly=True

For this configuration, if you have a document that contains multibyte text in both Japanese and Greek, Content tokenizes the Japanese text according to the `NGram` parameter. It does not tokenize the Greek multibyte text.

Copyright 2024 Open Text
Version 24.4 | Last updated September 2024

Send documentation feedback

Close

Send documentation feedback

To send feedback using your default email client, open an email window.

Otherwise, send your feedback and the information below to MFI-swpdl.idoldocsfeedback@opentext.com.

We appreciate your feedback!

Product: IDOL IDOL Category Component 24.4

Topic Title: Character Tokenization