Knowledge Discovery 25.1
Knowledge Discovery Expert

# Advanced Tokenization

This section discusses some exceptions to the usual language tokenization. You might need to consider these sections if you have documents in Asian languages, or if the standard Content component configuration options do not provide adequate support for your custom scenarios.

## Sentence Breaking

In most writing systems, spaces and punctuation characters define the gap between one word token and the next. Content uses this information to divide text up into searchable tokens.

In Chinese, Japanese, Korean, and Thai languages, there are no spaces between words in the text, and each word symbol is a multiple byte character. This property of the text stream means that Content cannot easily split the text up into searchable tokens.

For these languages, additional rules and methods exist to divide the text up into tokens. The Content installation includes these methods in sentence breaking libraries, which are specific to a particular language.

You can also use stemming files with the sentence breaking libraries to allow you to create user-defined dictionaries, which allow you to define how the sentence breaking library handles new words (such as new company names), and words specific to your use case (such as product names or jargon).

## Custom Tokenization

In rare cases, you might want to use a different tokenization system to the one that Content uses as standard, for example if the default stemming rules are not appropriate for your documents.

In these cases, you can create a custom sentence breaking library. You use the sentence breaking library to define the language-specific rules for custom tokenization in your language. For more information, refer to the Knowledge Discovery Sentence Breaking API Technical Note.

## N-Gram Tokenization

N-Gram tokenization is another approach that you can use for non-tokenized languages such as Chinese, Japanese, Korean and Thai. With N-Grams, Content splits all text up into tokens that contain `N` characters, where `N` is an integer value that you can configure.

This method is primarily for simple tokenization of Asian languages.

You can also use this option to provide consistent tokenization between multiple languages (Cross-Lingual Search). For example, if you want to be able to search for both Chinese and Japanese text in a single query, you must ensure that tokenization is consistent for both languages. Sentence breaking libraries are language-specific, so you must use NGram tokenization instead.

When you have documents that contain text from multiple languages, you can use the `NGramMultibyteOnly` and `NGramSentenceBrokenScriptOnly` configuration parameters. These parameters allow you to limit the N-Gram tokenization to only multibyte characters, or only Chinese, Japanese, and Korean characters.

Related Topics

- Order of Language Processes

- Unicode Script Ranges

Copyright 2024 Open Text
Version 25.1 | Last updated January 2025

Send documentation feedback

Close

Send documentation feedback

To send feedback using your default email client, open an email window.

Otherwise, send your feedback and the information below to MFI-swpdl.idoldocsfeedback@opentext.com.

We appreciate your feedback!

Product: Knowledge Discovery 25.1

Topic Title: Advanced Tokenization