Knowledge Discovery 25.1
Knowledge Discovery Expert

# Linguistic Adjustments

After the Content component has processed the tokens to remove any that you do not want to index, it indexes the remaining tokens. During this process, it modifies the tokens to make it easier to return the correct documents. This involves the following processes:

- Stemming

- Character Normalization

- Transliteration

## Stemming

Main Topic: Stemming

In many languages, a related set of words have a common root. For example, help, helping, helpful and helped, all stem from the common root help. You can reduce words to their common root without losing meaning.

The Content stemming algorithm reduces all words to their stem, and indexes the stem. This process allows you to search for a word, and return documents with related concepts that do not specifically include that word.

## Character Normalization

In some languages, there is more than one way to represent a character. For example:

- the Roman alphabet has uppercase and lowercase forms of all letters.

- the Japanese katakana script can have full width or half width characters.

- the Chinese language has two scripts, usually known as Chinese Traditional and Chinese Simplified.

- languages with multibyte scripts sometimes use multibyte versions of Latin characters such as punctuation or numeric digits.

Content uses canonicalization to ensure that it treats all character forms equally. It automatically converts to an internationally recognized canonical form. Retrieval then matches all versions of a character.

Character normalization is controlled by the same configuration settings as Transliteration.

## Transliteration

Transliteration is like Character Normalization in that it aims to map sets of characters to a standard form so that a search for different forms match documents containing any of those forms. An important example is the removal of accents from letters so that a search for cafe matches documents containing café, and the reverse. Similarly, the German letter ß is transliterated to ss.

Transliteration schemes differ by language. For example in German, the letter ö transliterates to oe, whereas in Swedish it transliterates to o. Several languages that use non-Roman scripts can also be transliterated to Roman. For example in Russian, Content transliterates Владимир to Vladimir. For details of the transliteration schemes used, see Transliteration Tables.

When you have multiple languages in your server, OpenText recommends setting `GenericTransliteration` to `True`. See Cross-Lingual Search.

NOTE: Transliteration affects only the characters used to represent a word. Content does not translate terms from one language to another.

Turn Transliteration Off

Several languages have unusual linguistic behavior, such that even if you set `Transliteration` to `False` for that language, some characters are still transliterated. To prevent all transliteration, set:

    [LanguageTypes]
    GenericTransliteration=True

and then set `Transliteration` to `False` in the individual language sections.

Related Topics

- Advanced Tokenization

- Stemming

- Order of Language Processes

Copyright 2024 Open Text
Version 25.1 | Last updated January 2025

Send documentation feedback

Close

Send documentation feedback

To send feedback using your default email client, open an email window.

Otherwise, send your feedback and the information below to MFI-swpdl.idoldocsfeedback@opentext.com.

We appreciate your feedback!

Product: Knowledge Discovery 25.1

Topic Title: Linguistic Adjustments