Open topic with navigation

# Prepare Text

It is important to ensure that the text used for custom language model building is cleaned up to a reasonable level.

- Remove any material that does not occur in standard written English, such as HTML tags, and anything that would not usually occur in spoken language, such as tables.
- Ensure that sentence breaks (periods) are present in the text.
- Ensure that there are no duplicated sections in the text.
- Ensure that the text encoding is UTF-8.

NOTE:

Japanese, Korean, Mandarin, and Taiwanese Mandarin languages require text segmentation before IDOL Speech Server can process them. Text segmentation inserts whitespace between words. The `LanguageModelBuild` task segments text if you set the `DoSegment` parameter to `True` (see Build the Language Model).

## Normalize Text

You must normalize the text used for language model building before processing so that word representations are standardized. For example, ‘1’ and ‘one’ are treated as two different representations of the same digit. For more information about the normalization scheme used in IDOL Speech Server, see Audio Transcript Requirements.

Many of the IDOL Speech Server text operations require you to normalize input text as an initial step. For details of the normalization procedure, see Run Text Normalization.

© 2015-2018 EntIT Software LLC, a Micro Focus company | Send documentation feedback to Micro Focus Close Send documentation feedback To open the configured email client on this computer, open an email window. Otherwise, copy the information below to a web mail client, and send this email to swpdl.idoldocsfeedback@microfocus.com. We appreciate your feedback! Product: Topic Title: Help Topic: Feedback:

_FT_HTML5_bannerTitle.htm