Open topic with navigation

# Optimize the Language Identification Set

After you prepare the classifier list, you can run a language identification task. However, before you do so, Micro Focus recommends that you optimize the language weights and score thresholds in the classifier list.

Optimization balances the language models. After training, some classifiers might be stronger than others owing to the properties of the training material and the languages themselves. The optimization process weights the language models so that weaker languages have increased recall, without compromising the precision for stronger language models.

Optimization also calculates score thresholds for open set language identification. In open set language identification, IDOL Speech Server checks whether a particular identified language matches the audio with a score above the specified threshold. If the language score is lower than the threshold, it returns the language identification as unknown.

The optimization process requires several audio files, each from a known language. It analyzes sections throughout each file, and determines whether each section was identified correctly. If a particular language is frequently not identified, IDOL Speech Server usually increases the weighting of the language classifier. Conversely, if IDOL Speech Server frequently identifies a particular language more often than it occurs, it usually reduces the weighting of the language classifier.

To optimize a language set

1. Gather the data to use. You need at least one audio file for each language represented in the classifier list. This audio data must not be the same data that you used to train the language classifiers.
2. Convert each audio file into a language identification feature file (see Create Language Identification Feature Files).

3. Create a list that contains the name of each feature file and a label that specifies the language that the file represents. Use the following format:

       LanguageName;LifFilename

   You can specify multiple files for each language. Either type the file names in a comma-separated list, or add multiple instances of a language. For example:

       ENUK;english1.lif
       ENUK;english2.lif
       ENUS;us-english1.lif,us-english2.lif
       ESES;spanish.lif
       FRFR;french.lif

   You can also use data for unknown languages, to optimize open set language identification. In this case, use the tag unknown. For example:

       Unknown;unknown1.lif
       Unknown;unknown2.lif

   For more information about IDOL Speech Server's list manager, see Create and Manage Lists.

4. If you intend to write the optimized weights to a list in the list manager, you must create the list before you specify it in the following step.

5. Run the optimization task. Send an `AddTask` action to IDOL Speech Server, and set the following parameters:

   `Type` | The task name. Set to `LangIdOptimize`.
   `DataList` | The list that specifies the feature files.
   `DataPath` | The path to the directory that contains the feature files listed in the `DataList`.
   `ClassList` | The list that specifies the classifiers (see Combine Classifiers into a Language Identification Set).
   `ClassPath` | The path to the directory that contains the language classifiers.
   `Out` | The classifier list to write the optimized weights to.

For example:

    http://localhost:15000/action=AddTask&Type=LangIdOptimize&DataList=ListManager/OptList&DataPath=C:/LangID/Data&ClassList=ListManager/ClassList&ClassPath=C:/LangID/class&Out=ListManager/OptClassSet

This action tests the feature files listed in the `OptList` list against the classifiers specified in the `ClassList` list and then generates an updated classifier list–`OptClassSet`.

This action returns a token. You can use the token to:

- Check the task status. See Check the Status of a Task.
- Retrieve the updated classifier file. See Get Task Results.

The `LangIdOptimize` action also generates a log file that contains information about the optimization process. See Review the Optimization Log File.

© 2015-2018 EntIT Software LLC, a Micro Focus company | Send documentation feedback to Micro Focus Close Send documentation feedback To open the configured email client on this computer, open an email window. Otherwise, copy the information below to a web mail client, and send this email to swpdl.idoldocsfeedback@microfocus.com. We appreciate your feedback! Product: Topic Title: Help Topic: Feedback:

_FT_HTML5_bannerTitle.htm