The Index Process

After all the pre-index processing, IDOL Server indexes the documents. This section describes the processes that IDOL Server performs when it indexes your content.

Index Action Process

See alsoThe Index Process in a Distributed System

The following diagram describes how the IDOL Server Content component processes an incoming index action. The colored boxes correspond to the status that the IndexerGetStatus action returns for that stage of the process:

For more information about IndexerGetStatus and the status messages that it returns, refer to the IDOL Server Reference.

Index Process for the IDOL Server Content Component

The following notes provide some additional information:

Document Processing

The following process describes the stages of document processing that IDOL Server performs on documents, and the order in which these occur. Some of the steps might not be performed, according to your configuration.

  1. IDOL Server determines whether to index the document at all. This step depends on your distribution settings.

  2. It identifies basic metadata:

    It processes these according to your field processing rules.

  3. It runs Automatic Language Detection  on data extracted from the configured LangDetectType fields.

  4. IDOL Server converts the entire document internally to a UTF-8 representation, according to the encoding information from It runs Automatic Language Detection  on data extracted from the configured LangDetectType fields. (if required).

  5. It processes the reference fields, and removes duplicate documents according to your deduplication settings.

    If you have set up your deduplication process to preserve fields, IDOL Server processes these and adds the preserved fields to the document.

  6. IDOL Server processes the conceptual information from the index fields:

    1. Sentence Breaking.

    2. Stop word removal.

    3. It stores information for each term:

    4. Advanced processing, including Soundex, synonyms, and proper names.

    This information is initially added to the index cache and made available when the cache is flushed. Flushing occurs after every index action if you have set DelayedSync to False in your configuration file. Otherwise, IDOL Server flushes the cache periodically, as determined by your MaxSyncDelay configuration parameter. You can also flush the index cache to disk by using the Sync function on the Service Control tab of the Console page in the Control section of IDOL Admin.

  7. IDOL Server processes the other field and subindex types (numeric, parametric, match, sort, security, and so on).

  8. IDOL Server stores the document content.