Open topic with navigation

The Index Process

After all the pre-index processing, IDOL Server indexes the documents. This section describes the processes that IDOL Server performs when it indexes your content.

Index Action Process

See also: The Index Process in a Distributed System

The following diagram describes how the IDOL Server Content component processes an incoming index action. The colored boxes correspond to the status that the IndexerGetStatus action returns for that stage of the process:

For more information about IndexerGetStatus and the status messages that it returns, refer to the IDOL Server Reference.

Index Process for the IDOL Server Content Component

The following notes provide some additional information:

• IDOL Server rejects an index action if it is not from an accepted index client, or if it does not have enough disk space to process the operation.

• As an alternative to indexing data by submitting index actions, you can also use the wizard on the Index tab on the Console page in the Control section of IDOL Admin to submit data for IDOL Server to index. You can also use the wizard on the Replace tab on the Console page in the Control section of IDOL Admin to change field values or delete fields from indexed documents.

• You can use the IndexerGetStatus action to pause, and restart the current index action by adding the IndexAction parameter (set to Pause or Restart). You can pause only the DREADD, DREADDDATA, DREREPLACE, DRECOMPACT, and DRETAGDOCCLUSTERS index actions.

• You can use the IndexerGetStatus action to cancel an index action by adding the IndexAction parameter (set to Cancel). You can cancel any queued index action, which means that IDOL Server does not start to process that operation. When you cancel the currently processing index action, where possible IDOL Server stops processing the operation and reverts any changes.

• You can view the index queue and associated information on the Index Queue tab of the Status page in IDOL Admin.

• The indexing thread automatically pauses the current operation if there is insufficient disk space.

• The DREFLUSHANDPAUSE index action automatically pauses when the flush is complete.

• The DREEXPORTREMOTE operation with Blocking set to False gives an index status of -38 "Processing in remote engine" until the target server indexes and syncs the exported data, or fails.

• The DRESHUTDOWN index action causes the server to flush and exit as soon as the current index operation is complete.

Document Processing

The following process describes the stages of document processing that IDOL Server performs on documents, and the order in which these occur. Some of the steps might not be performed, according to your configuration.

1. IDOL Server determines whether to index the document at all. This step depends on your distribution settings.

2. It identifies basic metadata:

   • Date

   • Database

   • SecurityType

   • AutnRank

   • LanguageType

   It processes these according to your field processing rules.

3. It runs Automatic Language Detection  on data extracted from the configured LangDetectType fields.

4. IDOL Server converts the entire document internally to a UTF-8 representation, according to the encoding information from It runs Automatic Language Detection  on data extracted from the configured LangDetectType fields. (if required).

5. It processes the reference fields, and removes duplicate documents according to your deduplication settings.

   If you have set up your deduplication process to preserve fields, IDOL Server processes these and adds the preserved fields to the document.

6. IDOL Server processes the conceptual information from the index fields:

   1. Sentence Breaking.

   2. Stop word removal.

   3. It stores information for each term:

      • the stem of the term

      • positional information (used for proximity searches)

      • case

      • unstemmed form

      • AdvancedSearch, AdvancedCaseSearch, and AdvancedPlus functions

   4. Advanced processing, including Soundex, synonyms, and proper names.

   This information is initially added to the index cache and made available when the cache is flushed. Flushing occurs after every index action if you have set DelayedSync to False in your configuration file. Otherwise, IDOL Server flushes the cache periodically, as determined by your MaxSyncDelay configuration parameter. You can also flush the index cache to disk by using the Sync function on the Service Control tab of the Console page in the Control section of IDOL Admin.

7. IDOL Server processes the other field and subindex types (numeric, parametric, match, sort, security, and so on).

8. IDOL Server stores the document content.

Related Topics

• The Index Process in a Distributed System

• The Index Flush Process

• Order of Language Processes