Knowledge Discovery 25.1
Knowledge Discovery Expert

# The Index Process

After all the pre-index processing, the Content component indexes the documents. This section describes the processes that Content performs when it indexes your content.

## Index Action Process

See also: The Index Process in a Distributed System

The following diagram describes how the Content component processes an incoming index action. The colored boxes correspond to the status that the `IndexerGetStatus` action returns for that stage of the process:

For more information about `IndexerGetStatus` and the status messages that it returns, refer to the Content Component Help.

Index Process for the Content component

The following notes provide some additional information:

- Content rejects an index action if it is not from an accepted index client, or if it does not have enough disk space to process the operation.

- As an alternative to indexing data by submitting index actions, you can also use the wizard on the Index tab on the Console page in the Control section of Knowledge Discovery Admin to submit data for Content to index. You can also use the wizard on the Replace tab on the Console page in the Control section of Knowledge Discovery Admin to change field values or delete fields from indexed documents.

- You can use the `IndexerGetStatus` action to pause, and restart the current index action by adding the `IndexAction` parameter (set to `Pause` or `Restart`). You can pause only the `DREADD`, `DREADDDATA`, `DREREPLACE`, `DRECOMPACT`, and `DRETAGDOCCLUSTERS` index actions.

- You can use the `IndexerGetStatus` action to cancel an index action by adding the `IndexAction` parameter (set to `Cancel`). You can cancel any queued index action, which means that Content does not start to process that operation. When you cancel the currently processing index action, where possible Content stops processing the operation and reverts any changes.

- You can view the index queue and associated information on the Index Queue tab of the Status page in Knowledge Discovery Admin.

- The indexing thread automatically pauses the current operation if there is insufficient disk space.

- The `DREFLUSHANDPAUSE` index action automatically pauses when the flush is complete.

- The `DREEXPORTREMOTE` operation with `Blocking` set to `False` gives an index status of -38 "Processing in remote engine" until the target server indexes and syncs the exported data, or fails.

- The `DRESHUTDOWN` index action causes the server to flush and exit as soon as the current index operation is complete.

## Document Processing

The following process describes the stages of document processing that Content performs on documents, and the order in which these occur. Some of the steps might not be performed, according to your configuration.

1. Content determines whether to index the document at all. This step depends on your distribution settings.

2. It identifies basic metadata:

   - Date

   - Database

   - `SecurityType`

   - `AutnRank`

   - `LanguageType`

   It processes these according to your field processing rules.

3. It runs Automatic Language Detection on data extracted from the configured LangDetectType fields.

4. Content converts the entire document internally to a UTF-8 representation, according to the encoding information from It runs Automatic Language Detection on data extracted from the configured LangDetectType fields. (if required).

5. It processes the reference fields, and removes duplicate documents according to your deduplication settings.

   If you have set up your deduplication process to preserve fields, Content processes these and adds the preserved fields to the document.

6. Content processes the conceptual information from the index fields:

   1. Sentence Breaking.

   2. Stop word removal.

   3. It stores information for each term:

      - the stem of the term

      - positional information (used for proximity searches)

      - case

      - unstemmed form

      - AdvancedSearch, AdvancedCaseSearch, and AdvancedPlus functions

   4. Advanced processing, including Soundex, synonyms, and proper names.

   This information is initially added to the index cache and made available when the cache is flushed. Flushing occurs after every index action if you have set `DelayedSync` to `False` in your configuration file. Otherwise, Content flushes the cache periodically, as determined by your `MaxSyncDelay` configuration parameter. You can also flush the index cache to disk by using the Sync function on the Service Control tab of the Console page in the Control section of Knowledge Discovery Admin.

7. Content processes the other field and subindex types (numeric, parametric, match, sort, security, and so on).

8. Content stores the document content.

Related Topics

- The Index Process in a Distributed System

- The Index Flush Process

- Order of Language Processes

Copyright 2024 Open Text
Version 25.1 | Last updated January 2025

Send documentation feedback

Close

Send documentation feedback

To send feedback using your default email client, open an email window.

Otherwise, send your feedback and the information below to MFI-swpdl.idoldocsfeedback@opentext.com.

We appreciate your feedback!

Product: Knowledge Discovery 25.1

Topic Title: The Index Process