Troubleshooting document content indexing and searching

  1. The Workgroup Server runs the select query based on the search to identify the electronic documents to index, taking into account the documents that are excluded due to their file type, Record Type, and size. When complete, the reindex window appears and you can start the index process.
  2. The Workgroup Server selects the documents in order of URI and passes them to the KeyView text converter. The number of text converters (KVOOP.exe) depends on the number of cores available on the Workgroup Server that is executing the indexing process.
  3. Each core creates a separate TRIMtrans_TES_x.txt file, which adds text converted documents until the number of processed records has reached the number specified in the field Size of transaction queue in the IDOL Record Reindex dialog box, Protocol tab.
    These transaction files are stored in the directory C:\Users\<username>\AppData\Local\\Micro Focus\Content ManagerTEMP.
    Note that crashes place crash dump files in the same directory. ZIP files for text extraction will be opened and extracted to the same location .
  4. When reaching the limit, the transaction text file is passed to the IDOL Server as a .dat file along with a .cmd file. The .cmd file is a DREADD command for the IDOL Server.
    The IDOL Server then passes the .dat file and .cmd file to every content service engine for processing.
    At this point, Content Manager has completed its processing and now, IDOL processes the data into the IDOL databases.
  1. Each content engine receives the text extraction and command file. Taking into account the options set in IDOL, they then process the command file and update the content engine database.

    The IDOL databases are distributed so that each of them processes separate parts of the text file as determined by the IDOL Server. They hold the information in memory until the synchronization limit is reached. At that point, they commit the changes to the IDOL database.

    The defaults for IDOL are:

    DelayedSync=true
    MaxSyncDelay=120

    This means that the IDOL database waits 120 seconds before committing all changes to the IDOL content database. When multiple threads are processing text extractions, this commit submits all changes to the database. Increasing the parameter reduces the number of commits during an indexing process. Memory use increases, but updates happen faster.

  2. If the IDOL content service is busy, the command and text extraction file are copied to the Status directory and queued for processing. When disk write speeds are slow, the likelihood of queuing is much greater. The content service processes from the queue as soon as it is available again.
    Before processing a queued command, the IDOL service checks whether the batch has already been processed by another content service. If so, it discards the batch and picks up the next queued batch file for processing.
  • You can see main updates in an IDOL database in the Dynterm and Nodetable directories.
  • Content Manager is oblivious to the state of the IDOL content databases and continues to pass text extractions and .cmd files to the IDOL Server.

When you cannot use document content searching in your Content Manager client or cannot find any records by using document content searching, check that your system meets these requirements:

  • You have created an IDOL document content index - to check, in Content Manager Enterprise Studio, expand Datasets. Right-click the dataset and on the Content Index menu, if the commands under Create IDOL are available, there is an IDOL content index. If there is no IDOL content index, use Create IDOL to create one. This is also what you need to do after you have upgraded Content Manager from version 7.1 (TRIM) or earlier. See Setting up document content indexing and searching.
  • You have content indexed your dataset - see Re-indexing the dataset using IDOL
  • The IDOL Service is running - to check, on the computer running the Workgroup Server, make sure that the services Content Manager IDOL Service and at least one Content Manager Content Service have the status Started
  • There is sufficient disk space - the Content Manager IDOL content engine requires a minimum of 1024MB of free space for the operation of the content service. You can change that figure in the IDOL configuration files. If there is insufficient space when starting the Content Manager IDOL Content Service, the message Error 1067: The process terminated unexpectedly may appear.

    During a reindexing process, the Content Manager error DCI server is unreachable may appear.

    The log file dih_index.log may also report I/O errors.

    In normal day-to-day operations, document content indexing is handled through the event processor. If the document content service has insufficient space, the following appears in the event processor log file:

    04:54:06 0 21 Error : 21 User: <account> - DCI server TRIMIDOL1 is unreachable

    The IDOL application log in the content service\log directory may report that there is insufficient space.

    To resolve the issue, increase the available space and restart the process.

  • Content Manager IDOL database set to read-only - there may be circumstances that make the IDOL services start up without details of the current Content Manager IDOL database configuration. In that case, the IDOL service detects that a database existed that was not in the configuration file and sets the database to read-only by appending it to the end of the Content Service configuration file.

    To resolve this issue, edit the TRIM Content Service.cfg configuration file and change the value to False:

    [TRIM_PM]
    DatabaseReadOnly=FALSE

    PM represents the Content Manager database ID of the dataset.

  • Index commands that can be sent to the IDOL Server by using a Web browser - contact Content Manager product support for details:
    • DREADD - indexes content into IDOL Server
    • DREADDDATA - indexes content over socket into IDOL Server
    • DREBACKUP - backs up IDOL Server's data index
    • DRECHANGEMETA - changes documents' meta fields
    • DRECOMPACT - compacts IDOL Server's data index
    • DRECREATEDBASE - creates an IDOL Server database
    • DREDELDBASE - deletes all documents from an IDOL Server database
    • DREDELETEDOC - deletes documents by ID
    • DREDELETEREF - deletes documents by reference
    • DREDUPLICATE - removes or tags duplicates in IDOL Server's data index
    • DREEXPIRE - expires documents from IDOL Server
    • DREEXPORTIDX - exports IDX files from IDOL Server
    • DREEXPORTREMOTE - xports XML files from one IDOL Server and indexes them into another
    • DREEXPORTXML- exports XML files from IDOL Server
    • DREFLUSHANDPAUSE - prepares IDOL Server for a snapshot (hot backup)
    • DREINITIAL - resets IDOL Server's data index
    • DREREGENERATE - fixes the parent-child relationships for documents that were indexed out of order
    • DREREMOVEDBASE - deletes an IDOL Server database
    • DRERENAMEDBASE - renames an IDOL Server database
    • DREREPLACE - changes documents' field values
    • DRERESET - activates configuration file changes
    • DRERESIZEINDEXCACHE - dynamically resizes the index cache
    • DRESPELLCHECKCACHE - controls the spell check correction cache
    • DRESYNC - flushes to disk the index cache
    • DRETAGDOCCLUSTERS - tags documents that are conceptually similar

Related Topics Link IconRelated information