Re-indexing the dataset using IDOL

Re-indexing records

You can use this function to select existing Content Manager records with electronic documents to index the document content for the first time or to re-index if they have not been indexed correctly, for example because of a network failure.

You should also use it when an entire Content Manager document content index has been lost or damaged, for an upgrade of a Content Manager 7.2 IDOL index and when the indexing parameters have been changed, for example which Additional Fields to index.

To improve the very first document content indexing process, see Options to improve the initial indexing process below.

Follow the instructions in the Content Manager Help topic IDOL Record Reindex.

Re-indexing user Locations

You can use this function to update the user security data in IDOL by using the CFS connector. This is useful when an administrator has made changes to user security data like Content Manager Access Controls and user permissions.

Follow the instruction in the Content Manager Help topic IDOL Record Reindex.

Options to improve the initial indexing process

The longest process in any installation of Content Manager IDOL is the initial indexing of the electronic documents in the dataset. However, there are a number of ways to optimize this process.

Exclude unwanted file types from indexing. See Suppressing document content indexing.
If there are specific Record Types in your dataset that do not need to have the content of their electronic documents indexed, ensure that you have suppressed document content indexing in the options for the Record Type. In Content Manager, see Content Manager Help about the Record Type Electronic tab.
In Content Manager, ensure that you have set the maximum size for archive and standard files in the IDOL Record Reindex dialog box, Select tab.
You can spread the indexing load across multiple Workgroup Servers. It is now possible to process document content from more than one Workgroup Server and write to the same content database.
Redundant Workgroup Servers can be configured to process chunks of data at the same time. The Workgroup Servers would be configured separately and not be part of the same deployment. They would be set up to be identical and configured to use the same IDOL Server. You could use virtual machines for this purpose, with enough disk space to handle the text extraction process and processing of archive files.
The process uses minimal resources to access the database and document stores and should not have an undue impact on production performance.
Each Workgroup Server would run as many text extraction processes as it has CPU cores available. Content Manager IDOL includes a deduplication function so it is not possible to create duplicates for the same document. However, using the document content indexing options above, it is possible to set up each Workgroup Server to process a unique set of records by either using the date fields to specify a range or a URI range.
Below is an example Oracle script that could be used to divide the data up into three groups of URIs based on the size of the electronic document stores. This would ensure that they share the load evenly. This script could be modified for the other databases or to break up the data into the number of groups required.
```
Set serveroutput on;
```
```
DECLARE
```
```
CurrentValue NUMBER := 0;
```
```
RunningSum NUMBER := 0;
```
```
MaxLimit NUMBER := 0;
```
```
MaxUri NUMBER := 0;
```
```
LowerUri NUMBER := 0;
```
```
UpperUri NUMBER := 0;
```
```
CurrentUri NUMBER := 0;
```
```
bSet BOOLEAN;
```
```
BEGIN
```
```
SELECT Round(Sum(reBytes)/3)
```
```
INTO MaxLimit
```
```
FROM TSRECELEC;
```
```
SELECT Max(uri)
```
```
INTO MaxUri
```
```
FROM TSRECELEC;
```
```
FOR i IN 1..3
```
```
LOOP
```
```
RunningSum := 0;
```
```
bSet := TRUE;
```
```
LowerUri := 0;
```
```
UpperUri := 0;
```
```
LOOP
```
```
SELECT uri, reBytes
```
```
INTO UpperUri, CurrentValue
```
```
FROM ( SELECT * FROM TSRECELEC
```
```
WHERE uri > CurrentUri
```
```
ORDER BY uri)
```
```
WHERE ROWNUM <= 1;
```
```
IF bSet THEN
```
```
LowerUri := UpperUri;
```
```
bSet := FALSE;
```
```
END IF;
```
```
RunningSum := RunningSum + CurrentValue;
```
```
CurrentUri := UpperUri;
```
```
IF CurrentUri = MaxUri THEN
```
```
EXIT;
```
```
ELSIF RunningSum < MaxLimit THEN
```
```
CONTINUE;
```
```
ELSE
```
```
EXIT;
```
```
END IF;
```
```
END LOOP;
```
```
DBMS_OUTPUT.PUT_LINE ('WGS ' || TO_CHAR(i) ||' Min Uri = ' || TO_CHAR(LowerUri));
```
```
DBMS_OUTPUT.PUT_LINE ('WGS ' || TO_CHAR(i) ||' Max Uri = ' || TO_CHAR(UpperUri));
```
```
END LOOP;
```
```
END;
```
When employing this approach, it is important to ensure that the IDOL Server has been correctly sized to handle the store content.