Content Manager 23.3 Content Manager

Elasticsearch Metadata Index

NOTE: This option is only available if an Elasticsearch Content Index has been configured.

Elasticsearch allows organisations to index metadata fields, as well as document content. To index metadata fields, from the Administration ribbon, in the Elasticsearch Index group, click Metadata. The Elasticsearch Metadata Configuration dialogue is displayed.

From the Available Metadata Fields column, select the fields to be indexed and click Add. These will be added to the Current Metadata Fields column.

NOTE: When adding a new metadata field(s) to be indexed, it will only be indexed from that point in time onwards. If you want to index all metadata associated to that field you will need to reindex the metadata.

NOTE: Once fields have been added to the Current Metadata Fields list they cannot be removed unless you delete the entire Elasticsearch index, then remove the field from the list, recreate the Elasticsearch index and then complete a reindex.

To allow searching of content and metadata fields that have been indexed using Elasticsearch, the System Option - Miscellaneous - Optimize queries containing content clauses option must be enabled.

Options to improve the initial indexing process

The longest process in any installation of Content Manager Content Index is the initial indexing of the electronic documents in the dataset. However, there are a number of ways to optimise this process.

Exclude unwanted file types from indexing. See Suppressing document content indexing in Content Manager Enterprise Studio Help.
If there are specific Record Types in your dataset that do not need to have the content of their electronic documents indexed, ensure that you have suppressed document content indexing in the options for the Record Type. In Content Manager, see Content Manager Help about the Record Type Electronic tab.
In Content Manager Enterprise Studio, ensure that you have set the maximum size for archive and standard files in the Content Index properties dialogue box.
You can spread the indexing load across multiple Workgroup Servers. It is now possible to process document content from more than one Workgroup Server and write to the same content database.
Redundant Workgroup Servers can be configured to process chunks of data at the same time. The Workgroup Servers would be configured separately and not be part of the same deployment. They would be set up to be identical and configured to use the same Content Index Server. You could use virtual machines for this purpose, with enough disk space to handle the text extraction process and processing of archive files.
The process uses minimal resources to access the database and document stores and should not have an undue impact on production performance.
Each Workgroup Server would run as many text extraction processes as it has CPU cores available. Content Manager Content Index includes a deduplication function so it is not possible to create duplicates for the same document. However, using the document content indexing options above, it is possible to set up each Workgroup Server to process a unique set of records by either using the date fields to specify a range or a URI range.
Below is an example Oracle script that could be used to divide the data up into three groups of URIs based on the size of the electronic document stores. This would ensure that they share the load evenly. This script could be modified for SQL Server or to break up the data into the number of groups required.
```
Set serveroutput on;
```
```
DECLARE
```
```
CurrentValue NUMBER := 0;
```
```
RunningSum NUMBER := 0;
```
```
MaxLimit NUMBER := 0;
```
```
MaxUri NUMBER := 0;
```
```
LowerUri NUMBER := 0;
```
```
UpperUri NUMBER := 0;
```
```
CurrentUri NUMBER := 0;
```
```
bSet BOOLEAN;
```
```
BEGIN
```
```
SELECT Round(Sum(reBytes)/3)
```
```
INTO MaxLimit
```
```
FROM TSRECELEC;
```
```
SELECT Max(uri)
```
```
INTO MaxUri
```
```
FROM TSRECELEC;
```
```
FOR i IN 1..3
```
```
LOOP
```
```
RunningSum := 0;
```
```
bSet := TRUE;
```
```
LowerUri := 0;
```
```
UpperUri := 0;
```
```
LOOP
```
```
SELECT uri, reBytes
```
```
INTO UpperUri, CurrentValue
```
```
FROM ( SELECT * FROM TSRECELEC
```
```
WHERE uri > CurrentUri
```
```
ORDER BY uri)
```
```
WHERE ROWNUM <= 1;
```
```
IF bSet THEN
```
```
LowerUri := UpperUri;
```
```
bSet := FALSE;
```
```
END IF;
```
```
RunningSum := RunningSum + CurrentValue;
```
```
CurrentUri := UpperUri;
```
```
IF CurrentUri = MaxUri THEN
```
```
EXIT;
```
```
ELSIF RunningSum < MaxLimit THEN
```
```
CONTINUE;
```
```
ELSE
```
```
EXIT;
```
```
END IF;
```
```
END LOOP;
```
```
DBMS_OUTPUT.PUT_LINE ('WGS ' || TO_CHAR(i) ||' Min Uri = ' || TO_CHAR(LowerUri));
```
```
DBMS_OUTPUT.PUT_LINE ('WGS ' || TO_CHAR(i) ||' Max Uri = ' || TO_CHAR(UpperUri));
```
```
END LOOP;
```
```
END;
```
When employing this approach, it is important to ensure that the Content Index Server has been correctly sized to handle the store content.