IDOL Content Component 23.2

# Deduplication Options—KillDuplicates

Use the following parameters to specify deduplication options. The IDOL Content component uses these parameters to determine whether documents match.

- The KillDuplicates parameter specified in either the `[Server]` section of the IDOL Content component configuration file or in the DREADD or DREADDDATA index action.

- The `KillDuplicates` parameter specified in the `[Indexing]` section of the CFS configuration file.

The following options are available for the deduplication parameters.

`NONE` | Allows duplicate documents in Content index. Content does not replace or delete documents.
`REFERENCE` | Replaces an existing document with the new document if the document to index has the same value in its `DREREFERENCE` field.
`REFERENCEMATCHN` | Replaces the existing document with the new document if the content of the document to index is more than `N` percent similar to the existing document. Content determines the similarity by comparing the content of the SourceType fields in the document, or the Index fields if no `SourceType` fields are configured. NOTE: This method can deduplicate only documents that are already synced in the IDOL Server index. It cannot deduplicate similar documents in the same index job.
`FieldName` | Replaces the existing document with the new document if the document to index contains a ReferenceType field named `FieldName` that has the same content as the `FieldName` field in the existing document. You can specify multiple `ReferenceType` fields in this option (separated by a plus symbol or space), in which case Content deletes documents that contain any of the specified fields with identical content. You must percent-encode any punctuation characters in the field name. NOTE: You identify fields as `ReferenceType` fields through field processes in the IDOL Content component configuration file. If you list multiple fields in the same PropertyFieldCSVs parameter where you list the `FieldName` for deduplication, Content uses all the fields to eliminate duplicate documents. If you want to define multiple `ReferenceType` fields but do not want to use all fields for duplicate elimination, set up multiple field processes.
`ReferenceField,GREATER:VersionField` | Replaces the existing document with the new document if the document to index contains a `ReferenceType` field named `ReferenceField` that has the same content as the `ReferenceField` field in the existing document, and if the `VersionField` field in the document to index has a higher value than the `VersionField` in the existing document. For XML documents, you must fully qualify the path of the XML field that you want to use as the version field (you cannot use wildcard values). `VersionField` must contain a positive integer value, but you do not need to configure it as a numeric field. If only one of the incoming and current documents has a valid value in the `VersionField`, IDOL Server keeps the version with a valid `VersionField`. When both documents have the same `VersionField`, IDOL Server keeps the existing document. NOTE: When you index IDX documents, for the version comparison to work correctly, the value in the field that you use as the `VersionField` must be listed in quotation marks (`""`). That is, the field must have the following format in the IDX:
    #DREFIELD MyField="N"
IDOL Server treats existing documents with a missing or non-numeric value in the `VersionField` as having a version number of negative infinity. It treats a new document with a missing or non-numeric value in the `VersionField` as having a version number of 0.
`NOOP` (DREADDDATA only) | Use the KillDuplicates parameter in the `[Server]` section of the IDOL Content component configuration file to determine how to treat duplicate documents. NOTE: This option is available only for the DREADDDATA action.

Related Topics

- ReferenceType Fields

- Enable Deduplication for all Index Jobs

When you specify a deduplication option, note that:

- If you postfix any of these options with `=2`, Content applies the `KillDuplicates` process to all databases, rather than just the database into which the current IDX or XML file indexes. For example:

      KillDuplicates=REFERENCE=2

- The setting in the KillDuplicates option in either the DREADD or DREADDDATA index action overrides the setting in the KillDuplicates configuration parameter.

Copyright 2023 Open Text
Version 23.2 | Last updated March 2023

Send documentation feedback

Close

Send documentation feedback

To send feedback using your default email client, open an email window.

Otherwise, send your feedback and the information below to swpdl.idoldocsfeedback@microfocus.com.

We appreciate your feedback!

Product: IDOL IDOL Content Component 23.2

Topic Title: Deduplication Options—KillDuplicates