Open topic with navigation

# Deduplication Options—KillDuplicates

Use the following parameters to specify deduplication options. The IDOL Content component uses these parameters to determine whether documents match.

- The `KillDuplicates` parameter specified in either the `[Server]` section of the IDOL Content component configuration file or in the `DREADD` or `DREADDDATA` index action.

- The `KillDuplicates` parameter specified in the `[Indexing]` section of the CFS configuration file.

The following options are available for the deduplication parameters.

`NONE` | Allows duplicate documents in Content index. Content does not replace or delete documents.
`REFERENCE` | Replaces an existing document with the new document if the document to index has the same value in its `DREREFERENCE` field.
`REFERENCEMATCHN` | Replaces the existing document with the new document if the content of the document to index is more than `N` percent similar to the existing document. Content determines the similarity by comparing the content of the `SourceType` fields in the document, or the `Index` fields if no `SourceType` fields are configured. NOTE: This method can deduplicate only documents that are already synced in the IDOL Server index. It cannot deduplicate similar documents in the same index job.
`FieldName` | Replaces the existing document with the new document if the document to index contains a `ReferenceType` field named `FieldName` that has the same content as the `FieldName` field in the existing document. You can specify multiple `ReferenceType` fields in this option (separated by a plus symbol or space), in which case Content deletes documents that contain any of the specified fields with identical content. You must percent-encode any punctuation characters in the field name. NOTE: You identify fields as `ReferenceType` fields through field processes in the IDOL Content component configuration file. If you list multiple fields in the same `PropertyFieldCSVs` parameter where you list the `FieldName` for deduplication, Content uses all the fields to eliminate duplicate documents. If you want to define multiple `ReferenceType` fields but do not want to use all fields for duplicate elimination, set up multiple field processes.
`ReferenceField,GREATER:VersionField` | Replaces the existing document with the new document if the document to index contains a `ReferenceType` field named `ReferenceField` that has the same content as the `ReferenceField` field in the existing document, and if the `VersionField` field in the document to index has a higher value than the `VersionField` in the existing document. For XML documents, you must fully qualify the path of the XML field that you want to use as the version field (you cannot use wildcard values). `VersionField` must contain a positive integer value, but you do not need to configure it as a numeric field. If only one of the incoming and current documents has a valid value in the `VersionField`, IDOL Server keeps the version with a valid `VersionField`. When both documents have the same `VersionField`, IDOL Server keeps the existing document. NOTE: When you index IDX documents, for the version comparison to work correctly, the value in the field that you use as the `VersionField` must be listed in quotation marks (`""`). That is, the field must have the following format in the IDX:
    #DREFIELD MyField="N"
IDOL Server treats existing documents with a missing or non-numeric value in the `VersionField` as having a version number of negative infinity. It treats a new document with a missing or non-numeric value in the `VersionField` as having a version number of 0.
`NOOP` (`DREADDDATA` only) | Use the `KillDuplicates` parameter in the `[Server]` section of the IDOL Content component configuration file to determine how to treat duplicate documents. NOTE: This option is available only for the `DREADDDATA` action.

Related Topics

- ReferenceType Fields

- Enable Deduplication for all Index Jobs

When you specify a deduplication option, note that:

- If you postfix any of these options with `=2`, Content applies the `KillDuplicates` process to all databases, rather than just the database into which the current IDX or XML file indexes. For example:

      KillDuplicates=REFERENCE=2

- The setting in the `KillDuplicates` option in either the `DREADD` or `DREADDDATA` index action overrides the setting in the `KillDuplicates` configuration parameter.

© 2017 Hewlett Packard Enterprise Development LP | Send documentation feedback to HPE Close Send documentation feedback To open the configured email client on this computer, open an email window. Otherwise, copy the information below to a web mail client, and send this email to AutonomyTPFeedback@hpe.com. We appreciate your feedback! Product: Topic Title: Help Topic: Feedback:

_HP_HTML5_bannerTitle.htm