Knowledge Discovery 25.1
Knowledge Discovery Expert

# Deduplicate Indexed Content

The `DREDUPLICATE` index action performs duplicate detection on a set of documents in the index. While the `KillDuplicates` parameter in the `DREADD` index action detects duplicates during the index process, `DREDUPLICATE` acts on sets of documents after they have been indexed. In addition, while the `KillDuplicates` parameter primarily prevents duplicate documents from being indexed, `DREDUPLICATE` can delete duplicate documents, tag them with a prescribed value, or move them to a different database.

All `DREDUPLICATE` index actions must include a `ReferenceField` parameter. The process detects duplicates when two documents have the same value in this field. You can also add secondary filters to compare the values of an additional field.

You might check whether the content of a document has changed when it is seen for the second time.
In this case the `ReferenceField` might be the document URL, and the secondary filter might be the document content field.

The Content component processes the documents in increasing `docid` order. It keeps the latest version of a set of duplicates, and it moves, deletes, or tags the matching documents with a lower `docid` . Therefore, for any set of duplicates, only the version of the document that was indexed last does not get deleted, moved, or tagged.

TIP: If you are using the wizard on the Index tab on the Console page in the Control section of Knowledge Discovery Admin to submit data for Content to index, you can specify how Content should treat duplicate documents in the Kill Duplicates page of the wizard..

## Example Usage

The following example moves all duplicate documents in the `News` and `Blog` databases to the `Duplicates` database:

    DREDUPLICATE?ReferenceField=DREREFERENCE&DuplicateAction=database&Database=Duplicates&DatabaseMatch=News+Blog

The following example tags all duplicate documents in the `News` database by setting a field `DuplicateFound` with the value `1`:

    DREDUPLICATE?ReferenceField=DREREFERENCE&DuplicateAction=tag&TagField=DuplicateFound&TagValue=1&DatabaseMatch=News

Related Topics

- Remove Duplicates

Copyright 2024 Open Text
Version 25.1 | Last updated January 2025

Send documentation feedback

Close

Send documentation feedback

To send feedback using your default email client, open an email window.

Otherwise, send your feedback and the information below to MFI-swpdl.idoldocsfeedback@opentext.com.

We appreciate your feedback!

Product: Knowledge Discovery 25.1

Topic Title: Deduplicate Indexed Content