IDOL Category Component 12.13

# Use KillDuplicates and Combine on ReferenceType Fields

When you instruct the IDOL Content component to eliminate duplicate document copies at index time using a specific `ReferenceType` field (by setting the `KillDuplicates` parameter in the IDOL Content component configuration file), it automatically uses any field listed for `PropertyFieldCSVs` alongside this `ReferenceType` field in the IDOL Content component configuration to eliminate duplicate document copies as well.

However, Content cannot use the same field for deduplication as for the `Combine` action parameter, because the `Combine` operation clashes (carried out at query time) with Content eliminating duplicate fields. This clash means that, if you want to eliminate duplicate document copies and use the `Combine` action parameter, you must set up separate `ReferenceType` fields for these processes.

Related Topics

- Prevent Duplicate Documents

To use KillDuplicates and Combine on ReferenceType fields

1. Open the IDOL Content component configuration file in a text editor.

2. In the `[FieldProcessing]` section, add two processes that identify `ReferenceType` fields (note that you must set up a field process to identify `ReferenceType` fields before you start indexing documents into Content). One of them is used to eliminate duplicate copies of documents, and the other one is used for the `Combine` operation.

   For example:

       [FieldProcessing]
       0=MyFirstProcess
       1=MySecondProcess
       2=SetUpReferenceFields
       3=SetUpMoreReferenceFields

3. Create a section for the processes that you added, and in each section, create a property for the respective process (you define the property later by setting one or more applicable configuration parameters). Identify the fields that you want to associate with each process.

   NOTE: The properties that you create must not have the same name as the processes.

   For example:

       [MyFirstProcess]
       Property=MyFirstProperty
       PropertyFieldCSVs=*/MyField,*/MySecondField

       [MySecondProcess]
       Property=MySecondProperty
       PropertyFieldCSVs=*/MyThirdField

       [SetUpReferenceFields]
       Property=ReferenceFields
       PropertyFieldCSVs=*/DREREFERENCE,*/URL

       [SetUpMoreReferenceFields]
       Property=MoreReferenceFields
       PropertyFieldCSVs=*/DRETITLE

4. Create a section for each of the properties and specify appropriate configuration settings for each. These configuration parameters define the processes that are applied to all the fields (or all documents that contain the fields) that you previously associated with the processes. For example:

       [MyFirstProperty]
       HiddenType=True

       [MySecondProperty]
       Index=True

       [ReferenceFields]
       ReferenceType=True
       TrimSpaces=True

       [MoreReferenceFields]
       ReferenceType=True
       TrimSpaces=True

5. Save and close the configuration file.

6. Restart the IDOL Content component for your changes to take effect.

After you index documents into Content, you can use, for example, the `*/DREREFERENCE` field to eliminate duplicate copies of documents. (Content then automatically also uses the `*/URL` field for deduplication because it is listed alongside `*/DREREFERENCE` for `PropertyFieldCSVs`.) This leaves you free to use the `*/DRETITLE` field for the `Combine` operation.

© Copyright 2022 Micro Focus or one of its affiliates
Version 12.13 | Last updated October 2022

Send documentation feedback

Close

Send documentation feedback

To send feedback using your default email client, open an email window.

Otherwise, send your feedback and the information below to swpdl.idoldocsfeedback@microfocus.com.

We appreciate your feedback!

Product: IDOL IDOL Category Component 12.13

Topic Title: Use KillDuplicates and Combine on ReferenceType Fields