CategorizeDocument

Provided in: idol-nifi-core-nar (see Install NiFi Ingest).

The CategorizeDocument processor makes requests (action=CategorySuggestFromText and action=BinaryCatQuery) to a Category component, to categorize incoming documents based on their text content.

For each matching category, the processor writes a new field to the document metadata:

<Category>
    <Name>...</Name>
    <Weight>...</Weight>
</Category>

The processor has an advanced configuration interface that you can use to:

• list, add, and modify binary categories
• list, add, and modify categories, and create taxonomies (nested sets of categories)
• categorize text, for testing purposes

For more information about categorization, refer to the Knowledge Discovery Administration Guide.

Properties

Name	Default Value	Description
IDOL License Service		An IdolLicenseServiceImpl that provides a way to communicate with a Knowledge Discovery License Server.
Category Host		The host name or IP address of the Category component.
Category Port		The Category component ACI port.
Request Timeout	60	The maximum amount of time to wait, in seconds, for a response from the Category component.
Minimum Weight		The minimum threshold that must be met for a document to be categorized (for the category to be added to the document metadata).
SSL Config Service		An optional IdolSSLConfigServiceImpl that specifies the settings to use to communicate with the Category component over SSL/TLS. Set this property if your Category component has been configured to accept connections over SSL.
Text Source	Content	The source to use for categorization. This property can take one of the following values:  Content. The processor uses the content text to query categories. It queries each content part separately. Metadata. The processor uses text values from fields to query categories. You define the metadata to use by setting Metadata Source XPath to the appropriate XPath expression. The processor queries each value separately, and adds all matched categories to the document. Document. The processor writes document metadata and content to XML and sends the whole document in the query, with the TextParse option set to True. When you use this option, you can optionally set the AgentBoolean option to match with Boolean restrictions in your categories.
Taxonomy Name		The name of the taxonomy to use for categorization. TIP: You can create taxonomies (nested sets of categories) in the "Categories" tab of the advanced configuration interface.
Binary Categories		A list of binary categories to query, or * to test incoming documents against all binary categories. TIP: You can view and train binary categories in the "Binary Categories" tab of the advanced configuration interface.
Metadata Source XPath		This option is required when you set Text Source to Metadata. It is not available for other text sources. Set this option to an XPath expression that defines the metadata fields that you want to extract source text from.
AgentBoolean	False	This option is available only when you set Text Source to Document. Whether to use Boolean restrictions on categories to check whether the categories match the query document also match the Boolean restrictions that you have set for a category.

Relationships

Name	Description
success	FlowFiles that were processed successfully.
failure	FlowFiles that were not processed successfully.

Advanced Configuration

The CategorizeDocument processor has an advanced configuration interface, which provides guided setup for configuration. It also has tabs that allow you to configure binary categories, categories and taxonomies, and to test the categorization process.

Binary Categories

A binary category is a special kind of category, designed to answer yes/no questions about documents, files, and text, such as Is this document spam?, Does this violate company policy?, or Is this work-related?.

When you train a binary category, you provide positive and negative training. Positive training provides examples of text that meets the 'yes' criteria for the category, while negative training provides examples that meet the 'no' criteria.

In the advanced configuration interface, the Binary Categories page allows you to create and train a binary category.

To add and train a new binary category

1. On the Binary Categories tab, click add .

2. In the Name box, type the name for the new binary category. Click Create.

3. Find the new category in the list, and click edit .

4. In the Positive Training box, add text that represents the documents that you want the binary category to match with a positive response. For example, for a spam category, add examples of spam documents.

5. In the Negative Training box, add text that represents the documents that you want the binary category to match with a negative response. For example, for a spam category, add examples of legitimate documents.

6. If the binary category contains existing training, you can select Replace Existing to delete the existing training and replace it with the new training. By default, the new training is added to the existing training.

7. Click Train.

Categories

Categories allow you to automatically classify documents into different groupings, based on training that you provide. Unlike binary categories, documents can belong to multiple categories, or none. When the processor matches a document to a particular category, it adds the category to the document metadata, and you can use that information in onward processing.

A taxonomy is a nested set of categories. For example, you might have a top level category sports, and subcategories for different individual sports, such as football, tennis, and hockey. You provide training text and Boolean expressions for each category.

On the categories tab, the left panel displays the taxonomy of categories, while the right panel displays the details of a selected category.

To add a new taxonomy

1. At the bottom of the list of categories click add . The New Category dialog box opens.

2. In the Name box, type the name for the new taxonomy.

3. Select Taxonomy Root to make the new category the root of the new taxonomy.

4. Click Create.

To add a new child category

1. In the list of categories, select the parent category to which you want to add a category. The details for the category are displayed in the right panel.

2. In the details panel, click add .

   The New Category dialog box opens. The description includes the name of the parent category that you selected.

3. In the Name box, type the name for the new child category.

4. Click Create.

To train a category

1. Find and select the category that you want to train in the taxonomy. The details for the category are displayed in the right panel.

2. In the details panel, click edit . The Edit Category dialog box opens.

3. Add training for the category. You can add the following types of training: 

   • Boolean Expression. A Boolean expression that describes the text that you want to match. For example sports OR games. For more information about Boolean categories, refer to the Knowledge Discovery Administration Guide.

     TIP: Boolean training is optional. To match training that uses Boolean expressions, you must configure the processor with Text Source set to Document, and AgentBoolean set to True.

   • Training Text. Plain text that represents the kind of text that you want the category to match. You can provide paragraphs of text from a document that would match the category. The Category component selects the best terms from your training text to use as the category matching values, so the more text you provide the better.

4. Click Train.

Tester

The Tester tab allows you to submit text and see what categories it matches. By default, the test matches against all categories in your system, including binary categories if you have set the Binary Categories configuration option. You can optionally restrict the test to a single taxonomy, which categorizes the text you provide against only categories in that taxonomy.

To test categorization

1. In the Text to be categorized box, type the text you want to categorize.

2. Optionally, in the Taxonomy Name box, type the name of a taxonomy.

3. Click Test.

The processor runs categorization on the text, and returns a list of matching categories.