NiFi Ingest 25.1
NiFi Ingest

# PutKafka

A processor that accepts document FlowFiles, and sends the data to Apache Kafka.

## Properties

Name | Default Value | Description
--- | --- | ---
IDOL License Service |  | An IdolLicenseServiceImpl that provides a way to communicate with a Knowledge Discovery License Server.
Document Registry Service |  | A DocumentRegistryServiceImpl that manages and updates a document registry database.
Indexer Batch Size | 100 | The maximum number of FlowFiles to accumulate before indexing documents. Indexing documents in larger batches can be more efficient; indexing smaller batches more frequently means that an individual document might spend less time in the queue. Documents might be indexed in incomplete batches if the Indexing Interval Duration is reached first.
Indexing Interval Duration | 5 min | The maximum amount of time to wait between indexing batches of documents. Documents might be indexed more frequently than this if the Indexer Batch Size is reached first. Waiting longer and indexing documents in larger batches can be more efficient; indexing smaller batches more frequently means that an individual document might spend less time in the queue.
Brokers |  | A comma-separated list of Kafka brokers to send data to. Specify each broker in the format `host:port`.
Topic |  | The name of the topic to append messages to.
Partition |  | The partition number to use for distribution of messages. The default is to use a round-robin method of distribution.
Producer Option Names |  | You can use these properties to configure producer options. The options that you can set are described in the documentation for the librdkafka library. If you want to set these properties, OpenText recommends using the guided setup wizard (click ADVANCED and follow the instructions on the GUIDED SETUP tab).
Producer Option Values |
Key Field Path | If not set, the processor uses the document reference. | The path of the field in the document metadata to use as the Kafka message key. You can select a sub-field by using slash-separated field names such as `FieldName/SubFieldName`.
Value Field Path | If not set, the processor uses the document content. | The path of the field in the document metadata to use as the Kafka message body. You can select a sub-field by using slash-separated field names such as `FieldName/SubFieldName`.
Timestamp Field Path | DREDATE | The path of the field in the document metadata to use as the Kafka message timestamp. You can select a sub-field by using slash-separated field names such as `FieldName/SubFieldName`.
Timestamp Field Format | EPOCHSECONDS | The date format used in the field specified by "Timestamp Field Path". Specify the format using a standard Knowledge Discovery date format string.
Metadata Mapping Mode | Full | Specifies how to convert document metadata into Kafka message headers: Full - Document metadata is converted into Kafka message headers. The header names are full XPath-like indexed field paths, for example `FIELD1[1]/@attr`. Simple - Document metadata is converted into Kafka message headers. The header names are generated without the indexes, for example `FIELD1_attr`. None - Document metadata is not converted by default. Use "Header Field Names" and "Header Field Paths" instead.
Header Field Names |  | You can use these properties to map document fields to Kafka message headers. If you want to set these properties, OpenText recommends using the guided setup wizard (click ADVANCED and follow the instructions on the GUIDED SETUP tab). Knowledge Discovery document metadata field paths are slash-separated and can reference attributes using the "@" character. For example, to reference an attribute called "name" on a field named "MyField", enter `MyField/@name`. The header name can contain placeholders, `$1`, `$2`, ... `$N` that correspond to the Nth asterisk wildcard (`*`) in the corresponding document metadata field path. An asterisk wildcard (`*`) that is used in a path can match only a whole field name or attribute name. Example header names and field paths
Header name | Field path | Description
--- | --- | ---
`myfield` | `myfield` | The value of the field "myfield" is written to the "myfield" header.
`$1` | `*` | All fields at the root of the document are added as headers with matching names.
`myfield_$1_$2` | `myfield/*/@*` | All of the attributes of subfields of "myfield" are added as headers. For example, if "myfield" has a sub-field named "subfield" with an attribute "color", then the processor creates a header named "myfield_subfield_color".
Header Field Paths |

## Relationships

Name | Description
--- | ---
success | FlowFiles that were indexed successfully.
failure | FlowFiles that were not indexed successfully.

Copyright 2018-2024 Open Text
Version 25.1 | Last updated January 2025

Send documentation feedback

Close

Send documentation feedback

To send feedback using your default email client, open an email window.

Otherwise, send your feedback and the information below to MFI-swpdl.idoldocsfeedback@opentext.com.

We appreciate your feedback!

Product: Knowledge Discovery NiFi Ingest 25.1

Topic Title: PutKafka