NiFi Ingest 24.4
NiFi Ingest

# Advanced Connector Properties

IDOL NiFi Ingest Connectors support some properties which you can use to customize the behavior of the processor. These properties are supported by all connectors.

Property Name | Description | Default
--- | --- | ---
`adv:AllowedClusterNodesRegex` | A regular expression that matches the host names of the NiFi instances to run the connector on (if you have a cluster of NiFi instances). | Run on all nodes of the cluster.
`adv:ConnectionValidation` | A Boolean value that specifies whether the connector validates its connection to the repository using any connection details or credentials you have provided. When this feature is enabled the connector provides a warning if the connection is not successful. | True
`adv:EnableDebugStatistics` | A Boolean value that specifies whether to collect additional statistics about the number and frequency of API calls that are being sent to the repository. Some connectors can store these statistics, to help you monitor the requests being made. Some connectors do not store any additional statistics. See also StatisticsIntervals. | False
`adv:FlowFileEmbedFiles` | A Boolean value that specifies whether to include binary file content in FlowFiles, rather than including only file paths. For example, if you set this property to `TRUE` for a File System Connector, a FlowFile created by the connector can include a file that was retrieved from the file system. Setting this property to `TRUE` can reduce performance and is not necessary if you are using a single NiFi instance. If you run a NiFi cluster, the files retrieved by a connector must be available to all of the nodes in the cluster. You can either configure an ingest shared path (by setting the property `adv:IngestSharedPath`), or embed the files in the FlowFiles (by setting this property to `TRUE`). | False
`adv:IngestBatchSize` | The number of documents to create before releasing a batch of documents to the outgoing queue. This property only applies to processors, such as GetFileSystem, that perform the synchronize action and have names beginning with "Get". | 100
`adv:IngestSharedPath` | The directory in which the connector saves files that were downloaded from the data repository. You can specify the same directory for multiple connectors, because each processor creates a subfolder within the specified path. If you run a NiFi cluster, the files retrieved by a connector must be available to all of the nodes in the cluster. You can either configure an ingest shared path (by setting this property to a directory that is accessible from every node in the cluster), or embed the files in the FlowFiles (by setting `adv:FlowFileEmbedFiles`). |
`adv:PurgeDatastore` | A Boolean value that specifies whether to purge (clear) the datastore after each scheduled synchronize cycle. By default, a connector stores information about the items it has retrieved from a repository so that it does not need to retrieve them again unless they are modified. If you set this property to `TRUE` the connector performs a full synchronize on every scheduled synchronize cycle. NOTE: If you set this property to `TRUE`, the connector cannot issue delete commands for items that were previously synchronized but have been deleted from the source repository. This property only applies to processors, such as GetFileSystem, that perform the synchronize action and have names beginning with "Get". | False
`adv:StatisticsIntervals` | The time intervals for which the connector returns statistics. The connector stores statistics, including the number of ingest-adds, ingest-updates, and ingest-deletes that have occurred. You can view these statistics through the advanced configuration UI (see View Connector Statistics). You can set this parameter to a comma-separated list of one or more of the following values (in seconds): 10,60,300,1800,3600,10800,86400,604800 See also EnableDebugStatistics. | The connector returns statistics for the last 10 seconds, 1 minute, 5 minutes, 30 minutes, 3 hours, 1 day, and 1 week.
`adv:TaskMaxDuration` | The maximum amount of time that the processor should run for. If the time limit is reached the connector stops until the next scheduled task begins. Specify the duration with time units, for example 2 hours or 30 min. |
`adv:Threads` | The number of threads to use. By default a connector uses up to five threads for synchronization. The connector might be able to synchronize with the repository in less time if you increase the value, but this places additional load on the repository because it must respond to more frequent requests. This property applies to all types of connector processors. | 5
`ATTR:name` | Adds a FlowFile attribute, with the specified `name`, to every FlowFile produced by the connector. |
`META:fieldname` | Adds a metadata field to every document produced by the connector. The field has the name `fieldname` and the value of the property. For example, to add a field named `SecurityType` to every document, set a property named `META:SecurityType`. |

To set an advanced property

1. Stop the connector, if it is running. To do this, right-click the processor and click Stop.

2. Right-click the processor and click Configure.

   The Configure Processor dialog box opens.

3. Click the Properties tab.

4. Click Add .

   The Add Property dialog box opens.

5. Type the name of the property that you want to add and click OK.
6. Set a value for the property.
7. Click APPLY.

To remove an advanced property, find it in the properties list and click the delete icon .

Copyright 2018-2024 Open Text
Version 24.4 | Last updated September 2024

Send documentation feedback

Close

Send documentation feedback

To send feedback using your default email client, open an email window.

Otherwise, send your feedback and the information below to MFI-swpdl.idoldocsfeedback@opentext.com.

We appreciate your feedback!

Product: IDOL NiFi Ingest 24.4

Topic Title: Advanced Connector Properties