NiFi Ingest 25.1
NiFi Ingest

# Create the Data Flow

This section describes how to build a dataflow to perform connector actions in response to HTTP requests.

A NiFi Ingest dataflow that listens for requests and performs multiple actions can be complex, so OpenText recommends building your dataflow from the following templates that are provided with NiFi Ingest:

- The Connector ACI Server template includes the components necessary to listen for, and respond to, HTTP requests. This template also includes a ConnectorGroupRouter processor, which is designed to route requests to a suitable connector group. For example, a request to fetch a web page must go to a Web Connector group, and a request to view an e-mail message in Exchange must be routed to an Exchange Connector group.
- A Connector group template for your chosen connector, for example a "File System Connector group" template or a "Web Connector group" template. A connector group contains all of the processors required to handle actions for one connector. For example, the File System Connector group includes a GetFileSystem processor to handle synchronize actions, a FetchFileSystem processor to handle collect actions, a ListFileSystem processor to handle identifiers actions, and so on. The group accepts input from the ConnectorGroupRouter and returns responses to it. It also has output relationships that receive FlowFiles, representing documents that were synchronized or collected, for example. These FlowFiles can then be routed to other NiFi processors.

To create a dataflow to handle connector actions

1. Add the components required to handle HTTP requests.

   1. Drag the template icon from the components toolbar to the canvas.

      The Add Template dialog box opens.

   2. In the list, select Connector ACI Server and click ADD.

      The components necessary to handle requests are added to the canvas.

      TIP: The Connector ACI group and the ConnectorGroupRouter added by this template are designed to handle requests for any number of connectors. In a later step you will configure the ConnectorGroupRouter to route requests to the correct connector.

2. Add a connector group. This is a group of processors that can handle all of the supported actions for a specific connector.

   1. Right-click the ConnectorGroupRouter processor, and click Configure.

   2. Click ADVANCED.

      A dialog box opens, showing every Connector that you have installed, along with the actions that it supports.

   3. (Optional) If you do not need to handle every action you can exclude the ones that you do not need. Click the check mark in the relevant column, so that it turns to a cross.

   4. Click INSTALL next to the relevant connector group. This example uses a File System Connector, so click INSTALL for the FileSystem Connector Group.

      The Template Name dialog box opens.

   5. Type a name for the template and click INSTALL.

      A template is installed that contains the components required to handle the selected actions.

   6. Close the dialog boxes and return to the canvas.

   7. Drag the template icon from the components toolbar to the canvas.

      The Add Template dialog box opens.

   8. In the list, select FileSystem Connector Group and click ADD.

      The components are added to the canvas.

3. Configure how to handle requests.

   1. Right-click the Connector ACI group and click Variables.

      The Variables dialog box opens.

   2. (Optional) You can choose the port to listen on by setting the variable connector.aci.port. The variable is set by default to use port 10000.
   3. Click APPLY.

   4. Right-click the Connector ACI group and click Configure.

      The Connector ACI Configuration dialog box opens.

   5. Click the Controller Services tab.

      An ACI Context Service already exists.

   6. (Optional) You can configure this service to customize how actions are handled.

   7. Enable the ACI Context Service (click Enable ).

      The Enable Controller Service dialog box opens.

   8. Click ENABLE, then CLOSE.
   9. Close the Connector ACI Configuration dialog box and return to the canvas.

4. Configure the ConnectorGroupRouter to route requests to your FileSystem Connector Group, and route responses back.

   1. Right-click the ConnectorGroupRouter and click Configure.
   2. Click the Properties tab.

   3. Click to add a new dynamic property.

      The Add Property dialog box opens.

   4. In the Property Name box, type `FileSystemConnectorGroup`. Then, click OK.

      The name that you choose becomes the name of an output relationship on the ConnectorGroupRouter processor. You could also create relationships for other connector groups, by creating properties such as `WebConnectorGroup` or `ExchangeConnectorGroup`.

   5. Set the value of the property to the name of the connector group, for example `FileSystem`.
   6. Click APPLY.

   7. Create a connection from the ConnectorGroupRouter processor to the FileSystem Connector Group.

      1. Hover the mouse pointer over the ConnectorGroupRouter until you see the connection icon - - and then drag the icon to the FileSystem Connector Group.

         The Create Connection dialog box opens.

      2. In the For Relationships area, select the FileSystemConnectorGroup relationship that you created and click ADD.

   8. Create a connection from the FileSystem Connector Group back to the ConnectorGroupRouter.

      1. Hover the mouse pointer over the FileSystem Connector Group until you see the connection icon - - and then drag the icon to the ConnectorGroupRouter.

         The Create Connection dialog box opens.

      2. In the From Output list, select the Response output and click ADD.

5. Configure the FileSystem Connector Group.

   1. Right-click the FileSystem Connector Group and click Variables.
   2. Set the license.server.hostname variable to the host name or IP address of your License Server, and click APPLY followed by CLOSE.

   3. Right-click the FileSystem Connector Group and click Configure.

      The FileSystem Connector Group Configuration dialog box opens.

   4. Click the Controller Services tab.

      An IDOL License Service already exists.

   5. Configure the IDOL License Service as necessary (for example, you might need to specify SSL settings if your License Server only accepts requests over SSL). Then click Enable and enable the service.

6. Open the FileSystem Connector Group and configure the connector processors as necessary.

   1. Double-click the FileSystem Connector Group (or right-click and choose Enter group).
   2. Configure the connector processors as necessary. For example, in a FileSystem Connector Group, you must set the Directory Paths property on the GetFileSystem processor, to set the starting point for the synchronize action. The other processors do not require any additional configuration.

   3. Configure the `Get*` processor, for example GetFileSystem in a File System Connector group, to add a field to each document, containing the name of the connector group. This is necessary so that front-end applications can determine which connector group retrieved a document in the index, and send subsequent actions (such as `View`) to the correct group. The name of the field must match the value of the parameter `ConnectorGroupField`, in your View Server configuration file. For example, you could add a field named `AUTN_GROUP`.

      1. Right-click the processor and click Configure.
      2. Click the Properties tab.
      3. Click to add a new dynamic property.
      4. Specify the property name as `META:AUTN_GROUP` and the value as `${idol.connector.group}`.
      5. Click APPLY.
   4. To return to the main dataflow, right-click a blank area of the canvas and click Leave group.

7. Route the output from the FileSystem Connector Group into your ingestion pipeline.

   1. Hover the mouse pointer over the FileSystem Connector Group until you see the connection icon - - and then drag the icon to the first processor in your ingest pipeline. In the image of the dataflow, at the beginning of this topic, an output port is used as a placeholder for the ingest pipeline.

      The Create Connection dialog box opens.

   2. In the From Output list, select Synchronize Output and click ADD.

   Depending on the type of connector that you are using, you can also create connections for other types of output, such as "Collect Output" or "Identifiers Output".

8. Start all of the processors in the dataflow. (Go to the Operate palette and click Start ).

   You can now begin sending actions (see Send Actions to NiFi).

Copyright 2018-2024 Open Text
Version 25.1 | Last updated January 2025

Send documentation feedback

Close

Send documentation feedback

To send feedback using your default email client, open an email window.

Otherwise, send your feedback and the information below to MFI-swpdl.idoldocsfeedback@opentext.com.

We appreciate your feedback!

Product: Knowledge Discovery NiFi Ingest 25.1

Topic Title: Create the Data Flow