NiFi Ingest 25.2
NiFi Ingest

# Create the Data Flow

This section describes how to build a dataflow to perform connector actions in response to HTTP requests.

A NiFi Ingest dataflow that listens for requests and performs multiple actions can be complex, so OpenText recommends building your dataflow from the following templates that are provided with NiFi Ingest:

- The Connector ACI Server template includes the components necessary to listen for, and respond to, HTTP requests. This template also includes a ConnectorGroupRouter processor, which is designed to route requests to a suitable connector group. For example, a request to fetch a web page must go to a Web Connector group, and a request to view an e-mail message in Exchange must be routed to an Exchange Connector group.
- A Connector group template for your chosen connector, for example a "File System Connector group" template or a "Web Connector group" template. A connector group contains all of the processors required to handle actions for one connector. For example, the File System Connector group includes a GetFileSystem processor to handle synchronize actions, a FetchFileSystem processor to handle collect actions, a ListFileSystem processor to handle identifiers actions, and so on. The group accepts input from the ConnectorGroupRouter and returns responses to it. It also has output relationships that receive FlowFiles, representing documents that were synchronized or collected, for example. These FlowFiles can then be routed to other NiFi processors.

To create a dataflow to handle connector actions

1. If you haven't already done so, create an IDOL License Service in the root process group by following the instructions in Create a License Service.

2. Add the components required to handle HTTP requests.

   1. Drag the process group icon from the components toolbar to the canvas.

      The Create Process Group dialog box opens.

   2. Click Upload and select the template `Connector_ACI_Server.json`. This is available in the NiFi Connector Routing zip package.

   3. Click Add.

      A process group is added to the canvas.

   4. Double-click the process group.

      The process group opens. You should see that the components necessary to handle requests have already been added to the canvas.

      TIP: The Connector ACI group and the ConnectorGroupRouter added by this template are designed to handle requests for any number of connectors. In a later step you will configure the ConnectorGroupRouter to route requests to the correct connector.

3. Add a connector group. This is a group of processors that can handle all of the supported actions for a specific connector.

   1. Right-click the ConnectorGroupRouter processor, and click Advanced.

   2. The advanced configuration page opens, showing every connector that you have installed, along with the actions that it supports.

   3. (Optional) If you do not need to handle every action you can exclude the ones that you do not need. Click the check mark in the relevant column, so that it turns to a cross.

   4. Click DOWNLOAD next to the relevant connector group. This example uses a File System Connector, so click DOWNLOAD for the FileSystem Connector Group.

      The Flow Name dialog box opens.

   5. Type a name for the template and click DOWNLOAD. Save the template on your machine. This template contains the components required to handle the selected actions.

   6. Return to the canvas.

   7. Drag the process group icon from the components toolbar to the canvas.

      The Create Process Group dialog box opens.

   8. Upload the FileSystem Connector Group template that you downloaded and click ADD.

      The new process group is added to the canvas.

4. Configure how to handle requests.

   1. Right-click the Connector ACI process group and click Controller Services.

      The Controller Services page opens. By right-clicking a process group rather than a blank area of the canvas, the page displays Controller Services that exist in the root process group and in the selected process group. You can determine where a Controller Service was defined by looking at the value in the Scope column.

      For example, if you defined your IDOL License Service at the root level, the scope will be "NiFi Flow". You should also see an AciContextServiceImpl in the scope "Connector ACI".

   2. Enable the ACI Context Service (Click followed by Enable).

   3. Make a note of the Controller Service IDs for the ACI Context Service and for your IDOL License Service. To find the IDs:

      1. At the end of the row for the Controller Service, click followed by View Configuration. (If the controller service is in a different scope, first choose Go To and then View Configuration).

         The Controller Service Details dialog opens.

      2. Click the Settings tab and make a note of the controller service Id.

   4. Return to the canvas.

   5. Right-click the Connector ACI process group and click Parameters.

      The Edit Parameter Context dialog box opens.

   6. Edit the ACI Context Service parameter and set the value to the ID of the ACI Context Service.

   7. Edit the IDOL License Service parameter and set the value to the ID of your IDOL License Service.

   8. (Optional) You can choose the port to listen on by setting Listening Port. The default value is port 10000.
   9. Click Apply followed by Close and return to the canvas.

5. Configure the ConnectorGroupRouter to route requests to your FileSystem Connector Group, and route responses back.

   1. Right-click the ConnectorGroupRouter and click Configure.
   2. Click the Properties tab.

   3. Click to add a new dynamic property.

      The Add Property dialog box opens.

   4. In the Property Name box, type `FileSystemConnectorGroup`. Then, click OK.

      The name that you choose becomes the name of an output relationship on the ConnectorGroupRouter processor. You could also create relationships for other connector groups, by creating properties such as `WebConnectorGroup` or `ExchangeConnectorGroup`.

   5. Set the value of the property to the name of the connector group, for example `FileSystem`.
   6. Click Apply.

   7. Create a connection from the ConnectorGroupRouter processor to the FileSystem Connector Group.

      1. Hover the mouse pointer over the ConnectorGroupRouter until you see the connection icon - - and then drag the icon to the FileSystem Connector Group.

         The Create Connection dialog box opens.

      2. In the Relationships area, select the FileSystemConnectorGroup relationship that you created and click Add.

   8. Create a connection from the FileSystem Connector Group back to the ConnectorGroupRouter.

      1. Hover the mouse pointer over the FileSystem Connector Group until you see the connection icon - - and then drag the icon to the ConnectorGroupRouter.

         The Create Connection dialog box opens.

      2. In the From Output list, select the Response output and click ADD.

6. Configure the FileSystem Connector Group.

   1. Right-click the FileSystem Connector Group and click Parameters.
   2. Edit the IDOL License Service parameter, setting the value to the ID of your IDOL License Service.
   3. Click Apply followed by Close.

7. Open the FileSystem Connector Group and configure the connector processors as necessary.

   1. Double-click the FileSystem Connector Group (or right-click and choose Enter group).
   2. Configure the connector processors as necessary. For example, in a FileSystem Connector Group, you must set the Directory Paths property on the GetFileSystem processor, to set the starting point for the synchronize action. The other processors do not require any additional configuration.

   3. Configure the `Get*` processor, for example GetFileSystem in a File System Connector group, to add a field to each document, containing the name of the connector group. This is necessary so that front-end applications can determine which connector group retrieved a document in the index, and send subsequent actions (such as `View`) to the correct group. The name of the field must match the value of the parameter `ConnectorGroupField`, in your View Server configuration file. For example, you could add a field named `AUTN_GROUP`.

      1. Right-click the processor and click Configure.
      2. Click the Properties tab.
      3. Click to add a new dynamic property.
      4. Specify the property name as `META:AUTN_GROUP` and the value as `${idol.connector.group}`.
      5. Click APPLY.
   4. To return to the connector routing dataflow, right-click a blank area of the canvas and click Leave group.

8. Route the output from the FileSystem Connector Group into your ingestion pipeline.

   1. Add an output port to the canvas.

   2. Hover the mouse pointer over the FileSystem Connector Group until you see the connection icon - - and then drag the icon to the output port.

      The Create Connection dialog box opens.

   3. In the From Output list, select Synchronize Output and click Add.

      TIP: Depending on the type of connector that you are using, you can also create additional output ports and connections for other types of output, such as "Collect Output" or "Identifiers Output".

   4. Right-click a blank area of the canvas and click Leave Group.

      You return to the root, "NiFi Flow", process group.

   5. You can now make a connection between the Connector_ACI_Server process group that you have configured and the first processor in your ingestion pipeline.

9. Start all of the processors in the dataflow. (Go to the Operate palette and click Start ).

   You can now begin sending actions (see Send Actions to NiFi).

Copyright 2018-2025 Open Text
Version 25.2 | Last updated April 2025

Send documentation feedback

Close

Send documentation feedback

To send feedback using your default email client, open an email window.

Otherwise, send your feedback and the information below to MFI-swpdl.idoldocsfeedback@opentext.com.

We appreciate your feedback!

Product: Knowledge Discovery NiFi Ingest 25.2

Topic Title: Create the Data Flow