Ingesting Sample CSV Data to the Input Data Table

SmartConnectors are applications that collect events from different devices, process them, and send them to the desired destinations.

If SmartConnectors are not available for a particular device of an Intelligence supported data type, you can create FlexConnectors that can read and parse information from the devices and map that information to ArcSight's event schema. FlexConnectors are custom connectors you define to gather security events from log files, databases, and other software and devices. For every FlexConnector that you create, you need to create a corresponding configuration file. A configuration file is a text file containing properties (name, value pairs) that describe how the FlexConnector parses event data.

This section provides guidance on ingesting sample CSV data of a supported data type (for example, Active Directory) to the default_secops_adm.events database input data table with the help of FlexConnectors.

Configuration File

The configuration file provided in this section is designed only for the sample data set provided here for the Active Directory data type. This configuration file is used by the FlexConnector to parse the CSV data and convert it to the CEF format. The configuration file must be in this format - <file_name>.sdkfilereader.properties. For example, testdata.sdkfilereader.properties.

Sample CSV Data of the Active Directory Data Type

destinationUserName,categoryOutcome,externalId,destinationHostName,deviceReceiptTime,deviceCustomString5
bennett.merry,Success,4659,OTTAWADC.interset.com,2016-04-01T08:00:04-05:00,
pamila.dankert,Success,4659,NFMC.interset.com,2016-04-01T08:00:18-05:00,
pamila.dankert,Failure,4777,NFMC.interset.com,2016-04-01T08:00:20-05:00,
lakendra.danielson,Success,4634,NFMC.interset.com,2016-04-01T08:00:27-05:00,3

Configuration File for the Sample CSV Data

delimeter=,
text.qualifier="
comments.start.with=\#
trim.tokens=true
contains.empty.tokens=true
			
token.count=6
			
token[0].name=destinationUserName
token[0].type=String
token[1].name=categoryOutcome
token[1].type=String
token[2].name=externalId
token[2].type=String
token[3].name=destinationHostName
token[3].type=String
token[4].name=deviceReceiptTime
token[4].type=String
token[5].name=deviceCustomString5
token[5].type=String
			
event.destinationNtDomain=__stringConstant("WIN-MP0VNBBQVSI")
event.categoryBehavior=__stringConstant("/Authentication/Verify")
event.categoryObject=__stringConstant("/Host/Operating System")
event.deviceProduct=__stringConstant("Microsoft Windows")
event.deviceVendor=__getVendor("Microsoft")
event.deviceReceiptTime=__createOptionalTimeStampFromString(deviceReceiptTime,"YYYY-MM-DDThh:mm:ss.SSSX")
event.destinationUserName=destinationUserName
event.categoryOutcome=categoryOutcome
event.externalId=externalId
event.destinationHostName=destinationHostName
event.deviceCustomString5=deviceCustomString5

You can also create or customize the configurations files for other data sets of the supported data types. For more information, see ArcSight FlexConnector Developer's Guide.

FlexConnector Installation and Configuration

To install and configure a FlexConnector, see ArcSight FlexConnector Developer's Guide.

Ensure the following when you install and configure the FlexConnector:

• Select ArcSight FlexConnector File as the Connector Type.
• When adding the parameters information, specify the following:
  • Select Log Unparsed Events as False.
  • Provide the absolute path and the CSV file name that the FlexConnector needs to read in the Log File Name field.
    For example, c:\temp\sample_data.csv.
  • For the Configuration File field, specify only the file name that you used in the configuration file.
    For example, if the configuration file is in this format - testdata.sdkfilereader.properties, then specify only testdata. The suffix .sdkfilereader.properties is appended automatically. The configuration file name now is testdata.sdkfilereader.properties.
• When configuring the destination, select either CEF File or Transformation Hub as the destination. For more information, see SmartConnector Installation and User Guide.

Post-Installation Tasks

After you install and configure the FlexConnector and before you run the FlexConnector, copy the configuration file in the ARCSIGHT_HOME\user\agent\flexagent location.

Sending Data to the Input Data Table

To send data to the input data table, you need to start the SmartConnector/FlexConnector. You can run the SmartConnector/FlexConnector in standalone mode or as a service, depending on the mode you selected during installation.

Running in Standalone Mode

If you have installed the SmartConnector/FlexConnector in the standalone mode, you need to start it manually (periodically or as per your requirement). Also, you need to start the SmartConnector/FlexConnector whenever the host on which it is installed is restarted, because the SmartConnector/FlexConnector is not automatically active when the host is restarted.

Perform the following steps to start the SmartConnector/FlexConnector agent so that it can send the CSV data to the Transformation Hub topic and which will then be loaded to the database events table.

1. Change to the following directory:

   cd $ARCSIGHT_HOME\current\bin\

2. Execute the following command:

   ./arcsight agents

Running as a Windows Service

To start or stop the SmartConnector/FlexConnector installed as a service on the Windows platform:

1. Right-click My Computer, then select Manage from the Context menu.

2. Expand the Services and Applications folder and select Services.

3. Right-click the SmartConnector/FlexConnector service name and select Start to run the SmartConnector/FlexConnector or Stop to stop the service.

To verify that the SmartConnector/FlexConnector service has started, view the following file:

$ARCSIGHT_HOME/logs/agent.out.wrapper.log

To reconfigure the SmartConnector/FlexConnector as a service, run the SmartConnectorConfiguration/FlexConnectorConfiguration Wizard again. Open a command window on $ARCSIGHT_HOME/current/bin and run:

./runagentsetup