Understanding Data Ingestion

ArcSight SaaS can receive data from your environment through SmartConnectors that you deploy or by your migrating data from ArcSight Logger.

SmartConnectors

SmartConnectors can parse individual events and normalize event values into Avro format for log consumers and it sends these events to Amazon S3 and ArcSight SaaS destinations. Based on their configuration, SmartConnectors receive events over the network using SNMP, HTTP, Syslog, or proprietary protocols such as OPSEC. Managing SmartConnectors via an ArcMC instance offers centralized management and monitoring of SmartConnectors and remotely-deployed on customer-provisioned hosts. You can deploy each managed SmartConnector and bulk-configure with a standard Amazon S3 bucket or ArcSight SaaS destinations.

To understand and plan for data ingestion options that best match your requirements to use the full capabilities of ArcSight, review SmartConnector Installation Overview in the Installation Guide for ArcSight SmartConnectors.

You can install the data ingestion components in one of the following ways:

Standalone SmartConnectors

Enables deployment of only the SmartConnectors that you need, on operating systems that you install. These connectors deliver their event stream to the designated Amazon S3 or ArcSight SaaS destinations for data ingestion into ArcSight. This data ingestion method is usually required only when you deploy the SmartConnector in a location that ArcMC cannot reach.

Although not a recommended method, it is possible to deploy connectors on customer-provisioned hosts, with each connector independently configured to deliver their event stream to the designated Amazon S3 or ArcSight SaaS destinations for data ingestion.

Standalone SmartConnectors managed and deployed by a standalone ArcMC

Enables deployment and management of only the SmartConnectors that you need and ArcMC, on operating systems that you install.

For maximum flexibility, you can deploy SmartConnectors in any mix of the SmartConnectors and ArcMC configurations, as determined by your environment and policy.

ArcSight Logger

If you have ArcSight Logger deployed, you can import the event data from your Loggers to ArcSight SaaS, thus allowing your users to run searches on those events. To do so, you must migrate first the metadata that defines the event archives, and then the event data. During the first phase of the migration process, the system temporarily stores the metadata files and archive catalogs in the AWS S3 bucket. After you complete the second phase of migration, the ArcSight Database stores the event data. Retention times in the Database depend on your product license. Users cannot search the event data until you complete the second phase.

For more information, see Importing Copied Logger Data to the SaaS Database.