Configuring Intelligence Analytics Targeted Events

This topic and the procedure that follows are applicable only if you are deploying both the ArcSight Intelligence and ArcSight Recon capabilities with the ArcSight Platform.

Elasticsearch filtering enables Intelligence analytics resources to be focused on the events that are most valuable for Intelligence analytics capabilities.

Through extensive research, ArcSight data scientists have identified a targeted set of events proven to be the best inputs into threat detection analytic models.

Other events, useful for different purposes (such as investigating all stages of a threat life-cycle), are not used as input into the Intelligence analytics capabilities.

In some deployments (especially when ArcSight Recon and Intelligence are both deployed), and depending on the ingested data, applying the correct filter can free up significant storage, CPU and memory resources in each server.

This procedure installs the Logstash-based event filter to limit what is stored in the Intelligence Elasticsearch component to those identified valuable events.

Once the filter is configured, the Intelligence Elasticsearch component will only ingest from Transformation Hub filtered events that are useful with Intelligence analytics.

To install the filter for Intelligence Elasticsearch:

  1. Log in to the master node of the OMT cluster.

  2. Edit the logstash-config-pipeline file using the following command:

    kubectl -n $(kubectl get namespaces | grep arcsight | cut -d ' ' -f1) edit configmaps logstash-config-pipeline
  3. Locate the code block below and add a line break at the end:

    if [destinationUserName] =~ "\$" {
       mutate {
    
    replace => {
    	  "did" => "1"
          }
       }
    }
    
  4. Add the following code block after the line break, while respecting code indentation:

    if [destinationUserName] and [externalId] in ['4624','4625','4648','4768','4769','4771','4776','4777'] {
    }
    else if [categoryDeviceGroup] =~ "/Network Equipment" and [categoryDeviceType] =~ "Network Monitoring" {
    }
    else if [sourceusername] and [externalId] in ['Squid','Microsoft','Blue Coat'] and [deviceProduct] in ['Proxy SG','Squid Web Proxy Server','ISA Server'] {
    }
    else if [categoryDeviceType] =~"Repository" and [destinationUserName] and [deviceAction] and [deviceCustomString1] {
    }
    else if [destinationUserName] and [categoryOutcome] and [fileName] and [deviceProduct] in ['','WORKGROUP','NT SERVICE','NT AUTHORITY'] {
    }
    else if [type] == 2 {
    }
    else if [categoryObject] =~ "/Host/Application/Service" {
    }
    else {
          drop {}
    }
    
  5. Run the following command:

    kubectl -n $(kubectl get namespaces | grep arcsight | cut -d ' ' -f1) scale statefulset interset-logstash --replicas=0
  6. After allowing time for the process to finish, validate that pods are successfully scaled down by running:

    kubectl -n $(kubectl get namespaces | grep arcsight | cut -d ' ' -f1) get pods | grep logstash

    You will know the process is finished when no pods are shown.

  7. Run the following command to complete the filter installation:

    kubectl -n $(kubectl get namespaces | grep arcsight | cut -d ' ' -f1) scale statefulset interset-logstash --replicas=X

    Were X stands for the number of Logstash instances you want to run across the cluster and generally equals the number of worker nodes.