Large Workload
This section describes the system sizing and tuning results from tests of the ArcSight Platform and deployed capabilities Transformation Hub, Fusion, Command Center for ESM, Intelligence, Recon, and the ArcSight Database that has been confirmed in our testing lab to maintain satisfactory performance of the system under a large workload.
Workloads
This section describes the workload that was placed on the tested system.
Event Workload
| Application | Events per second |
|---|---|
| Microsoft Windows | 54,000 |
| InfoBlox NIOS | 54,000 |
| Intelligence Data (VPN, AD, Proxy) | 12,000 |
| Total | 120,000 |
Other Workload
| Category | Level |
|---|---|
| Storage Groups | 10 |
| Searches | 3 per hour (concurrent) |
| Reports | 1 scheduled every hour |
System Sizing
This section describes the system sizing of the tested system.
AWS Deloyment
The OMT Worker (Platform) system resources are where the core platform, Transformation Hub, Fusion, Command Center for ESM, and Recon components were deployed on the tested system. However, Intelligence components were deployed on the OMT Worker (Intelligence) system resources because they utilize a significant amount of resources when running analytics jobs. When using this information as guidance for your own system sizing, the OMT Worker (Platform) system resources are always needed, the Database system resources are only needed when deploying Recon or Intelligence, and the OMT Worker (Intelligence) system resources are only needed when deploying ArcSight Intelligence.
| Category | OMT Worker (Platform) | Database | OMT Worker (Intelligence) |
|---|---|---|---|
| Instance Type | m5.8xlarge | m5d.8xlarge | m5.8xlarge |
|
Instance Count |
3 | 18 | 6 |
| Disks (per node) | 3 TB - EBS storage | 2 x 600 NVMe SSD | 3 TB - EBS storage |
System Tuning
This section describes the system tuning of the tested system.
Database Tuning:
| Category | Property | AWS |
|---|---|---|
| Core Database |
shard_count |
18 |
| Core Database | depot_size | 60% |
| Tuple Mover | tm_concurrency | 5 |
| Tuple Mover | tm_memory | 10G |
| Tuple Mover | plannedconcurrency | 5 |
| Tuple Mover | tm_memory_usage | 10000 |
| Tuple Mover | maxconcurrency | 10 |
| Ingest Resource pools |
ingest_pool_memory_size |
30% |
| Ingest Resource pools | ingest_pool_planned_concurrency | 6 |
| Backup |
Backup Interval (hours) |
1 |
| Communal Storage | Server-side Encryption | disabled |
Transformation Hub Tuning
| Property | AWS |
|---|---|
| # of Kafka broker nodes in the Kafka cluster | 3 |
| # of ZooKeeper nodes in the ZooKeeper cluster | 3 |
| # of Partitions assigned to each Kafka Topic* | 108 |
| # of replicas assigned to each Kafka Topic | 2 |
| # of message replicas for the __consumer_offsets Topic | 3 |
| Schema Registry nodes in the cluster | 3 |
| # of CEF-to-Avro Stream Processor instances to start** | 0 |
|
# of Enrichment Stream Processor Group instances to start |
6 |
*Kafka topics - th-arcsight-avro; mf-event-avro-enriched; and th-cef, if connectors are configured to send to Transformation Hub in CEF format
**If connectors are configured to send Avro format to Transformation Hub, you can set the # of CEF-to-Avro Stream Processor instances to start quantity to 0 because there is no need to convert CEF to Avro.
| Kafka Override Parameters | AWS |
|---|---|
| arcsight.eventbroker.kafka.KAFKA_NUM_IO_THREADS | 256 |
| arcsight.eventbroker.kafka.KAFKA_NUM_NETWORK_THREADS | 52 |
| arcsight.eventbroker.kafka.KAFKA_NUM_REPLICA_FETCHERS | 145 |
Intelligence Tuning
| Property | AWS |
|---|---|
| Elasticsearch Shard Count | 6 |
| Elasticsearch data processing Instances | 6 |
| Elasticsearch Index Replica Count | 1 |
| Elasticsearch Memory (GB) | 24 |
| Elasticsearch number of cores | 12 |
| Elasticsearch Size Per Batch | 10mb |
| Logstash Instances | 108 |
| Logstash pipeline workers per instance | 2 |
| Logstash Pipeline Batch size | 2000 |
| Spark Parallelism | 64 |
| Spark number of executors | 24 |
| Spark executor memory | 12g |
| Spark number of executor cores | 1 |
| Spark Driver Memory | 8g |
| Spark Memory Overhead Factor | 0.2 |
| Intelligence Job per day | 1 |
Fusion Tuning
| Category | All Deployments |
|---|---|
| Event Integrity Check Task Count | 6 |
| Event Integrity Check Chunk Size | 1000 |
| Use Event Integrity Check Resource Pool |
false |
SmartConnector Tuning
| Category | All Deployments |
|---|---|
| SmartConnector version that we tested | 8.3.0.14008.0 |
| Instance Count | 5 |
| Acknowledgement Mode | none |
| usessl (Transformation Hub Destination Param) | false |
| contenttype (Transformation Hub Destination Param) | Avro |
| topic (Transformation Hub Destination Param) | th-arcsight-avro |
| compression.type | gzip |
| transport.batchqueuesize | 20000 |
| transport.cefkafka.batch.size | 50000 |
| transport.cefkafka.linger.ms | 10 |
| transport.cefkafka.max.request.size | 4194304 |
| transport.cefkafka.multiplekafkaproducers | true |
| transport.cefkafka.threads | 6 |