Small Workload
This section describes the system sizing and tuning results from tests of the ArcSight Platform and deployed capabilities Transformation Hub, Fusion, Command Center for ESM, Intelligence, Recon, and the ArcSight Database that has been confirmed in our testing lab to maintain satisfactory performance of the system under a small workload.
Workloads
This section describes the workload that was placed on the tested system.
Event Workload
| Application | Events per second |
|---|---|
| Microsoft Windows | 700 |
| InfoBlox NIOS | 700 |
| Intelligence Data (VPN, AD, Proxy) | 100 |
| Total | 1,500 |
Other workload
| Category | Level |
|---|---|
| Storage Groups | 10 |
| Searches | 3 per hour (concurrent) |
| Reports | 1 scheduled every hour |
System Sizing
This section describes the sizing of the tested system.
On-Premises Deployment
The OMT Master/Worker Node/Database Node system resources are where the core platform, Transformation Hub, Fusion, Command Center for ESM, Recon, and Database compute components were deployed in an all-in-one collocated configuration on the tested system. However, the Database Communal Storage components were deployed on a separate node because they are not embedded within the ArcSight Platform. When using this information as guidance for your own system sizing, the OMT Master/Worker Node/Database Node system resources are always needed, but the Database Communal Storage system resources are only needed when deploying Recon or Intelligence.
| Category | 1 x OMT Master/Worker Node/Database Node | 1 x Communal Storage |
|---|---|---|
| Processor | Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz | Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz |
| vCPU(s) (# threads) | 24 | 6 |
| RAM (per node) | 128 GB | 32 GB |
| Disks (per node) | ESX data store | ESX data store |
| Storage per day (1x) | 7 GB (depot) + 15 GB (ES) | 27 GB (MinIO) |
| Total disk space (5 Billion events) | 1 TB (holds up to 45 days of events) | 1 TB (holds up to 40 days of events) |
| K-safety level | 0 | N/A |
AWS Deployment
The OMT Worker (Platform) system resources are where the core platform, Transformation Hub, Fusion, Command Center for ESM, and Recon components were deployed on the tested system. However, Intelligence components were deployed on the OMT Worker (Intelligence) system resources because they utilize a significant amount of resources when running analytics jobs. When using this information as guidance for your own system sizing, the OMT Worker (Platform) system resources are always needed, the Database system resources are only needed when deploying Recon or Intelligence, and the OMT Worker (Intelligence) system resources are only needed when deploying ArcSight Intelligence.
| Category | OMT Worker (Platform) | Database | OMT Worker (Intelligence) |
|---|---|---|---|
| Instance Type | m5.2xlarge | m5d.4xlarge | m5.2xlarge |
| Instance Count | 3 | 3 | 3 |
| Disks (per node) | 500 GB - EBS storage (gp2) | 2 x 300 NVMe SSD | 500 GB - EBS storage (gp2) |
Azure Deployment
The OMT Worker (Platform) system resources are where the core platform, Transformation Hub, Fusion, Command Center for ESM, and Recon components were deployed on the tested system. However, Intelligence components were deployed on the OMT Worker (Intelligence) system resources because they utilize a significant amount of resources when running analytics jobs. When using this information as guidance for your own system sizing, the "OMT Worker (Platform)" system resources are always needed, the Database system resources are only needed when deploying Recon or Intelligence, and the OMT Worker (Intelligence) system resources are only needed when deploying ArcSight Intelligence.
| Category | OMT Worker (Platform) | Database | OMT Worker (Intelligence) |
|---|---|---|---|
| Instance Type | D2s_V3 | D4s_V3 | D2s_V3 |
| Instance Count | 3 | 3 | 3 |
| Disks (per node) | 1 x 500 GB - Premium SSD | 2 x 300 GB - Premium SSD | 1 x 500 GB - Premium SSD |
System Tuning
This section describes the system tuning of the tested system.
Database Tuning
| Category | Property | On-Premises | AWS | Azure |
|---|---|---|---|---|
| Core Database |
shard_count |
3 |
3 |
3 |
| Core Database | depot_size | 40% | 60% | 60% |
| Tuple Mover | tm_concurrency | 5 | 6 | 5 |
| Tuple Mover | tm_memory | 10G | 10G | 10G |
| Tuple Mover | plannedconcurrency | 5 | 6 | 5 |
| Tuple Mover | tm_memory_usage | 10000 | 10000 | 10000 |
| Tuple Mover | maxconcurrency | 10 | 7 | 10 |
| Ingest Resource pools |
ingest_pool_memory_size |
30% |
30% |
30% |
| Ingest Resource pools | ingest_pool_planned_concurrency | 6 | 6 | 6 |
| Backup |
Backup Interval (hours) |
1 |
1 |
1 |
| Communal Storage | Server-side Encryption | Disabled | Disabled | Yes (MMK) |
Transformation Hub Tuning
| Property | On-Premises | AWS | Azure |
|---|---|---|---|
| # of Kafka broker nodes in the Kafka cluster | 1 | 3 | 3 |
| # of ZooKeeper nodes in the ZooKeeper cluster | 1 | 3 | 3 |
| # of Partitions assigned to each Kafka Topic* | 12 | 24 | 24 |
| # of replicas assigned to each Kafka Topic | 1 | 2 | 2 |
| # of message replicas for the __consumer_offsets Topic | 1 | 3 | 3 |
| Schema Registry nodes in the cluster | 1 | 3 | 3 |
| # of CEF-to-Avro Stream Processor instances to start** | 0 | 0 | 0 |
| # of Enrichment Stream Processor Group instances to start | 2 | 2 | 2 |
*Kafka topics - th-arcsight-avro; mf-event-avro-enriched; and th-cef, if connectors are configured to send to Transformation Hub in CEF format
**If connectors are configured to send Avro format to Transformation Hub, you can set the # of CEF-to-Avro Stream Processor instances to start quantity to 0 because there is no need to convert CEF to Avro.
Intelligence Tuning
| Property | On-Premises | AWS | Azure |
|---|---|---|---|
| Elasticsearch Shard Count | 6 | 6 | 6 |
| Elasticsearch data processing Instances | 1 | 3 | 3 |
| Elasticsearch Index Replica Count | 0 | 1 | 1 |
| Elasticsearch Memory (GB) | 10 | 4 | 4 |
| Elasticsearch number of cores | 6 | 2 | 2 |
| Elasticsearch Size Per Batch | 5mb | 5mb | 5mb |
| Logstash Instances | 2 | 3 | 3 |
| Logstash pipeline workers per instance | 2 | 1 | 1 |
| Logstash Pipeline Batch size | 500 | 500 | 500 |
| LogStash Filter Applied | yes | yes | yes |
| Spark parallelism | 32 | 32 | 32 |
| Spark number of executors | 3 | 3 | 3 |
| Spark executor memory | 5g | 4g | 4g |
| Spark number of executor cores | 1 | 1 | 1 |
| Spark driver memory | 4g | 4g | 3g |
| Spark memory overhead factor | 0.2 | 0.2 | 0.2 |
| Intelligence Job per day | 1 | 1 | 1 |
Fusion Tuning
| Category | All Deployments |
|---|---|
| Event Integrity Check Task Count | 1 |
| Event Integrity Check Chunk Size | 1000 |
| Use Event Integrity Check Resource Pool | false |
SmartConnector Tuning
| Category | All Deployments |
|---|---|
| SmartConnector version that we tested | 8.3.0.14008.0 |
| Instance Count | 1 |
| Acknowledgement Mode | leader |
| usessl (Transformation Hub Destination Param) | false |
| contenttype (Transformation Hub Destination Param) | Avro |
| topic (Transformation Hub Destination Param) | th-arcsight-avro |
| compression.type | gzip |
| transport.batchqueuesize | 20000 |
| transport.cefkafka.batch.size | 50000 |
| transport.cefkafka.linger.ms | 10 |
| transport.cefkafka.max.request.size | 4194304 |
| transport.cefkafka.multiplekafkaproducers | true |
| transport.cefkafka.threads | 3 |