Open topic with navigation

# Tuning Your Deployment for Recon or Intelligence

This section describes tuning your deployment for Recon or Intelligence. Skip this section if you have not deployed Recon or Intelligence.

## Verifying Recon `cron` Jobs

After deployment, check Recon to verify that the corresponding `cron` jobs are running, as follows:

1. In Recon, browse to INSIGHT >Data Timeseries and Source Agents and Hourly Event Volume. If there is no information displayed after an hour, the `cron` job `events_quality.sh` is not running.

2. Go to DASHBOARD > Data Processing monitoring and Health and Performance Monitoring. If there is no information displayed after an hour, the `cron` job `events_hourly_rate.sh` is not running.

If either of these `cron` jobs is not running, then restart `fusion-db-adm-schema-mgmt`, as follows:

1. Connect to the first master node.

2. Run the following commands:

    PODS=`kubectl get pods -A | grep fusion-db-adm-schema-mgmt | awk '{print $1, $2}'
    kubectl delete pods -n $PODS

## Updating Event Topic Partition Number

Refer to the Technical Requirements for ArcSight Platform 23.3, section entitled System Hardware Sizing and Tuning Guidelines to determine an appropriate event topic partition number for your workload.

To update the topic partition number from the master node1:

Select one of the following commands, based on your encryption and authentication configuration:

Note: The commands contain two variables that will need to be replaced before the execution:

- `{topic_to_update}`: replace with `th-arcsight-avro`, `mf-event-avro-esmfiltered`, `th-cef` and `mf-event-avro-enriched` and for each iteration of the command

- {number_of_partitions}: For Recon - database node count * 12

For example, for a 3 nodes database cluster, the partition number would be = 3 * 12 = 36

For FIPS (or non-FIPS) Encryption with Client Authentication:

    kubectl exec th-kafka-0 -n $(kubectl get ns|awk '/arcsight/ {print $1}') -- sh -c 'sed -ir "s/^[#]*\s*ssl.truststore.password=.*/ssl.truststore.password=$STORES_SECRET/" /etc/kafka/client.properties && \
    sed -ir "s/^[#]*\s*ssl.keystore.password=.*/ssl.keystore.password=$STORES_SECRET/" /etc/kafka/client.properties && \
    sed -ir "s/^[#]*\s*ssl.key.password=.*/ssl.key.password=$STORES_SECRET/" /etc/kafka/client.properties && \
    kafka-topics --bootstrap-server th-kafka-svc:9093 --alter --topic {topic_to_update} --partitions {number_of_partitions} --command-config /etc/kafka/client.properties'

After executing the above, Copy and then paste the following command block:

    kubectl exec th-kafka-0 -n $(kubectl get ns|awk '/arcsight/ {print $1}') -- sh -c 'sed -ir "s/^[#]*\s*ssl.truststore.password=.*/ssl.truststore.password=/" /etc/kafka/client.properties && \
    sed -ir "s/^[#]*\s*ssl.keystore.password=.*/ssl.keystore.password=/" /etc/kafka/client.properties && \
    sed -ir "s/^[#]*\s*ssl.key.password=.*/ssl.key.password=/" /etc/kafka/client.properties'

For FIPS Encryption Without Client Authentication

    kubectl exec th-kafka-0 -n $(kubectl get ns|awk '/arcsight/ {print $1}') -- sh -c 'KAFKA_OPTS+=" -Djavax.net.ssl.trustStore=/etc/kafka/secrets/th-kafka.truststore " && \
    KAFKA_OPTS+="-Djavax.net.ssl.trustStorePassword=$STORES_SECRET " && \
    KAFKA_OPTS+="-Djavax.net.ssl.trustStoreProvider=BCFIPS " && \
    KAFKA_OPTS+="-Djavax.net.ssl.trustStoreType=BCFKS " && \
    kafka-topics --bootstrap-server th-kafka-svc:9093 --alter --topic {topic_to_update} --partitions {number_of_partitions} --command-config /etc/kafka/client2.properties'

For non-FIPS Encryption Without Client Authentication

    kubectl exec th-kafka-0 -n $(kubectl get ns|awk '/arcsight/ {print $1}') -- sh -c 'KAFKA_OPTS+=" -Djavax.net.ssl.trustStore=/etc/kafka/secrets/th-kafka.truststore " && \
    KAFKA_OPTS+="-Djavax.net.ssl.trustStorePassword=$STORES_SECRET " && \
    kafka-topics --bootstrap-server th-kafka-svc:9093 --alter --topic {topic_to_update} --partitions {number_of_partitions} --command-config /etc/kafka/client2.properties'

Copy the selected command (or commands in case of For FIPS (or non-FIPS) Encryption with Client Authentication) and execute it 4 times according to the following table:

Command Execution | Replace the `{topic_to_update}` variable with: | Replace the `{number_of_partitions}` variable with:
First |
    th-arcsight-avro
| A number of partitions that will comply with your Recon requirements
Second |
    mf-event-avro-esmfiltered
| Use the same number as in the first execution of the command
Third |
    th-cef
| Use the same number as in the first execution of the command
Fourth |
    mf-event-avro-enriched
| Use the same number as in the first execution of the command

Standard Kafka topics settings only permit increasing the number of partitions, not decreasing them.

5. Use the Kafka manager to verify that the partition number for the `th-cef, th-arcsight-avro,``mf-event-avro-enriched` and `mf-event-avro-esmfiltered` topics have been updated to the desired partition number.

##

## Updating the OMT Hard Eviction Policy

You need to update the Kubernetes hard eviction policy from 15% (default) to 100 GB to maximize disk usage.

To update the OMT Hard Eviction Policy, perform the following steps on each worker node, after deployment has been successfully completed. Please verify the operation is successfully executed on one work node first, then proceed on the next worker node.

The eviction-hard can be defined as either a percentage or a specific amount. The percentage or the specific amount will be determined by the volume storage.

To update the policy:

1. Run the following commands:

       cp /usr/lib/systemd/system/kubelet.service /usr/lib/systemd/system/kubelet.service.orig
       vim /usr/lib/systemd/system/kubelet.service

2. In the file, after `ExecStart=/usr/bin/kubelet \`, add the following line:

       --eviction-hard=memory.available<100Mi,nodefs.available<100Gi,imagefs.available<2Gi \

3. Save your change to the file.

4. To activate the change, run the following command:

       systemctl daemon-reload ; systemctl restart kubelet

5. To verify the change, run:

       systemctl status kubelet

   No error should be reported.