Configuring the Retention Settings for New Kafka Topics

The mf-event-avro-enriched topic, which was added with ArcSight Platform 21.1, occupies storage space on worker nodes where you apply the 'kafka:yes' label. To ensure that the nodes have enough storage space for each topic, you must tune the settings for topic retention. By tuning the storage settings, you allocate storage space for this topic, the existing topics, and other components that use storage on these worker nodes. Please note that this procedure does not interrupt the flow of events through the system.

  1. Log in to an ArcSight master node as root.

  2. To determine the current topic retention storage size for the th-arcsight-avro topic, run the following command:
    3. kubectl exec -n $( kubectl get pods --all-namespaces | grep kafka-0 | cut -d ' ' -f1 ) th-kafka-0 -- /usr/bin/kafka-configs --bootstrap-server th-kafka-svc:9092 --describe --topic th-arcsight-avro | grep retention.bytes
  3. To calculate the retention size for each topic, divide the retention.bytes value from Step 2 by two.
    The th-arcsight-avro and mf-event-avro-enriched topics will share the same retention size.

  4. To set the retention size for the th-arcsight-avro topic, in the following command replace {number} with the value calculated in Step 3, and then run it:

    5. kubectl exec -n $( kubectl get pods --all-namespaces | grep kafka-0 | cut -d ' ' -f1 ) th-kafka-0 -- /usr/bin/kafka-configs --bootstrap-server th-kafka-svc:9092 --alter --topic th-arcsight-avro --add-config retention.bytes={number}
  5. To set the retention size for the mf-event-avro-enriched topic, in the following command replace {number} with the value calculated in Step 3, and then run it:
    6. kubectl exec -n $( kubectl get pods --all-namespaces | grep kafka-0 | cut -d ' ' -f1 ) th-kafka-0 -- /usr/bin/kafka-configs --bootstrap-server th-kafka-svc:9092 --alter --topic mf-event-avro-enriched --add-config retention.bytes={number}