Open topic with navigation

# Stopping the Intelligence Event Consumers

Applies only when you have Intelligence in your environment

You will need to stop the Intelligence consumers after they have read past the offsets that the database has read. Then you must remove older events so the database does not receive duplicates after the upgrade. Although this procedure prevents the database from receiving duplicate events, Intelligence might receive duplicate events but then it would automatically correct this.

Complete the following steps to prevent the loss of events during the upgrade. This procedure requires the `db_scheduler_offset.csv` file created in Stopping the Database Event Consumer.

1. Log in to a bastion host. Ensure that you have copied the `db_scheduler_offset.csv` script to this node.
2. Create an executable shell script file, named `compare-current-offsets.sh`, with the following content:

       #!/bin/bash
       dof="$1"
       if [[ -z $dof || ! -f "$dof" ]]; then
            echo "Database offsets not found: $vof"
            exit 1
       fi
       NS=$(kubectl get ns | awk '/^arcsight-installer/{print $1}')
       t="mf-event-avro-enriched"
       ig="interset-logstash-es"
       ic=Intelligence
       tmp=$(mktemp -u --tmpdir offsets.XXXXX)
       printf "\nGetting current %s offsets for Intelligence\n\n" $t
       kubectl exec -it -n "$NS" th-kafka-0 -- bash -c "kafka-consumer-groups --bootstrap-server localhost:9092 --offsets --describe --all-groups" | grep "$t\s" | sed -re "s/^${ig}/$ic/" | awk '{c=$1; p=$3; o=$4; printf( "%s,%s,%s\n", p, o, c );}' | sort -n > "$tmp"
       partns=$( cut -d',' -f1 "$tmp" | sort -u)
       for p in $partns ; do for c in $ic ; do
            vo=$( awk -v FS="," "/^$p,.*,DB/"'{print $2}' $dof)
            if [[ -n $vo ]]; then
                desc="no $c offset data"
                co=$(awk -v FS="," "/^$p,.*,${c}$/"'{print $2}' "$tmp")
       	  if [[ -n $co && $co =~ ^[0-9]+$ ]]; then
                      lead=$(( co - vo ))
                      desc="$c is $(( - lead )) messages behind the db"
       		if [[ $lead -ge 0 ]]; then
       			desc="$c is caught up to the db and is $lead messages ahead"
       		fi
       	fi
       	printf "partition %d: %s\n" "$p" "$desc"
            fi
       done ; done
       rm -f $tmp

3. Using the CSV file and script that you created previously, run the following command:

       ./compare-current-offsets.sh db_scheduler_offset.csv

   This command generates output that shows the offsets between the database and the data consumers for all partitions:

   `Getting current offsets for Intelligence`

   `partition 0: Intelligence is caught up to the db and is 100231 messages ahead`

   `partition 1: Intelligence is caught up to the db and is 100123 messages ahead`

   `partition 2: Intelligence is 1002 messages behind the db`

   `partition 3: Intelligence is caught up to the db and is 100141 messages ahead`

   `partition 4: Intelligence is caught up to the db and is 100343 messages ahead`

   `partition 5: Intelligence is caught up to the db and is 100999 messages ahead`

4. Monitor the output from the command, which can indicate the following scenarios:

   - Whether a consumer has caught up to the database offset. For example, Intelligence is reading the mf-event-avro-enriched topic events ahead of the database:

     partition 0: Intelligence is caught up to the db and is 9995 messages ahead

   - Whether the database offset is ahead of the consumers reading the mf-event-avro-enriched topic. For example, Intelligence has not caught up to the database:

     partition 0: Intelligence is 1002 messages behind the db

   - Whether a consumer is not reading the mf-event-avro-enriched topic. For example, when this occurs, the output indicates no offset data:

     partition 0: no Intelligence offset data

   If a capability is not reading messages from mf-event-avro-enriched, you do not need to compare the offsets.

5. When Intelligence has caught up to the database, complete the following steps to stop the Intelligence consumers:
   1. Log in the master node of the cluster.
   2. Run the following command:

          NS=$(kubectl get namespaces | awk '/arcsight-installer/{print $1}')
          kubectl scale statefulset interset-logstash -n $NS  --replicas=0

   3. Run the following command periodically until you see a response indicating that none of the instances are ready:

          kubectl -n $NS get sts interset-logstash

      For example, you would want to see a response that indicates 0 out of 3 instances are ready:

   4. In the READY column, note the second number, which represents the number of instances. For example, `3`. You will need this number for the Restarting the Event Consumers procedure.