Workaround for Schema Registry or Broker ID Mismatch Errors
-
kubectl exec th-zookeeper-0 -n $NS -- /bin/bash -c "/usr/bin/zookeeper-shell th-zookeeper-0.th-zk-sts:2181 ls /brokers/ids"
Note any broker IDs that are mismatched compared to those recorded in the data collection table.
-
If ALL broker IDs are mismatched from the values in the data collection table, then restart all the Transformation Hub pods by running the following command:
kubectl delete pods -n $NS $(kubectl get pods -n $NS -o wide | grep "th-"| cut -d ' ' -f1)
Proceed to Step 4.
-
If only SOME of the worker nodes are mismatched from the values in the data collection table, then do the following:
-
Identify the worker nodes where you must restart all pods. On the jump host, save the following script in a file with the
name
:identifyBrokerIdsMismatches.sh
If you choose to copy and paste the following script text into a text editor, in Adobe Acrobat, select the text for the script and choose Copy with Formatting. Then paste the text into your editor. Some minor manual editing, such as the removal of blank lines, may be necessary after pasting.#!/bin/bash NS=$(kubectl get namespaces | grep arcsight-installer | awk ' {print $1} ') if [ ! -z "$1" ] && [ ! -z "$2" ];then POD_NAME=$(kubectl get pods -n $NS -o wide | grep -v "manager" | grep -E "th-kafka-.*$1\s" | cut -d ' ' -f1) if [ ! -z $POD_NAME ]; then BROKER_IDS_PATTERN="brokerId=$2\]" if [ ! -z "$(kubectl logs "${POD_NAME}" -n $NS | grep "$BROKER_IDS_PATTERN")" ]; then echo "The specified node $1 is running with the specified brokerId $2"; else echo "The specified node $1 is not running with the specified brokerId $2"; fi else echo "The specified node $1 is not running kafka"; fi else echo "Missing arguments, please check"; fi unset POD_NAME
-
Give the script execution privileges by running:
chmod +x identifyBrokerIdsMismatches.sh
-
Run the script once for each row (node) of your data collection table, each time passing as argument the <node-name> and its corresponding <broker-id> accordingly:
If the script returns the message:./identifyBrokerIdsMismatches.sh <node-name> <broker-id>
The specified node <node-name> is not running with the specified brokerId <broker-id>
then there is a broker ID mismatch on the node. Restart all Transformation Hub pods on any mismatched node that returns this message by running the following command:
kubectl delete pods -n $NS $(kubectl get pods -n $NS -o wide | grep <Node FQDN> | grep "th-" | cut -d ' ' -f1)
-
-
Validate that the procedure was successful by checking that all services are up and running, by performing steps 15a through 15g of Performing the AKS Upgrade .
The assigned broker IDs must match the original broker IDs you recorded in the data collection table (usually 1001,1002, and 1003 in a 3-workers deployment).