Workaround for Cluster ID Mismatches
If cluster ID mismatches are encountered, use the following workaround to resolve the issue.
-
On the jump host, save the namespace in an environment variable for execution of some commands later:
NS=$(kubectl get namespaces | grep arcsight-installer | awk ' {print$1} ')
-
Get the correct cluster ID from the error log in the problematic Kafka pod (in CrashLoopBackOff status) by running the following command:
kubectl logs <th-kafka-N> -n $NS | sed -n -e 's/^.*\ (InconsistentClusterIdException: The Cluster ID \)/\1/p'
Where <th-kafka-N> is the problematic pod.
Example output:
InconsistentClusterIdException: The Cluster ID LU79-NxHQRmtgFAfcqmGjw doesn’t match stored clusterId Some(VLAfF6HgTeiHeuMPXLLcCw) in meta.properties
The leftmost cluster ID that is mismatching (shown in bold in the example) is the Cluster ID to set in the meta.properties file in Step 3.
-
In a text editor, open the meta.properties file on the failed pod in the
/opt/arcsight/k8s-hostpath-volume/th/kafka
directory. Edit the section after the stringcluster.id=
, as in the following example:#Wed January 26 11:21:15 EET 2023
cluster.id=<mismatching cluster ID from Step 2> version=0
broker.id=<node broker ID>
-
Restart the problematic Kafka pod using the following command:
kubectl delete pod th-kafka-N -n $NS
Where NS is the value you obtained from step 1.
-
Validate that the procedure was successful by checking that all services are up and running, by performing steps 15a through 15g of Performing the AKS Upgrade .
The assigned broker IDs must match the original broker IDs you recorded in the data collection table you prepared earlier (usually 1001, 1002, and 1003 in a 3 worker deployment).