(Conditional - For Intelligence Deployments only) If the Elasticsearch recovery in step 8 failed

  1. If the Elasticsearch recovery in step 8 of the EKS upgrade procedure failed for the Intelligence pods, you must proceed to delete the unassigned shards that may impede the recovery process, by executing this command:

    kubectl exec -it -n $(kubectl get ns |awk '/arcsight/ {print $1}') elasticsearch-master-0 -c  elasticsearch -- curl -k -XGET https://elastic:<password>@localhost:9200/_cat/shards | grep UNASSIGNED | awk {'print $1'} | xargs -i curl -k -XDELETE 'https://elastic:<password>@localhost:9200/{}'

  2. Log into the pod again and execute this command:

    kubectl exec -n $(kubectl get ns |awk '/arcsight/ {print $1}') elasticsearch-master-0 -c  elasticsearch -it bash

    Monitor the Snapshot progress using the following command:

    curl -k -XGET 'https://elastic:<password>@localhost:9200/_cat/health?v=true'

    Example command and output:

    curl -k -XGET 'https://elastic:changeme@localhost:9200/_cat/health?v=true'
    epoch      timestamp cluster  status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1671118161 15:29:21  interset green           6         3   1128 583    0    0        0             0                  -                100.0%

    In the example above, the value has reached a 100% and the status is green. If the result of the command is a yellow status, and a value under 100%, apply the following procedure:

    1. Login with system-admin role to the interset UI https://<CLUSTER FQDN>/interset

    2. Click on the gear icon on the top right corner and select Search Manager.

    3. Click on the Job History list box.

    4. Select Submit a Job.

    5. Click on the Job type list box and select Restore.

    6. Enter 0 for the Customer to apply Snapshot to field.

    7. Click the SUBMIT JOB button.

    Verify the job status

    Go to the Job history page and check the Snapshot job ID status. Click the REFRESH button until the status becomes either COMPLETED_SUCCESS or COMPLETED_FAILED.

    If the final status is COMPLETED_FAILED, execute these commands:

    Log into any of the elasticsearch-masters and execute the following command:

    kubectl exec -n $(kubectl get ns |awk '/arcsight/ {print $1}') elasticsearch-master-0 -c  elasticsearch -it bash

    Monitor the Snapshot progress using the following command:

    curl -k -XGET 'https://elastic:<password>@localhost:9200/_cat/health?v=true'

    Example command and output:

    curl -k -XGET 'https://elastic:changeme@localhost:9200/_cat/health?v=true'
    epoch      timestamp cluster  status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1671118161 15:29:21  interset green           6         3   1128 583    0    0        0             0                  -                100.0%

    In the example above, the value has reached a 100% and the status is green. If the result of the command is a yellow status, and a value under 100%, you may wait 5 minutes and then repeat the command until the green status is achieved.

  3. Scale up logstash using the following command:

    kubectl -n $(kubectl get ns |awk '/arcsight/ {print $1}') scale statefulset interset-logstash --replicas=<replica count>

    Where you must replace the <replica count> value.

  4. Run Analytics on Demand to make sure that Analytics will resume after the EKS upgrade:

    Before you run Analytics on demand, do the following:

    • Ensure that Analytics is not already running because running Analytics on demand can cause Analytics in progress to fail.
    • If the previous Analytics execution failed, check whether the properties in the Intelligence tab are set correctly. If this does not solve the issue, contact Micro Focus Customer Support.

    To run Analytics on demand:

    1. Launch a terminal session and log in to the NFS node.

    2. Navigate to the following directory:

      cd <NFSVolume>/interset/analytics

    3. (Conditional) Delete the blackhawk_down file, if present. This is an error file and it is generated if the previous Analytics execution fails.

      rm blackhawk_down

    4. When prompted whether you want to delete the file, enter yes.

    5. Execute the following command to delete the latest AnalyticsStarted.mk and AnalyticsCompleted files:

      rm -rf AnalyticsStarted-0-<Today's_date>.mk AnalyticsCompleted-0-<Today's_date>.mk

    6. When prompted whether you want to delete the files, enter yes.

      After 30 seconds of deletion of the files, Analytics is triggered automatically.

If Analytics were to fail after the EKS upgrade, there are two things you can check to remedy it:

Suppressed: org.elasticsearch.client.ResponseException: method [POST], 
host [https://elasticsearch-svc:9200], URI [/_aliases?master_timeout=30s&timeout=30s], 
status line [HTTP/1.1 503 Service Unavailable]{"error":{"root_cause":[{"type":"not_master_exception",
"reason":"no longer master. source: [index-aliases]"}],"type":"master_not_discovered_exception",
"reason":"NotMasterException[no longer master. source: [index-aliases]]",
"caused_by":{"type":"not_master_exception","reason":"no longer master. source: [index-aliases]"}},"status":503}

If this error is found, execute the following commands:

kubectl -n $(kubectl get ns |awk '/arcsight/ {print $1}') scale statefulset elasticsearch-master --replicas=0
kubectl -n $(kubectl get ns |awk '/arcsight/ {print $1}') scale statefulset elasticsearch-data --replicas=0
kubectl -n $(kubectl get ns |awk '/arcsight/ {print $1}') scale statefulset elasticsearch-master --replicas={replica_count}
kubectl -n $(kubectl get ns |awk '/arcsight/ {print $1}') scale statefulset elasticsearch-data --replicas={replica_count}