(Conditional – Intelligence) Resolving Issues before Running Analytics on Demand

If the Elasticsearch recovery actions in Step 8 of the EKS upgrade procedure failed for the Intelligence pods, you must delete the unassigned shards that might imede the recovery process.

• Ensuring that Analytics Can Be Run on Demand

• Applying Workarounds for Two Issues

Ensuring that Analytics Can Be Run on Demand

1. Run the following command, being sure to replace the <password> value with yours:

   kubectl exec -it -n $(kubectl get ns |awk '/arcsight/ {print $1}') elasticsearch-master-0 -c  elasticsearch -- curl -k -XGET https://elastic:<password>@localhost:9200/_cat/shards | grep UNASSIGNED | awk {'print $1'} | xargs -i curl -k -XDELETE 'https://elastic:<password>@localhost:9200/{}'

2. To monitor the delete progress, run the following command:

   kubectl exec -n $(kubectl get ns |awk '/arcsight/ {print $1}') elasticsearch-master-0 -c  elasticsearch -it bash curl -k -XGET 'https://elastic:<password>@localhost:9200/_cat/health?v=true'

   Example command and output:

   curl -k -XGET 'https://elastic:changeme@localhost:9200/_cat/health?v=true'

   epoch      timestamp cluster  status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
   1671118161 15:29:21  interset green           6         3   1128 583    0    0        0             0                  -                100.0%	
   

   In the example above, the value has reached a 100% and the status is green.

3. If the result of the command is a yellow status, and a value under 100%, apply the following procedure:

   1. Login with system-admin role to the interset UI https://<CLUSTER FQDN>/interset

   2. Click on the gear icon on the top right corner and select Search Manager.

   3. Click on the Job History list box.

   4. Select Submit a Job.

   5. Click on the Job type list box and select Restore.

   6. Enter 0 for the Customer to apply Snapshot to field.

   7. Click the SUBMIT JOB button.

4. To verify the job status, complete the following steps:

   1. On the Job history page, check the Snapshot job ID status.

   2. Click the REFRESH button until the status becomes either COMPLETED_SUCCESS or COMPLETED_FAILED.

   3. If the final status is COMPLETED_FAILED, execute the following commands to monitor the health:

      kubectl exec -n $(kubectl get ns |awk '/arcsight/ {print $1}') elasticsearch-master-0 -c  elasticsearch -it curl -k -XGET 'https://elastic:<password>@localhost:9200/_cat/health?v=true'

      Example command and output:

      curl -k -XGET 'https://elastic:changeme@localhost:9200/_cat/health?v=true'

      epoch      timestamp cluster  status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
      1671118161 15:29:21  interset green           6         3   1128 583    0    0        0             0                  -                100.0%	
      

      In the example above, the value has reached a 100% and the status is green.

   4. If the result of the command is a yellow status, and a value under 100%, you could wait 5 minutes and then repeat the command until the green status is achieved.

5. Scale up logstash using the following command:

   kubectl -n $(kubectl get ns |awk '/arcsight/ {print $1}') scale statefulset interset-logstash --replicas=<replica count>

   Where you must replace the <replica count> value.

6. To prepare for running Analytics on Demand, complete the following steps:

   1. To ensure that Analytics is not currently running, run the following command on any worker or master node to confirm:

      ANALYTICS_POD=$(kubectl get pods -n $(kubectl get namespaces | awk '/arcsight/ {print $1}') |awk '/interset-analytics/ {print $1}') ; kubectl exec -it -n $(kubectl get namespaces | awk '/arcsight/ {print $1}') ${ANALYTICS_POD} -c interset-analytics -- ls -l /tmp/interset_lock/

   2. If Analytics is currently running, the output of this command will be:

      => tenant_0.lock

      In this case, wait for Analytics to complete running before proceeding. The command can be run periodically to monitor the status of Analytics.

   3. If the previous Analytics execution failed, check whether the properties in the Intelligence tab are set correctly. If this does not solve the issue, contact Micro Focus Customer Support.

7. To run Analytics on Demand and make sure that Analytics will resume after the EKS upgrade, complete the following steps:

   1. Launch a terminal session and log in to the NFS node.

   2. Navigate to the following directory:

      cd <NFSVolume>/interset/analytics

   3. (Conditional) Delete the blackhawk_down file, if present. This is an error file and it is generated if the previous Analytics execution fails.

      rm blackhawk_down

   4. When prompted whether you want to delete the file, enter yes.

   5. Execute the following command to delete the latest AnalyticsStarted.mk and AnalyticsCompleted files:

      rm -rf AnalyticsStarted-0-<Today's_date>.mk AnalyticsCompleted-0-<Today's_date>.mk

      1. When prompted whether you want to delete the files, enter yes.

         After 30 seconds of deletion of the files, Analytics is triggered automatically.

         You can monitor the delete progress using the following command:

         kubectl exec -n $(kubectl get ns |awk '/arcsight/ {print $1}') elasticsearch-master-0 -c  elasticsearch -it bash curl -k -XGET 'https://elastic:<password>@localhost:9200/_cat/health?v=true'

Applying Workarounds for Two Issues

If you encounter any of the following issues, perform the workarounds.

Issue: If Analytics were to fail after the EKS upgrade, there are two things you can check to remedy it:

• Review the logs for the following error:

  Suppressed: org.elasticsearch.client.ResponseException: method [POST], 
  host [https://elasticsearch-svc:9200], URI [/_aliases?master_timeout=30s&timeout=30s], 
  status line [HTTP/1.1 503 Service Unavailable]{"error":{"root_cause":[{"type":"not_master_exception",
  "reason":"no longer master. source: [index-aliases]"}],"type":"master_not_discovered_exception",
  "reason":"NotMasterException[no longer master. source: [index-aliases]]",
  "caused_by":{"type":"not_master_exception","reason":"no longer master. source: [index-aliases]"}},"status":503}

• If this error is found, execute the following commands:

  kubectl -n $(kubectl get ns |awk '/arcsight/ {print $1}') scale statefulset elasticsearch-master --replicas=0

  kubectl -n $(kubectl get ns |awk '/arcsight/ {print $1}') scale statefulset elasticsearch-data --replicas=0

  kubectl -n $(kubectl get ns |awk '/arcsight/ {print $1}') scale statefulset elasticsearch-master --replicas={replica_count}

  kubectl -n $(kubectl get ns |awk '/arcsight/ {print $1}') scale statefulset elasticsearch-data --replicas={replica_count}

Issue: If Analytics fails after the EKS upgrade because the HDFS namenode has entered safe mode.

Workaround: Perform the following steps:

1. Execute the following command to restart HDFS pods:

   kubectl delete pods -n $(kubectl get ns |awk '/arcsight/ {print $1}')  $(kubectl get pods -n $(kubectl get ns |awk '/arcsight/ {print $1}') -o wide |  grep "hdfs-" | cut -d ' ' -f1)

2. Execute the following command:

   kubectl exec -n $(kubectl get ns |awk '/arcsight/ {print $1}') hdfs-namenode-0 -c  hdfs-namenode -it bash

   hdfs dfsadmin -safemode leave

3. Run Analytics on demand.