(Conditional – Intelligence) Running Analytics on Demand after Recovering Elasticsearch

If the Elasticsearch recovery actions in Step 8 of the EKS upgrade procedure succeeded for the Intelligence pods, complete the following procedure.

Ensuring that Analytics Can Be Run on Demand
Applying Workarounds for Two Issues

Ensuring that Analytics Can Be Run on Demand

Scale up logstash using the following command:

kubectl -n $(kubectl get ns |awk '/arcsight/ {print $1}') scale statefulset interset-logstash --replicas=<replica count>

Where you must replace the <replica count> value.

To prepare for running Analytics on Demand, complete the following steps:
1. To verify that Analytics is not currently running. run the following command on any worker or master node to confirm:
```
kubectl -n $(kubectl get namespaces | awk '{print $1}' | grep arcsight-installer) exec -it $(kubectl)
```
2. (Conditional) If this command displays a lock file, then Analytics is currently running. Wait for Analytics to complete before proceeding . You can run the command periodically to monitor the state of Analytics.
3. (Conditional) If the previous Analytics execution failed, check whether the properties in the Intelligence tab are set correctly. If this does not solve the issue, contact Micro Focus Customer Support.
To run Analytics on Demand and make sure that Analytics will resume after the EKS upgrade, complete the following steps:
1. Launch a terminal session and log in to the NFS node.
2. Navigate to the following directory:
  
  cd <NFSVolume>/interset/analytics
3. (Conditional) Delete the blackhawk_down file, if present. This is an error file and it is generated if the previous Analytics execution fails.
  
  rm blackhawk_down
4. When prompted whether you want to delete the file, enter yes.
5. Execute the following command to delete the latest AnalyticsStarted.mk and AnalyticsCompleted files:
  
  rm -rf AnalyticsStarted-0-<Today's_date>.mk AnalyticsCompleted-0-<Today's_date>.mk
6. When prompted whether you want to delete the files, enter yes.
  
  After 30 seconds of deletion of the files, Analytics is triggered automatically.
7. (Conditional) If Analytics fails after the EKS upgrade, perform the workarounds.

Applying Workarounds for Two Issues

If you encounter any of the following issues, perform the workarounds.

Issue: If Analytics were to fail after the EKS upgrade, there are two things you can check to remedy it.

Review the logs for the following error:

Suppressed: org.elasticsearch.client.ResponseException: method [POST], 
host [https://elasticsearch-svc:9200], URI [/_aliases?master_timeout=30s&timeout=30s], 
status line [HTTP/1.1 503 Service Unavailable]{"error":{"root_cause":[{"type":"not_master_exception",
"reason":"no longer master. source: [index-aliases]"}],"type":"master_not_discovered_exception",
"reason":"NotMasterException[no longer master. source: [index-aliases]]",
"caused_by":{"type":"not_master_exception","reason":"no longer master. source: [index-aliases]"}},"status":503}

If this error is found, execute the following commands:

kubectl -n $(kubectl get ns |awk '/arcsight/ {print $1}') scale statefulset elasticsearch-master --replicas=0

kubectl -n $(kubectl get ns |awk '/arcsight/ {print $1}') scale statefulset elasticsearch-data --replicas=0

kubectl -n $(kubectl get ns |awk '/arcsight/ {print $1}') scale statefulset elasticsearch-master --replicas={replica_count}

kubectl -n $(kubectl get ns |awk '/arcsight/ {print $1}') scale statefulset elasticsearch-data --replicas={replica_count}

Issue: If Analytics fails after the EKS upgrade because the HDFS namenode has entered safe mode.

Workaround: Complete the following steps:

To restart HDFS pods, execute the following command:

kubectl delete pods -n $(kubectl get ns |awk '/arcsight/ {print $1}')  $(kubectl get pods -n $(kubectl get ns |awk '/arcsight/ {print $1}') -o wide |  grep "hdfs-" | cut -d ' ' -f1)

Execute the following command:

kubectl exec -n $(kubectl get ns |awk '/arcsight/ {print $1}') hdfs-namenode-0 -c  hdfs-namenode -it bash

hdfs dfsadmin -safemode leave

Run Analytics on demand.