21.11 Snapshot and Restore

Ensure to use a shared file system repository "type": "fs" to store snapshots on a shared file system. To register a shared file system repository, first mount the file system to the /var/elasticsearch/snapshot on all the dashboard nodes.

NOTE:It is highly recommended to take snapshots to avoid data loss.

21.11.1 What is a Snapshot?

A snapshot is a backup taken from a running Elasticsearch cluster. Snapshots are incremental. Each snapshot only stores data that is not part of an earlier snapshot. This enables you to take frequent snapshots with minimal overhead. Snapshot lifecycle management (SLM) is used to automatically take and manage snapshots.

21.11.2 Setting up a Snapshot Policy

Run the ELKBackup.sh located in /opt/novell/nam/scripts/ELKBackup.sh. The execution of this script ends up in creating a snapshot schedule, which will take snapshots every day at 1.30 AM. You can change this default schedule time using ELKBackup.sh file. For example, you can change it to “schedule”: "0 30 1 * * ?" .

NOTE:Setting up a snapshot policy does not take a snapshot. This will only create a scheduler. To take instant snapshot, refer to Section 21.11.3, Executing the Snapshot Policy Manually.

21.11.3 Executing the Snapshot Policy Manually

You can manually execute a snapshot policy to take an instant snapshot. This is useful for taking snapshots before making a configuration change, upgrading, or to test a new policy. Manually executing a policy does not affect the configured schedule. Instead of waiting for the policy to run, use SLM to take a snapshot using the configuration instantly instead of waiting for scheduled time using the command curl -X PUT "http://{elastic_ip}:9200/_slm/policy/nightly-snapshots/_execute?pretty".

NOTE:The elastic_ip can be either the IP of the node in case you are in dashboard cluster or 127.0.0.1, in case you are not using a cluster configuration.

21.11.4 Getting Status of the Snapshot Policy

After scheduling the nightly-snapshots policy to run or nightly-snapshots run as per schedule, you can retrieve the policy to get success or failure information. You can ensure periodically that the snapshots are successfully executed as per schedule using command: curl -X GET http://{elastic_ip}:9200/_slm/policy/nightly-snapshots?human&pretty.

NOTE:The elastic_ip can be either the IP of the node in case you are in dashboard cluster or 127.0.0.1, in case you are not using cluster configuration.

21.11.5 Deleting a Snapshot Policy

If you do not require the policy anymore, you can delete using the command: curl -X DELETE "http://{elastic_ip}:9200/_slm/policy/ nightly-snapshots.

NOTE:The elastic_ip can be either the IP of the node in case you are in dashboard cluster or 127.0.0.1, in case you are not using cluster configuration.

NOTE:Note that this command will not delete snapshots taken, this will only delete the snapshot policy.

21.11.6 Deleting Individual Snapshot Policy

You can retrieve the snapshots using command curl -X GET " http://{elastic_ip}:9200/_snapshot/as_repo/_all?pretty". The output similar to the following is displayed:

{
  "snapshots": [
    {
      "snapshot": "snapshot_2",
      "uuid": "vdRctLCxSketdKb54xw67g",
      "version_id": <version_id>,
      "version": <version>,
      "indices": [],
      "data_streams": [],
      "include_global_state": true,
      "state": "SUCCESS",
      "start_time": "2020-07-06T21:55:18.129Z",
      "start_time_in_millis": 1593093628850,
      "end_time": "2020-07-06T21:55:18.876Z",
      "end_time_in_millis": 1593094752018,
      "duration_in_millis": 0,
      "failures": [],
      "shards": {
        "total": 0,
        "failed": 0,
        "successful": 0
      }
    }
  ]
}

You can get the snapshot name using the command curl -X GET " http://{elastic_ip}:9200/_snapshot/as_repo/_all?pretty" and use it to DELETE API curl -X DELETE " http://{elastic_ip}:9200/_snapshot/as_repo/ <SNAPSHOT_NAME>?pretty". When a snapshot is deleted from a repository, Elasticsearch deletes all files associated with the snapshot that are not in use by other snapshots. If the delete snapshot operation starts while the snapshot is being created, the snapshot process stops and all files created as part of this process are removed. Use the delete snapshot API to cancel long running snapshot operations that might have been started by mistake.

To delete multiple snapshots from a repository, separate snapshot names by commas use, curl -X DELETE " http://{elastic_ip}:9200/_snapshot/as_repo/snapshot_2,snapshot_3?pretty".

21.11.7 Restoring the Snapshot

You require Python language in the target box to execute the restore script. Ensure that the restore script brings up a single node cluster and you will also need to import or install new nodes later. In case all nodes are inactive and you need to restore the data that from a snapshot, mount the shard directory to /var/elasticsearch/snapshot and run the command "sh /opt/novell/nam/scripts/ELKRestore.sh".

NOTE:If you are using RHEL, ensure you install python3 using yum install python3 command.

After restoring the snapshot successfully, run the following command to create template for Elasticsearch:

curl -XPUT "http://{elastic_ip}:9200/_template/dashboard" -H 'Content-Type: application/json' -d'{ "template" : "dashboard-*", "settings" : {"number_of_shards" : 1 }, "mappings": {"properties": {"@timestamp": {"type": "date"},"createDate": {"type": "date"},"@version": {"type": "keyword"},"appdata": {     "type":"nested","include_in_parent":true,"properties": { "accessType": { "type": "text", "norms": false, "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "appName": { "type": "text", "norms": false, "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "eventTime": { "type": "text", "norms": false, "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }}},"browserName": {"type": "text","norms": false,"fields": { "keyword": { "type": "keyword", "ignore_above": 256 }}},"healthClusterID": {"type": "text","norms": false,"fields": { "keyword": { "type": "keyword", "ignore_above": 256 }}},"contractName": {"type": "text","norms": false,"fields": { "keyword": { "type": "keyword", "ignore_above": 256 }}},"countryCode": {"type": "text","norms": false,"fields": { "keyword": { "type": "keyword", "ignore_above": 256 }}},"healthClusterName": {"type": "text","norms": false,"fields": { "keyword": { "type": "keyword", "ignore_above": 256 }}},"healthDeviceID": {"type": "text","norms": false,"fields": { "keyword": { "type": "keyword", "ignore_above": 256 }}},"deviceName": {"type": "text","norms": false,"fields": { "keyword": { "type": "keyword", "ignore_above": 256 }}},"healthDeviceName": {"type": "text","norms": false,"fields": { "keyword": { "type": "keyword", "ignore_above": 256 }}},"health": {"type": "text","norms": false,"fields": { "keyword": { "type": "keyword", "ignore_above": 256 }}},"eventType": {"type": "text","norms": false,"fields": { "keyword": { "type": "keyword", "ignore_above": 256 }}},"riskLevel": {"type": "text","norms": false,"fields": { "keyword": { "type": "keyword", "ignore_above": 256 }}},"eventID": {"type": "text","norms": false,"fields": { "keyword": { "type": "keyword", "ignore_above": 256 }}},"failedCount": {"type": "long"},"geoIP": {"dynamic": "true","properties": { "ip": { "type": "ip" }, "latitude": { "type": "half_float" }, "location": { "type": "geo_point" }, "longitude": { "type": "half_float" }}},"loginCount": {"type": "long"}, "usersCount": {"type": "long"}, "sessionscount": {"type": "long"}, "agRequest": {"type": "long"},"cacheUtil": {"type": "long"},"sessionID": {"type": "text","norms": false,"fields": { "keyword": { "type": "keyword", "ignore_above": 256 }}},"sourceIP": {"type": "text","norms": false,"fields": { "keyword": { "type": "keyword", "ignore_above": 256 }}},"timestamp": {"type": "text","norms": false,"fields": { "keyword": { "type": "keyword", "ignore_above": 256 }}},"userName": {"type": "text","norms": false,"fields": { "keyword": { "type": "keyword", "ignore_above": 256 }}} } },"aliases" : {"historic" : {},"realtime":{"filter" : {"range" : { "createDate": { "gt" : "now-7d" }}} }}}'

NOTE:The elastic_ip can be either the IP of the node in case you are in dashboard cluster or 127.0.0.1, in case you are not using a cluster configuration.