Resource monitoring allows OES Cluster Services to detect a the resource failure independently of its ability to detect node failures. Monitoring is disabled by default. It is enabled separately for each cluster resource.
When you enable resource monitoring, you must specify a polling interval, a failure rate, a failure action, and a timeout value. These settings control how error conditions are resolved for the resource.
The monitor script runs at a frequency specified by the polling interval. By default, it runs every minute when the resource is online. You can specify the polling interval in minutes or seconds. The polling interval applies only to a given resource.
The failure rate is the maximum number of failures () detected by the monitor script during a specified amount of time ( ).
A failure action is initiated when the resource monitor detects that the resource fails more times than the maximum number of local failures allowed to occur during the specified time interval. For failures that occur before it exceeds the maximum, Cluster Services automatically attempts to unload and load the resource. The progress and output of executing a monitor script are appended to /var/opt/novell/log/ncs/<resource_name>.monitor.out file.
For example, if you set the failure rate to 3 failures in 10 minutes, the failure action is initiated if it fails 4 times in a 10 minute period. For the first 3 failures, Cluster Services automatically attempts to unload and load the resource.
Theindicates whether you want the resource to be set to a comatose state, to migrate to another server, or to reboot the hosting node (without synchronizing or unmounting the disks) if a failure action initiates. With resource monitoring, the , , and Modes have no effect on where the resource migrates. This means that a resource that has been migrated by the resource monitoring failure action does not migrate back (fail back) to the node it migrated from unless you manually migrate it back.
Set Resources as Comatose: (Default) If the failure action initiates, the resource is placed in a comatose state. Administrator action is required to take the resource offline, resolve the issue, and bring it online again on the same or different node.
Migrate the Resource Based on the Preferred Nodes List: If the failure action initiates and the resource is on its most preferred node, the resource migrates to the next available node in itslist, which you previously ordered according to your failover order preferences. The resource is not automatically failed back to the original node. Administrator action is required to cluster migrate the resource to the node, as desired. Each time a failure action triggers a failover, the resource migrates to a different node, according to the order in its list and availability of the nodes.
Reboot the Hosting Node without Syncing or Unmounting Disks: If the failure action initiates, each of the resources on the hosting node will fail over to the next available node in itslist because of the reboot. All resources on the node are failed over. This is a hard reboot, not a graceful one. The reboot option is normally used only for a mission-critical cluster resource that must remain available. The resources are not automatically failed back to the original node. Administrator action is required to cluster migrate them back to the node, as desired.
The timeout value determines how much time the script is given to complete. If the script does not complete within the specified time, the configured failure action is initiated. Cluster Services marks the process as failed right after the defined timeout expires, but it must wait for the process to conclude before it can start other resource operations.
The timeout value is applied only when the resource is migrated to another node. It is not used during resource online/offline procedures.
The monitor script runs at the frequency you specify as the polling interval.
There are two conditions that trigger a response by OES Cluster Services:
Cluster Services tallies the error occurrence, compares it to the configured failure rate, then does one of the following:
Total errors in the interval are less than or equal to the Maximum Local Failures: Cluster Services tries to resolve the error by offlining the resource, then onlining the resource.
If this problem resolution effort fails, Cluster Services goes to Step 4 immediately regardless of the failure rate condition at that time.
Total errors in the interval are more than the Maximum Local Failures: Go to Step 4.
Cluster Services initiates the configured failure action. Possible actions are:
Puts the resource in a comatose state
Migrates the resource to another server
Reboots the hosting node (without synchronizing or unmounting the disks)
The resource monitoring function allows you to monitor the health of a specified resource by using a script that you create or customize. If you want OES Cluster Services to check the health status of a resource, you must enable and configure resource monitoring for that resource. Enabling resource monitoring requires you to specify a polling interval, a failure rate, a failure action, and a timeout value.
If you are creating a new cluster resource, the Monitor Script page should already be displayed. You can start with Step 6.
In iManager, select> .
Select the cluster that you want to manage.
If the cluster does not appear in your list, add the cluster to your list as described in Section 8.2, Setting Up a Personalized List of Clusters to Manage.
Click the cluster resource to open its Properties page.
You can also select the check box next to the resource, then click.
On the Properties page, click thetab.
Select thecheck box to enable resource monitoring for the selected resource.
Resource monitoring is disabled by default.
For the polling interval, specify how often you want the resource monitor script for this resource to run.
You can specify the value in minutes or seconds.
Specify the number of failures () for the specified amount of time ( ).
See Failure Rate.
Specify theby indicating whether you want the resource to be set to a comatose state, to migrate to another server, or to reboot the hosting node (without synchronizing or unmounting the disks) if a failure action initiates. The reboot option is normally used only for a mission-critical cluster resource that must remain available.
See Failure Action.
Click thetab, then click the link.
Edit or add the necessary commands to the script to monitor the resource on the server.
The resource templates included with Cluster Services for Linux include resource monitor scripts that you can customize.
You also need to personalize the script by replacing variables with actual values for your specific configuration, such as the mount point, IP address, volume group name, file system type, and mount device.
You can use the same commands that would be used at the Linux terminal console. For example, see Section 10.7.3, Monitoring Services that Are Critical to a Resource.
Specify thevalue, then click to save the script.
The timeout value determines how much time the script is given to complete. If the script does not complete within the specified time, the failure action you chose in Step 9 initiates.
Do one of the following:
If you are configuring a new resource, click Section 10.9.2, Setting the Start, Failover, and Failback Modes for a Resource., then continue with
Clickto save your changes.
Changes for a resource’s properties are not applied while the resource is loaded or running on a server. Apply the updated script by taking the resource offline and then bringing it online on the same node. Alternatively, after the updated scripts have been synchronized from eDirectory to the source and destination nodes, the updated scripts are used automatically on system failover or cluster migration. For more information, see Section 10.8, Applying Updated Resource Scripts by Offline/Offline, Failover, and Migration.
In addition to monitoring the clustered service or storage objects, a resource monitor script can be used to monitor the status of services that are critical to the resource, such as Linux User Management (LUM), eDirectory, or other services.
IMPORTANT:NCS provides the ability to monitor the status of the eDirectory daemon (ndsd) at the NCS level. It is disabled by default. The monitoring can be set independently on each node. It runs whenever NCS is running on the node. See Section 8.8, Configuring NCS to Monitor the eDirectory Daemon (ndsd).
If you enable NDSD monitoring at the NCS level, we recommend that you remove (or comment out) the eDirectory status check in individual monitor scripts to avoid excessive checking.
The resource monitor script runs only on the cluster server where the cluster resource is currently online. The script does not monitor the critical services on its assigned cluster server when the resource is offline. The monitor script does not monitor critical services for any other cluster node. Each monitor script acts independently, so be aware of the potential traffic you might generate if you include a status check in multiple monitoring scripts.
You can use the systemctl status namcd.service command to monitor whether the Linux User Management service is running. Add the following command to the resource monitor script:
# (optional) status of the Linux User Management service exit_on_error systemctl status namcd.service
Alternatively, you use the namcd status command to monitor whether the Linux User Management service is running and to automatically restart namcd if its daemon is not running. However, namcd creates messages in /var/log/messages with each check. Add the following command to the resource monitor script:
# (optional) status of the LUM service and restart if it is not loaded or running exit_on_error namcd status
You can monitor the state of the eDirectory service in an individual monitoring script by adding the following command.
# (optional) status of the eDirectory service exit_on_error systemctl status ndsd.service
For OES 11 SP2 and later, you can alternatively monitor NDSD at the NCS level on each node. See Section 8.8, Configuring NCS to Monitor the eDirectory Daemon (ndsd).
The resource templates included with OES Cluster Services for Linux include resource monitor scripts that you can customize.
Example monitor scripts are available in the following sections:
Clustered NSS pool:
Clustered LVM volume group: