Event monitors (seemonitor.exe processes) constantly monitor a running Enterprise Server for .NET system. If a SEP pool (w3wp.exe or seesep.exe) process dies for any reason (e.g. due to a StackOverflowException or a user decides to "Stop Debugging"), the event monitor running on the same machine will detect that and instigate the recovery of any orphaned running tasks from that process. All resources associated with an orphaned task are released (e.g. ENQ locks, initiators, or printers). See Local SEP Pool Recovery below.
Event monitors may also be configured to monitor the health of SEP pools running on other machines in an Enterprise Server for .NET scale-out. When configured in this way, an event monitor will check whether a SEP pool is still active as part of its periodic system health check, and if it deems that the SEP pool is no longer alive, it will instigate the recovery of any orphaned running tasks associated with it. See Remote SEP Pool Recovery below.
In rare circumstances it is possible that a request gets dispatched, but does not get associated with a SEP for execution (e.g. communication failure between dispatcher and SEP pool processes). Each time this happens, an available SEP is effectively allocated, until a point is reached where all SEPs are assigned and no further requests can get processed by the region. Such requests now get detected as part of the event monitor system health checking and re-dispatched to an available SEP. See Dispatched Request Recovery below.
An event monitor running on the same machine as a listener (seelistener.exe process) will also periodically check that the EZ listener associated with a region is still responsive. If it is not, the region's EZ listener is reset to allow subsequent EZ socket operations to be routed to a different listener process.
The Registered Event Monitors and Registered SEP Pools view in the Enterprise Server for .NET Administration UI are provided to allow the manual monitoring of the health of event monitors and SEP pools in the system respectively, and take action if an event monitor or SEP pool is deemed to be no longer active.
SEEEM0058W SEP pool process (Host: MY-MACHINE; PID:3868) terminated with orphaned tasks SEEEP0173I SEP pool process (Host: MY-MACHINE; PID:3868) recovery started SEEEP0174I SEP pool process (Host: MY-MACHINE; PID:3868) recovery completedCorresponding Windows event log entries will also be created.
SEP pool processes periodically emit a heartbeat to indicate that they are still alive. When a SEP pool process stops emitting heartbeats it can be assumed that the machine on which it is running has either died, been powered down or has lost network connectivity. Any tasks that were running on a SEP pool no longer heartbeating will have been orphaned (e.g. a job previously running in the SEP pool would still appear Active in the Enterprise Server for .NET Administration UI's Spool view and hold resource locks, but no application would actually be running).
<MicroFocus.SEE.Monitor> <general> ... <add key="EM.HealthCheck.Interval" value="20" /> <add key="EM.SepPool.Recovery" value="true" /> <add key="EM.SepPool.Recovery.HeartbeatElapsed" value="10" /> ... </general> </MicroFocus.SEE.Monitor>Messages similar to those for a local SEP pool recovery get written to the region console and Windows event log when an event monitor automatically recovers a SEP pool.
<MicroFocus.SEE.Monitor> <general> ... <add key="EM.ActiveRequest.Elapsed" value="15" /> ... </general> </MicroFocus.SEE.Monitor>