Transformation Hub Liveness Probes

A liveness probe is a Kubernetes feature that can be configured to detect problematic pods. Once detected, Kubernetes will take action to restart a problematic pod. Liveness probes help ensure higher availability of pods as well as a more robust cluster environment. Consult the Kubernetes documentation for a more detailed explanation of liveness probes. Transformation Hub supports these liveness probe types:

• TCP/IP port-socket connection
• HTTP request
• Log scanning

Each container or pod supports the listed liveness probes, with their default parameter values shown.

Container/Pod	Probe	initialDelaySeconds	periodSeconds	timeoutSeconds	failureThreshold
Kafka	tcp socket :9092 and log scanning	240	60	30	3
Zookeeper	tcp socket :2181 and log scanning	240	60	30	3
Web Service	https GET :8080 and log scanning	240	300	30	3
Schema Registry	https GET :8081 config and log scanning	240	300	30	3
Kafka Manager	https GET :9000 and log scanning	240	600	30	3
Routing/Enrichment Processor	log scanning	240	60	30	3
C2AV (CEF-to-Avro) Processor	log scanning	240	60	30	3

Probe parameters are defined as follows:

Parameter	Definition
initialDelaySeconds	Number of seconds after the container has started before liveness probes are initiated. The first probe execution after startup is not until initialDelaySeconds + periodSeconds.
periodSeconds	How often to perform the probe.
timeoutSeconds	Number of seconds after which the probe times out.
failureThreshold	When a Pod starts and the probe fails, Kubernetes will try failureThreshold times before giving up and restarting the pod.

Managing Liveness Probes

To check if a pod has a liveness probe configured:

1. Run:
   kubectl -n <namespace> describe pod <podname>
2. Review the output. Look (or grep) for the line starting with the string Liveness...This will show some of the probe's configuration.

To check for probe failures:

1. Run:
   kubectl get pods --all-namespaces
2. If any pod shows 1 or more restarts, run:
   kubectl -n <namespace> describe pod <podname>
3. Review any list of events at the end of the output. Liveness probe failures will be shown here.

Configuring Liveness Probes

The default values for liveness probes can be overriden by changing the values of the appropriate properties on the Configuration page.

1. Log in to the CDF Management Portal.
2. Click Administration.
3. Click the ... (Browse) icon to the right of the main window.
4. From the drop-down, click Reconfigure. The post-deployment settings page is displayed.
5. Browse the configuration properties list to find the desired property, and specify the new value.

5. Click Save.

Configuring Log Scanning Liveness Probes

Log scanning probes scan the application's output for a match to a configured pattern, such as a known error message. If the pattern is found, the pod is restarted.

In addition to the four parameters described in the table above, log scanning probes have two additional properties:

literal	A literal expression for matching against the application's log output.
regex	A regular expression for matching against the application's log output.

• The literal property specifies a literal (exact match) search string. If the value matches a portion of the log text, the liveness probe, on its next periodic check, will report a failure and restart the pod.
• The regex property is similar, except that a regular expression can be specified for the match. This regex must conform to Java regex rules. To specify a regex escape value within the regex, use 2 backslashes to escape it (\\).
• Multiple search patterns can be specified per property, separated by 4 vertical bars (||||). A match on any of the patterns will trigger the probe failure.
• There are no default values for these parameters. Log scanning is disabled in the default configuration.
• Matching across multiple rows is not supported. The match must occur on one log line.

• For example, to restart the CEF-to-Avro Routing Stream Processor pod when the value, Setting stream threads to d (where d could be any single digit), is found in the log, change the configuration property "CEF-to-Avro Routing Stream Processor liveness probes regular expression" to the following value .

    Setting stream threads to \\d

Verification

To verify that log scanning is configured as intended, review the pod's log and look for entries containing InputStreamScanner.

For example, to view the c2av-processor pod log, run:

   kubectl -n <namespace> logs th-c2av-processor-0 | more

For the previous property example, the corresponding log line would be:

   InputStreamScanner: Will scan for RegEx pattern [Setting stream threads to \d]