Felix Health Checks

Based on a simple HealthCheck SPI interface, Felix Health Checks are used to check the health/availability of Apache Felix instances at runtime based on inputs like

• OSGi framework status
• JMX MBean attribute values
• OSGi service(s) / SCR component(s) availablility
• ... or any context information retrievable via any API

Health checks are easily extensible either by

• configuring the default HealthCheck services in bundle general checks (can be configuration or scripts, see out-of-the-box checks). For simple setups, the out-of-the-box health checks are often sufficient
• or by implementing your own HealthCheck to cater special requirements (this is done by just registering a service for this interface)

There are various ways to execute health checks - this is a good starting point to get familiar with how health checks work.

See also:

• Source code for the HealthCheck modules
• adaptTo() slides about Health Checks (from the time when they were part of Apache Sling):
  • adaptTo() 2019 - Felix Health Checks
  • adaptTo() 2014 - New features of the Sling Health Check
  • adaptTo() 2013 - Automated self-testing and health check of live Sling instances

Use cases

Generally health checks have two high level use cases:

• Load balancers/orchestration engines can query the health of an instance and decide when to route requests to it
• Operations teams checking instances for their internal state manually

The strength of Health Checks are to surface internal state for external use:

• Check that all OSGi bundles are up and running
• Verify that performance counters are in range
• Ping external systems and raise alarms if they are down
• Run smoke tests at system startup
• Check that demo content has been removed from a production system
• Check that demo accounts are disabled

The health check subsystem uses tags to select which health checks to execute so you can for example execute just the performance or security health checks once they are configured with the corresponding tags.

The out of the box health check services also allow for using them as JMX aggregators and processors, which take JMX attribute values as input and make the results accessible via JMX MBeans.

Implementing HealthChecks

Health checks can be contributed by any bundle via the provided SPI interface. It is best practice to implement a health check as part of the bundle that contains the functionality being checked.

The HealthCheck SPI interface

A HealthCheck is just an OSGi service that returns a Result.

    public interface HealthCheck {
        
        /** Execute this health check and return a {@link Result} 
         *  This is meant to execute quickly, access to external
         *  systems, for example, should be managed asynchronously.
         */
        public Result execute();
    }

A simple health check implementation might look like follows:

    public class SampleHealthCheck implements HealthCheck {

        @Override
        public Result execute() {
            FormattingResultLog log = new FormattingResultLog();
            ...
            log.info("Checking my context {}", myContextObject);
            if(myContextObject.retrieveStatus() != ...expected value...) {
                log.warn("Problem with ...");
            }
            if(myContextObject.retrieveOtherStatus() != ...expected value...) {
                log.critical("Cricital Problem with ...");
            }
            return new Result(log);
        }

    }

The Result is a simple immutable class that provides a Status via getStatus() (OK, WARN, CRITICAL etc.) and one or more log-like messages that can provide more info about what, if anything, went wrong.

Instead of using Log4j side by side with ResultLog/FormattingResultLog it is recommended to turn on autoLogging in the health check executor config in order to keep the implementation classes DRY. NOTE: This feature does not work with checks implemented against the legacy Sling interface.

Semantic meaning of health check results

In order to make health check results aggregatable in a reasonable way, it is important that result status values are used in a consistent way across different checks. When implementing custom health checks, comply to the following table:

*) The health check executor automatically turns checks that coninuosly return TEMPORARILY_UNAVAILABLE into CRITICAL after a certain grace period, see Configuring the Health Check Executor

Configuring Health Checks

HealthCheck services are created via OSGi configurations. Generic health check service properties are interpreted by the health check executor service. Custom health check service properties can be used by the health check implementation itself to configure its behaviour.

The following generic Health Check properties may be used for all checks (all service properties are optional):

Annotations to simplify configuration of custom Health Checks

To configure the defaults for the service properties above, the following annotations can be used:

// standard OSGi
@Component 
@Designate(ocd = MyCustomCheckConfig.class, factory = true)  

// to set `hc.name` and `hc.tags`
@HealthCheckService(name = "Custom Check Name", tags= {"tag1", "tag2"})

// to set `hc.async.cronExpression` or  `hc.async.intervalInSec`
@Async(cronExpression="0 0 12 1 * ?" /*, intervalInSec = 60 */)

// to set `hc.resultCacheTtlInMs`:
@ResultTTL(resultCacheTtlInMs = 10000)

// to set `hc.mbean.name`:
@HealthCheckMBean(name = "MyCustomCheck")

// to set `hc.keepNonOkResultsStickyForSec`:
@Sticky(keepNonOkResultsStickyForSec = 10)
public class MyCustomHealthCheck implements HealthCheck {
...

General purpose health checks available out-of-the-box

The following checks are contained in bundle org.apache.felix.healthcheck.generalchecks and can be activated by simple configuration:

Constraints

The JMX Attribute Check and Http Requests Check allow to check values against contraints. See the following examples:

• value string value (checks for equality)
• value = 0
• value > 0
• value < 100
• value BETWEEN 3 AND 7
• value CONTAINS a string to find in value (searches for a string to find in value in value)
• value STARTS_WITH prefix string (checks for prefix prefix string)
• value ENDS_WITH suffixStr (checks for suffix suffixStr )
• value MATCHES ^.*SomeRegEx[0-9]$ (checks for the given RegEx)
• value OLDER_THAN 100 ms (checks a time value to be older than given value, other units s, min, h, days are supported as well, e.g. OLDER_THAN 10 days)
• NOT prefix works for all expressions, e.g. NOT 20, NOT > 20, NOT BETWEEN 3 AND 7, NOT MATCHES ^.*SomeRegEx[0-9]$

Also see class org.apache.felix.hc.generalchecks.util.SimpleConstraintsChecker and its JUnit test.

Request Spec Syntax

The Http Requests Check allows to configure a list of request specs. Requests specs have two parts: Before => can be a simple URL/path with curl-syntax advanced options (e.g. setting a header with -H "Test: Test val"), after the => it is a simple response code that can be followed && MATCHES <RegEx> to match the response entity against or other matchers like HEADER, TIME or JSON.

Examples:

• /path/example.html: assumes 200 for the request to localhost:
• http://www.example.com/path/example.html => 200: explicitly checking 200 response code for full URL
• /protected/example.html => 401: protected page
• -u admin:admin /protected/example.html => 200: protected page with password (only use for non-sensitive credentials)
• /path/example.html => 200 && MATCHES <title>html title.*</title>: ensure 200 response and matching content
• /path/example.html => 200 && HEADER Content-Type MATCHES text/html.*: Checks for content type
• /path/example.json => 200 && JSON [3].prop = myval: checks JSON response's third element for property prop to equal to myval
• -H "Test: Test val" /path/example.json => 200 && JSON .city STARTS_WITH New: checks JSON response's city property to start with New
• /path/example-timing-important.html => 200 && TIME < 5 ms: Checks if the response time is smaller than specified

All constraints from Constraints can be used.

Adjustable Status Health Check

This is a health check that can be dynamically controlled via JMX bean org.apache.felix.healthcheck:type=AdjustableStatusHealthCheck. It allows to dynamically add a health check that returns WARN (operation addWarnResultForTags(String)), CRITICAL (operation addCriticalResultForTags(String)) or TEMPORARILY_UNAVAILABLE (operation addTemporarilyUnavailableResultForTags(String)) for certain tags. This is useful for testing purposes or go-live sequences. The operation reset() removes the dynamic result again.

Executing Health Checks

Health Checks can be executed via a webconsole plugin, the health check servlet or via JMX. HealthCheck services can be selected for execution based on their hc.tags multi-value service property.

The HealthCheckFilter utility accepts positive and negative tag parameters, so that osgi,-security selects all HealthCheck having the osgi tag but not the security tag, for example.

For advanced use cases it is also possible to use the API directly by using the interface org.apache.felix.hc.api.execution.HealthCheckExecutor.

Configuring the Health Check Executor

The health check executor can optionally be configured via service PID org.apache.felix.hc.core.impl.executor.HealthCheckExecutorImpl:

JMX access to health checks

Health checks that define the service property hc.mbean.name will automatically get the JMX bean with that name, the domain org.apache.felix.healthcheck and with the type HealthCheck registered. The bean provides access to the Result (status, logs, etc.)

Health Check Servlet

The health check servlet allows to query the checks via http. It provides similar features to the Web Console plugin described above, with output in HTML, JSON (plain or jsonp) and TXT (concise or verbose) formats (see HTML format rendering page for more documentation).

The Health Checks Servlet is disabled by default, to enable it create an OSGi configuration like

PID = org.apache.felix.hc.core.impl.servlet.HealthCheckExecutorServlet
servletPath = /system/health

which specifies the servlet's base path. That URL then returns an HTML page, by default with the results of all active health checks and with instructions at the end of the page about URL parameters which can be used to select specific Health Checks and control their execution and output format.

Note that by design the Health Checks Servlet doesn't do any access control by itself to ensure it can detect unhealthy states of the authentication itself. Make sure the configured path is only accessible to relevant infrastructure and operations people. Usually all /system/* paths are only accessible from a local network and not routed to the Internet.

By default the HC servlet sends the CORS header Access-Control-Allow-Origin: * to allow for client-side browser integrations. The behaviour can be configured using the OSGi config property cors.accessControlAllowOrigin (a blank value disables the header).

Webconsole plugin

If the org.apache.felix.hc.webconsole bundle is installed, a webconsole plugin at /system/console/healthcheck allows for executing health checks, optionally selected based on their tags (positive and negative selection, see the HealthCheckFilter mention above).

The DEBUG logs of health checks can optionally be displayed, and an option allows for showing only health checks that have a non-OK status.

Gogo Console

The Gogo command hc:exec can be used as follows:

hc:exec [-v] [-a] tag1,tag2
  -v verbose/debug
  -a combine tags with and logic (instead of or logic)

The command is available without installing additional bundles (it is included in the core bundle org.apache.felix.healthcheck.core)

Monitoring Health Checks

Setting up a monitor configuration

By default, health checks are only executed if explicitly triggered via one of the mechanisms as described in Executing Health Checks (servlet, web console plugin, JMX, executor API). With the HealthCheckMonitor, Health checks can be regularly monitored by configuring the the factory PID org.apache.felix.hc.core.impl.monitor.HealthCheckMonitor with the following properties:

Marker Service to depend on a health status in SCR Components

It is possible to use OSGi service references to depend on the health status of a certain
tag or name. For that to work, a HealthCheckMonitor needs to be configured for the relevant tag or name. To depend on a health status in a component, use a @Reference to one of the marker services Healthy, Unhealthy and SystemReady - this will then automatically activate/deactivate the component based on the certain health status. To activate a component only upon healthiness of a certain tag/name use the following code:

   @Reference(target="(tag=dbavail)")
   Healthy healthy;
 
   @Reference(target="(name=My Health Check)")
   Healthy healthy;

For the special tag systemready, there is a convenience marker interface available:

   @Reference
   SystemReady systemReady;

It is also possible to depend on a unhealthy state (e.g. for fallback functionality or self-healing):

   @Reference(target="(tag=dbavail)")
   Unhealthy unhealthy;

NOTE: This does not support the RFC 242 Condition Service yet - however once final the marker services will also be able to implement the Condition interface.

OSGi events for Health Check status changes and updates

OSGi events with topic org/apache/felix/health/* are sent for tags/names that a HealthCheckMonitor is configured for and if sendEvents is set to STATUS_CHANGES or ALL:

• STATUS_CHANGES notifies only of status changes with suffix /STATUS_CHANGED
• ALL sends events whenever the monitor runs, depending on status will either send the event with suffix /UPDATED or /STATUS_CHANGED

All events sent generally carry the properties executionResult, status and previousStatus.

Event listener example:

@Component(property = { EventConstants.EVENT_TOPIC + "=org/apache/felix/health/*"})
public class HealthEventHandler implements EventHandler {
    private static final Logger LOG = LoggerFactory.getLogger(HealthEventHandler.class);

    public void handleEvent(Event event) {
        LOG.info("Received event: "+event.getTopic());
        LOG.info("    previousStatus:  "+event.getProperty("previousStatus"));
        LOG.info("    status:  "+event.getProperty("status"));
        HealthCheckExecutionResult executionResult = (HealthCheckExecutionResult) event.getProperty("executionResult");
        LOG.info("    executionResult:  "+executionResult);
        LOG.info("    result:  "+executionResult.getHealthCheckResult());
    }
}

Servlet Filters

Service Unavailable Filter

For health states of the system that mean that requests can only fail it is possible to configure a Service Unavailable Filter that will cut off all requests if certain tags are in a CRITICAL or TEMPORARILY_UNAVAILABLE status. Typical usecases are startup/shutdown and deployments. Other scenarios include maintenance processes that require request processing of certain servlets to be stalled (the filter can be configured to be active on arbitrary paths). It is possible to configure a custom response text/html.

Configure the factory configuration with PID org.apache.felix.hc.core.impl.filter.ServiceUnavailableFilter with specific parameters to activate the Service Unavailable Filter:

[1] https://felix.apache.org/documentation/subprojects/apache-felix-http-service.html#filter-service-properties

[2] Choose a combination of osgi.http.whiteboard.filter.regex/ osgi.http.whiteboard.context.select wisely, e.g. osgi.http.whiteboard.context.select=(osgi.http.whiteboard.context.name=*) and osgi.http.whiteboard.filter.regex=.* would also cut off all admin paths.

Adding ad hoc results during request processing

For certain scenarios it is useful to add a health check dynamically for a specific tag durign request processing, e.g. it can be useful during deployment requests (the tag(s) being added can be queried by e.g. load balancer or Service Unavailable Filter.

To achieve this configure the factory configuration with PID org.apache.felix.hc.core.impl.filter.AdhocResultDuringRequestProcessingFilter with specific parameters: