Introducing go-health

Today, InVision is excited to introduce go-health — the scalable health checking library for Golang services. go-health is designed to help developers set up Kubernetes health check probes that are safe from DoS attacks and scalable for large deployments.

The challenge

Kubernetes has two kinds of health check probes: liveness probes and readiness probes. Readiness probes signal to the kubelet agent that a pod can begin handling traffic. When a readiness probe is defined, traffic will only reach a pod after a 2xx status response is received from the readiness probe endpoint. Liveness probes work similarly to readiness probes but they help your service communicate that a running pod has deadlocked or become unresponsive. Kubernetes will restart a pod after it fails to receive a positive response from a liveness probe.

When we wrote our first health checks, InVision services used the same HTTP endpoint for both liveness and readiness probes. This approach can work well for small services and simple health checks. However, as InVision grew, we found ourselves creating longer, more complex health checks that put strain on downstream dependencies. A typical request to a /healthcheck endpoint led to a database query, network calls to other health check endpoints and test API calls to dependent services. These checks were often performed synchronously, in which case the /healthcheck endpoint could not return a response until they completed.

go-health to the rescue

Daniel Selans, Tatsuro Alpert, and Steve High on the InVision team developed go-health to improve the reliability of our core services - providing a standard, reliable way for all services to verify their own state. InVision now uses go-health in most of our production Go services. go-health helped solve our problem with complex health checks by performing them asynchronously in the background. When a request is made for a health check, the last known state is returned.

This approach helps protect our health check dependencies from stress caused by restarting a large deployment of pods. It also insulates us from a DoS attack, should a malicious actor gain access to the health check endpoint. go-health comes with three types of checkers built in: HTTP checks, SQL queries and Redis queries. You can also create your own checker by implementing the ICheckable interface.

h := health.New()
goodTestURL, _ := url.Parse("https://invisionapp.com")
goodHTTPCheck, _ := checkers.NewHTTP(&checkers.HTTPConfig{
        URL: goodTestURL,
})
h.AddChecks([]*health.Config{
        {
                Name:     "good-check",
                Checker:  goodHTTPCheck,
                Interval: time.Duration(2) * time.Second,
                Fatal:    true,
        },
})
h.Start()

In the example above, the health check consists of a GET HTTP request to InVision’s homepage. An HTTP dependency check can send a GET, HEAD, POST or PUT request, with or without a request payload. You can configure the health check to fail based on a bad status code or a timeout (which is the default) or based on the contents of the response body.

This library comes with built-in checkers for two types of datastores: SQL and Redis. You can use go-health to ping the specified datastore and verify that it is reachable by your service, or execute a read or write operation to test that the datastore can be accessed.

Using go-health, you can also register status listeners. Status listeners will call a function when a dependency begins to fail or when a dependency recovers from failure. Listeners are a great place to add logging functionality to your health check.

Adding go-health to our toolkit did not completely solve InVision’s challenges with complex health checks. We also created guidelines to reduce the number and cost of the dependency checks within InVision’s health check probes. In particular, InVision engineers cut back on dependency checks that were performed in liveness probes and moved them to the readiness probe. Readiness probes are usually the best place to verify that your service can connect to datastores and other APIs that it requires to do its job.

Get started

go-health is available on GitHub under the MIT license. Take a look at the code examples, try it out in your own Golang services and let us know what you think. Have questions about the library? Leave us a comment below!