Operations

Health checks

The startup, liveness and readiness routes each Telark service exposes, and how to check the whole install.

Check the whole install

kubectl get pods -n telark

Every pod should be Running and READY: a service's pod is ready only while its readiness route answers.

Probe routes

Every API service serves both routes on its http port (8080). They need no authentication.

ProbeRoute
StartupGET /api/v1/status/live
LivenessGET /api/v1/status/live
ReadinessGET /api/v1/status/ready

What "ready" means per service:

ServiceReady when
exporterRedis is reachable and its application cache has synced. Before it serves, it runs any migration steps and pins the bootstrap account; a failed step, or data two or more releases newer than the exporter, stops it before it is ready
discoveryRedis is reachable. Startup, exporter and Kubernetes problems show as degraded, not unready
authThe process is up
notifierConnected to NATS
analyzerRedis answers a ping
uinginx answers GET /healthz itself, with no upstream call. The chart points all three probes at that route (services.ui.healthCheck)

To call a route yourself:

kubectl port-forward -n telark svc/telark-discovery-service 8080:8080
curl -s localhost:8080/api/v1/status/ready

Probe timings

All services share one block, app.shared.healthCheck:

SettingStartupLivenessReadiness
initialDelaySeconds0155
periodSeconds5155
timeoutSeconds51515
failureThreshold6033

Liveness and readiness start once the startup probe passes. The exporter, notifier and auth open their port only after Redis answers, so the startup probe allows 5 minutes, for example while Redis moves to another node with its volume, before liveness can restart them.

The defaults allow for slow cold starts. Lower them only after you have measured pod start times in your cluster.

Next