Operations

Troubleshooting

Common problems, how to confirm the cause, and how to fix them.

Most checks below need the discovery leader. Find it with:

kubectl exec -n telark telark-redis-master-0 -- sh -c 'REDISCLI_AUTH="$(cat "$REDIS_PASSWORD_FILE")" redis-cli GET election:prewarm'

If your problem is not here, open an issue with the symptom and the relevant kubectl logs and kubectl get output.

The chart refuses to render

Error mentionsFix
bootstrap.adminAdd --set app.auth.bootstrap.admin=test@example.com.
services.exporter.replicas, ReadWriteManySet app.persistence.storageClass to a ReadWriteMany class, or keep one exporter replica.
storageClass, does not existName a class that kubectl get storageclass lists.
passkey id or originWith an Ingress or Gateway, set app.auth.passkey.id and app.auth.passkey.origin.
failOpenSet kyverno.features.forceFailurePolicyIgnore.enabled, the key Kyverno reads, and drop the deprecated app.kyverno.failOpen.
Redis must require a passwordKeep redis.auth.enabled=true, leave redis.auth.password empty and redis.auth.existingSecret at its default, or name your own Secret.
never shrinks a claimKeep the Ollama volume's size on every upgrade with the --set ollama.persistentVolume.size=... it names.
volume size of an existing StatefulSetRun the kubectl delete statefulset ... --cascade=orphan command it prints, then upgrade again, or pass the current Redis and NATS sizes (Upgrading).
app.webhooks.targetsTwo targets share a name, a local or cluster host lacks allowPrivate: true, or a field fails the schema (name, https URL, whsec_ secret, event types). Fix that target (Webhooks).

Passkey registration or sign-in fails

Confirm: the dashboard shows a warning about a secure origin, or the browser reports a WebAuthn error.

Fix:

  • Serve the dashboard over HTTPS with a certificate the browser trusts. Only https:// and http://localhost allow passkeys.
  • Set app.auth.passkey.id to the hostname (telark.example.com) and app.auth.passkey.origin to the full origin (https://telark.example.com), then helm upgrade.
  • A passkey registered on one hostname does not work on another. Register again on the new host, or use Settings → Security → Passkeys → Add on another device.
  • Back on the sign-in page right after the passkey prompt, in Safari on http://localhost: Safari drops the Secure session cookie there. Use Chrome, Edge or Firefox, or serve https. Open a port-forward at localhost, not 127.0.0.1.
  • this account is being recovered with break-glass: try again shortly (409): a recovery of this account is running. Break-glass holds the account until a passkey is registered with the link it printed, at most about 12 minutes.

A link from Add on another device or from break-glass --enroll lasts 10 minutes. Create a new link, or run the command again: until the bootstrap account's first passkey it prints a new link, and after that it opens or prints a recovery request (next section).

A link created from Members registers one passkey and lasts an hour by default (services.auth.env.ENROLL_INVITE_TTL_SEC). Once it lapses, the account's row shows Invite expired. Ask an admin to choose Create enrollment link again, which also revokes any earlier link. Links created before the upgrade to 0.2.0 stop working: create new ones.

The bootstrap account's first passkey pins it, and from then on break-glass --enroll opens a recovery request instead of printing a link.

Fix:

  • A recovery can't be approved: wait with no sign-in by the bootstrap account (24 hours by default, app.security.breakGlass.waitHours), then run the command again. It prints a 10-minute link and waits for a passkey registered with it, so keep it running and open the link. The account keeps its other passkeys until the new one is saved. If the link expires unused, run the command again for a new one.
  • A replacement, after a change of app.auth.bootstrap.admin, completes once the bootstrap account approves it with a passkey at the link the command prints (https://<host>/break-glass#request=<id>&kind=replace), or after the same wait. Then run the command again.
  • A sign-in by the bootstrap account cancels the request. Run the command again to open a new one.
  • If someone else holds a passkey of the bootstrap account, their sign-ins cancel every request, so you cannot recover through Telark. A cluster admin then resets the pin, an audited write to the custom resources. Add your identity to app.security.crdGuard.extraAllowedUsers first, and take it off afterwards:
ID=$(kubectl get tcfg default -n telark -o jsonpath='{.status.bootstrap.userID}')
for kind in passkeys sessions; do
  kubectl delete "$kind" -n telark $(kubectl get "$kind" -n telark -o jsonpath="{.items[?(@.spec.userId=='$ID')].metadata.name}")
done
# Removes the pin and keeps the requests: the next break-glass --enroll acts at once,
# and the new passkey pins the account again and ends a request left open.
kubectl patch tcfg default -n telark --subresource=status --type=merge -p '{"status":{"bootstrap":{"pinnedAt":null}}}'

Applications page stays empty

Confirm:

  • The namespace is not excluded: check Settings → Discovery → Excluded namespaces. default and the kube-* namespaces are excluded by default.
  • The workloads carry a grouping label such as app.kubernetes.io/name or app.
  • Discovery synced its informers: kubectl logs -n telark deploy/telark-discovery-service | grep 'informers synced'.
  • GET /api/v1/discovery/status shows a pass that finishes.

Fix: Use a non-excluded namespace or remove it from the list. With a custom install, give the discovery ServiceAccount the chart's cluster-wide read permissions.

Plan stuck in scheduled

Confirm: the start time has passed, and the discovery leader logs This replica is now leader without repeated leadership losses.

Fix: Wait one controller tick (31 seconds by default). If the phase still does not move, delete the leader pod. Another replica takes over within about 15 seconds.

A repeating plan is scheduled between windows by design: its card shows the Next window. A time range that passed whole while discovery was down ends terminated without activating.

Plan health is degraded or drifted

Confirm: open the plan's Health panel. Each row says which policy is missing, not ready, or different from what the plan expects. List the deployed policies:

kubectl get namespacedvalidatingpolicies -A -l telark.io/protection-plan=<plan-id>

Fix:

  • Drifted: someone edited a policy. Telark redeploys it on the next tick. If drift keeps coming back, find what in the cluster keeps editing it.
  • Degraded: a policy is missing or not ready. Check Kyverno: kubectl get pods -n telark | grep kyverno. Telark redeploys missing policies on the next tick once Kyverno is healthy.
  • Degraded on every active plan, with your own Kyverno: it is older than 1.18 and serves no policies.kyverno.io kind. Discovery logs why on each pass. Upgrade Kyverno to 1.18 or later.
  • Degraded right after the upgrade to 0.2.0: a new policy is still not ready 2 minutes after its creation, while its 0.1.x predecessor keeps enforcing (Upgrade notes).
  • Refresh health on the plan page recomputes the status now; repairs still happen on the tick.

An enforce plan did not block a change

Confirm: Kyverno's admission pods were ready at the time, and the resource is in the plan's scope and not excluded.

Fix: With the default kyverno.features.forceFailurePolicyIgnore.enabled=true, requests are admitted while Kyverno is down or rolling out. For strict enforcement, see Install Telark for production.

Rollback stuck in pending

Confirm: the leader's logs: kubectl logs -n telark <leader-pod> | grep -i rollback. picked up pending entry should appear within seconds. lock for <app> held elsewhere; deferring pickup means another rollback of the same application is still running.

Fix: Wait for the other rollback. If nothing is picked up, restart the discovery leader.

Rollback stuck in in_progress or failed

Confirm: the rollback's error field, and kubectl describe on the resource it names. Common causes are an admission rejection (often an enforce protection plan on the same scope), an immutable field, or a missing namespace.

Fix: Remove the cause and roll back again. A rollback still in_progress after 5 minutes is marked failed.

Snapshot volume is full

Confirm: Settings → Storage → Snapshot storage is near 100%, and the exporter logs failed to write snapshot to disk:

kubectl logs -n telark deploy/telark-exporter-service | grep -i snapshot

Fix:

  • Now: lower the snapshots kept per application. Each application is pruned at its next change. See Change snapshot retention.
  • Next: raise app.persistence.snapshotsSize with helm upgrade if your storage class allows expansion.
  • Later: exclude noisy namespaces.

Google sign-in fails

See the troubleshooting table in Set up Google sign-in.

Notifications stop appearing

Confirm: the exporter stores in-app notifications in Redis. kubectl logs -n telark deploy/telark-exporter-service | grep -i redis.

Fix: Make sure telark-redis-master-0 is running, then restart the exporter.

The bell keeps a live stream (text/event-stream) open. Behind a proxy that buffers it, the bell falls back to polling every 30 seconds, so notifications arrive late. Make the proxy pass the stream unbuffered and allow more than 15 seconds of idle time, since the services ping every 15 seconds. ingress-nginx honors the X-Accel-Buffering: no header the services send.

Webhook deliveries fail

Confirm: the notifier logs one line per attempt, with its status and outcome:

kubectl logs -n telark deploy/telark-notifier-service | grep 'webhook target='

Fix:

  • A 4xx other than 408 and 429, or a redirect, fails at once and is not retried: fix the receiver, for example a signature check that refuses or a wrong path.
  • A private or cluster address needs allowPrivate: true on the target.
  • A receiver that stays down is given up after 8 attempts over about 8 hours.

Insights show rule text without an explanation

Confirm: the model has not been downloaded yet, or the install is air-gapped.

Fix: In connected installs, wait for the download or choose Install model in Settings → Insights. Air-gapped installs must pre-load the model.

CRDs or the namespace stuck in Terminating

Users, groups and access roles carry cleanup finalizers that only the auth service removes. The uninstall hooks clear them. After a --no-hooks uninstall, clear them as in Uninstall Telark → Uninstalled with --no-hooks; the deletions then complete.

The namespace also hangs when it was deleted before helm uninstall. The release's cluster-wide objects stay behind, and the bundled metrics-server's v1beta1.metrics.k8s.io API points at a service that no longer exists.

Confirm: kubectl get namespace telark -o yaml shows a NamespaceDeletionDiscoveryFailure condition that names metrics.k8s.io.

Fix: Delete the release's leftover cluster-wide objects. The namespace then finishes on its own. The label selects only the Telark release, so a metrics-server you run yourself stays:

kubectl delete apiservices,clusterroles,clusterrolebindings -l app.kubernetes.io/instance=telark