Install Telark for production
Choose a size and storage, expose the dashboard over HTTPS, install, and enroll the first admin.
The Quickstart gets Telark running on defaults. This guide covers the choices to make before a production install. Check the system requirements first, including the bundled dependencies you can skip.
Every setting is a --set key=value flag on the same command. Helm does not remember them, so keep your flags in a values file or script and pass them again on every helm upgrade.
1. Choose a size
app.mode sets replicas, resource requests, API rate limits and disruption budgets for every Telark service.
| You have | Mode |
|---|---|
| An evaluation or a single node | minimal |
| Small and mid-size production | standard (default) |
| A large cluster, or a high change rate across many namespaces | performance |
You have outgrown a mode when the discovery pod restarts with OOMKilled, the Applications page shows "Discovering applications…" most of the time, or bulk actions such as force sync on many applications crawl.
Add --set vpa.enabled=true in standard or performance to install the Vertical Pod Autoscaler, so requests follow real usage. Its objects appear on the first helm upgrade after the install.
2. Choose storage
By default you need no storage flag: the exporter runs one replica on ReadWriteOnce claims from the default StorageClass. app.persistence.snapshotsSize and app.persistence.reportsSize set the volume sizes (defaults per mode).
A second exporter replica keeps the exporter available through node failures and rolling upgrades. Both replicas share the snapshot and report volumes, so they need a ReadWriteMany class (efs-sc on EKS with the EFS CSI driver):
--set services.exporter.replicas=2 \
--set app.persistence.storageClass=<rwx-class>More than one replica without a class fails the render, and so does a class the cluster does not have. minimal always runs one replica.
Upgrades keep the existing claims, and naming a different class on a later helm upgrade copies the data to new claims; see Upgrade Telark.
3. Choose how to reach the dashboard
The dashboard is a ClusterIP Service on port 8080. The chart ships no ingress or gateway controller; it uses the one you run.
Passkeys only work on a secure origin (https://, or http://localhost) and are bound to the hostname. For anything other than a port-forward, serve HTTPS and pin the relying party to your hostname:
--set app.auth.passkey.id=telark.example.com \
--set app.auth.passkey.origin=https://telark.example.comWith ingress.enabled or gateway.enabled the chart refuses to render until both are set. Pin them yourself for NodePort and LoadBalancer, and terminate TLS in front of those, since they expose plain HTTP.
Sign-in sets a Secure session cookie. On a port-forward, open http://localhost:3000 (not 127.0.0.1) in Chrome, Edge or Firefox: Safari drops Secure cookies on http://localhost.
| Option | Flags |
|---|---|
| Port-forward (evaluation) | none; kubectl port-forward -n telark svc/telark-ui-service 3000:8080 |
| NodePort | services.ui.serviceType=NodePort, optional services.ui.nodePort=30080 |
| LoadBalancer | services.ui.serviceType=LoadBalancer |
| Ingress | ingress.enabled=true, ingress.className, ingress.host, ingress.tls, ingress.annotations |
| Gateway API | gateway.enabled=true, gateway.parentRefs[0].name, gateway.hostnames[0]. Needs Gateway API v1.0+ CRDs and a controller that implements HTTPRoute v1 (Envoy Gateway, NGINX Gateway Fabric, Cilium, Istio). ingress-nginx does not. |
Example: Ingress with a Let's Encrypt certificate
The hostname must already resolve to your ingress controller. A self-signed certificate is not enough: browsers disable WebAuthn on pages with certificate errors.
-
Install ingress-nginx and cert-manager if you do not run them:
helm upgrade --install ingress-nginx ingress-nginx \ --repo https://kubernetes.github.io/ingress-nginx \ --namespace ingress-nginx --create-namespace helm upgrade --install cert-manager cert-manager \ --repo https://charts.jetstack.io \ --namespace cert-manager --create-namespace \ --set crds.enabled=true -
Create a
ClusterIssuer. Replace the email with the address Let's Encrypt should notify:kubectl apply -f - <<'EOF' apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: admin@example.com privateKeySecretRef: name: letsencrypt solvers: - http01: ingress: ingressClassName: nginx EOF -
Use these flags in step 4:
--set ingress.enabled=true \ --set ingress.className=nginx \ --set ingress.host=telark.example.com \ --set 'ingress.tls[0].secretName=telark-tls' \ --set 'ingress.tls[0].hosts[0]=telark.example.com' \ --set ingress.annotations."cert-manager\.io/cluster-issuer"=letsencrypt \ --set app.auth.passkey.id=telark.example.com \ --set app.auth.passkey.origin=https://telark.example.com
4. Install
helm install telark oci://ghcr.io/telark/charts/telark -n telark --create-namespace \
--set app.mode=standard \
--set app.auth.bootstrap.admin=test@example.com \
<your dashboard flags from step 3>app.auth.bootstrap.admin is required: the chart refuses to render without it, and the auth service refuses to start. It names the one bootstrap admin, which gets the Admin role only from a break-glass enrollment (step 6). Once that account's first passkey pins it, changing the value only lets break-glass open a replacement request.
The CRDs install with the release as a bundled subchart and are kept on uninstall. If something else manages them, add --set crds.enabled=false.
Already running Kyverno? Install with the bundled Kyverno disabled and Telark uses yours:
helm install telark oci://ghcr.io/telark/charts/telark -n telark --create-namespace \
--set app.mode=standard \
--set app.auth.bootstrap.admin=test@example.com \
--set kyverno.enabled=false \
<your dashboard flags from step 3>Telark is tested with Kyverno v1.19.2, the version the chart bundles. Your own Kyverno must be 1.18 or later, serve the policies.kyverno.io CRDs (plan policies are NamespacedValidatingPolicy objects, and without them every active plan reads Degraded) and keep its PolicyViolation Events, as its default omitEvents does: the plan's admission decisions come from them. Upgrade Telark to 0.2.0 before you move Kyverno to 1.20 (Upgrade notes). With your own Kyverno, change history names no author, because the chart installs the policy that records authors only with the bundled Kyverno.
5. Run one upgrade to enable change authors
Kyverno stamps telark.io/last-modified-by on workloads so change history can name who made each change. That policy, a Kyverno MutatingPolicy, renders only once the MutatingPolicy API (policies.kyverno.io/v1) exists, which is not yet true during the first install. Run the upgrade once, with the same flags:
helm upgrade telark oci://ghcr.io/telark/charts/telark -n telark <same flags as step 4>Changes made before this carry no author. Secrets are never stamped, so history does not name who changed a Secret. With your own Kyverno (kyverno.enabled=false) the chart does not install the policy, so history names no author and this step changes nothing.
6. Enroll the first admin
kubectl exec -n telark deploy/telark-auth-service -- ./main break-glass --email test@example.com --enrollIt prints an enrollment link that starts with app.auth.passkey.origin, such as https://telark.example.com/register#enroll=<token>&email=test%40example.com, valid for 10 minutes, and one step: open it in a browser that is not signed in to Telark, such as a private window. Register a passkey on the page it opens.
That first passkey pins the bootstrap account. From then on the same command, for example after losing the passkey, opens a recovery request instead of acting. A recovery can't be approved: it completes once the wait has passed since it opened (24 hours by default, app.security.breakGlass.waitHours), unless the bootstrap account signs in (which cancels it) or denies it with a passkey at https://telark.example.com/break-glass#request=<id>&kind=recover. Then run the command again: it prints a 10-minute link and waits, up to about 12 minutes, for a passkey registered with it. Once that passkey is saved, the account's other passkeys and sessions are removed. See Recovery requests.
A second passkey for the bootstrap admin is recommended, not required. Kept on another device or in a password manager, it signs you in when one device is lost, without break-glass. Once signed in, open Settings → Security → Passkeys and choose Add on another device. Passkeys are bound to the relying-party ID (app.auth.passkey.id, step 3), so changing it means registering every passkey again, the bootstrap admin's through break-glass (a recovery request).
Other people join in one of three ways:
- Enrollment link. Create their account under Administration → Members and choose Create enrollment link on its row. The link registers one passkey and expires after an hour. A canceled passkey prompt doesn't use it up. See Enrollment links.
- Google sign-in. See Set up Google sign-in.
- Passkey self-registration. A toggle in Settings, next to Single Sign-On, off by default. Leave it off unless only people you trust can reach the dashboard.
Only the bootstrap admin can turn on Google sign-in or self-registration. Self-registered and first-time Google accounts get the ReadOnly role, and Google sign-in never grants Admin, even to the bootstrap admin's email. Once they have signed in, give at least two of them the Admin role under Administration, directly or through a group, so access never depends on one account. Telark refuses a change that would leave no active Admin.
7. Verify
kubectl get pods -n telarkEvery pod should be Running and Ready.
Security defaults
These are on by default. Change them only with a reason.
CRD write guard (app.security.crdGuard.enabled, app.security.crdGuard.enforce). A ValidatingAdmissionPolicy rejects writes to every telark.io resource, and to the OIDC trust Secret, from anything but Telark's own service accounts, in every namespace. Edit rights on another namespace therefore do not turn into Telark Admin. Rights on telark itself can: helm upgrade, pods/exec, Secret read and an entry in the allowlist below each let their holder act as any Telark user, so keep them to cluster operators. Allow a break-glass identity with --set 'app.security.crdGuard.extraAllowedUsers={system:serviceaccount:ops:breakglass}'. enforce=false only audits.
NetworkPolicies (app.security.networkPolicy.enabled). Default deny for every Telark pod. The service APIs accept traffic only from Telark pods of the same release, the dashboard port is open, and NATS accepts only discovery and the notifier on port 4222. Egress is not restricted. They need a CNI that enforces NetworkPolicy.
NATS users. Discovery publishes with its own user and the notifier consumes with another. The chart generates both passwords.
Redis password. Redis requires a password, which the chart generates. Every Telark service that uses Redis authenticates with it, and the chart refuses to render with redis.auth.enabled=false.
Kyverno fails open (kyverno.features.forceFailurePolicyIgnore.enabled=true). While Kyverno is down or rolling out, the API server admits every request, including ones an enforce plan would block. Workloads stay deployable at the cost of best-effort enforcement. For strict enforcement turn it off:
--set kyverno.features.forceFailurePolicyIgnore.enabled=falseFail-closed means an unavailable Kyverno blocks writes to everything its webhooks match, so keep its two admission replicas.
With your own Kyverno (kyverno.enabled=false), the key does not apply: plan policies set no failurePolicy, so they fail closed unless your Kyverno runs with forceFailurePolicyIgnore.
GitOps (cluster-less renders)
The chart generates the service token, both NATS passwords, the Redis password and the OIDC trust Secret, and reads them back on upgrade with lookup. Argo CD, Flux and other helm template pipelines render without a cluster, so every sync would generate new values and break the running pods. Create the Secrets yourself:
kubectl create secret generic telark-service-token -n telark \
--from-literal=token="$(openssl rand -hex 32)"
kubectl create secret generic telark-nats-publisher -n telark \
--from-literal=username=publisher --from-literal=password="$(openssl rand -hex 24)"
kubectl create secret generic telark-nats-consumer -n telark \
--from-literal=username=consumer --from-literal=password="$(openssl rand -hex 24)"
kubectl create secret generic telark-redis-password -n telark \
--from-literal=redis-password="$(openssl rand -hex 24)"
kubectl create secret generic telark-oidc-trust -n telark --from-literal=googleJwkJson=''Then point the chart at them:
--set app.security.serviceToken.existingSecret=telark-service-token \
--set nats.existingSecrets.publisher=telark-nats-publisher \
--set nats.existingSecrets.consumer=telark-nats-consumer \
--set redis.auth.existingSecret=telark-redis-password \
--set app.auth.oidc.existingSecret=telark-oidc-trustWebhook targets are credentials too, so keep them out of values files in Git: create a Secret with the key targets.json and pass --set app.webhooks.existingSecret=<name> (Webhooks).
Insights runtime
Insights runs on Ollama inside the cluster and is on after install, with the granite4:350m model. Change the model, turn on automatic analysis or turn the analyzer off in Settings → Insights. See Insights for when analysis runs.
| Setup | Flags |
|---|---|
| Connected (default) | none. Ollama downloads the model (about 708 MB) after start and gets HTTPS egress for it. |
| Air-gapped | app.ollama.autoPull=false. No download and no egress; pre-load the model on the volume or in the image. |
| Your own Ollama endpoint | ollama.enabled=false, app.ollama.runtimeUrl=http://<host>:11434. A URL only, no key. |
| No Insights | ollama.enabled=false, and turn the analyzer off in Settings → Insights. The rest of Telark works without it. |
Autoscaling
In standard and performance, auth, discovery, the notifier and the dashboard scale on CPU from 1 replica up to 3 or 5. The exporter and the analyzer never autoscale. Tune one service:
--set services.discovery.autoscaling.minReplicas=2 \
--set services.discovery.autoscaling.maxReplicas=8 \
--set services.discovery.autoscaling.targetCPUUtilizationPercentage=70Set the same keys under app.serviceDefaults.autoscaling to change every service.
Other settings
metrics-server on kind, minikube or self-signed kubelets
If HPAs show <unknown>, the bundled metrics-server cannot verify kubelet certificates. There, and only there, add:
--set 'metrics-server.args={--kubelet-insecure-tls}'Sign-in rate limits
Auth limits the four sign-in starts anyone can call without a session: passkey sign-in, passkey registration, the break-glass approval and the Google sign-in nonce. In each one-minute window, a client gets RATE_LIMIT_PER_CLIENT starts on each of them (default 30), and all clients together RATE_LIMIT_GLOBAL (default 3000). Past either limit, auth answers 429 with Retry-After: 60 and the message "too many attempts right now: wait a minute, then try again". The finishes, logout and every other route are not limited.
Auth finds the client in X-Forwarded-For: the dashboard's nginx appends the address it got the request from, and every proxy in front of it appends one more, so auth takes the entry RATE_LIMIT_TRUSTED_HOPS from the right and ignores what the client wrote to the left of it. An IPv6 client counts per /64. The chart sets 1 with ingress.enabled or gateway.enabled, else 0. Set it yourself when your path differs:
-
One more L7 proxy or cloud load balancer in front of the ingress controller or gateway adds a hop:
--set services.auth.env.RATE_LIMIT_TRUSTED_HOPS=2 -
NodePort or LoadBalancer: keep 0, and keep the client's address on the
uiService, or every client counts as the node that forwarded it. The same goes for your ingress controller's own Service:kubectl patch svc -n telark telark-ui-service -p '{"spec":{"externalTrafficPolicy":"Local"}}'
A wrong hop count weakens only the per-client limit: too high trusts an address the client wrote, too low counts a proxy as the client. The per-route limit holds either way.
Large installs
Redis keeps everything in memory and never evicts. Its use grows with the number of applications, so on large installs raise redis.master.resources.limits.memory (512 MiB by default) in proportion.
Every value, with its default: Helm values.
Known limitations
- The API is
telark.io/v1alpha1and may change before 1.0. Pin the chart version. - Single cluster per install.
- By default the chart bundles Kyverno, Redis, NATS, metrics-server and Ollama. The bundled Kyverno can be turned off to use one you already run (Install), and the bundled metrics-server with
metrics-server.enabled=false. - The bundled Kyverno stays on 1.19. Kyverno 1.20 is untested.