Operations

Upgrade Telark

Upgrade the Helm release in place and handle the changes that need extra steps.

One chart version pins every Telark service and the CRDs. An upgrade is a helm upgrade of the same release.

Before you upgrade

  1. Back up first. After an upgrade that ran a migration step, helm rollback to the previous release still works, but an exporter two or more releases older refuses to start on the data, so the way back that far is the backup. Take an etcd snapshot (etcdctl snapshot save) where you run the control plane yourself, or a Velero backup of the namespace:

    velero backup create telark-before-upgrade --include-namespaces telark --wait

    Telark keeps state in .status (plan phases and approvals, application history, the migration record), and a Velero restore leaves it out unless it names those resources: --status-include-resources applications.telark.io,protectionplans.telark.io,telarkconfigs.telark.io. See Back up Telark.

  2. Read the upgrade notes and the changelog of every version between yours and the target.

  3. Let any rollback in progress finish.

Upgrade

Pass every --set and -f flag you used at install. Helm does not remember them, and a flag you leave out returns to its default.

helm upgrade telark oci://ghcr.io/telark/charts/telark -n telark \
  --version <target-version> \
  <your install flags>

Do not use --reuse-values: it keeps the previous release's defaults instead of the new chart's.

The upgrade also updates the CRDs, which ship as templates of the bundled telark-crds subchart. If you manage the CRDs out of band (crds.enabled=false), upgrade telark-crds before telark: an older schema refuses repeating plans and silently drops kindRef and status.bootstrap. Deployments with two or more replicas roll one pod at a time. Discovery's new leader takes over within one lease (15 seconds), and plans and rollbacks resume on it. Sessions are custom resources, so an upgrade signs nobody out, except the upgrade to 0.2.0 (Upgrade notes).

Check the result

kubectl get pods -n telark

Every pod should be Running and Ready. In the dashboard, protection plans should show the same phases as before, and a Force Sync on one application should complete.

Migrations

Before it serves, the new exporter runs the migration steps the upgrade notes list and records the last one in the TelarkConfig status (status.lastMigration).

  • A failed step stops the exporter, which retries it on every restart. Its log names the step: kubectl logs -n telark deploy/telark-exporter-service.
  • After an upgrade that ran a step, helm rollback to the previous release still works: its exporter lowers the record to its own last step and logs lowered the record, and the next upgrade runs the newer steps again.
  • An exporter two or more releases older than the data refuses to start and says so in its log. Run a newer release, or restore the backup you took before the upgrade.

Upgrade notes

Each minor version lists here the changes an upgrade brings that need your attention. A version not listed needs nothing beyond the steps above.

0.2.0

No migration step: 0.2.0 adds the migration record itself (status.lastMigration and status.rollbackMigration on the TelarkConfig).

  • Upgrade with --reset-then-reuse-values, or with your full -f and --set flags, not --reuse-values. That flag keeps 0.1.x's defaults, which lack the keys 0.2.0 adds, so the render fails (nil pointer evaluating interface {}.existingSecret).
  • Some values moved. The old paths still work in 0.2.0 and win when set, but a later release removes them. The notes Helm prints after the upgrade list any your values still use. An old path an earlier release set keeps winning under --reset-then-reuse-values, so change it on the old path, or upgrade with --reset-values and your flags on the new paths.
    Old pathNew path
    app.crdGuard.*app.security.crdGuard.*
    app.networkPolicy.enabledapp.security.networkPolicy.enabled
    app.serviceToken.*app.security.serviceToken.*
    app.kubectlImageapp.image.kubectl
    app.kyverno.enabledkyverno.enabled
    app.ollama.enabledollama.enabled
    app.kyverno.failOpenRemoved: kyverno.features.forceFailurePolicyIgnore.enabled alone sets fail-open. A different failOpen still fails the render.
  • Everyone signs in once. The session moves from the dashboard's browser storage into an HttpOnly cookie, and the dashboard revokes the old token at its first load. Upgrade the dashboard and the services together, in one chart release: a new dashboard on an old backend gets no cookie. API clients keep sending X-Session-Token; a change that relies on the cookie needs X-Requested-With.
  • Sign-in starts can answer 429. Auth limits the public sign-in starts per client and per route (Sign-in rate limits). With another proxy in front of your ingress or gateway, or with a NodePort or LoadBalancer, check RATE_LIMIT_TRUSTED_HOPS and the ui Service's externalTrafficPolicy there.
  • Old enrollment links stop working, because auth now signs the entries behind them. Create new links from Members, and run break-glass again for the bootstrap admin. A passkey registration open during the upgrade starts again.
  • The bootstrap account is pinned. At each start, before it serves, the exporter pins the bootstrap account that has a passkey (the app.auth.bootstrap.admin one first, else the oldest) and clears the bootstrap marker of every other account, which keeps its roles: check Members afterwards. From then on break-glass opens a recovery request instead of acting, and a changed app.auth.bootstrap.admin no longer makes a second bootstrap account.
  • Break-glass alerts. The TelarkConfig gets spec.breakGlassAlerts.notifyAdmins, the bootstrap account's switch for whether Admins on ALL see recovery requests and hear when one opens or is canceled. A link break-glass is issuing and a completed request reach them either way. The CRD defaults it to true, so nothing changes and nothing migrates. After a rollback to 0.1.x and a new upgrade, the switch is on again.
  • The bootstrap account is masked. Other users, Admins included, see it as Bootstrap account, with no username, full name or email (Sign-in). Notifications sent before the upgrade keep the names they were sent with.
  • Kept reports. The reports volume gets a kept/ directory for the reports of deleted plans that you chose to keep. Reports already stored are not kept, so nothing migrates. Kept reports are never swept, so they count against app.persistence.reportsSize until someone deletes them.
  • New plan policy kind. Plan policies move from Kyverno's deprecated kyverno.io/v1 Policy to policies.kyverno.io/v1 NamespacedValidatingPolicy, under the same names. Within a controller tick, discovery deploys each active plan's new policies, so enforcement never stops. A new policy still not ready 2 minutes after its creation makes its plan read Degraded while the old one keeps enforcing. Telark leaves the old policies in place: once every plan reads Healthy, delete them with kubectl delete policies.kyverno.io -A -l app.kubernetes.io/managed-by=telark. Until then both refuse, so a refusal can show twice in a plan's admission decisions.
  • Refusal messages now name the plan by its name instead of its ID and end with (requested by "<username>"), and the plan's admission decisions and reports show that username.
  • Before you move Kyverno to 1.20, check that this returns nothing: kubectl get policies.kyverno.io -A -l app.kubernetes.io/managed-by=telark.
  • Last-modified policy. The chart's last-modified policy becomes a MutatingPolicy. The upgrade leaves the 0.1.x ClusterPolicy of the same name: delete it next to the old plan policies with kubectl delete clusterpolicies.kyverno.io telark-inject-modifier. Until then both stamp the same last-modified annotations, which does no harm. The new policy writes telark.io/last-modified-at with fractional seconds (RFC 3339, such as 2026-10-20T09:15:02.123456789Z) where 0.1.x wrote whole seconds, so a script that reads the annotation must accept a fraction.
  • Plan kinds and repeating plans. ProtectionPlan gets spec.kindRef, spec.repeat (with the new timeMode: repeating) and status.window, and categories get the plan-kinds scope, with four built-in kinds (QA, Testing, Release, Maintenance window) added at the exporter's first start. Existing plans get no kind and keep their schedule. Each repeating plan keeps its newest 100 window reports, and one report can reach 16 MiB, so a busy Enforce series can hold about 1.6 GiB of app.persistence.reportsSize.

Upgrades that need extra steps

Kubernetes never changes the volume size of an existing StatefulSet. If an upgrade changes the Redis or NATS volume size, it stops before it changes anything and prints the command to run once:

kubectl delete statefulset -n telark telark-redis-master telark-nats --cascade=orphan

The pods, claims and data stay, the claims keep their size, and the next upgrade re-creates both StatefulSets. To skip the step, pass the current sizes on every upgrade (redis.master.persistence.size, nats.persistence.size). With Argo CD, Flux or another helm template pipeline the chart cannot check the live size, so run the command before the sync.

Kubernetes never shrinks a claim either. If ollama.persistentVolume.size is below the model volume's size, the upgrade stops and names the size to keep on every upgrade.

Exporter storage

Kubernetes never changes a claim's class or access mode, so upgrades keep the exporter's claims as they are:

  • An empty app.persistence.storageClass keeps the class the claims already use, so leaving the flag out changes nothing.
  • app.persistence.snapshotsSize and app.persistence.reportsSize, which the mode also sets, only grow, and only where the class allows volume expansion. A smaller value keeps the current size.

To change the class, name the new one, and the replica count, on helm upgrade. The chart creates new claims named after the class, such as telark-exporter-snapshots-pvc-efs-sc-rwx, and the exporter copies the snapshots and reports into them before it starts, so it restarts once. The old claims are kept, and the upgrade's notes list them. Delete them after the next upgrade:

kubectl delete pvc -n telark telark-exporter-snapshots-pvc telark-exporter-reports-pvc
  • If the new claims stay Pending, for example ReadWriteMany on a block-storage class, helm rollback telark -n telark returns to the old claims.
  • A rollback across a class change returns the exporter to the old claims as they were at the change. Upgrading to the new class again copies over what the exporter wrote in between; for each file, the newer copy wins.
  • Uninstall keeps the claims, and a reinstall picks them up: the ones named after its storage settings, else the original ones, copied like an upgrade when the class differs. After a class change, reinstall with the same settings, or the original claims come back as they were at the change.
  • Cluster-less renders (GitOps) cannot read the live claims, so they always render the original names and the requested class. Keep the class you installed with.

Roll back an upgrade

helm rollback telark -n telark

The Deployments return to the previous images and the CRDs to the previous definitions. An upgrade never touches the snapshot volume. Across a storage class change, the rollback returns the exporter to the old claims (Exporter storage). Two or more releases back, restore the backup instead (Migrations).

Back to 0.1.x

  1. Before the rollback, cancel repeating plans or give them another schedule: 0.1.x treats a repeating plan as permanent and never opens or closes its windows.

  2. After the rollback, once its controller has redeployed the old policies (about a minute), delete the new ones, which 0.1.x leaves enforcing:

    kubectl delete namespacedvalidatingpolicies.policies.kyverno.io -A -l app.kubernetes.io/managed-by=telark
    kubectl delete mutatingpolicies.policies.kyverno.io telark-inject-modifier
  3. helm rollback runs no post-upgrade hook, so 0.1.x's last-modified policy doesn't come back. Upgrade once to the same 0.1.x chart, with the flags you installed with, and its hook recreates it:

    helm upgrade telark oci://ghcr.io/telark/charts/telark --version <0.1.x> -n telark --reset-then-reuse-values

The rollback brings back 0.1.x's CRDs, whoever manages them. They keep neither a plan's kind, repeat and window nor the migration record, so plans show no kind, and an object 0.1.x writes in between loses those fields for good: note each plan's kind first. Kept reports stay on the volume, but 0.1.x cannot list or download them. They come back with the next upgrade. 0.1.x also drops the bootstrap account's pin and any open recovery request, and the next upgrade pins it again.