Features

Notifications

In-app notifications for the events that need a person, such as a finished rollback, an approval request or a role change, and signed webhooks for your own tools.

Some events need a specific person to act or to know: a plan waiting for their approval, the outcome of the rollback they started, a change to their own access. Telark sends those to that person inside the dashboard, and can send some of them to your own endpoints as signed webhooks.

What you get

A bell in the dashboard header with an unread count, and a drawer that lists your notifications. Clicking one marks it read and opens what it points to. Mark all read and Clear all act on the whole list.

EventSent toWhen
plan.approval.requestedEvery user who may approve the plan, except the requesterA plan that requires approval is created, duplicated or reactivated.
plan.approval.decidedThe requesterAn approver approves (success) or rejects (warning, with the comment).
rollback.completedThe user who triggered the rollback, or the user who aborted itThe rollback succeeded, failed (with the error) or was aborted while pending.
role.changedThat userRoles were added to or removed from their account.
group.membership.changedEach user added or removedGroup membership changed.
enroll.link.createdThe account ownerSomeone created an enrollment link for their account while it already has a passkey (an account recovery).
enroll.link.usedThe account ownerA passkey was added to their account through such a link.
incident.criticalEvery user who may read InsightsA new Critical incident, once per episode. It opens the card on the Insights page.
bootstrap.requestThe bootstrap account and every Admin on ALL: always for a link break-glass is issuing or a completed request, and only while break-glass alerts are on for an opened or canceled oneA break-glass recovery request was opened, canceled or completed, or break-glass is issuing an enrollment link for it.

Other plan lifecycle transitions are logged, not notified.

How it works

The exporter stores notifications in Redis, per user. Other services emit them through an internal endpoint with the service token, and emission is best-effort: a failed notification never blocks the operation that caused it. The bell keeps a live stream open (GET notifications/events) and refetches the list when it changes. It polls every 30 seconds only while the stream is down, for example behind a proxy that buffers it.

  • An unread notification is kept 30 days, a read one 7 days, and each user keeps at most the newest 500.
  • A new notification with the same type and target as an unread one updates it instead of adding a row.
  • Titles are capped at 100 characters and messages at 500.
  • Each user reads and clears only their own notifications.

Webhooks

The notifier sends four event types to endpoints you name, signed the Standard Webhooks way: rollback.completed, plan.approval.requested, plan.approval.decided and incident.critical. Each event goes once to every target that lists its type, however many users see it in the bell. Role, group and enrollment-link notifications stay in the bell: a target that lists them fails the render.

# webhooks.yaml, passed with -f on install and on every upgrade
app:
  webhooks:
    targets:
      - name: ops                        # [a-z0-9-], unique, shown in the notifier log
        url: https://hooks.example.com/telark
        secret: whsec_REPLACE_ME         # echo "whsec_$(openssl rand -base64 32)"
        events: [rollback.completed, plan.approval.requested, plan.approval.decided, incident.critical]
KeyDescription
nameLowercase letters, digits and -, unique.
urlhttps only, with no user or password in it.
secretwhsec_ and at least 32 base64 characters. Signs every delivery.
previousSecretOptional, while receivers move to a new secret: each delivery then carries a second signature.
eventsOne or more of the four types above.
allowPrivatetrue lets the target be a private or cluster address and use http.
  • Secrets. The chart renders the targets into the Secret telark-webhooks, which the notifier re-reads about a minute after a change, with no restart. A URL like Slack's is itself a credential, so keep targets out of values files in Git: for Argo CD, Flux or any cluster-less render, create a Secret with the key targets.json holding the same list as JSON, and set app.webhooks.existingSecret. One bad target refuses the whole file, and the targets already running keep going.
  • Verify. Each POST carries webhook-id, webhook-timestamp and webhook-signature (v1, and the base64 HMAC-SHA256 of <id>.<timestamp>.<body>, keyed by the base64-decoded part of the secret). The Standard Webhooks libraries check them. Refuse a timestamp more than 5 minutes off, and an id you already processed.
  • Retries. A 2xx counts as delivered. A network error, a timeout (10 seconds), 408, 429 or a 5xx is retried, for 8 attempts over about 8 hours, and Retry-After is honored. Any other status fails at once, and redirects are never followed. A delivery still owed survives a notifier restart.
  • Addresses. The notifier checks the address it actually dials, after DNS. Loopback, link-local (including the cloud metadata address), multicast, broadcast and unspecified addresses are always refused, and private ranges need allowPrivate: true.
  • Log. One line per attempt, never with the URL or a secret: kubectl logs -n telark deploy/telark-notifier-service | grep 'webhook target='.

Limits

  • No email or push delivery. Webhooks carry only the four event types above.
  • Not an audit log: notifications expire. Use plan reports and change history for records.

Reference