Notifications
In-app notifications for the events that need a person, such as a finished rollback, an approval request or a role change, and signed webhooks for your own tools.
Some events need a specific person to act or to know: a plan waiting for their approval, the outcome of the rollback they started, a change to their own access. Telark sends those to that person inside the dashboard, and can send some of them to your own endpoints as signed webhooks.
What you get
A bell in the dashboard header with an unread count, and a drawer that lists your notifications. Clicking one marks it read and opens what it points to. Mark all read and Clear all act on the whole list.
| Event | Sent to | When |
|---|---|---|
plan.approval.requested | Every user who may approve the plan, except the requester | A plan that requires approval is created, duplicated or reactivated. |
plan.approval.decided | The requester | An approver approves (success) or rejects (warning, with the comment). |
rollback.completed | The user who triggered the rollback, or the user who aborted it | The rollback succeeded, failed (with the error) or was aborted while pending. |
role.changed | That user | Roles were added to or removed from their account. |
group.membership.changed | Each user added or removed | Group membership changed. |
enroll.link.created | The account owner | Someone created an enrollment link for their account while it already has a passkey (an account recovery). |
enroll.link.used | The account owner | A passkey was added to their account through such a link. |
incident.critical | Every user who may read Insights | A new Critical incident, once per episode. It opens the card on the Insights page. |
bootstrap.request | The bootstrap account and every Admin on ALL: always for a link break-glass is issuing or a completed request, and only while break-glass alerts are on for an opened or canceled one | A break-glass recovery request was opened, canceled or completed, or break-glass is issuing an enrollment link for it. |
Other plan lifecycle transitions are logged, not notified.
How it works
The exporter stores notifications in Redis, per user. Other services
emit them through an internal endpoint with the service token, and
emission is best-effort: a failed notification never blocks the
operation that caused it. The bell keeps a live stream open
(GET notifications/events) and refetches the list when it changes. It
polls every 30 seconds only while the stream is down, for example behind
a proxy that buffers it.
- An unread notification is kept 30 days, a read one 7 days, and each user keeps at most the newest 500.
- A new notification with the same type and target as an unread one updates it instead of adding a row.
- Titles are capped at 100 characters and messages at 500.
- Each user reads and clears only their own notifications.
Webhooks
The notifier sends four event types to endpoints you name, signed the
Standard Webhooks way:
rollback.completed, plan.approval.requested, plan.approval.decided
and incident.critical. Each event goes once to every target that lists
its type, however many users see it in the bell. Role, group and
enrollment-link notifications stay in the bell: a target that lists them
fails the render.
# webhooks.yaml, passed with -f on install and on every upgrade
app:
webhooks:
targets:
- name: ops # [a-z0-9-], unique, shown in the notifier log
url: https://hooks.example.com/telark
secret: whsec_REPLACE_ME # echo "whsec_$(openssl rand -base64 32)"
events: [rollback.completed, plan.approval.requested, plan.approval.decided, incident.critical]| Key | Description |
|---|---|
name | Lowercase letters, digits and -, unique. |
url | https only, with no user or password in it. |
secret | whsec_ and at least 32 base64 characters. Signs every delivery. |
previousSecret | Optional, while receivers move to a new secret: each delivery then carries a second signature. |
events | One or more of the four types above. |
allowPrivate | true lets the target be a private or cluster address and use http. |
- Secrets. The chart renders the targets into the Secret
telark-webhooks, which the notifier re-reads about a minute after a change, with no restart. A URL like Slack's is itself a credential, so keep targets out of values files in Git: for Argo CD, Flux or any cluster-less render, create a Secret with the keytargets.jsonholding the same list as JSON, and setapp.webhooks.existingSecret. One bad target refuses the whole file, and the targets already running keep going. - Verify. Each
POSTcarrieswebhook-id,webhook-timestampandwebhook-signature(v1,and the base64 HMAC-SHA256 of<id>.<timestamp>.<body>, keyed by the base64-decoded part of the secret). The Standard Webhooks libraries check them. Refuse a timestamp more than 5 minutes off, and anidyou already processed. - Retries. A 2xx counts as delivered. A network error, a timeout
(10 seconds), 408, 429 or a 5xx is retried, for 8 attempts over about
8 hours, and
Retry-Afteris honored. Any other status fails at once, and redirects are never followed. A delivery still owed survives a notifier restart. - Addresses. The notifier checks the address it actually dials,
after DNS. Loopback, link-local (including the cloud metadata address),
multicast, broadcast and unspecified addresses are always refused, and
private ranges need
allowPrivate: true. - Log. One line per attempt, never with the URL or a secret:
kubectl logs -n telark deploy/telark-notifier-service | grep 'webhook target='.
Limits
- No email or push delivery. Webhooks carry only the four event types above.
- Not an audit log: notifications expire. Use plan reports and change history for records.