Operations and alerting
This page answers the on-call questions: what should page me, what happens when the database goes away, and how do I run this with the availability I need.
Probes
| Endpoint | What it checks | Used as |
|---|---|---|
/healthz | Nothing beyond the process being up — it always answers 200. A live process with a dead dependency is a readiness problem, not a liveness one. | Liveness probe (initial delay 10s, period 20s) |
/readyz | The database, with a 3-second timeout. Unreachable → 503 and the replica leaves the Service. The reasoning is in the code: a replica that cannot read the verdict cache would admit everything. | Readiness probe (initial delay 5s, period 10s) |
Both Deployments — operator and scanner — carry both probes. Rollouts, ArgoCD sync waves and helm --wait all key off readiness, so a deploy of Attestkeep itself does not report healthy until it can actually decide.
What a database outage does — and does not do
The verdict store and scan queue live in PostgreSQL, so with the database gone the operator cannot decide. What happens next is deliberate, and we have run it, not reasoned about it:
- Replicas turn unready and leave the Service. They do not crash-loop; they come back the moment the database does, with no restart.
- With the default
webhook.failurePolicy: Ignore, the API server admits pods unchecked for the duration. That window is not silent: on recovery the operator finds the heartbeat gap and writes it to the outage record, and the next reconciliation sweep names every digest that entered without a decision. - Admission never fails closed because of the database alone: quota metering explicitly admits on a database error rather than blocking deploys, and a failed cache read falls back to a cold verification instead of a deny.
- Scanning stops with the database and resumes with it — the queue is in the database, so nothing is lost, only delayed.
If an unchecked window is unacceptable for you, set webhook.failurePolicy: Fail and read Admission for what that trades away. Evidence documents report the outage either way: the availability block lists every window with its start, end and cause, and continuous is true only under Fail with zero outages.
Availability of the pieces
| Component | Default | How to harden |
|---|---|---|
| Operator / webhook | 2 replicas, preferred anti-affinity across nodes, PodDisruptionBudget minAvailable: 1 | Raise replicaCount; the replicas serialise controller work through a PostgreSQL advisory lock, so scaling out is safe. |
| Scanner | 1 replica, Recreate strategy | Raise scanner.replicas — the chart refuses the combination of multiple replicas and a ReadWriteOnce cache volume, and tells you why. |
| Bundled PostgreSQL | Single replica by design — it is a convenience, not an HA database | Point externalDsn (or externalDsnExistingSecret) at the PostgreSQL you already operate with replication and backups. The bundled StatefulSet then does not render at all. |
What should page you
Metrics are served on port 9090 at /metrics, exposed through the chart's Service. Series for every label combination exist from startup at zero, so absence of data is distinguishable from absence of denials.
| Condition | Signal | Why it matters |
|---|---|---|
| Admission is being bypassed | attestkeep_webhook_distrusted == 1 | The operator dials its own webhook through the CA bundle published on the ValidatingWebhookConfiguration, exactly as the API server would. While this gauge is 1 under failurePolicy: Ignore, deployments are proceeding unchecked — the one failure mode that otherwise looks like a quiet cluster. This is the alert to treat as a page, not a ticket. |
| Something got in | attestkeep_runtime_unmatched_digests > 0 | The reconciliation sweep found running digests with no allowed admission behind them — side-loaded images, or arrivals during an outage window. Also raised as the unverified_workload notification, once per newly seen digest. |
| Database trouble | Readiness: pods unready on /readyz 503 | There is deliberately no separate database gauge — unready replicas are the signal, and your existing kube_pod_status_ready alerting already covers it. |
| Scanning is falling behind | attestkeep_scan_queue_depth sustained high; attestkeep_scans_failed_total rate vs attestkeep_scans_completed_total | First-day queues on a large cluster drain over hours; a queue that only grows means the scanner cannot reach a registry or its database. |
| Vulnerability data going stale | attestkeep_vulnerability_db_age_seconds | Verdicts are only as fresh as the database behind them. The trivy_db_failed notification fires on failed updates. |
| Licence quota approaching | attestkeep_image_quota{kind="count"} against {kind="limit"} | Exhaustion denies only image names never seen this period — running workloads keep running. threshold_80 / threshold_100 notification events fire at the boundaries. |
| Denials worth reviewing | attestkeep_admission_denials_total{reason=…} | Reasons are labelled: threshold, signature, tag, quota, unscanned and friends — a spike in one reason is a story, a spike across all of them is a policy change. |
Latency, if you watch it: attestkeep_admission_duration_seconds is a histogram labelled path="cached" or "cold" — the cached path is the one your rollouts feel. We do not publish latency numbers yet, because we have not measured them at a scale worth quoting; measure your own p99 from this series.
The outage record
Windows in which the control was not operating are recorded in the database (webhook_outages) and reported in every evidence document's availability block. Two things create an entry, both written when the operator can see again, with honest wording about what it cannot know:
- A controller heartbeat gap of more than two minutes — every replica down, the database unreachable, or the cluster itself out; the record says exactly that.
- A window in which the API server could not have verified the webhook against its published CA bundle — recorded from the moment distrust began to the moment the probe recovered.
There is no dedicated console page for outages yet; they surface in evidence documents and are queryable in the database. The live counterpart is the attestkeep_webhook_distrusted gauge above.
GitOps, ArgoCD, and who applies things
Enforcement sits at pod admission, so it does not matter who does the applying: kubectl, ArgoCD, Flux, a Job, an operator reconciling its CRD — every pod CREATE and UPDATE outside the bypass namespaces passes the same ValidatingWebhookConfiguration. There is nothing to integrate and no per-tool plugin: if your GitOps controller tries to sync a workload whose image fails policy, the sync fails with the denial message as the error, which is exactly where you want to read it.
Policies themselves are cluster-scoped CRDs and belong in git; Policies covers how console edits and git-managed policies coexist.