Attestkeep docs

Runtime events from Falco

Falco watches what containers do after they start. Attestkeep can receive Falco's events, store each one against the image and pod that raised it, and act on it through runtime policies you write in the console.

Attestkeep does not ship, install, configure or write to Falco. Detection is done by your Falco and its rules. Attestkeep only receives the events. A runtime event never changes an admission decision. See what Attestkeep does not do.

How it works

  1. A Falco rule fires on a node, for example because someone opened a shell in a container.
  2. The event is sent as JSON to the operator's integrations port over mutual TLS. Falco sends it itself from http_output, or Falcosidekick forwards it from its webhook output. Either way the sender presents a client certificate that the operator issued.
  3. The operator stores the event and ties it to the image and digest the container runs.
  4. Your runtime policies are checked in order. The first one that matches decides what happens: keep a record, send a notification, open an issue in your tracker, stop the pod.
  5. The event and the decision show up under Falco > Events, next to the image in the images list, and in the evidence pack.

Example

An admin creates a policy under Falco > Policies > New policy:

Saving asks once more, because the policy deletes pods. Later someone runs:

kubectl -n shop exec -it deploy/web -- sh

Falco's upstream rule Terminal shell in container (priority Notice, source syscall) fires for the interactive shell. The event reaches the operator, which stores it and matches the policy. Then:

Requirements

Direct or through Falcosidekick

The Attestkeep chart value integrations.falco.connection picks one of two ways in:

The listener and the client certificate are the same in both cases. The setting decides which Falco chart values the console and the install notes show you. Any value other than mtls or sidekick makes the chart fail.

How the connection is authenticated

Falco's http_output cannot add a header, so there is no token. The only credential is a client certificate.

Setup

1. Turn the integration on

In the Attestkeep chart values:

integrations:
  port: 8444
  falco:
    enabled: true
    connection: mtls   # or sidekick
helm upgrade attestkeep oci://ghcr.io/attestkeep/charts/attestkeep \
  -n attestkeep -f values.yaml

The chart opens the integrations port on the pod and the Service. The operator creates the Secret when it starts. After the upgrade, helm prints the Falco chart values for the connection you chose.

With enabled: false, the default, there is no listener and no CA. The Falco menu then shows only Falco integration, and only to admins.

2. Download the certificates

Open Falco > Falco integration. The page shows the status, the connection, the endpoint URL, when the client certificate expires and when the last event arrived. It also shows the commands and Helm values for the next two steps, with your endpoint filled in.

Download certificates saves attestkeep-falco-mtls.tar.gz with three files:

FileWhat it is
ca.crtThe CA of the operator's server certificate. Falco uses it to verify the operator.
client.crtThe client certificate Falco presents.
client.keyIts private key. Anyone holding it can post events that your policies act on, so treat it as a secret.

The same bundle is available from GET /api/v1/integrations/falco/bundle (admin). Every download is written to the audit log as runtime_integration_change.

3. Create the Secret in the Falco namespace

tar xzf attestkeep-falco-mtls.tar.gz
kubectl -n falco create secret generic attestkeep-falco-client \
  --from-file=ca.crt --from-file=client.crt --from-file=client.key

If Falco runs in another namespace, change -n falco. With connection: sidekick the Secret has to be in the namespace where Sidekick runs, which is the Falco release namespace when Sidekick comes from the Falco chart.

4a. Falco values for connection: mtls

# values for the falcosecurity/falco chart
falco:
  json_output: true
  http_output:
    enabled: true
    url: https://attestkeep.attestkeep.svc:8444/api/v1/integrations/falco
    mtls: true
    ca_cert: /etc/falco/certs/client/ca.crt
    client_cert: /etc/falco/certs/client/client.crt
    client_key: /etc/falco/certs/client/client.key
certs:
  existingClientSecret: attestkeep-falco-client

4b. Falco values for connection: sidekick

Use these instead of the http_output values in 4a.

# values for the falcosecurity/falco chart
falco:
  json_output: true
falcosidekick:
  enabled: true
  config:
    mutualtlsclient:
      certfile: /etc/attestkeep/falco-client/client.crt
      keyfile: /etc/attestkeep/falco-client/client.key
      cacertfile: /etc/attestkeep/falco-client/ca.crt
    webhook:
      address: https://attestkeep.attestkeep.svc:8444/api/v1/integrations/falco
      mutualtls: true
      minimumpriority: ""
  extraVolumes:
    - name: attestkeep-falco-client
      secret:
        secretName: attestkeep-falco-client
  extraVolumeMounts:
    - name: attestkeep-falco-client
      mountPath: /etc/attestkeep/falco-client
      readOnly: true

5. Check that events arrive

Apply the Falco values with helm upgrade on your Falco release, then trigger a rule. An interactive shell is the simplest:

kubectl -n <namespace> exec -it <pod> -- sh

Without -t there is no terminal and Terminal shell in container does not fire.

Then look at Falco > Falco integration > Last event. Next to it the page says whether the event came from Falco or from Falcosidekick. If that does not match integrations.falco.connection, the page shows a warning, because the values it gives you are not the ones your Falco is using.

Policies, Rules and Events appear in the Falco menu once the integration is on. Each of them starts with a status line, either "Falco connected · last event <time>" or the reason it is not connected.

The same check through the API is GET /api/v1/runtime/status, which returns enabled, last_event_at, events_24h and by_source.

Runtime policies

Every stored event is checked against the policies on Falco > Policies, top to bottom. The first enabled policy that matches decides. If none matches, the event is kept with action record and nothing is sent. Use the arrows on the page to change the order.

A policy change reaches the replica that saved it at once. Other operator replicas pick it up within 15 seconds.

The policy form

FieldWhat it does
NameRequired, at most 128 characters, unique.
Falco rulesThe policy matches an event from any of these rules, by exact name. Pick from the Rules list or type the name of your own rule. At least one is required.
All namespaces / Only in namespacesLimits the policy to events from pods in the listed namespaces.
Send a notificationSends the event to every enabled channel subscribed to Runtime alert.
Open an issueOpens an issue through the tracker channels subscribed to Runtime alert.
Stop the podEvicts the pod the event came from. Kill at once or Use the application's shutdown time. See stopping the pod.
StateOn or Off. A new policy is Off until you switch it on. An Off policy is kept and does nothing.

With any of the three actions on, the policy is saved with action notify. With all three off it is saved as record: matching events are recorded against the policy and nothing else happens.

Notifications and issues

A message is titled Runtime alert: <Falco rule>. Its body is Falco's output line. It also lists the rule, priority, source, namespace, pod, container, image, digest, node, time, the policy and its action. If the policy stopped the pod, the message says so. Events with priority emergency, alert or critical are sent at level critical, everything else at warning.

A policy sends at most one message per image digest per hour. For an event with no digest the limit is per namespace and pod. Events held back this way are counted in attestkeep_runtime_events_dropped_total{reason="throttled"}. The limit applies to messages and issues. It does not delay stopping a pod.

An issue is opened only when the event is tied to an image digest. There is one issue per digest and Falco rule. It is titled RUNTIME: <rule> in <image>, carries the label runtime and counts toward the channel's daily cap. It closes by itself as quiet when the rule has not fired on that image for 7 days, or as image_retired when no running pod has used that digest for 24 hours. The closing comment gives the reason. Neither reason means the problem was fixed. See issue trackers for setting up the channel.

An event counts as notified only if at least one enabled channel is subscribed to receive it. With no such channel nothing is sent, and the evidence pack does not count the alert as sent. When you add a channel in the console, Runtime alert is switched on by default.

Conditions available through the API

The API (/api/v1/runtime/rules, admin for writes) accepts more than the form shows. A policy saved with any of these keeps them when it is edited in the console, and the form shows a notice that the policy has settings it does not display.

FieldMeaning
match.rulesFalco rule names, exact, any of the list. This is the form's Falco rules field.
match.tagsFalco tags, with mode anyOf or allOf.
match.priorityop >= or = with value, or in with values, in Falco's priority order.
match.sourceFalco's event source, exact, for example syscall.
match.fieldsConditions on output_fields, for example proc.cmdline, with =, !=, in, contains, startswith, >, <. A condition on a field the event does not carry never matches, != included.
match.imagecompliance (Compliant, NonCompliant, Unknown) is the verdict admission last gave the image. signed is whether its signature verified. An image Attestkeep has never seen matches no image condition.
match.namespacesin and not_in lists.
windowcount events within duration before the action fires, counted per image digest (per namespace and pod without a digest), the current event included. Earlier events in the window are recorded against the policy with action record. The form saves 1 event in 1h, which fires on the first event.
action.kindallow, record or notify. allow keeps the event and does nothing else. Placed first, it quiets a noisy source without changing Falco's rules.
action.notify, action.issue, action.stop, action.stop_modeThe form's switches. Allowed only with notify. stop_mode is immediate (default) or graceful.
action.throttleThe message limit per image. One hour if not set.
noteFree text, stored with the policy and its revisions.

A policy with no condition is refused with 400, and a name already in use with 409. The action block was removed in 1.3.1 and is refused. Existing block policies were changed to notify with issue on.

What is recorded

Each event row carries the policy that matched and the action taken. The policy's name, the hash of its content and the hash of the whole policy set are stored with the decision, so renaming or deleting the policy later does not change what past events show. Every policy change stores the whole set with its hash as a revision and writes an audit entry of kind runtime_rule_change.

Stopping the pod

A policy with Stop the pod on stops the pod named in the event as soon as the event arrives, or as soon as the window is reached if the policy has one. The operator asks the API server to evict the pod, so PodDisruptionBudgets apply. If a budget does not allow the eviction, the pod keeps running and the audit log records why.

The pod's controller starts a new pod. A pod without a controller is gone for good. If the cause is in the image, the new pod can trigger the same event and be stopped again.

Every attempt is written to the audit log as runtime_pod_stop. The actor is policy:<name> and the target is the pod. The detail holds the Falco rule, pod UID, container, image, digest, node, event id and result. Stopping a pod does not change admission: new pods of the same image are admitted as before.

Writing Falco rules in the console

Falco > Rules > New rule (admins) opens an editor for a Falco rules file: rules, macros, lists and overrides in Falco's own YAML, saved under a file name. The editor is in beta.

Completion and checks follow the Falco version running in your cluster. The editor finds Falco by the label the Falco chart sets, app.kubernetes.io/name=falco, in any namespace. If no running pod has that label, you can still write rules but not validate them.

Your Falco loads the file through falcoctl. Add the reference to the falcoctl values of your Falco chart and add the file to rules_files after Falco's own rules. The editor shows these values with your repository and file name filled in:

# values for the falcosecurity/falco chart
falcoctl:
  config:
    artifact:
      install:
        refs:
          - falco-rules:5
          - registry.example.com/falco/my-rules:latest
      follow:
        refs:
          - falco-rules:5
          - registry.example.com/falco/my-rules:latest
        every: 5m
falco:
  rules_files:
    - /etc/falco/falco_rules.yaml
    - /etc/falco/my-rules.yaml

Validating, saving, deleting and publishing are written to the audit log as falco_rule_validate, falco_rule_file_change and falco_rule_file_publish.

Where events show up

Console

Evidence pack

The runtime section contains:

Two timestamps are kept apart. event_time is Falco's own timestamp. received_at is when the operator received the event. Coverage measures whether the feed reached the operator, so it uses received_at, and coverage.clock says so. In the PDF, the decisions and raw events tables name the clock in each time column heading. The longest gap is measured from the start of the period or from the first event this installation ever received, whichever is later.

Include raw runtime events in the Generate dialog (include_runtime_raw: true on POST /api/v1/evidence) adds the stored events themselves, up to 5000. It is off by default.

The runtime-observed attestation is met when the integration is on and at least one event arrived in the period. It is partial when the integration is on and no event arrived, because a quiet cluster and a Falco that stopped sending look the same from the operator's side. It is a gap when the integration is off. See compliance frameworks for the rest of the pack.

Metrics

attestkeep_runtime_events_total{source, priority}
attestkeep_runtime_events_dropped_total{reason="duplicate|unresolved_image|throttled"}
attestkeep_runtime_alerts_notified_total
attestkeep_runtime_rule_matches_total{rule, action}

In attestkeep_runtime_rule_matches_total, rule is the runtime policy's name. unresolved_image does not mean the event was lost. Falco also watches processes on the host, outside any container. Such an event is stored without an image and appears in the evidence pack without one.

Reference

Priorities

Falco's order, most severe first. Names are matched without regard to case, and info is read as informational.

emergency  alert  critical  error  warning  notice  informational  debug

An event with a priority outside this list is stored and matches no priority condition.

Every event is stored

Attestkeep has no priority threshold of its own. Your policies decide what an event leads to, not its priority. To stop Falco sending events you never want to keep, set Falco's own falco.priority. That saves the delivery as well.

Retention

Stored events are kept for runtime.retentionDays days, 90 by default, and removed by an hourly job. 0 keeps them indefinitely. Only events are removed. Issue history, audit entries and policy revisions are kept.

Response codes

CodeMeaning
202Stored. The policy decision is written before this is returned.
204Accepted but not stored, because the same event is already on record.
400The body is not a Falco event, the Content-Type is not JSON, or rule, priority or time is missing.
401The client certificate is from the Falco CA but its CN is not falco.
413The body is larger than 64 KiB.
500The event could not be written. The sender logs a failed delivery.

A connection without a client certificate, or with one from another CA, gets no status code. The TLS handshake fails.

Falco logs every non-2xx answer as a failed delivery. A duplicate is not the sender's fault, so it gets 204, not a 4xx.

Delivering the same event twice is safe. Falco's payload has no delivery id, and Sidekick's uuid is new on every send, so the operator identifies an event by its source, rule, Falco timestamp, container id and pod. The second copy is answered with 204 and not stored.

How an event is tied to an image

The operator reads the image from container.image.repository, container.image.tag and container.image.digest in output_fields. Falco reports the image as the runtime pulled it, for example nginx:1.27. Attestkeep stores images by canonical name, here docker.io/library/nginx, so the event's name is normalised with the same code admission uses before it is stored.

If the event has no digest, the operator uses the digest it most recently recorded for that name and tag. If it has none, the digest stays empty. Without a digest the event still triggers messages and pod stops, but no issue.

Rotating the client certificate

Rotate on Falco > Falco integration, or POST /api/v1/integrations/falco/rotate (admin, audited as runtime_integration_change), issues a new client certificate from the same CA. The old certificate is not revoked and works until it expires, so events keep arriving while you switch.

Download the bundle again, replace the Secret and restart the pods that mount it so they load the new files:

tar xzf attestkeep-falco-mtls.tar.gz
kubectl -n falco create secret generic attestkeep-falco-client \
  --from-file=ca.crt --from-file=client.crt --from-file=client.key \
  --dry-run=client -o yaml | kubectl apply -f -

# connection: mtls
kubectl -n falco rollout restart daemonset/falco
# connection: sidekick
kubectl -n falco rollout restart deployment/falco-falcosidekick

The workload names above are the ones the Falco chart creates for a release named falco. The client certificate is valid for one year. The integration page shows its expiry date, and after it expires the Falco pages say Falco is not connected. Rotating the Falco CA itself is not offered in this version.

When the operator's server certificate rotates

The ca.crt in the bundle is the CA of the operator's server certificate, which is renewed about once a year. The bundle contains the current CA and, while it is still valid, the previous one. Falco therefore keeps connecting through one renewal but not through a second.

Once the CA has changed since the bundle was last downloaded, Falco > Falco integration shows "Certificates changed on <date>". Download the bundle again and replace the Secret as above before the next renewal.

Upgrading from the token endpoint

Releases before 1.2.0 accepted events on the API port with a bearer token (integrations.falco.existingSecret). That is gone. The API port no longer serves POST /api/v1/integrations/falco, and neither the token nor integrations.falco.minPriority is read. Events stop arriving after the upgrade until Falco is connected as described above. Follow steps 1 to 4, then delete the old token Secret.