APIServer Audit Feature
The audit feature enables the cluster administrator to answer the following questions:
- What happened?
- When did it happen?
- Who triggered it?
- On which object(s) did the activity happen?
- Where was it observed?
- Where was it triggered from?
- What was the subsequent behavior of the activity?
The audit log record function will increase the memory consumption of the API server as it needs to store some context required for the audit for each request. In addition, the memory consumption depends on the configuration of the audit log recording.
1. Audit Policy
Set your own audit policy by editing the file /etc/kubernetes/audit-policy.yaml.
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
- "RequestReceived"
rules:
# The cluster contains a large number of the following low-risk requests, it is recommended not to audit (do not log)
# Watch request of kube-proxy
- level: None
users: ["system:kube-proxy"]
verbs: ["watch"]
resources:
- group: "" # core
resources: ["endpoints", "services", "services/status"]
# Get request for configmap under kube-system namespace
- level: None
users: ["system:unsecured"]
namespaces: ["kube-system"]
verbs: ["get"]
resources:
- group: "" # core
resources: ["configmaps"]
# kubelet's get request for node
- level: None
users: ["kubelet"] # legacy kubelet identity
verbs: ["get"]
resources:
- group: "" # core
resources: ["nodes", "nodes/status"]
# system:node user group's get request for node
- level: None
userGroups: ["system:nodes"]
verbs: ["get"]
resources:
- group: "" # core
resources: ["nodes", "nodes/status"]
# Get/update requests for endpoints in kube-system namespace by system components
- level: None
users:
- system:kube-controller-manager
- system:kube-scheduler
- system:serviceaccount:kube-system:endpoint-controller
verbs: ["get", "update"]
namespaces: ["kube-system"]
resources:
- group: "" # core
resources: ["endpoints"]
# apiserver's get request for namespace
- level: None
users: ["system:apiserver"]
verbs: ["get"]
resources:
- group: "" # core
resources: ["namespaces", "namespaces/status", "namespaces/finalize"]
# Get/update requests for configmap, endpoint in kube-system namespace by cluster-autoscaler
- level: None
users: ["cluster-autoscaler"]
verbs: ["get", "update"]
namespaces: ["kube-system"]
resources:
- group: "" # core
resources: ["configmaps", "endpoints"]
# HPA's requests for metrics information through controller manager
- level: None
users:
- system:kube-controller-manager
verbs: ["get", "list"]
resources:
- group: "metrics.k8s.io"
# The following read-only URL
- level: None
nonResourceURLs:
- /healthz*
- /version
- /swagger*
# event
- level: None
resources:
- group: "" # core
resources: ["events"]
# Update and patch requests for nodes by kubelet, system:node-problem-detector and system:nodes, level set to Request, recording metadata and request body
- level: Request
users: ["kubelet", "system:node-problem-detector", "system:serviceaccount:kube-system:node-problem-detector"]
verbs: ["update","patch"]
resources:
- group: "" # core
resources: ["nodes/status", "pods/status"]
- level: Request
userGroups: ["system:nodes"]
verbs: ["update","patch"]
resources:
- group: "" # core
resources: ["nodes/status", "pods/status"]
# The log level for Secrets, ConfigMaps, tokenreviews interfaces that may contain sensitive information or binary files is set to Metadata
- level: Metadata
resources:
- group: "" # core
resources: ["secrets", "configmaps", "serviceaccounts/token"]
- group: authentication.k8s.io
resources: ["tokenreviews"]
# For some get, list, watch requests with a large return body, set to Request
- level: Request
verbs: ["get", "list", "watch"]
resources:
- group: "" # core
- group: "admissionregistration.k8s.io"
- group: "apiextensions.k8s.io"
- group: "apiregistration.k8s.io"
- group: "apps"
- group: "authentication.k8s.io"
- group: "authorization.k8s.io"
- group: "autoscaling"
- group: "batch"
- group: "certificates.k8s.io"
- group: "extensions"
- group: "metrics.k8s.io"
- group: "networking.k8s.io"
- group: "node.k8s.io"
- group: "policy"
- group: "rbac.authorization.k8s.io"
- group: "scheduling.k8s.io"
- group: "storage.k8s.io"
# Set to RequestResponse for Known Kubernetes API
- level: RequestResponse
resources:
- group: "" # core
- group: "admissionregistration.k8s.io"
- group: "apiextensions.k8s.io"
- group: "apiregistration.k8s.io"
- group: "apps"
- group: "authentication.k8s.io"
- group: "authorization.k8s.io"
- group: "autoscaling"
- group: "batch"
- group: "certificates.k8s.io"
- group: "extensions"
- group: "metrics.k8s.io"
- group: "networking.k8s.io"
- group: "node.k8s.io"
- group: "policy"
- group: "rbac.authorization.k8s.io"
- group: "scheduling.k8s.io"
- group: "storage.k8s.io"
# Set all other requests as Metadata
- level: Metadata1.1 Phase (omitStages)
| Stage | Indicates |
|---|---|
| RequestReceived | This stage corresponds to the event generated after the audit processor receives the request and before delegating to other processors |
| ResponseStarted | The event generated after the response message header is sent and before the response body is sent. Only long-running requests (such as watch) generate this stage |
| ResponseComplete | When the response body is complete and no more data needs to be transferred |
| Panic | Generated when panic occurs |
1.2 Audit Level (level)
| Level | Indicates |
|---|---|
| None | Logs that comply with this rule will not be recorded |
| Metadata | Records the metadata of the request (user, timestamp, resource, verb, etc.) but does not record the request or response body |
| Request | Records the metadata of the event and request body, but not the response body. This does not apply to non-resource requests |
| RequestResponse | Records event metadata, request and response body. This does not apply to non-resource requests |
2. Audit Log Configuration
Log in to the 3 Master nodes separately, add the following parameters in the APIServer configuration file /etc/kubernetes/apiserver, and restart APIServer with systemctl restart kube-apiserver:
# Specifies the log file path to write the audit events. Not specifying this flag will disable the log backend.
--audit-log-path=/var/log/audit.log
# Specifies the audit policy configuration file
--audit-policy-file=/etc/kubernetes/audit-policy.yaml
# Specifies the maximum number of days to keep old audit log files.
--audit-log-maxage=7
# Specifies the maximum number of audit log files to keep.
--audit-log-maxbackup=10
# Specifies the maximum size of an audit log file in megabytes.
--audit-log-maxsize=1000