Auto-Gen Rules for Pod Controllers

Pods are one of the most common object types in Kubernetes and as such are the focus of most types of validation rules. But creation of Pods directly is almost never done as it is considered an anti-pattern. Instead, Kubernetes has many higher-level controllers that directly or indirectly manage Pods, namely the Deployment, DaemonSet, StatefulSet, Job, and CronJob resources. Writing policy that targets Pods but must be written for every one of these controllers would be tedious and inefficient. Kyverno solves this issue by supporting automatic generation of policy rules for higher-level controllers from a rule written for a Pod.

For example, when creating a validation policy like below which checks that all images come from an internal, trusted registry, the policy applies to all resources capable of generating Pods.

 1apiVersion : kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: restrict-image-registries
 5spec:
 6  validationFailureAction: enforce
 7  rules:
 8  - name: validate-registries
 9    match:
10      any:
11      - resources:
12          kinds:
13          - Pod
14    validate:
15      message: "Images may only come from our internal enterprise registry."
16      pattern:
17        spec:
18          containers:
19          - image: "registry.domain.com/*"

Once the policy is created, these other resources can be shown in auto-generated rules which Kyverno adds to the policy.

 1spec:
 2  background: true
 3  failurePolicy: Fail
 4  rules:
 5  - match:
 6      any:
 7      - resources:
 8          kinds:
 9          - Pod
10    name: validate-registries
11    validate:
12      message: Images may only come from our internal enterprise registry.
13      pattern:
14        spec:
15          containers:
16          - image: registry.domain.com/*
17  - match:
18      any:
19      - resources:
20          kinds:
21          - DaemonSet
22          - Deployment
23          - Job
24          - StatefulSet
25    name: autogen-validate-registries
26    validate:
27      message: Images may only come from our internal enterprise registry.
28      pattern:
29        spec:
30          template:
31            spec:
32              containers:
33              - image: registry.domain.com/*
34  - match:
35      any:
36      - resources:
37          kinds:
38          - CronJob
39    name: autogen-cronjob-validate-registries
40    validate:
41      message: Images may only come from our internal enterprise registry.
42      pattern:
43        spec:
44          jobTemplate:
45            spec:
46              template:
47                spec:
48                  containers:
49                  - image: registry.domain.com/*
50  validationFailureAction: enforce

This auto-generation behavior is controlled by the pod-policies.kyverno.io/autogen-controllers annotation.

By default, Kyverno inserts an annotation pod-policies.kyverno.io/autogen-controllers=DaemonSet,Deployment,Job,StatefulSet,CronJob, to generate additional rules that are applied to these controllers.

You can change the annotation pod-policies.kyverno.io/autogen-controllers to customize the target Pod controllers for the auto-generated rules. For example, Kyverno generates a rule for a Deployment if the annotation of policy is defined as pod-policies.kyverno.io/autogen-controllers=Deployment.

Kyverno skips generating Pod controller rules whenever the following resources fields/objects are specified in a match or exclude block as these filters may not be applicable to Pod controllers:

• names
• selector
• annotations

To disable auto-generating rules for Pod controllers set pod-policies.kyverno.io/autogen-controllers to the value none.

When disabling auto-generation rules for select Pod controllers, Kyverno still applies policy matching on Pods to those spawned by those controllers. To exempt these Pods, use preconditions with an expression similar to the below which may allow Pods created by a Job controller to pass.

1- key: Job
2  operator: AnyNotIn
3  value: "{{ request.object.metadata.ownerReferences[].kind }}"

Exclusion by Metadata

In some cases it may be desirable to use an exclude block applied to Pods that uses either labels or annotations. For example, the following match and exclude statement may be written, the purpose of which would be to match any Pods except those that have the annotation policy.test/require-requests-limits=skip.

 1rules:
 2  - name: validate-resources
 3    match:
 4      any:
 5      - resources:
 6          kinds:
 7            - Pod
 8    exclude:
 9      any:
10      - resources:
11          annotations:
12            policy.test/require-requests-limits: skip

When Kyverno sees these types of fields as mentioned above it skips auto-generation for the rule. The next choice may be to use preconditions to achieve the same effect but by writing an expression that looks at request.object.metadata.*. As part of auto-generation, Kyverno will see any variables from AdmissionReview such as that beginning with request.object and translate it for each of the applicable Pod controllers. The result may be that the auto-generated rule for, as an example, Deployments will get translated to request.object.spec.template.metadata.* which references the metadata object inside the Pod template and not the metadata object of the Deployment itself. To work around this and have preconditions which are not translated for these metadata use cases, double quote the object portion of the variable as shown below.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: require-requests-limits
 5spec:
 6  validationFailureAction: enforce
 7  background: true
 8  rules:
 9    - name: validate-resources
10      match:
11        any:
12        - resources:
13            kinds:
14              - Pod
15      preconditions:
16        all:
17        - key: "{{ request.\"object\".metadata.annotations.\"policy.test.io/require-requests-limits\" || '' }}"
18          operator: NotEquals
19          value: skip
20      validate:
21        message: "CPU and memory resource requests and limits are required."
22        pattern:
23          spec:
24            containers:
25              - resources:
26                  requests:
27                    memory: "?*"
28                    cpu: "?*"
29                  limits:
30                    memory: "?*"

The result will have the same effect as the first snippet which uses an exclude block and have the benefit of auto-generation coverage.