In our process to improve security within our Kubernetes Clusters, we go through various steps in understanding the overall flow of the processes which are generally executed and then apply security best practices.

Let's take an example of creating a pod in a cluster.

This process would start out with a user using a kubectl command(kubectl run nginx --image nginx) having his/her kubeconfig file stored in the laptop.

That kubeconfig file would have the certificate information about the user which will serve the purpose of Authentication, next in the process we have RBAC, RBAC serves the purpose of providing Authorization which dictates what that user is authorized to performed in the cluster once authenticated. Roles and Role/ClusterRole Bindings will describe the type of API Groups or API resources that user has access to and goes 1 step further describing what type of actions(verbs) can the user perform on those API objects. You can tke this even further by providing specific resourceNames to which the user can perform actions on using the resourceNames field.

With RBAC, we are able to serve the purpose of authorization, however we are not able to enforce policies such as - restricting images which certain tags(latest tag), or pulling from a public docker registry, etc. This is where Admission Controllers come to the rescue.

I wrote a primer on it at https://amitrathod.postach.io/post/admission-controllers-dynamic-webhooks-and-opa-gatekeeper

Let's describe this a bit further.

To see what admission controllers are enabled by default, the easiest aproach is running kube-apiserver -h | grep enable-admission-plugins command, this will show the list which is enabled.

In a kubeadm setup run the same command within the kube-apiserver pod

kubectl -n kube-system exec kube-apiserver -- kube-apiserver -h | grep enable-admission-plugins

If you want to add an admission controller , add the name of admission controller in the --enable-admission-plugins list within the apiserver's manifest file. Alternately, if you wish to disable one, add that to the list --disable-admission-plugins.

Now let's take a step further to understand types of admission controllers, Validating and Mutating Admission Controllers.

Let's take the example of creating a pvc(kubectl create pvc mypvc ...), where you dont specify a storageClass by default, there is a default Admission Controller called as DefaultStorageClass which will add a default storage class named as default in the spec of the pvc. This is a type of a mutating admission controller which mutates the pvc object to add the defaultstorageclass.

Those admission controllers which validate a request are called as validating Admission controllers.

Mutating admission controllers are run first and then the validating ones this is a good idea since validating admission controllers can see the final version after the mutating admission controllers have made the necessary modifications.

Also remember mutating admission controllers need to be idempotent so that the order in which they are called shouldnt play too much of a role while being executed.

If you would like to run your own set of admission controllers, we can do so by using WebHooks. These webhooks are provided to support external Admission Controllers, there are two types of webhooks provided , mutating and validation admission webhooks.

Once the object creation goes through all the default Admission Controllers, it hits the mutating and validating admission webhooks which passes in an AdmissionReview object to the Admission Webhook Server. This AdmissionReview object has all the details about that object in a json format.

Upon review of the AdmissionReview request , the Admission Webhook server sends out a response in the same AdmissionReview object type with a reponse in the json format telling the Webhook Controllers whether the request is allowed or denied.

Assuming that the admission webhook server is deployed within your cluster as a deployment, it will need a service created for it to be reachable , so we create a service for the deployment.

Now we need to create an Object of the type ValidatingWebhookConfiguration/MutatingWebhookConfiguration with

    apiVersion: admissionregistration.k8s.io/v1
    kind: ValidatingWebhookConfiguration
    metadata: 
        name: podcreatevalidation
    webhooks: 
    - name: podcreatevalidation
      admissionReviewVersions: ["v1"]
    clientConfig: 
        service:
        name: validation-service
        namespace: webhook
        caBundle: "insert certificate here"
    rules:
    - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE"]
        resources: ["pods"]  

So in this WebhookConfiguration, we are advising that for every pod to be created, the Admission Webhook Server(deployed as a Deployment and exposed as service called as a validation-service) to be consulted every time.

When an apiserver receives a request that matches one of the rules, the apiserver sends an admissionReview request to webhook as specified in the clientConfig.

In case of a Mutating Admission controller upon receipt of an AdmissionReview from the API Server, they will alter the objects before passing on to the API Server. Think how Istio injects sidecar containers when the namespace has the label istio-injection=enabled .

The Admission Controller itself doesnt alter the object, however when it sends in a response to the API server , it will describe how to modify the object with JSONPatch objects.

Ok, so from a system design perspective, you should always think about what should happen if the Admission webhook is unreachable, in those cases, we should be setting the failurePolicy to either Fail or Ignore. For the critical security based webhooks, I would advise setting them up as Fail , however for the non-critical one's use precaution and you can set them to Ignore.

Also think of High Availability , in case of critical weebhooks think about how will you provide HA to the webhooks.