In this post, I will talk about admission controller, one of the key component of API server(document reference). In the past few posts, we deployed PKI to secure the communication between each component. It’s time to secure/validate the requests itself. In previous post, I illustrated how API server deals the request, and it can be summarized as follows:
- Authentication … Is the requester a valid account?
- Authorization … Is the requester allowed to do what it request?
- Admission Control … Allow/Reject/Modify the original request based on various criteria.
You may encounter problems like pods not launching with kubectl run command with the recent kubernetes packages. It might be the case related to this admission controller, because kubernetes apiserver launches admission controller enabled by default. In order to successfully launch pods, service account needs to be correctly setup.
Service Account Setup
At this step, we do have service account, which is created by default(and named default) when we launch the cluster. But it doesn’t have any token which is used for authorization.
[ Controller-1 ]
|
1 2 3 4 5 6 7 8 9 10 11 12 |
root@controller-1:/var/lib/kubernetes/cert# kubectl get sa NAME SECRETS AGE default 0 22d root@controller-1:/var/lib/kubernetes/cert# kubectl describe sa/default Name: default Namespace: default Labels: <none> Annotations: <none> Image pull secrets: <none> Mountable secrets: <none> Tokens: <none> Events: <none> |
1. Create key files for Service Account setup
In order for admission controller to successfully serve ServiceAccount plugin, both Token controller(which resides in controller-manager) and API server needs to load the key files as per the official document.
First we create key files:
[ controller-1 ]
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
root@controller-1:/var/lib/kubernetes/cert# cat << EOF > sa-csr.json > { > "CN": "admin", > "key": { > "algo": "rsa", > "size": 2048 > }, > "names": [ > { > "C": "UK", > "L": "London", > "O": "system:masters", > "OU": "k8s Demo" > } > ] > } > EOF root@controller-1:/var/lib/kubernetes/cert# root@controller-1:/var/lib/kubernetes/cert# cfssl gencert \ > -ca=ca.pem \ > -ca-key=ca-key.pem \ > -config=ca-config.json \ > -profile=kubernetes \ > sa-csr.json | cfssljson -bare sa 2018/10/18 22:31:07 [INFO] generate received request 2018/10/18 22:31:07 [INFO] received CSR 2018/10/18 22:31:07 [INFO] generating key: rsa-2048 2018/10/18 22:31:07 [INFO] encoded CSR 2018/10/18 22:31:07 [INFO] signed certificate with serial number |
2. Modify API server system service file
Modify API server service systemd file so that it loads the key file for service account setup to be loaded on launch.
[ Controller-1 ]
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
root@controller-1:/var/lib/kubernetes/cert# cat /etc/systemd/system/kube-apiserver.service [Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes [Service] ExecStart=/usr/local/bin/kube-apiserver --etcd-servers=http://127.0.0.1:2379 --service-cluster-ip-range=10.32.0.0/24 --insecure-bind-address=0.0.0.0 --client-ca-file=/var/lib/kubernetes/cert/ca.pem --kubelet-certificate-authority=/var/lib/kubernetes/cert/ca.pem --kubelet-client-certificate=/var/lib/kubernetes/cert/apiserver.pem --kubelet-client-key=/var/lib/kubernetes/cert/apiserver-key.pem --tls-cert-file=/var/lib/kubernetes/cert/apiserver.pem --tls-private-key-file=/var/lib/kubernetes/cert/apiserver-key.pem --service-account-key-file=/var/lib/kubernetes/cert/sa.pem Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target root@controller-1:/var/lib/kubernetes/cert# systemctl daemon-reload root@controller-1:/var/lib/kubernetes/cert# systemctl restart kube-apiserver root@controller-1:/var/lib/kubernetes/cert# systemctl status kube-apiserver ● kube-apiserver.service - Kubernetes API Server Loaded: loaded (/etc/systemd/system/kube-apiserver.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2018-10-18 22:46:10 UTC; 6s ago Docs: https://github.com/kubernetes/kubernetes Main PID: 3067 (kube-apiserver) Tasks: 6 Memory: 283.1M CPU: 6.147s CGroup: /system.slice/kube-apiserver.service └─3067 /usr/local/bin/kube-apiserver --etcd-servers=http://127.0.0.1:2379 --service-cluster-ip-range=10.32.0.0/24 --insecure-bind-address=0.0.0.0 --client-ca-file=/var/lib/ku Oct 18 22:46:11 controller-1 kube-apiserver[3067]: W1018 22:46:11.569043 3067 genericapiserver.go:342] Skipping API batch/v2alpha1 because it has no resources. Oct 18 22:46:11 controller-1 kube-apiserver[3067]: W1018 22:46:11.581813 3067 genericapiserver.go:342] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources. Oct 18 22:46:11 controller-1 kube-apiserver[3067]: W1018 22:46:11.585110 3067 genericapiserver.go:342] Skipping API storage.k8s.io/v1alpha1 because it has no resources. Oct 18 22:46:11 controller-1 kube-apiserver[3067]: W1018 22:46:11.596088 3067 genericapiserver.go:342] Skipping API admissionregistration.k8s.io/v1alpha1 because it has no resources. Oct 18 22:46:11 controller-1 kube-apiserver[3067]: [restful] 2018/10/18 22:46:11 log.go:33: [restful/swagger] listing is available at https://10.240.0.11:6443/swaggerapi Oct 18 22:46:11 controller-1 kube-apiserver[3067]: [restful] 2018/10/18 22:46:11 log.go:33: [restful/swagger] https://10.240.0.11:6443/swaggerui/ is mapped to folder /swagger-ui/ Oct 18 22:46:13 controller-1 kube-apiserver[3067]: [restful] 2018/10/18 22:46:13 log.go:33: [restful/swagger] listing is available at https://10.240.0.11:6443/swaggerapi Oct 18 22:46:13 controller-1 kube-apiserver[3067]: [restful] 2018/10/18 22:46:13 log.go:33: [restful/swagger] https://10.240.0.11:6443/swaggerui/ is mapped to folder /swagger-ui/ Oct 18 22:46:13 controller-1 kube-apiserver[3067]: W1018 22:46:13.823297 3067 admission.go:68] PersistentVolumeLabel admission controller is deprecated. Please remove this controller Oct 18 22:46:13 controller-1 kube-apiserver[3067]: I1018 22:46:13.825063 3067 plugins.go:149] Loaded 9 admission controller(s) successfully in the following order: NamespaceLifecycle root@controller-1:/var/lib/kubernetes/cert# |
3. Modify Controller-Manager system service file
Modify Controller Manager systemd file so that it loads the service account private key file on launch.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
root@controller-1:/var/lib/kubernetes/cert# cat /etc/systemd/system/kube-controller-manager.service [Unit] Description=Kubernetes Controller Manager Server Documentation=https://github.com/kubernetes/kubernetes [Service] ExecStart=/usr/local/bin/kube-controller-manager --master=http://127.0.0.1:8080 --service-account-private-key-file=/var/lib/kubernetes/cert/sa-key.pem Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target root@controller-1:/var/lib/kubernetes/cert# systemctl daemon-reload root@controller-1:/var/lib/kubernetes/cert# systemctl restart kube-controller-manager root@controller-1:/var/lib/kubernetes/cert# systemctl status kube-controller-manager ● kube-controller-manager.service - Kubernetes Controller Manager Server Loaded: loaded (/etc/systemd/system/kube-controller-manager.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2018-10-18 22:43:39 UTC; 7s ago Docs: https://github.com/kubernetes/kubernetes Main PID: 2994 (kube-controller) Tasks: 5 Memory: 66.1M CPU: 102ms CGroup: /system.slice/kube-controller-manager.service └─2994 /usr/local/bin/kube-controller-manager --master=http://127.0.0.1:8080 --service-account-private-key-file=/var/lib/kubernetes/cert/sa-key.pem Oct 18 22:43:39 controller-1 systemd[1]: Stopped Kubernetes Controller Manager Server. Oct 18 22:43:39 controller-1 systemd[1]: Started Kubernetes Controller Manager Server. Oct 18 22:43:39 controller-1 kube-controller-manager[2994]: I1018 22:43:39.311807 2994 controllermanager.go:116] Version: v1.10.8 Oct 18 22:43:39 controller-1 kube-controller-manager[2994]: W1018 22:43:39.313264 2994 authentication.go:55] Authentication is disabled Oct 18 22:43:39 controller-1 kube-controller-manager[2994]: I1018 22:43:39.313528 2994 insecure_serving.go:44] Serving insecurely on [::]:10252 Oct 18 22:43:39 controller-1 kube-controller-manager[2994]: I1018 22:43:39.313925 2994 leaderelection.go:175] attempting to acquire leader lease kube-system/kube-controller-manager. root@controller-1:/var/lib/kubernetes/cert# |
4. Confirmation
Now we can see if it works well. First check if token is generated for default user.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
root@controller-1:/var/lib/kubernetes/cert# kubectl get sa NAME SECRETS AGE default 1 22d root@controller-1:/var/lib/kubernetes/cert# kubectl describe sa/default Name: default Namespace: default Labels: <none> Annotations: <none> Image pull secrets: <none> Mountable secrets: default-token-wcvlj Tokens: default-token-wcvlj Events: <none> |
Looks good. And let’s see if we can launch pod and how the token is used.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 |
# create deployment root@controller-1:/var/lib/kubernetes/cert# kubectl run nginx --image=nginx deployment.apps "nginx" created root@controller-1:/var/lib/kubernetes/cert# kubectl get deploy NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE nginx 1 1 1 1 7s root@controller-1:/var/lib/kubernetes/cert# kubectl get pods NAME READY STATUS RESTARTS AGE nginx-65899c769f-pgr7r 1/1 Running 0 18s root@controller-1:/var/lib/kubernetes/cert# kubectl describe pods Name: nginx-65899c769f-pgr7r Namespace: default Node: worker-2/10.240.0.22 Start Time: Thu, 18 Oct 2018 22:46:44 +0000 Labels: pod-template-hash=2145573259 run=nginx Annotations: <none> Status: Running IP: 10.200.2.3 Controlled By: ReplicaSet/nginx-65899c769f Containers: nginx: Container ID: docker://7d624d1fd19762c02538ad00e712822c39d8dfa6b04ed643ea5f05504e37a200 Image: nginx Image ID: docker-pullable://nginx@sha256:5704bcdeec8715eb71c95a425f4c4a14264d8c6f92a0b105c23933a2eb503b63 Port: <none> Host Port: <none> State: Running Started: Thu, 18 Oct 2018 22:46:46 +0000 Ready: True Restart Count: 0 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-wcvlj (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: default-token-wcvlj: Type: Secret (a volume populated by a Secret) SecretName: default-token-wcvlj Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 31s default-scheduler Successfully assigned nginx-65899c769f-pgr7r to worker-2 Normal Pulling 30s kubelet, worker-2 pulling image "nginx" Normal Pulled 29s kubelet, worker-2 Successfully pulled image "nginx" Normal Created 29s kubelet, worker-2 Created container Normal Started 29s kubelet, worker-2 Started container |
It looks ok, serviceaccount token is mounted on the container. And the container uses this credential if it needs to communicate with Kubernetes service.
Let’s see the contents of this directory.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
root@controller-1:/var/lib/kubernetes/cert# kubectl exec -it nginx-65899c769f-pgr7r -- /bin/bash root@nginx-65899c769f-pgr7r:/# ls -l /var/run/secrets/kubernetes.io/serviceaccount total 0 lrwxrwxrwx 1 root root 16 Oct 18 22:46 namespace -> ..data/namespace lrwxrwxrwx 1 root root 12 Oct 18 22:46 token -> ..data/token # we can see the namespace root@nginx-65899c769f-pgr7r:/# cat /var/run/secrets/kubernetes.io/serviceaccount/namespace default # and this is the actual token to be used root@nginx-65899c769f-pgr7r:/# cat /var/run/secrets/kubernetes.io/serviceaccount/token eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9...... root@nginx-65899c769f-pgr7r:/# exit |