k8s 12: Admission Controller - Service Account

In this post, I will talk about admission controller, one of the key component of API server(document reference). In the past few posts, we deployed PKI to secure the communication between each component. It’s time to secure/validate the requests itself. In previous post, I illustrated how API server deals the request, and it can be summarized as follows:

Authentication … Is the requester a valid account?
Authorization … Is the requester allowed to do what it request?
Admission Control … Allow/Reject/Modify the original request based on various criteria.

You may encounter problems like pods not launching with kubectl run command with the recent kubernetes packages. It might be the case related to this admission controller, because kubernetes apiserver launches admission controller enabled by default. In order to successfully launch pods, service account needs to be correctly setup.

Service Account Setup

At this step, we do have service account, which is created by default(and named default) when we launch the cluster. But it doesn’t have any token which is used for authorization.

[ Controller-1 ]

root@controller-1:/var/lib/kubernetes/cert# kubectl get sa
NAME      SECRETS   AGE
default   0         22d
root@controller-1:/var/lib/kubernetes/cert# kubectl describe sa/default
Name:                default
Namespace:           default
Labels:              <none>
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   <none>
Tokens:              <none>
Events:              <none>

1. Create key files for Service Account setup

In order for admission controller to successfully serve ServiceAccount plugin, both Token controller(which resides in controller-manager) and API server needs to load the key files as per the official document.

First we create key files:

[ controller-1 ]

root@controller-1:/var/lib/kubernetes/cert# cat << EOF > sa-csr.json
> {
>   "CN": "admin",
>   "key": {
>     "algo": "rsa",
>     "size": 2048
>   },
>   "names": [
>     {
>       "C": "UK",
>       "L": "London",
>       "O": "system:masters",
>       "OU": "k8s Demo"
>     }
>   ]
> }
> EOF
root@controller-1:/var/lib/kubernetes/cert# 
root@controller-1:/var/lib/kubernetes/cert# cfssl gencert \
>   -ca=ca.pem \
>   -ca-key=ca-key.pem \
>   -config=ca-config.json \
>   -profile=kubernetes \
>   sa-csr.json | cfssljson -bare sa
2018/10/18 22:31:07 [INFO] generate received request
2018/10/18 22:31:07 [INFO] received CSR
2018/10/18 22:31:07 [INFO] generating key: rsa-2048
2018/10/18 22:31:07 [INFO] encoded CSR
2018/10/18 22:31:07 [INFO] signed certificate with serial number

2. Modify API server system service file

Modify API server service systemd file so that it loads the key file for service account setup to be loaded on launch.

[ Controller-1 ]

root@controller-1:/var/lib/kubernetes/cert# cat /etc/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-apiserver --etcd-servers=http://127.0.0.1:2379 --service-cluster-ip-range=10.32.0.0/24 --insecure-bind-address=0.0.0.0 --client-ca-file=/var/lib/kubernetes/cert/ca.pem --kubelet-certificate-authority=/var/lib/kubernetes/cert/ca.pem --kubelet-client-certificate=/var/lib/kubernetes/cert/apiserver.pem --kubelet-client-key=/var/lib/kubernetes/cert/apiserver-key.pem --tls-cert-file=/var/lib/kubernetes/cert/apiserver.pem --tls-private-key-file=/var/lib/kubernetes/cert/apiserver-key.pem --service-account-key-file=/var/lib/kubernetes/cert/sa.pem
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
root@controller-1:/var/lib/kubernetes/cert# systemctl daemon-reload
root@controller-1:/var/lib/kubernetes/cert# systemctl restart kube-apiserver
root@controller-1:/var/lib/kubernetes/cert# systemctl status kube-apiserver
● kube-apiserver.service - Kubernetes API Server
   Loaded: loaded (/etc/systemd/system/kube-apiserver.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-10-18 22:46:10 UTC; 6s ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 3067 (kube-apiserver)
    Tasks: 6
   Memory: 283.1M
      CPU: 6.147s
   CGroup: /system.slice/kube-apiserver.service
           └─3067 /usr/local/bin/kube-apiserver --etcd-servers=http://127.0.0.1:2379 --service-cluster-ip-range=10.32.0.0/24 --insecure-bind-address=0.0.0.0 --client-ca-file=/var/lib/ku

Oct 18 22:46:11 controller-1 kube-apiserver[3067]: W1018 22:46:11.569043    3067 genericapiserver.go:342] Skipping API batch/v2alpha1 because it has no resources.
Oct 18 22:46:11 controller-1 kube-apiserver[3067]: W1018 22:46:11.581813    3067 genericapiserver.go:342] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
Oct 18 22:46:11 controller-1 kube-apiserver[3067]: W1018 22:46:11.585110    3067 genericapiserver.go:342] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
Oct 18 22:46:11 controller-1 kube-apiserver[3067]: W1018 22:46:11.596088    3067 genericapiserver.go:342] Skipping API admissionregistration.k8s.io/v1alpha1 because it has no resources.
Oct 18 22:46:11 controller-1 kube-apiserver[3067]: [restful] 2018/10/18 22:46:11 log.go:33: [restful/swagger] listing is available at https://10.240.0.11:6443/swaggerapi
Oct 18 22:46:11 controller-1 kube-apiserver[3067]: [restful] 2018/10/18 22:46:11 log.go:33: [restful/swagger] https://10.240.0.11:6443/swaggerui/ is mapped to folder /swagger-ui/
Oct 18 22:46:13 controller-1 kube-apiserver[3067]: [restful] 2018/10/18 22:46:13 log.go:33: [restful/swagger] listing is available at https://10.240.0.11:6443/swaggerapi
Oct 18 22:46:13 controller-1 kube-apiserver[3067]: [restful] 2018/10/18 22:46:13 log.go:33: [restful/swagger] https://10.240.0.11:6443/swaggerui/ is mapped to folder /swagger-ui/
Oct 18 22:46:13 controller-1 kube-apiserver[3067]: W1018 22:46:13.823297    3067 admission.go:68] PersistentVolumeLabel admission controller is deprecated. Please remove this controller
Oct 18 22:46:13 controller-1 kube-apiserver[3067]: I1018 22:46:13.825063    3067 plugins.go:149] Loaded 9 admission controller(s) successfully in the following order: NamespaceLifecycle
root@controller-1:/var/lib/kubernetes/cert#

3. Modify Controller-Manager system service file

Modify Controller Manager systemd file so that it loads the service account private key file on launch.

root@controller-1:/var/lib/kubernetes/cert# cat /etc/systemd/system/kube-controller-manager.service 
[Unit]
Description=Kubernetes Controller Manager Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-controller-manager  --master=http://127.0.0.1:8080 --service-account-private-key-file=/var/lib/kubernetes/cert/sa-key.pem
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
root@controller-1:/var/lib/kubernetes/cert# systemctl daemon-reload
root@controller-1:/var/lib/kubernetes/cert# systemctl restart kube-controller-manager
root@controller-1:/var/lib/kubernetes/cert# systemctl status kube-controller-manager
● kube-controller-manager.service - Kubernetes Controller Manager Server
   Loaded: loaded (/etc/systemd/system/kube-controller-manager.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-10-18 22:43:39 UTC; 7s ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 2994 (kube-controller)
    Tasks: 5
   Memory: 66.1M
      CPU: 102ms
   CGroup: /system.slice/kube-controller-manager.service
           └─2994 /usr/local/bin/kube-controller-manager --master=http://127.0.0.1:8080 --service-account-private-key-file=/var/lib/kubernetes/cert/sa-key.pem

Oct 18 22:43:39 controller-1 systemd[1]: Stopped Kubernetes Controller Manager Server.
Oct 18 22:43:39 controller-1 systemd[1]: Started Kubernetes Controller Manager Server.
Oct 18 22:43:39 controller-1 kube-controller-manager[2994]: I1018 22:43:39.311807    2994 controllermanager.go:116] Version: v1.10.8
Oct 18 22:43:39 controller-1 kube-controller-manager[2994]: W1018 22:43:39.313264    2994 authentication.go:55] Authentication is disabled
Oct 18 22:43:39 controller-1 kube-controller-manager[2994]: I1018 22:43:39.313528    2994 insecure_serving.go:44] Serving insecurely on [::]:10252
Oct 18 22:43:39 controller-1 kube-controller-manager[2994]: I1018 22:43:39.313925    2994 leaderelection.go:175] attempting to acquire leader lease  kube-system/kube-controller-manager.
root@controller-1:/var/lib/kubernetes/cert#

4. Confirmation

Now we can see if it works well. First check if token is generated for default user.

root@controller-1:/var/lib/kubernetes/cert# kubectl get sa
NAME      SECRETS   AGE
default   1         22d
root@controller-1:/var/lib/kubernetes/cert# kubectl describe sa/default
Name:                default
Namespace:           default
Labels:              <none>
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   default-token-wcvlj
Tokens:              default-token-wcvlj
Events:              <none>

Looks good. And let’s see if we can launch pod and how the token is used.

# create deployment
root@controller-1:/var/lib/kubernetes/cert# kubectl run nginx --image=nginx
deployment.apps "nginx" created
root@controller-1:/var/lib/kubernetes/cert# kubectl get deploy
NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx     1         1         1            1           7s
root@controller-1:/var/lib/kubernetes/cert# kubectl get pods
NAME                     READY     STATUS    RESTARTS   AGE
nginx-65899c769f-pgr7r   1/1       Running   0          18s
root@controller-1:/var/lib/kubernetes/cert# kubectl describe pods
Name:           nginx-65899c769f-pgr7r
Namespace:      default
Node:           worker-2/10.240.0.22
Start Time:     Thu, 18 Oct 2018 22:46:44 +0000
Labels:         pod-template-hash=2145573259
                run=nginx
Annotations:    <none>
Status:         Running
IP:             10.200.2.3
Controlled By:  ReplicaSet/nginx-65899c769f
Containers:
  nginx:
    Container ID:   docker://7d624d1fd19762c02538ad00e712822c39d8dfa6b04ed643ea5f05504e37a200
    Image:          nginx
    Image ID:       docker-pullable://nginx@sha256:5704bcdeec8715eb71c95a425f4c4a14264d8c6f92a0b105c23933a2eb503b63
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Thu, 18 Oct 2018 22:46:46 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wcvlj (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-wcvlj:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-wcvlj
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  31s   default-scheduler  Successfully assigned nginx-65899c769f-pgr7r to worker-2
  Normal  Pulling    30s   kubelet, worker-2  pulling image "nginx"
  Normal  Pulled     29s   kubelet, worker-2  Successfully pulled image "nginx"
  Normal  Created    29s   kubelet, worker-2  Created container
  Normal  Started    29s   kubelet, worker-2  Started container

It looks ok, serviceaccount token is mounted on the container. And the container uses this credential if it needs to communicate with Kubernetes service.

Let’s see the contents of this directory.

root@controller-1:/var/lib/kubernetes/cert# kubectl exec -it nginx-65899c769f-pgr7r -- /bin/bash
root@nginx-65899c769f-pgr7r:/# ls -l /var/run/secrets/kubernetes.io/serviceaccount
total 0
lrwxrwxrwx 1 root root 16 Oct 18 22:46 namespace -> ..data/namespace
lrwxrwxrwx 1 root root 12 Oct 18 22:46 token -> ..data/token
# we can see the namespace
root@nginx-65899c769f-pgr7r:/# cat /var/run/secrets/kubernetes.io/serviceaccount/namespace
default
# and this is the actual token to be used
root@nginx-65899c769f-pgr7r:/# cat /var/run/secrets/kubernetes.io/serviceaccount/token    
eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9......
root@nginx-65899c769f-pgr7r:/# exit

k8s 12: Admission Controller – Service Account

Service Account Setup

Related

Service Account Setup

Share this:

Related