In this post, we’re going to modify the communication between worker nodes and API server.
The communication between these services on worker nodes and master nodes requires two way.
Secure communication between Kubelet and API Server
1. Create kubeconfig file
These keys in this kubeconfig file are used for authentication during the request from worker node to API server.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 |
# again, node authorization requires default user credentail(system:node:xx), and we need to make one kubeconfig each for worker-node root@controller-1:/var/lib/kubernetes/cert# for num in 1 2; do > kubectl config set-cluster k8s-demo \ > --certificate-authority=ca.pem \ > --embed-certs=true \ > --server=https://10.240.0.11:6443 \ > --kubeconfig=worker-${num}-kubelet.kubeconfig > > kubectl config set-credentials system:node:worker-${num} \ > --client-certificate=worker-${num}-kubelet.pem \ > --client-key=worker-${num}-kubelet-key.pem \ > --embed-certs=true \ > --kubeconfig=worker-${num}-kubelet.kubeconfig > > kubectl config set-context default \ > --cluster=k8s-demo \ > --user=system:node:worker-${num} \ > --kubeconfig=worker-${num}-kubelet.kubeconfig > done Cluster "k8s-demo" set. User "system:node:worker-1" set. Context "default" created. Cluster "k8s-demo" set. User "system:node:worker-2" set. Context "default" created. root@controller-1:/var/lib/kubernetes/cert# root@controller-1:/var/lib/kubernetes/cert# kubectl config view --kubeconfig=worker-1-kubelet.kubeconfig apiVersion: v1 clusters: - cluster: certificate-authority-data: REDACTED server: https://10.240.0.11:6443 name: k8s-demo contexts: - context: cluster: k8s-demo user: system:node:worker-1 name: default current-context: default kind: Config preferences: {} users: - name: system:node:worker-1 user: client-certificate-data: REDACTED client-key-data: REDACTED |
2. Modify system service file
This change is required for worker nodes to serve valid HTTPS service so that API server can retrieve the data from worker nodes.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
root@worker-1:~# cat /lib/systemd/system/kubelet.service [Unit] Description=kubelet: The Kubernetes Node Agent Documentation=https://kubernetes.io/docs/home/ [Service] ExecStart=/usr/bin/kubelet --kubeconfig=/var/lib/kubelet/worker.kubeconfig --client-ca-file=/var/lib/kubelet/cert/ca.pem --tls-cert-file=/var/lib/kubelet/cert/worker-1-kubelet.pem --tls-private-key-file=/var/lib/kubelet/cert/worker-1-kubelet-key.pem --cluster-dns=10.32.0.10 --cluster-domain="cluster.local" Restart=always StartLimitInterval=0 RestartSec=10 [Install] WantedBy=multi-user.target root@worker-1:~# systemctl daemon-reload root@worker-1:~# systemctl restart kubelet root@worker-1:~# systemctl status kubelet ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled) Active: active (running) since Sun 2018-10-14 22:41:25 UTC; 4s ago Docs: https://kubernetes.io/docs/home/ Main PID: 11261 (kubelet) Tasks: 10 Memory: 35.1M CPU: 569ms CGroup: /system.slice/kubelet.service └─11261 /usr/bin/kubelet --kubeconfig=/var/lib/kubelet/worker.kubeconfig --client-ca-file=/var/lib/kubelet/cert/ca.pem --tls-cert-file=/va Oct 14 22:41:26 worker-1 kubelet[11261]: I1014 22:41:26.008299 11261 kubelet_node_status.go:79] Attempting to register node worker-1 Oct 14 22:41:26 worker-1 kubelet[11261]: I1014 22:41:26.021699 11261 kubelet_node_status.go:123] Node worker-1 was previously registered Oct 14 22:41:26 worker-1 kubelet[11261]: I1014 22:41:26.022060 11261 kubelet_node_status.go:82] Successfully registered node worker-1 Oct 14 22:41:26 worker-1 kubelet[11261]: I1014 22:41:26.188147 11261 cpu_manager.go:155] [cpumanager] starting with none policy Oct 14 22:41:26 worker-1 kubelet[11261]: I1014 22:41:26.188636 11261 cpu_manager.go:156] [cpumanager] reconciling every 10s Oct 14 22:41:26 worker-1 kubelet[11261]: I1014 22:41:26.188799 11261 policy_none.go:42] [cpumanager] none policy: Start Oct 14 22:41:26 worker-1 kubelet[11261]: Starting Device Plugin manager Oct 14 22:41:26 worker-1 kubelet[11261]: W1014 22:41:26.198923 11261 pod_container_deletor.go:75] Container "74be015696350d458d07a84141e2c9d7ac7963 Oct 14 22:41:26 worker-1 kubelet[11261]: I1014 22:41:26.294844 11261 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started fo Oct 14 22:41:26 worker-1 kubelet[11261]: I1014 22:41:26.295407 11261 reconciler.go:154] Reconciler: start to sync state root@worker-1:~# |
3. Modify API server
Modify API server to communicate with kubelet in HTTPS.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
k_shogo@controller-1:~$ cat /etc/systemd/system/kube-apiserver.service [Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes [Service] ExecStart=/usr/local/bin/kube-apiserver --etcd-servers=http://127.0.0.1:2379 --service-cluster-ip-range=10.32.0.0/24 --insecure-bind-address=0.0.0.0 --client-ca-file=/var/lib/kubernetes/cert/ca.pem --kubelet-certificate-authority=/var/lib/kubernetes/cert/ca.pem --kubelet-client-certificate=/var/lib/kubernetes/cert/apiserver.pem --kubelet-client-key=/var/lib/kubernetes/cert/apiserver-key.pem --tls-cert-file=/var/lib/kubernetes/cert/apiserver.pem --tls-private-key-file=/var/lib/kubernetes/cert/apiserver-key.pem Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target k_shogo@controller-1:~$ sudo systemctl daemon-reload k_shogo@controller-1:~$ sudo systemctl restart kube-apiserver k_shogo@controller-1:~$ sudo systemctl status kube-apiserver ● kube-apiserver.service - Kubernetes API Server Loaded: loaded (/etc/systemd/system/kube-apiserver.service; enabled; vendor preset: enabled) Active: active (running) since Sun 2018-10-14 22:04:58 UTC; 6s ago Docs: https://github.com/kubernetes/kubernetes Main PID: 7012 (kube-apiserver) Tasks: 5 Memory: 282.8M CPU: 6.161s CGroup: /system.slice/kube-apiserver.service └─7012 /usr/local/bin/kube-apiserver --etcd-servers=http://127.0.0.1:2379 --service-cluster-ip-range=10.32.0.0/24 --insecure-bind-address=0.0.0.0 --disable-admission-plugins= Oct 14 22:04:58 controller-1 kube-apiserver[7012]: W1014 22:04:58.777809 7012 authentication.go:377] AnonymousAuth is not allowed with the AllowAll authorizer. Resetting AnonymousAu Oct 14 22:04:59 controller-1 kube-apiserver[7012]: I1014 22:04:59.266542 7012 master.go:228] Using reconciler: master-count Oct 14 22:04:59 controller-1 kube-apiserver[7012]: W1014 22:04:59.454386 7012 genericapiserver.go:342] Skipping API batch/v2alpha1 because it has no resources. Oct 14 22:04:59 controller-1 kube-apiserver[7012]: W1014 22:04:59.466897 7012 genericapiserver.go:342] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources. Oct 14 22:04:59 controller-1 kube-apiserver[7012]: W1014 22:04:59.468811 7012 genericapiserver.go:342] Skipping API storage.k8s.io/v1alpha1 because it has no resources. Oct 14 22:04:59 controller-1 kube-apiserver[7012]: W1014 22:04:59.479568 7012 genericapiserver.go:342] Skipping API admissionregistration.k8s.io/v1alpha1 because it has no resources. Oct 14 22:04:59 controller-1 kube-apiserver[7012]: [restful] 2018/10/14 22:04:59 log.go:33: [restful/swagger] listing is available at https://10.240.0.11:6443/swaggerapi Oct 14 22:04:59 controller-1 kube-apiserver[7012]: [restful] 2018/10/14 22:04:59 log.go:33: [restful/swagger] https://10.240.0.11:6443/swaggerui/ is mapped to folder /swagger-ui/ Oct 14 22:05:01 controller-1 kube-apiserver[7012]: [restful] 2018/10/14 22:05:01 log.go:33: [restful/swagger] listing is available at https://10.240.0.11:6443/swaggerapi Oct 14 22:05:01 controller-1 kube-apiserver[7012]: [restful] 2018/10/14 22:05:01 log.go:33: [restful/swagger] https://10.240.0.11:6443/swaggerui/ is mapped to folder /swagger-ui/ k_shogo@controller-1:~$ |
4. Confirm everything is working fine
Now let’s check if all components is still working.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
# create one busybox instance to test k_shogo@controller-1:~$ kubectl run busybox --image=busybox -- sleep 3600 deployment.apps "busybox" created # it seems kubelet -> API server is working well k_shogo@controller-1:~$ kubectl get pods NAME READY STATUS RESTARTS AGE busybox-5ccc978d8d-clm4t 1/1 Running 0 4s # as the connection from api server to kubelet is in HTTPS, the write operation also succeed # with HTTP, kubectl logs should be timed out k_shogo@controller-1:~$ kubectl exec -it busybox-5ccc978d8d-clm4t -- sh / # ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 16: eth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue link/ether 02:42:0a:c8:01:03 brd ff:ff:ff:ff:ff:ff inet 10.200.1.3/24 brd 10.200.1.255 scope global eth0 valid_lft forever preferred_lft forever / # exit # And the connection is made on port 6443(secure port) rather than 8080(insecure port) root@worker-1:~# lsof -nPi tcp:6443 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME kubelet 8596 root 6u IPv4 71488 0t0 TCP 10.240.0.22:55324->10.240.0.11:6443 (ESTABLISHED) # Kube-Proxy is still using port 8080 root@worker-1:~# lsof -nPi tcp:8080 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME kube-prox 9281 root 9u IPv4 76963 0t0 TCP 10.240.0.22:45702->10.240.0.11:8080 (ESTABLISHED) kube-prox 9281 root 10u IPv4 76964 0t0 TCP 10.240.0.22:45704->10.240.0.11:8080 (ESTABLISHED) root@worker-2:~# |
Secure communication between Kubelet and API Server
1. Create kubeconfig file
This step is almost identical to kubelet. But this config file doesn’t have any node specific info.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
root@controller-1:/var/lib/kubernetes/cert# { > kubectl config set-cluster k8s-demo \ > --certificate-authority=ca.pem \ > --embed-certs=true \ > --server=https://10.240.0.11:6443 \ > --kubeconfig=kube-proxy.kubeconfig > > kubectl config set-credentials system:kube-proxy \ > --client-certificate=kube-proxy.pem \ > --client-key=kube-proxy-key.pem \ > --embed-certs=true \ > --kubeconfig=kube-proxy.kubeconfig > > kubectl config set-context default \ > --cluster=k8s-demo \ > --user=system:kube-proxy \ > --kubeconfig=kube-proxy.kubeconfig > } Cluster "k8s-demo" set. User "system:kube-proxy" set. Context "default" created. |
2. Modify system service file
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
oot@worker-2:~# cat /etc/systemd/system/kube-proxy.service [Unit] Description=Kubernetes Proxy Server Documentation=https://github.com/kubernetes/kubernetes [Service] ExecStart=/usr/local/bin/kube-proxy --kubeconfig=/var/lib/kube-proxy/kube-proxy.kubeconfig Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target root@worker-2:~# systemctl daemon-reload root@worker-2:~# systemctl restart kube-proxy root@worker-2:~# systemctl status kube-proxy ● kube-proxy.service - Kubernetes Proxy Server Loaded: loaded (/etc/systemd/system/kube-proxy.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2018-10-15 23:02:11 UTC; 2s ago Docs: https://github.com/kubernetes/kubernetes Main PID: 11447 (kube-proxy) Tasks: 0 Memory: 9.7M CPU: 109ms CGroup: /system.slice/kube-proxy.service ‣ 11447 /usr/local/bin/kube-proxy --kubeconfig=/var/lib/kube-proxy/kube-proxy.kubeconfig Oct 15 23:02:11 worker-2 kube-proxy[11447]: I1015 23:02:11.947620 11447 conntrack.go:52] Setting nf_conntrack_max to 131072 Oct 15 23:02:11 worker-2 kube-proxy[11447]: I1015 23:02:11.947829 11447 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400 Oct 15 23:02:11 worker-2 kube-proxy[11447]: I1015 23:02:11.947991 11447 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600 Oct 15 23:02:11 worker-2 kube-proxy[11447]: I1015 23:02:11.948606 11447 config.go:202] Starting service config controller Oct 15 23:02:11 worker-2 kube-proxy[11447]: I1015 23:02:11.948813 11447 controller_utils.go:1019] Waiting for caches to sync for service config controller Oct 15 23:02:11 worker-2 kube-proxy[11447]: I1015 23:02:11.949087 11447 config.go:102] Starting endpoints config controller Oct 15 23:02:11 worker-2 kube-proxy[11447]: I1015 23:02:11.949238 11447 controller_utils.go:1019] Waiting for caches to sync for endpoints config controller Oct 15 23:02:12 worker-2 kube-proxy[11447]: I1015 23:02:12.049560 11447 controller_utils.go:1026] Caches are synced for endpoints config controller Oct 15 23:02:12 worker-2 kube-proxy[11447]: I1015 23:02:12.050029 11447 controller_utils.go:1026] Caches are synced for service config controller Oct 15 23:02:12 worker-2 kube-proxy[11447]: E1015 23:02:12.068422 11447 proxier.go:1319] Failed to delete stale service IP 10.32.0.10 connections, error: error deleting connection tra root@worker-2:~# |
3. Confirmation
|
1 2 3 4 5 |
# now both kubelet and kube-proxy is using secure port root@worker-2:~# lsof -nPi tcp:6443 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME kubelet 8596 root 6u IPv4 71488 0t0 TCP 10.240.0.22:55324->10.240.0.11:6443 (ESTABLISHED) kube-prox 11447 root 6u IPv4 93449 0t0 TCP 10.240.0.22:59980->10.240.0.11:6443 (ESTABLISHED) |