We’ll explore how to expose container services so that it can be accessible from other containers and outside network. Kubernetes service details is here in the official document, but in this entry I show you NodePort and ClusterIP only.
First of all, we need to change the cluster config again. Since we don’t have any overlay network yet, I’m going to configure each worker node to have unique container ip ranges so that they can be accessible via their IP address directly. This is, again, not ideal and it will be addressed later.
Prepare for container routing
1. Change IP address range for containers
[ worker-1 ]
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 |
root@worker-1:/home/k_shogo# docker network list NETWORK ID NAME DRIVER SCOPE 1a6da5939ab8 bridge bridge local f33806236084 host host local ba28deac8b49 none null local root@worker-1:/home/k_shogo# docker network inspect bridge | grep -i gateway "Gateway": "172.17.0.1" root@worker-1:/home/k_shogo# root@worker-1:/home/k_shogo# cat /etc/docker/daemon.json cat: /etc/docker/daemon.json: No such file or directory root@worker-1:/home/k_shogo# cat << EOF >> /etc/docker/daemon.json > { > "bip": "10.200.1.1/24" > } > EOF root@worker-1:/home/k_shogo# systemctl restart docker root@worker-1:/home/k_shogo# systemctl status docker ● docker.service - Docker Application Container Engine Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2018-09-29 22:30:12 UTC; 6s ago Docs: https://docs.docker.com Main PID: 5910 (dockerd) Tasks: 29 Memory: 48.1M CPU: 453ms CGroup: /system.slice/docker.service ├─5910 /usr/bin/dockerd -H fd:// ├─5938 docker-containerd --config /var/run/docker/containerd/containerd.toml └─6107 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/694478b80a3684f5cc71fff3bc61032196a7c415b34b0c4a1b3a3eca7 Sep 29 22:30:12 worker-1 dockerd[5910]: time="2018-09-29T22:30:12.442861847Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc Sep 29 22:30:12 worker-1 dockerd[5910]: time="2018-09-29T22:30:12.443043590Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420189cc0, CONNECTING" module=grpc Sep 29 22:30:12 worker-1 dockerd[5910]: time="2018-09-29T22:30:12.443812760Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420189cc0, READY" module=grpc Sep 29 22:30:12 worker-1 dockerd[5910]: time="2018-09-29T22:30:12.444012860Z" level=info msg="Loading containers: start." Sep 29 22:30:12 worker-1 dockerd[5910]: time="2018-09-29T22:30:12.645679205Z" level=info msg="Loading containers: done." Sep 29 22:30:12 worker-1 dockerd[5910]: time="2018-09-29T22:30:12.664604761Z" level=info msg="Docker daemon" commit=e68fc7a graphdriver(s)=overlay2 version=18.06.1-ce Sep 29 22:30:12 worker-1 dockerd[5910]: time="2018-09-29T22:30:12.665023812Z" level=info msg="Daemon has completed initialization" Sep 29 22:30:12 worker-1 systemd[1]: Started Docker Application Container Engine. Sep 29 22:30:12 worker-1 dockerd[5910]: time="2018-09-29T22:30:12.677158468Z" level=info msg="API listen on /var/run/docker.sock" Sep 29 22:30:12 worker-1 dockerd[5910]: time="2018-09-29T22:30:12Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/694478b80a3684f5cc71fff3bc61032196a7c415b root@worker-1:/home/k_shogo# docker network list NETWORK ID NAME DRIVER SCOPE c4fdf5ec80a8 bridge bridge local f33806236084 host host local ba28deac8b49 none null local root@worker-1:/home/k_shogo# docker network inspect bridge | grep -i gateway "Gateway": "10.200.1.1" root@worker-1:/home/k_shogo# docker network inspect bridge | grep -i subnet "Subnet": "10.200.1.1/24", |
You need to do the same thing on each worker node, just change the bip to respective ip address.
Deploy Kube-Proxy
1. Install Kube-proxy service
[ worker-1 ]
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
root@worker-1:/home/k_shogo/work# mv ./kubernetes/server/ root@worker-1:/home/k_shogo/work# mv ./kubernetes/server/bin/kube-proxy /usr/local/bin/ root@worker-1:/home/k_shogo/work# cat << EOF >> /etc/systemd/system/kube-proxy.service > [Unit] > Description=Kubernetes Proxy Server > Documentation=https://github.com/kubernetes/kubernetes > > [Service] > ExecStart=/usr/local/bin/kube-proxy --master=http://10.240.0.11:8080 > Restart=on-failure > RestartSec=5 > > [Install] > WantedBy=multi-user.target > EOF root@worker-1:/home/k_shogo/work# systemctl start kube-proxy root@worker-1:/home/k_shogo/work# systemctl status kube-proxy ● kube-proxy.service - Kubernetes Proxy Server Loaded: loaded (/etc/systemd/system/kube-proxy.service; disabled; vendor preset: enabled) Active: active (running) since Sat 2018-09-29 16:40:51 UTC; 8s ago Docs: https://github.com/kubernetes/kubernetes Main PID: 3058 (kube-proxy) Tasks: 0 Memory: 9.1M CPU: 107ms CGroup: /system.slice/kube-proxy.service ‣ 3058 /usr/local/bin/kube-proxy --master=http://10.240.0.11:8080 Sep 29 16:40:51 worker-1 kube-proxy[3058]: I0929 16:40:51.491012 3058 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072 Sep 29 16:40:51 worker-1 kube-proxy[3058]: I0929 16:40:51.491063 3058 conntrack.go:52] Setting nf_conntrack_max to 131072 Sep 29 16:40:51 worker-1 kube-proxy[3058]: I0929 16:40:51.491113 3058 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400 Sep 29 16:40:51 worker-1 kube-proxy[3058]: I0929 16:40:51.491134 3058 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600 Sep 29 16:40:51 worker-1 kube-proxy[3058]: I0929 16:40:51.491875 3058 config.go:202] Starting service config controller Sep 29 16:40:51 worker-1 kube-proxy[3058]: I0929 16:40:51.491900 3058 controller_utils.go:1019] Waiting for caches to sync for service config controller Sep 29 16:40:51 worker-1 kube-proxy[3058]: I0929 16:40:51.491936 3058 config.go:102] Starting endpoints config controller Sep 29 16:40:51 worker-1 kube-proxy[3058]: I0929 16:40:51.491941 3058 controller_utils.go:1019] Waiting for caches to sync for endpoints config controller Sep 29 16:40:51 worker-1 kube-proxy[3058]: I0929 16:40:51.592246 3058 controller_utils.go:1026] Caches are synced for endpoints config controller Sep 29 16:40:51 worker-1 kube-proxy[3058]: I0929 16:40:51.592935 3058 controller_utils.go:1026] Caches are synced for service config controller root@worker-1:/home/k_shogo/work# systemctl enable kube-proxy Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /etc/systemd/system/kube-proxy.service. |
Test – Node Port service
With Node Port, the container can be accessed by its host IP address. So it can be accessed from anywhere(even the internet) as long as they have access to this host IP address.
1. Deploy nginx and expose that service with node port
[ controller-1 ]
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
k_shogo@controller-1:~/work$ cat nginx-nodeport.yaml apiVersion: v1 kind: Pod metadata: name: nginxtest-nodeport labels: app: nginx lab: nodeport spec: containers: - name: nginx image: nginx ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: nginxtest-nodeport spec: type: NodePort selector: app: nginx lab: nodeport ports: - name: nginxtest-nodeport port: 80 targetPort: 80 k_shogo@controller-1:~/work$ kubectl apply -f nginx-nodeport.yaml pod "nginxtest-nodeport" created service "nginxtest-nodeport" created |
2. Confirm the service is running
[ controller-1 ]
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
k_shogo@controller-1:~/work$ kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE nginxtest-nodeport 1/1 Running 0 1m 10.200.1.2 worker-1 k_shogo@controller-1:~/work$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.32.0.1 <none> 443/TCP 5d nginxtest-nodeport NodePort 10.32.0.89 <none> 80:30070/TCP 12s k_shogo@controller-1:~/work$ kubectl describe svc/nginxtest-nodeport Name: nginxtest-nodeport Namespace: default Labels: <none> Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"nginxtest-nodeport","namespace":"default"},"spec":{"ports":[{"name":"nginxtest... Selector: app=nginx,lab=nodeport Type: NodePort IP: 10.32.0.89 Port: nginxtest-nodeport 80/TCP TargetPort: 80/TCP NodePort: nginxtest-nodeport 30070/TCP Endpoints: 10.200.1.2:80 Session Affinity: None External Traffic Policy: Cluster Events: <none> |
From this output, you can find the nginx container(its actual pod ip address is 10.200.1.2) is assigned node port 30070. Let’s check if it is the case.
3. Check if node port works
[ worker-2 ]
|
1 2 3 4 5 6 |
root@worker-2:/home/k_shogo# curl http://worker-1:30070 <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> ... |
[ controller-1 ]
|
1 2 3 4 5 6 |
shogo@controller-1:~/work$ curl http://10.240.0.21:30070 <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> ... |
It is working.
It’s because the host (in this case worker-1) makes the iptables rule to forward the request on 30070 port to the container ip address and port internally. Because it’s happening only on the destination host, it can be accessed from anywhere.
[ worker-1 ]
|
1 2 3 4 5 6 7 8 9 10 11 12 |
root@worker-1:~# iptables-save | grep 30070 -A KUBE-NODEPORTS -p tcp -m comment --comment "default/nginxtest-nodeport:nginxtest-nodeport" -m tcp --dport 30070 -j KUBE-MARK-MASQ -A KUBE-NODEPORTS -p tcp -m comment --comment "default/nginxtest-nodeport:nginxtest-nodeport" -m tcp --dport 30070 -j KUBE-SVC-Y4FZLDNSKOURDUGQ root@worker-1:~# iptables -L KUBE-SVC-Y4FZLDNSKOURDUGQ -t nat Chain KUBE-SVC-Y4FZLDNSKOURDUGQ (2 references) target prot opt source destination KUBE-SEP-QXCV3YI6FCBDVWQL all -- anywhere anywhere /* default/nginxtest-nodeport:nginxtest-nodeport */ root@worker-1:~# iptables -L KUBE-SEP-QXCV3YI6FCBDVWQL -t nat Chain KUBE-SEP-QXCV3YI6FCBDVWQL (1 references) target prot opt source destination KUBE-MARK-MASQ all -- 10.200.1.2 anywhere /* default/nginxtest-nodeport:nginxtest-nodeport */ DNAT tcp -- anywhere anywhere /* default/nginxtest-nodeport:nginxtest-nodeport */ tcp to:10.200.1.2:80 |
Test – Cluster IP service
With cluster ip, the container can be accessed from only those hosts with kube-proxy(, which is in the same cluster). The controller node is not the exception and it should have kube-proxy running if it needs access to those cluster ip address.
1. Deploy nginx and expose that service with cluster ip
[ controller-1 ]
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
shogo@controller-1:~/work$ cat nginx-clusterip.yaml apiVersion: v1 kind: Pod metadata: name: nginx-clusterip labels: app: nginx lab: clusterip spec: containers: - name: nginx image: nginx ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: nginx-clusterip spec: type: ClusterIP selector: app: nginx lab: clusterip ports: - name: nginx-clusterip protocol: TCP port: 8080 targetPort: 80 shogo@controller-1:~/work$ kubectl apply -f nginx-clusterip.yaml pod "nginx-clusterip" created service "nginx-clusterip" created |
2. Confirm the service is running
[ controller-1 ]
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
shogo@controller-1:~/work$ kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE nginx-clusterip 1/1 Running 0 36s 10.200.2.2 worker-2 nginxtest-nodeport 1/1 Running 0 15m 10.200.1.2 worker-1 shogo@controller-1:~/work$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.32.0.1 <none> 443/TCP 5d nginx-clusterip ClusterIP 10.32.0.50 <none> 8080/TCP 45s nginxtest-nodeport NodePort 10.32.0.89 <none> 80:30070/TCP 15m shogo@controller-1:~/work$ kubectl describe svc/nginx-clusterip Name: nginx-clusterip Namespace: default Labels: <none> Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"nginx-clusterip","namespace":"default"},"spec":{"ports":[{"name":"nginx-cluste... Selector: app=nginx,lab=clusterip Type: ClusterIP IP: 10.32.0.50 Port: nginx-clusterip 8080/TCP TargetPort: 80/TCP Endpoints: 10.200.2.2:80 Session Affinity: None Events: <none> |
From this output, you can find the nginx container(its actual pod ip address is 10.200.2.2) is assigned cluster ip 10.32.0.50 and port 8080. Let’s check if it is really working.
3. Check if cluster ip service works
[ worker-1 ]
|
1 2 3 4 5 6 |
root@worker-1:~# curl http://10.32.0.50:8080 <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> ... |
[ controller-1 ]
|
1 2 |
shogo@controller-1:~/work$ curl http://10.32.0.50:8080 curl: (7) Failed to connect to 10.32.0.50 port 8080: Connection timed out |
Yes, it’s working, and it can be accessed only from worker nodes. Since the controller node doesn’t have kube-proxy configured, it doesn’t know how to (pre-)forward the request to the cluster-ip. Since this cluster ip is not known to the external network, it’s dropped eventually.
As the result of the kube-rpoxy modification on iptables, actually the original request is re-written to the request to the backend container ip address and port when it leaves the host.
[ controller-1 ] so it’s technically same it sends request to the container ip address. And it can be reachable from the controller node as well.
|
1 2 3 4 5 6 |
shogo@controller-1:~/work$ curl http://10.200.2.2 <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> ... |
[ worker-1 ] Kube-proxy on this node changes the iptables, and it modifies the request.
|
1 2 3 4 5 6 7 8 9 10 11 |
root@worker-1:~# iptables-save | grep 10.32.0.50 -A KUBE-SERVICES -d 10.32.0.50/32 -p tcp -m comment --comment "default/nginx-clusterip:nginx-clusterip cluster IP" -m tcp --dport 8080 -j KUBE-SVC-A4IMIIJJXTXEHWFI root@worker-1:~# iptables -L KUBE-SVC-A4IMIIJJXTXEHWFI -t nat Chain KUBE-SVC-A4IMIIJJXTXEHWFI (1 references) target prot opt source destination KUBE-SEP-2V4CHBMLT6MH5RD7 all -- anywhere anywhere /* default/nginx-clusterip:nginx-clusterip */ root@worker-1:~# iptables -L KUBE-SEP-2V4CHBMLT6MH5RD7 -t nat Chain KUBE-SEP-2V4CHBMLT6MH5RD7 (1 references) target prot opt source destination KUBE-MARK-MASQ all -- 10.200.2.2 anywhere /* default/nginx-clusterip:nginx-clusterip */ DNAT tcp -- anywhere anywhere /* default/nginx-clusterip:nginx-clusterip */ tcp to:10.200.2.2:80 |
Housekeeping
Delete the pod/services we made during this lab.
|
1 2 3 4 5 6 |
shogo@controller-1:~/work$ kubectl delete -f nginx-nodeport.yaml pod "nginxtest-nodeport" deleted service "nginxtest-nodeport" deleted shogo@controller-1:~/work$ kubectl delete -f nginx-clusterip.yaml pod "nginx-clusterip" deleted service "nginx-clusterip" deleted |