Kubernetes is an orchestrator, or in other word it works like a conductor in the orchestra. In order to make the music happen, loads of other services are required. It includes not only direct service provider, such as API controller, but also in-direct service providers. Below are some example of them:
- Computer resource deployment … Previously I deployed compute node manually. You can also use external provisioning tools like terraform.
- Install system services(e.g. kubelet, container runtime) … For this as well I installed all of them manually, you can use automation tools like ansible.
- Network management … Kube-proxy does provide network connectivity for services, but it doesn’t provide direct connectivity from a container to the other. For this end-to-end connectivity purpose, I added routes in GCP manually for all three worker nodes to route respective pod network(of 24 bit).
In this post, we explore some network fundamentals of container network management. For routing concept, there is a good slide shared by “TimHockin’s Illustrated Guide To Kubernetes Networking“.
In many of kubernetes tutorials, you’re advised to create overlay networks so that kubernetes works correctly. But I didn’t create one in my cluster so far. And it comes with a great burden as shown below:
So I need to manually update routing table all the time. It’s not a realistic option.
Actually I didn’t use any overlay network, or more specifically I didn’t pass any flags for “network-plugin” when I launched kubelet, and it goes “noop” by default. It just modify the system not to stop the forwarding traffic, and nothing else. I can pass “cni” to this “network-plugin” flag, and kubelet tries to use network plugins as specified in config file(details can be found here). In the config file, you can create as many manifest files as you want with mixture of network plugins(it’s very confusing though). CNI just defines how network interfaces are created/removed, so the concept diagram is rather simple yet.
So far, it looks very simple and similar to noops mode. What network plugin shines are how it forwards the traffic, and I will illustrate now. The overlay network is a common term to address this kind of setup, but there are many kinds of network plugins are available and each plugins have their own way to achieve this. I explain flannel and calico for example here.
[ Flannel ]
Flannel is the most popular network plugins in use. It doesn’t have much dependency to deploy, and it can be deployed rather easy. And as it’s advertised it’s layer 2 connection, and from the application point of view it is very convenient because it’s something as if all the containers are working in the same network segment.
[ Calico ]
Compared to Flannel, calico requires additional setup of external router. Calico(actually BIRD in calico container) advertise the network changes to the router. This is almost the same thing what I did manually in my cluster, but with this setup Calico does those updates on behalf.
Personally, I prefer L3 routing than the L2 switching. I assume it’s because I’m from legacy Routing&Switching area of the world. Unlike L3 routing with Calico, VxLAN(like flannel) hides the actual traffic metadata from network point of view, and it often times not quite acceptable for legacy network admins(in security point of view as well as management). But it’s also true that the application development is so fast and service delivery must be as fast as possible. It’s a never ending fight.
For this study purpose, I use Calico as my choice at this moment. But in sometime next step, I will try layer 2 overlay network along with some security tools(e.g. stealthwatch to monitor anomaly behaviour of lateral move), which should address both security(which network admin requires) and agility(which application developer requires).