k8s 02: How kubelet works

I found it quite useful to make a kubernetes cluster from scratch, and check the outcome step by step. By doing so, I can see each component role very clearly.

For this purpose, I mainly followed excellent kubernetes how to from Kamal Marhabi’s blog.

First of all, I start exploring kubelet, which acts like a site supervisor of a pod.

An agent that runs on each node in the cluster. It makes sure that containers are running in a pod.

The kubelet takes a set of PodSpecs that are provided through various mechanisms and ensures that the containers described in those PodSpecs are running and healthy.

So we can make a pod running if we spin up kubelet. It can be illustrated as follows:

Here’s step by step guide:

Prepare for container runtime

1. Install docker


shogo@aio-node:~$ sudo apt-get install docker-ce
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  aufs-tools cgroupfs-mount libltdl7 pigz
Suggested packages:
The following NEW packages will be installed:
  aufs-tools cgroupfs-mount docker-ce libltdl7 pigz

2. Confirm if it’s installed. You can run example container by running

#docker run hello-world

shogo@aio-node:~$ which docker
shogo@aio-node:~$ sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
d1725b59e92d: Pull complete 
Digest: sha256:0add3ace90ecb4adbf7777e9aacf18357296e799f81cabc9fde470971e499788
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:

For more examples and ideas, visit:

I have running container runtime now.

Prepare for kubelet

1. Install kubelet

kubelet can be installed via apt if you are using Ubuntu. Follow official document to install kubelet.

shogo@aio-node:~$ sudo su -
root@aio-node:~# curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
root@aio-node:~# cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
> deb http://apt.kubernetes.io/ kubernetes-xenial main
root@aio-node:~# apt-get update
root@aio-node:~# apt-get install kubelet
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  ebtables kubernetes-cni socat
The following NEW packages will be installed:
  ebtables kubelet kubernetes-cni socat
0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 29.6 MB of archives.
After this operation, 210 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
root@aio-node:~# service kubelet status
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: 
   Active: active (running)
     Docs: https://kubernetes.io/docs/home/
 Main PID: 9562 (kubelet)
    Tasks: 11
   Memory: 44.6M
      CPU: 4.358s
   CGroup: /system.slice/kubelet.service
           └─9562 /usr/bin/kubelet


Now, kubelet is installed. But it’s not ready to serve yet.

2. Configure kubelet

Kubelet, as a supervisor of the pod, it receives a request from various ways. As depicted previously, kubelet receives requests usually from apiserver in complete kubernetes setup. But this time, the apiserver is not ready, so there are following options to interact with kubelet without apiserver.

– File … tell kubelet to check specific directory periodically for new manifest file. Change state if any changes in the directory.

– HTTP endpoint … tell kubelet to check specific URL periodically for new manifest file. Change state if any changes in the directory.

– HTTP server … it responds to some API requests for node/pod status

This time I tell kubelet to check local directory /work/manifests to manage pod.

Since I’m using systemd to start kubelet, I modify service file…

root@aio-node:~# cat /lib/systemd/system/kubelet.service 
Description=kubelet: The Kubernetes Node Agent

ExecStart=/usr/bin/kubelet --pod-manifest-path=/work/manifests

root@aio-node:~# systemctl daemon-reload
root@aio-node:~# systemctl restart kubelet

Now, kubelet is ready to serve for my request, which I’m just going to put a yaml file in /work/manifests directory.


Run a Pod

1. Create a test manifest in specified folder

Create a test pod manifest file. Refer official document for the elements. This example just run a simple nginx server.

root@aio-node:~# cat << EOF > /work/manifests/nginx-test.yaml
> apiVersion: v1
>  kind: Pod
>  metadata:
>    name: nginx-test
>    spec:
>      containers:
>      - name: nginx
>         image: nginx
>         ports:
>         - containerPort: 80

That’s all I need to do.

Kubelet checks this directory every 20 seconds(by default) to see if there are any changes. And once it detects a new file, it creates a new pod.

I can ask kubelet for this pod status.

root@aio-node:~# curl --stderr /dev/null http://localhost:10255/pods | jq .
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {},
  "items": [
      "metadata": {
        "name": "nginx-test-aio-node",
        "namespace": "default",
        "selfLink": "/api/v1/namespaces/default/pods/nginx-test-aio-node",
      "spec": {
        "containers": [
            "name": "nginx",
            "image": "nginx",
            "ports": [
                "containerPort": 80,
                "protocol": "TCP"

Or I can as well ask the container runtime directly

root@aio-node:~# docker ps
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS               NAMES
e96e99aa9aaf        nginx                  "nginx -g 'daemon of…"   5 seconds ago       Up 4 seconds                            k8s_nginx_nginx-test-aio-node_default_4fef8d0911b1723b38800c5c32b03d77_1
033b2c71b082        k8s.gcr.io/pause:3.1   "/pause"                 7 seconds ago       Up 6 seconds                            k8s_POD_nginx-test-aio-node_default_4fef8d0911b1723b38800c5c32b03d77_1

And finally I can confirm the nginx is serving on port80 as specified in the manifest file.

root@aio-node:~# docker inspect --format '{{ .NetworkSettings.IPAddress  }}' 033b2c71b082
root@aio-node:~# curl
<!DOCTYPE html>
<title>Welcome to nginx!</title>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>

Network configuration of Kubernetes and docker is very confusing, and it requires third party integration(it works ok with basic deployment, but you need third party plugin to utilize full functionality of kubernetes). So I’m not going to write in detail here, but I will make a separate entries for this network topics.


That’s all. I just need to delete manifest file from the directory, and the pod will also be deleted.