k8s ex04: Security daemon – Cisco Stealthwatch

Network security used to be only deployed to investigate the traffic between north and south(in other word external and internal), but as the cloud and virtualization progress, it is now required to have east to west (intra-site) security investigation. For this purpose, ISFW(inter-segment firewall) is deployed on-premise, but it’s quite difficult if the servers are in the cloud.

And with Kubernetes, it’s even more difficult. Because each pod can be connected each other over some kind of tunnel(e.g. overlay network) as I mentioned in the previous post. So all the communications are somewhat hidden and simple security rule/policy cannot be used to restrict the communication. We use network policy or tools like Istio to restrict these unexpected traffic. But similar to the legacy network, these restrictions are still rely on manual work, we need to make policy by ourselves and it needs to be updated every time new service are procured. This is very difficult, the developer wants to deploy the service as smooth as possible, but the security needs to be guaranteed even though some services are dealt by another team…

Cisco Stealthwatch can be used in these cases as a turnkey security monitor.  As the stealthwatch doesn’t require anything to be changed on kubernetes config, actually it deploys a special host network pod in each worker node as a daemonset. It visualizes the inter-node/external communication dynamically and can send an alert in case unexpected communication happens. Please note that I use Stealthwatch Cloud for this trial.


Deploy Stealthwatch in K8s

1. Access Stealthwatch portal, and navigate to Integration > Kubernetes. You can find manifest file for stealthwatch daemonset along with service-key, which identifies your account.

2. Apply manifest files.


3. After a few minutes, nodes with stealthwatch pod comes up as a sensor.

That’s all we need to do. With this setup, inter-pod/external communication are monitored and statistics information are sent to Stealthwatch cloud, where all the statistics data is processed.


Watch it works

1. After a few data being sent to the stealthwatch cloud, the portal starts to populate the valuable information. Dashboard by default shows the endpoint(in our case it includes pods on each nodes) and total in/out traffic.

2. And it shows any observations and alert as well. These are based on either static pattern(e.g. blacklisted), or behavioural basis(e.g. the traffic pattern is unusual compared to last 36 days).

Because this is my initial deployment it observes communication between controller-1( and other worker node are rather high. But this will be considered to be normal after some time.

And of course it can send an alert if there is any alert triggered.

3. You can explore network communication more if you would like to.

The good thing about this deployment is you don’t really have to care much of the things. Because the pod can see the traffic and it can also interact with the API server, it integrates all the real-time information with the historical data, and it nicely summarizes and show the outcome on the portal. And you can use them as a baseline to create security policies.

And it can be extended to GCP(with flow logs), AWS(flow logs), On-Premise(with onsite VM and sensor – e.g. Cat9300) to consolidate all the behavioural monitoring. Maybe I will introduce them in another post.