AWS Client VPN with SimpleAD

In this post, I’m going to guide how to set up AWS client VPN from scratch including Simple AD deployment.

AWS Client VPN can be used to connect to private segment directly from your client. It is well documented here in official document “AWS Client VPN Administrator Guide“.

In VPN settings, there are two main part you need to consider first:

  • Authentication … AWS Client VPN supports two types:
    • Active Directory Authentication
    • Mutual (Certificate) Authentication
  • Authorization
    • Security Groups
    • Network-based Authorization (work with Active Directory Authentication)

In my opinion, Active Directory Authentication is more flexible and intuitive compared to Mutual Authentication. It doesn’t support MFA yet, but it provides user/password authentication as well as it allows specific groups of users in AD to be able to connect to SSLVPN, which is a requirement of most clients.

Setup procedure is below:

  1. Generate certs and keys using easy-rsa, and register them on ACM
  2. Procure Simple AD
  3. Create users and groups on Simple AD
  4. Procure Client VPN Endpoint
  5. (Option) Configure a web server for connection test
  6. Configure OpenVPN client

By the end of this guide, the test environment looks like below:

1. Generate Certificate and Keys

You need to generate certificate and keys for servers to process client vpn first. You can follow the official steps here. It is not required to generate client certificate/key because we use Active Directory Authentication.

 git clone
 cd easy-rsa/easyrsa3
 ./easyrsa init-pki
 ./easyrsa build-ca nopass
 ./easyrsa build-server-full lab_server nopass
 mkdir /temp_folder
 cp pki/ca.crt /temp_folder/
 cp pki/issued/lab_server.crt /temp_folder/
 cp pki/private/lab_server.key /temp_folder/
 cd /temp_folder/

Once they are generated, register them into the AWS Certificate Manager(ACM). Please note you need to register these to the region you are going to have your VPN connection.

aws acm import-certificate --certificate file://lab_server.crt --private-key file://lab_server.key --certificate-chain file://ca.crt --region eu-west-1

If it returns arn, you are successfully registered certificate/key on ACM.

2. Procure Simple AD

Simple AD is not available yet in all regions. You can check the availability here.

First, select “Simple AD”, and select your VPC and subnet you want this service.

Once created, wait for a ew minutes till Directory service is ready. Note the DNS address listed in Directory detail, this information is required later to have management server join this domain.

3. Create Users and Groups for VPN access

In order to manage this AD, I procure another Windows server(management server) in public subnet.

For DNS configuration, you can either change DHCP option of your subnet or set DNS server statically in the servers.

After restart, reconnect to the server, this time with domain administrator account – password is the one you set during Simple AD setup. Install RSAT.

Launch “Active Directory Users and Computers”. If you logged in with domain administrator account, you should be able to see your domain is listed. I created two users “shogo.kobayashi” and “john.smith”, and only “shogo.kobayashi” is a member of “SSLVPN_USERS” group.

4. Procure Client VPN Endpoint

In Create Client VPN Endpoint wizard, you need to specify IPv4 CIDR which should be different from your existing VPC.

  • Server certificate ARN … Select arn, which you received in step 1.
  • Authentication Options … Select “Use Active Directory authentication”
  • Directory ID … select Directory ID you created on step 2.
  • DNS Server 1/2 IP Address … Use DNS IP address of your Simple AD

Associate this endpoint with your subnet in order to use it. Note that you will be charged once you associate endpoint with subnet. I used private subnet for this.

  • VPC … VPC you want to use this VPN Endpoint in
  • Subnet … Subnet you want to use this VPN Endpoint in

This is to authorize which network is reachable for each group of users. In this lab, I created only one rule that allows VPN connection to communicate anyone in the network as long as the authenticated user is in “SSLVPN_USERS” AD security group.

  • Destination network to enable … Which network this group of users can access
  • Grant access to … Allow access to users in a specific Active Directory group
  • Active Directory Group ID … SID of the SSLVPN_USERS group in AD

5. (Optional) Setup web server for connection test

We are going to setup web server for connection test. We use VPC endpoint to retrieve httpd packages because we want this server to be in private subnet, and hence there is no direct internet connectivity.

And minimum setup to bootstrap web server.

6. Configure OpenVPN Client

You can download configuration file from AWS console.

Launch OpenVPN client of your choice, and use the configuration file you just downloaded. Note you need to have a root CA certificate, which you generate in step 1, in the same folder you have your configuration file.

Now you should …

  • Be able to access web server if you login with user who is a member of “SSLVPN_USERS”, and
  • Not be able to access web server if you login with user who is NOT a member of “SSLVPN_USERS”.

Terraform Basics – AWS / GCP / Aliyun

What is Terraform?

It’s a tool to create, manage infrastructure as a code. Infrastructure includes not only servers but also network resources –e.g. DNS, loadbalancer. The benefit you can get is as follows:

  • Versioning of your changes
  • Management of all services as a whole (orchestration)
  • Single management of multi-cloud platform
  • and so on …

Let’s Try

I make two compute instances and make modifications, and finally delete all resources to demonstrate how to use Terraform.

  • on AWS (amazon web service), GCP (Google cloud platform) and Aliyun (Alibaba cloud)
  1. Install Terraform
  2. Get credentials
  3. Create servers
  4. Modify servers
  5. Delete all procured resources
Continue reading “Terraform Basics – AWS / GCP / Aliyun”

Kubernetes CKA certification – Where to Start


  • I passed CKA exam on December 2018
  • Prepared for 4 months, before that I had little production experience on kubernetes
  • Must read: Kubernets in Action
  • Must possess: patience, curiosity
  • You cannot pass the exam if you just remember all commands in Kubernetes The Hard Way.
  • To check if you are ready, look through all the document. And if you still feel not overwhelmed by the amount of new things, it should be good timing to give it a go.
Continue reading “Kubernetes CKA certification – Where to Start”

KTHW Reinvented – Agenda

From the next post, I will guide you how to bring up Kubernetes cluster locally.

I use Kubernetes The Hard Way as a guidepost, but I will re-order the procedure so that it goes component by component. If you are willing to take CKA(Kubernetes Certified Administrator) Certification, you should follow original kubernetes the hard way again after completing this agenda, so that you can improve your deployment speed.


  1. Compute resource procurement … I use my desktop pc to host 4 virtual ubuntu machines
  2. Etcd cluster bootstrap … Etcd is the base system of kubernetes to hold all the information
  3. Control plane bootstrap 01 … API server installation and flags investigation
  4. Control plane bootstrap 02 … Deploy LoadBalancer for API server
  5. Worker node bootstrap … kubelet and kube-proxy are installed on nodes.
  6. Control plane bootstrap 03 … Controller-Manager installation
  7. Control plane bootstrap 04 … Scheduller installation
  8. Pod network routes … configure network for inter-pod communication
  9. DNS … Deploy coredns in cluster
  10. Data encryption at rest … secure secret file encrypted

Continue reading “KTHW Reinvented – Agenda”

Kubernetes The Hard Way Picture

One of the most popular tutorial to bootstrap kubernetes components is¬†Kelsey Hightower’s “Kubernetes The Hard Way“. It is really helpful to understand the complicated component structure of kubernetes.

I saw some people asking if there is any equivalent tutorial which is not using GCP(e.g. on-prem, AWS).¬†Because Kubernetes The Hard Way is using GCP as its backend, it’s no wonder they think the tutorial is specific to GCP. But in fact, only a small bit of part is specific to GCP(maybe LoadBalancer, Swap configuration only), and most of the part is still applicable to any infrastructure.

Continue reading “Kubernetes The Hard Way Picture”

k8s ex04: Security daemon – Cisco Stealthwatch

Network security used to be only deployed to investigate the traffic between north and south(in other word external and internal), but as the cloud and virtualization progress, it is now required to have east to west (intra-site) security investigation. For this purpose, ISFW(inter-segment firewall) is deployed on-premise, but it’s quite difficult if the servers are in the cloud.

And with Kubernetes, it’s even more difficult. Because each pod can be connected each other over some kind of tunnel(e.g. overlay network) as I mentioned in the previous post. So all the communications are somewhat hidden and simple security rule/policy cannot be used to restrict the communication. We use network policy or tools like Istio to restrict these unexpected traffic. But similar to the legacy network, these restrictions are still rely on manual work, we need to make policy by ourselves and it needs to be updated every time new service are procured. This is very difficult, the developer wants to deploy the service as smooth as possible, but the security needs to be guaranteed even though some services are dealt by another team…

Cisco Stealthwatch can be used in these cases as a turnkey security monitor.  As the stealthwatch doesn’t require anything to be changed on kubernetes config, actually it deploys a special host network pod in each worker node as a daemonset. It visualizes the inter-node/external communication dynamically and can send an alert in case unexpected communication happens. Please note that I use Stealthwatch Cloud for this trial.


Deploy Stealthwatch in K8s

1. Access Stealthwatch portal, and navigate to Integration > Kubernetes. You can find manifest file for stealthwatch daemonset along with service-key, which identifies your account.

2. Apply manifest files.

root@controller-1:~/work# {
> echo -n "YOUR_SECRET_KEY" > obsrvbl-service-key.txt
> kubectl create secret generic obsrvbl --from-file=service_key=obsrvbl-service-key.txt
> rm obsrvbl-service-key.txt
> }
secret "obsrvbl" created
root@controller-1:~/work# {
> kubectl create serviceaccount --generator=serviceaccount/v1 obsrvbl
> kubectl create clusterrolebinding "obsrvbl" --clusterrole="view" --serviceaccount="default:obsrvbl"
> }
serviceaccount "obsrvbl" created "obsrvbl" created
root@controller-1:~/work# cat << EOF > obsrvbl-daemonset.yaml
> apiVersion: apps/v1
> kind: DaemonSet
> metadata:
>   name: obsrvbl-ona
> spec:
>   selector:
>     matchLabels:
>       name: obsrvbl-ona
>   template:
>     metadata:
>       labels:
>         name: obsrvbl-ona
>     spec:
>       serviceAccountName: obsrvbl
>       tolerations:
>         - key:
>           effect: NoSchedule
>       hostNetwork: true
>       containers:
>         - name: ona
>           image: obsrvbl/ona:3.1
>           env:
>             - name: OBSRVBL_SERVICE_KEY
>               valueFrom:
>                 secretKeyRef:
>                   name: obsrvbl
>                   key: service_key
>             - name: OBSRVBL_KUBERNETES_WATCHER
>               value: "true"
>             - name: OBSRVBL_HOSTNAME_RESOLVER
>               value: "false"
>               value: "false"
root@controller-1:~/work# kubectl apply -f obsrvbl-daemonset.yaml 
daemonset.apps "obsrvbl-ona" created
root@controller-1:~/work# kubectl get ds
obsrvbl-ona   2         2         2         2            2           <none>          1m
root@controller-1:~/work# kubectl get pods -o wide
NAME                       READY     STATUS    RESTARTS   AGE       IP            NODE
busybox-5ccc978d8d-2nzs4   1/1       Running   4          2d   worker-1
nginx-65899c769f-txgtm     1/1       Running   3          2d   worker-2
obsrvbl-ona-dm9gb          1/1       Running   0          1m   worker-1
obsrvbl-ona-fkjnz          1/1       Running   0          1m   worker-2


3. After a few minutes, nodes with stealthwatch pod comes up as a sensor.

That’s all we need to do. With this setup, inter-pod/external communication are monitored and statistics information are sent to Stealthwatch cloud, where all the statistics data is processed.


Watch it works

1. After a few data being sent to the stealthwatch cloud, the portal starts to populate the valuable information. Dashboard by default shows the endpoint(in our case it includes pods on each nodes) and total in/out traffic.

2. And it shows any observations and alert as well. These are based on either static pattern(e.g. blacklisted), or behavioural basis(e.g. the traffic pattern is unusual compared to last 36 days).

Because this is my initial deployment it observes communication between controller-1( and other worker node are rather high. But this will be considered to be normal after some time.

And of course it can send an alert if there is any alert triggered.

3. You can explore network communication more if you would like to.

The good thing about this deployment is you don’t really have to care much of the things. Because the pod can see the traffic and it can also interact with the API server, it integrates all the real-time information with the historical data, and it nicely summarizes and show the outcome on the portal. And you can use them as a baseline to create security policies.

And it can be extended to GCP(with flow logs), AWS(flow logs), On-Premise(with onsite VM and sensor – e.g. Cat9300) to consolidate all the behavioural monitoring. Maybe I will introduce them in another post.



k8s 14: Calico IP-in-IP

In this post, I’m going to replace the network plugin from default “noops” to “cni”, and use Calico to connect each pod.

We follow official installation manual “Installing Calico for policy and networking“.

There are basically two types of installation of available. One uses kubernetes API server (and eventually backend etcd) to store data, and the other uses other etcd datastore. I use the former to utilise existing kubernetes setup.

Continue reading “k8s 14: Calico IP-in-IP”

k8s ex03: Network Plugin

Kubernetes is an orchestrator, or in other word it works like a conductor in the orchestra. In order to make the music happen, loads of other services are required. It includes not only direct service provider, such as API controller, but also in-direct service providers. Below are some example of them:

  • Computer resource deployment … Previously I deployed compute node manually. You can also use external provisioning tools like terraform.
  • Install system services(e.g. kubelet, container runtime) … For this as well I installed all of them manually, you can use automation tools like ansible.
  • Network management … Kube-proxy does provide network connectivity for services, but it doesn’t provide direct connectivity from a container to the other. For this end-to-end connectivity purpose, I added routes in GCP manually for all three worker nodes to route respective pod network(of 24 bit).

In this post, we explore some network fundamentals of container network management. For routing concept, there is a good slide shared by “TimHockin’s Illustrated Guide To Kubernetes Networking“.

Continue reading “k8s ex03: Network Plugin”

k8s 13: affinity and taint/toleration

In k8s 05: scheduler, I used node selector to select which node to launch a pod, and we don’t need to use that node selector once we launched scheduler. It is because Scheduler choose the best node for the pod based on various criteria(e.g. resource balancing). But often times, we have specific requirement for a pod which environment it should be launched on. For this purpose, kubernetes uses two ways as follows:

  • Labels and Affinity …… This is used to specify the “Preference” of the launching pod to select which node/pod to be launched on/with.
  • Taints and Tolerations …… This is used to specify the “Requirement” of the node to allow launching pod to be deployed on it.


k8s 12: Admission Controller – Service Account

In this post, I will talk about admission controller, one of the key component of API server(document reference). In the past few posts, we deployed PKI to secure the communication between each component. It’s time to secure/validate the requests itself. In previous post, I illustrated how API server deals the request, and it can be summarized as follows:

  • Authentication … Is the requester a valid account?
  • Authorization … Is the requester allowed to do what it request?
  • Admission Control … Allow/Reject/Modify the original request based on various criteria.

Continue reading “k8s 12: Admission Controller – Service Account”