Background

When deploying applications to kubernetes, we should restrict access to resources like databases, queues, cache, etc. From a security prospective, we should not be able to connect to storage/database from outside the kubernetes deployment(applications). Only applications deployed to kubernetes cluster should be allowed access to databases,queues, etc.

Problem

But for debugging purpose its essential to be able to connect to such resources. For e.g. we might need to check some data in the database or verify if an item is getting stored in redis cache, etc

Solution

For that purpose we may want to deploy a pod in the dev cluster to be able to debug applications.

We may call it a bastion pod or a pod shell which gets deployed to the kubernetes cluster which is allowed to access cloud resources like database, etc. The pod behaves as a proxy in the k8s cluster.

Such a pod may support following use cases:

  • connect to database like sql, cache, queues,topics, etc
  • able to run HTTP requests from inside the cluster
  • be able to debug dns issues

Build image

Sample Dockerfile to build a container image: ravikrsingh20/bastion:0.1. The Dockerfile contains utilities that may be required for debugging purpose.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

FROM ubuntu:latest
WORKDIR /root
RUN apt-get update -qq && \
    apt-get install -y apt-transport-https \
                       ca-certificates \
                       software-properties-common \
                       man \
                       manpages-posix \
                       man-db \
                       vim \
                       screen \
                       curl \
                       atop \
                       htop \
                       dstat \
                       jq \
                       dnsutils \
                       tcpdump \
                       traceroute \
                       iputils-ping \
                       net-tools \
                       netcat \
                       iproute2 \
                       strace \
                       telnet \
                       conntrack \
                       mysql-client \
                       redis-tools \
                       stunnel4     
CMD [ "/bin/bash" ]

If needed we may add more utilites for improving debugging and trouble shooting.

Kubernetes deployment

Sample bastion pod deployment

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

apiVersion: v1
kind: Pod
metadata:
  name: bastion-pod
spec:
  containers:
    - name: bastion
      image: ravikrsingh20/bastion:0.1
      imagePullPolicy: Always
      command: [ "sleep", "infinity" ]
      resources:
        limits:
          memory: 512Mi
          cpu: "500m"
        requests:
          memory: 64Mi
          cpu: "125m"
      securityContext:
        runAsNonRoot: true
        runAsUser: 100000
        runAsGroup: 100000
  1. To create the pod
1

kubectl apply -n $namespace -f pod.yml
  1. To delete the pod
1

kubectl delete -n $namespace -f pod.yml

Usage example

  1. Connect to Database
1

kubectl exec -it bastion-pod -n $namespace -- mysql -u $username -p"$password" -h  $host $database
  1. Open a bash shell in the pod
1

kubectl exec -it bastion-pod -n $namespace  -- /bin/bash