This doc covers the learning exercise: deploying cert-manager as a manually applied ArgoCD Application, creating a self-signed CA issuer, and enabling TLS for eu-dev services via values file changes.

Current approach: in eu-dev-rancher, cert-manager is managed as wave 0 in the sync wave sequence - no manual kubectl apply needed. The config/cert-manager/ ClusterIssuer manifests used here are reused as-is by eu-dev-rancher. See docs/09-sync-waves-cluster-complete.md.

1
2
3
4
5
6
Browser (https) → Traefik (port 443) → svc1 Service (alpha-dev namespace)
                         ↑
               TLS cert provisioned by cert-manager
               stored as a Kubernetes Secret (svc1-eu-dev-tls)
               requested via Ingress annotation in eu-dev values file
               ApplicationSet auto-discovers the updated values file

How cert-manager Works
#

cert-manager runs as a controller inside the cluster. It watches for Certificate and Ingress resources and automatically creates TLS certificates.

Three key concepts:

Resource	What it is
`ClusterIssuer`	Cluster-scoped authority that signs certificates (e.g. self-signed, Let’s Encrypt)
`Certificate`	A request for a cert - cert-manager creates a `Secret` with the actual TLS key/cert
Ingress annotation	`cert-manager.io/cluster-issuer: <name>` - tells cert-manager to auto-issue a cert for this Ingress

For local development we use a self-signed CA issuer:

Create a root CA cert (signed by itself)
Create a ClusterIssuer backed by that CA
All app certs are signed by the CA
Trust the CA once in macOS Keychain → browser trusts all app certs automatically

Files Created
#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
config/
  cert-manager/
    self-signed-issuer.yaml   # Bootstraps the root CA cert
    ca-cert.yaml              # The root CA Certificate resource
    ca-issuer.yaml            # ClusterIssuer backed by the CA
    kustomization.yaml

environments/eu-dev/values/
  svc1.yaml                   # Add ingress.annotations + ingress.tls to enable TLS
  svc2.yaml                   # Same
  svc3.yaml                   # Same

The apps/cert-manager/ and apps/cert-manager-config/ Application manifests were applied manually as a learning exercise. In eu-dev-rancher, these are replaced by environments/eu-dev-rancher/platform/cert-manager.yaml (wave 0) and environments/eu-dev-rancher/platform/cert-manager-config.yaml (wave 1), which point at the same config/cert-manager/ ClusterIssuer manifests.

The ApplicationSet (environments/eu-dev/appset.yaml) auto-discovers values files - no changes to the AppSet manifest are needed. Updating a values file is enough to enable TLS.

Step-by-Step Instructions
#

1. Deploy cert-manager via ArgoCD
#

cert-manager is distributed as a Helm chart. We deploy it as an ArgoCD Application pointing at the official Helm repo.

Create apps/cert-manager/application.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cert-manager
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://charts.jetstack.io
    chart: cert-manager
    targetRevision: v1.16.1
    helm:
      values: |
        installCRDs: true
  destination:
    server: https://kubernetes.default.svc
    namespace: cert-manager
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

installCRDs: true - cert-manager’s Custom Resource Definitions (Certificate, ClusterIssuer, etc.) must exist before any cert resources can be created. This flag installs them as part of the Helm release.

Apply this Application to ArgoCD:

1
kubectl apply -f apps/cert-manager/application.yaml

Wait for cert-manager pods to be ready:

1
2
kubectl get pods -n cert-manager
# Expected: cert-manager, cert-manager-cainjector, cert-manager-webhook - all Running

2. Create a Self-Signed CA and ClusterIssuer
#

We use a two-step bootstrap:

A temporary selfsigned ClusterIssuer - no config needed, just signs certs with themselves
Use it to issue a root CA Certificate
A ca-issuer ClusterIssuer backed by that CA cert - all app certs are signed by this

This mirrors how real internal PKI works: you generate a root CA once, then use it to sign everything else.

Create config/cert-manager/self-signed-issuer.yaml:

1
2
3
4
5
6
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned
spec:
  selfSigned: {}

Create config/cert-manager/ca-cert.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: local-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: ravikrs.local CA
  secretName: local-ca-secret
  privateKey:
    algorithm: RSA
    size: 2048
  issuerRef:
    name: selfsigned
    kind: ClusterIssuer
    group: cert-manager.io

This tells cert-manager: use the selfsigned issuer to create a CA certificate, store the key+cert in local-ca-secret in the cert-manager namespace.

Create config/cert-manager/ca-issuer.yaml:

1
2
3
4
5
6
7
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: local-ca-issuer
spec:
  ca:
    secretName: local-ca-secret

This tells cert-manager: when signing certs, use the key stored in local-ca-secret as the signing CA.

Create config/cert-manager/kustomization.yaml:

1
2
3
4
5
6
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - self-signed-issuer.yaml
  - ca-cert.yaml
  - ca-issuer.yaml

Create apps/cert-manager-config/application.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cert-manager-config
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/ravikrs/learning-argocd
    targetRevision: HEAD
    path: config/cert-manager
  destination:
    server: https://kubernetes.default.svc
    namespace: cert-manager
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Commit and push, then apply:

1
2
3
4
5
git add apps/cert-manager-config/ config/cert-manager/
git commit -m "add cert-manager ClusterIssuers and CA cert"
git push

kubectl apply -f apps/cert-manager-config/application.yaml

Verify the CA cert was issued and the issuer is ready:

1
2
3
4
5
6
7
8
# ClusterIssuers should show READY=True
kubectl get clusterissuers

# CA Certificate should show READY=True
kubectl get certificate -n cert-manager

# The CA secret should exist
kubectl get secret local-ca-secret -n cert-manager

3. Enable TLS for eu-dev Services via the ApplicationSet
#

The eu-dev-alpha-services ApplicationSet watches environments/eu-dev/values/*.yaml. Each service is rendered through charts/backend-service, which already supports ingress.tls. To enable TLS for a service, update its values file - the AppSet picks up the change automatically on the next sync.

Pattern: updated values file with TLS (example: environments/eu-dev/values/svc1.yaml):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
nameOverride: svc1
fullnameOverride: svc1

replicaCount: 1

image:
  repository: nginx
  tag: "1.27"

ingress:
  enabled: true
  className: traefik
  annotations:
    cert-manager.io/cluster-issuer: local-ca-issuer
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
  hosts:
    - host: svc1.eu-dev.ravikrs.local
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: svc1-eu-dev-tls
      hosts:
        - svc1.eu-dev.ravikrs.local

Key changes vs the HTTP-only values file:

Field	Old (HTTP)	New (TLS)
`ingress.annotations`	`router.entrypoints: web`	`router.entrypoints: websecure` + `cluster-issuer` annotation
`ingress.tls`	absent	`secretName` + `hosts`

Apply by committing and pushing the updated values file:

1
2
3
git add environments/eu-dev/values/svc1.yaml
git commit -m "eu-dev/svc1: enable TLS via local-ca-issuer"
git push

ArgoCD will sync alpha-svc1-eu-dev automatically. cert-manager sees the cluster-issuer annotation on the Ingress and issues a certificate, storing it in the svc1-eu-dev-tls Secret.

Verify the cert was issued:

1
2
3
4
5
6
7
8
# Certificate should show READY=True
kubectl get certificate -n alpha-dev

# The TLS secret should exist
kubectl get secret svc1-eu-dev-tls -n alpha-dev

# Describe to see issuer details
kubectl describe certificate svc1-eu-dev-tls -n alpha-dev

Test HTTPS (with /etc/hosts entry for svc1.eu-dev.ravikrs.local → 127.0.0.1):

1
2
3
4
5
# Expect TLS handshake to succeed (CA trusted after Keychain step)
curl -v https://svc1.eu-dev.ravikrs.local

# Or with --cacert if not yet trusted in Keychain
curl --cacert /tmp/local-ca.crt https://svc1.eu-dev.ravikrs.local

Commands Reference
#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# Check cert-manager pods
kubectl get pods -n cert-manager

# List all ClusterIssuers and their ready status
kubectl get clusterissuers

# List all Certificates across namespaces
kubectl get certificate -A

# Describe a certificate (see events, error messages)
kubectl describe certificate <name> -n <namespace>

# Watch cert-manager controller logs (useful for debugging issuance failures)
kubectl logs -n cert-manager -l app.kubernetes.io/name=cert-manager -f

# Extract the CA cert and trust it in macOS Keychain
kubectl get secret local-ca-secret -n cert-manager \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/local-ca.crt

sudo security add-trusted-cert -d \
  -r trustRoot \
  -k /Library/Keychains/System.keychain \
  /tmp/local-ca.crt

# Verify the CA is trusted
security find-certificate -c "ravikrs.local CA" \
  /Library/Keychains/System.keychain

# Remove the CA from macOS Keychain if needed
sudo security delete-certificate -c "ravikrs.local CA" \
  /Library/Keychains/System.keychain

Gotchas
#

CRDs must exist before Certificate/ClusterIssuer resources are applied. If cert-manager-config syncs before cert-manager itself is fully running, you’ll see no matches for kind "ClusterIssuer" errors. ArgoCD will retry automatically - wait a minute and re-sync. For production, use sync waves to enforce ordering.
ca-issuer depends on local-ca-secret existing. The cert-manager-config application deploys all three resources together. The ca-cert.yaml must reconcile (and populate local-ca-secret) before ca-issuer.yaml can become ready. cert-manager handles this with retries - you may see ca-issuer show READY=False briefly before the CA cert is signed.
ClusterIssuer is cluster-scoped - it can sign certs in any namespace. The CA Certificate lives in cert-manager namespace but the issuers it backs can serve apps in alpha-dev, alpha-staging, or any other namespace.
Let’s Encrypt does not work for .local domains. ACME HTTP-01 and DNS-01 challenges require publicly resolvable DNS. For local-only hostnames, self-signed CA is the right approach.