Chapter 65: Kubernetes Domain Names and HTTPS: Ingress, DNS, ExternalDNS, cert-manager and Gateway API

65.4 cert-manager and Let's Encrypt: free HTTPS that renews itself

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Browsers now mark plain HTTP pages as not secure, payment rules such as PCI DSS demand encryption, and HTTP/2 in browsers needs TLS. So every public host from 65.2 needs a certificate. Buying one per year and copying the files into a Secret by hand is how outages happen: the person who did it leaves, the reminder email goes to an old inbox, and one morning PhonePe-style checkout pages show a red warning because the certificate expired at midnight. cert-manager removes the human from that loop. It is a Kubernetes controller that adds a few objects: an Issuer or ClusterIssuer says where certificates come from, a Certificate says which names you need and which Secret to store them in, and cert-manager does the rest. With Let's Encrypt, a free public certificate authority, it uses the ACME protocol: Let's Encrypt asks you to prove you control the name, cert-manager answers the challenge, receives a certificate valid for 90 days and renews it automatically about 30 days before expiry. There are two challenge types. HTTP-01 asks you to serve a token at http://your-name/.well-known/acme-challenge/...; cert-manager creates a tiny solver pod and a temporary Ingress for it, so the name must already point at your controller on port 80. DNS-01 asks you to create a TXT record under _acme-challenge; cert-manager does that through the Route 53 API. DNS-01 is the only way to get a wildcard such as *.shop.example.com and works even for internal names, but needs DNS permissions. Always test with the Let's Encrypt staging server first, because production has rate limits, such as 50 certificates per registered domain per week, and a broken loop can lock you out for days. On kind we cannot pass a public challenge, so we first learn the objects with a self-signed ClusterIssuer and a local CA, which behave exactly the same from the Ingress side. Then we write the real staging and production ACME issuers with HTTP-01 and DNS-01 for EKS.

What: every term in plain English

• TLS certificate — A signed file that proves a server owns a name and carries its public key.
• cert-manager — Controller that requests, stores and renews certificates as Kubernetes objects.
• Issuer and ClusterIssuer — Where certificates come from; Issuer works in one namespace, ClusterIssuer in all.
• Certificate — Object listing DNS names, the issuer and the target Secret name.
• ACME — Protocol used by Let's Encrypt to prove control of a name and issue certificates.
• HTTP-01 — Challenge answered by serving a token on port 80 at /.well-known/acme-challenge/.
• DNS-01 — Challenge answered by a TXT record at _acme-challenge.name; needed for wildcards.
• Staging server — Let's Encrypt test endpoint with high limits and untrusted certificates.
• kubernetes.io/tls Secret — Secret with tls.crt and tls.key that Ingress controllers read.
• ingress-shim — cert-manager feature that creates a Certificate from Ingress annotations and the tls block.

How: step by step

1. Install cert-manager from a pinned manifest. Three pods Running.
2. Create a self-signed ClusterIssuer and a local CA. Issuers Ready.
3. Request a Certificate for shop.example.com. Secret created.
4. Use the Secret in an Ingress tls block. HTTPS works with the local CA.
5. Let an annotation create the Certificate. ingress-shim at work.
6. Write Let's Encrypt staging and production issuers. HTTP-01 ready for EKS.
7. Add a DNS-01 Route 53 solver for a wildcard. *.shop.example.com.
8. Check renewal and expiry. cmctl status.

Under the hood cert-manager creates a chain of objects, and knowing that chain makes every problem easy to trace. A Certificate creates a CertificateRequest holding a signing request. For ACME issuers the request creates an Order with Let's Encrypt, the Order creates one Challenge per name, and each Challenge creates either a solver pod with a temporary Ingress (HTTP-01) or a TXT record (DNS-01). When the Challenge is valid, the Order is finalised, the signed certificate comes back, and cert-manager writes it into the Secret. If anything is stuck, run kubectl describe on each object in that order; the Events at the bottom always name the failing step.

Example 1: Install cert-manager and issue a certificate from a local CA

Goal. Install cert-manager and issue a certificate from a local CA.

install.sh

# cert-manager-v1.19.1.yaml: the release manifest saved from the official cert-manager docs, pinned and reviewed
kubectl apply -f cert-manager-v1.19.1.yaml
kubectl wait -n cert-manager --for=condition=available deploy --all --timeout=180s
kubectl get pods -n cert-manager

local-ca.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: lab-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: ch65-lab-ca
  secretName: lab-ca
  privateKey: {algorithm: ECDSA, size: 256}
  issuerRef: {name: selfsigned, kind: ClusterIssuer}
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: lab-ca
spec:
  ca:
    secretName: lab-ca
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: shop-tls
  namespace: shop
spec:
  secretName: shop-tls
  dnsNames: ["shop.example.com", "www.shop.example.com"]
  duration: 2160h
  renewBefore: 720h
  issuerRef: {name: lab-ca, kind: ClusterIssuer}

What each line does

• Three pods — controller, webhook and cainjector.
• selfsigned ClusterIssuer — Signs the CA certificate itself; only used to bootstrap.
• lab-ca — A CA certificate stored in a Secret, then used by the ca ClusterIssuer.
• duration and renewBefore — 90 days, renewed 30 days before expiry, matching Let's Encrypt behaviour.
• secretName shop-tls — Where tls.crt and tls.key land.

Green output

$ bash install.sh
deployment.apps/cert-manager condition met
deployment.apps/cert-manager-cainjector condition met
deployment.apps/cert-manager-webhook condition met
NAME                                       READY   STATUS
cert-manager-6c9b8d7f5-kq2xd               1/1     Running
cert-manager-cainjector-7f4b9c6d8-m8n2p    1/1     Running
cert-manager-webhook-5d8f7b9c4-t6w3z       1/1     Running
$ kubectl apply -f local-ca.yaml
$ kubectl get certificate -A
NAMESPACE      NAME       READY   SECRET     AGE
cert-manager   lab-ca     True    lab-ca     15s
shop           shop-tls   True    shop-tls   12s
$ kubectl get secret shop-tls -n shop -o jsonpath='{.type}'
kubernetes.io/tls

Expected result. cert-manager built a small CA and signed a 90-day certificate for two names into a TLS Secret, all from YAML. Swapping lab-ca for Let's Encrypt later changes only the issuerRef.

Example 2: Serve HTTPS from the Ingress and let an annotation create the Certificate

Goal. Serve HTTPS from the Ingress and let an annotation create the Certificate.

tls-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: storefront
  namespace: shop
  annotations:
    cert-manager.io/cluster-issuer: lab-ca
spec:
  ingressClassName: nginx
  tls:
    - hosts: ["shop.example.com", "www.shop.example.com"]
      secretName: shop-web-tls
  rules:
    - host: shop.example.com
      http:
        paths:
          - {path: /, pathType: Prefix, backend: {service: {name: web, port: {number: 80}}}}

test.sh

kubectl get secret lab-ca -n cert-manager -o jsonpath='{.data.ca\.crt}' | base64 -d > lab-ca.crt
curl -s --cacert lab-ca.crt --resolve shop.example.com:443:127.0.0.1 https://shop.example.com/
curl -sI --resolve shop.example.com:80:127.0.0.1 http://shop.example.com/ | head -3
echo | openssl s_client -connect 127.0.0.1:443 -servername shop.example.com 2>/dev/null | openssl x509 -noout -subject -issuer -dates

What each line does

• cert-manager.io/cluster-issuer — ingress-shim sees this and creates a Certificate named after secretName.
• tls block — Tells the controller which Secret to use for which hosts.
• --cacert lab-ca.crt — We trust our lab CA only for this curl; no system change.
• 308 redirect — Controllers usually redirect HTTP to HTTPS once a tls block exists.
• openssl s_client — Shows the subject, issuer and validity dates the server presents.

Green output

$ kubectl apply -f tls-ingress.yaml
$ kubectl get certificate -n shop
NAME           READY   SECRET         AGE
shop-tls       True    shop-tls       3m
shop-web-tls   True    shop-web-tls   9s
$ bash test.sh
web says hi
HTTP/1.1 308 Permanent Redirect
Location: https://shop.example.com
subject=
issuer=CN=ch65-lab-ca
notBefore=Oct 25 02:20:11 2026 GMT
notAfter=Jan 23 02:20:11 2027 GMT

Expected result. The annotation alone produced a Certificate, the controller served it on 443 and redirected port 80. The dates show a 90-day life, and cert-manager will renew it in December without anyone remembering.

Example 3: Let's Encrypt issuers for EKS: staging, production and a DNS-01 wildcard

Goal. Let's Encrypt issuers for EKS: staging, production and a DNS-01 wildcard.

letsencrypt.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: platform-team@example.com
    privateKeySecretRef: {name: le-staging-account}
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: platform-team@example.com
    privateKeySecretRef: {name: le-prod-account}
    solvers:
      - selector:
          dnsZones: ["shop.example.com"]
        dns01:
          route53:
            region: ap-south-1
            hostedZoneID: Z0123456789EXAMPLE
      - http01:
          ingress:
            ingressClassName: nginx

wildcard.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: stores-wildcard
  namespace: shop
spec:
  secretName: stores-wildcard-tls
  dnsNames: ["stores.shop.example.com", "*.stores.shop.example.com"]
  issuerRef: {name: letsencrypt-prod, kind: ClusterIssuer}

What each line does

• Staging first — Same flow, untrusted certificates, generous limits; switch the issuerRef once it works.
• email — Let's Encrypt sends expiry warnings here if renewal ever breaks.
• privateKeySecretRef — cert-manager creates the ACME account key in this Secret; never copy it into Git.
• dns01 route53 — For names in the shop zone it writes _acme-challenge TXT records; the cert-manager service account needs a scoped IAM role through IRSA, like ExternalDNS.
• http01 fallback — Other names use a temporary solver Ingress on port 80.

Green output

$ kubectl apply -f letsencrypt.yaml
$ kubectl get clusterissuer
NAME                  READY   AGE
letsencrypt-prod      True    20s
letsencrypt-staging   True    20s
$ kubectl apply -f wildcard.yaml
$ kubectl get challenges -n shop
NAME                                   STATE     DOMAIN                    AGE
stores-wildcard-1-2391077436-0         pending   stores.shop.example.com   30s
$ kubectl describe certificate stores-wildcard -n shop | tail -3
  Normal  Issuing    2m    cert-manager-certificates-issuing   The certificate has been successfully issued
$ cmctl status certificate stores-wildcard -n shop | head -4
Name: stores-wildcard
Conditions:
  Ready: True, Reason: Ready, Message: Certificate is up to date and has not expired
Renewal Time: 2026-12-24T03:10:42Z

Expected result. On EKS the DNS-01 solver proved control through Route 53 and a trusted wildcard certificate landed in the Secret. Renewal is scheduled for 24 December 2026 at 08:40 IST, about 30 days before expiry.

Every command, step by step (kind lab with kubectl)

1. Install

   kubectl apply -f cert-manager-v1.19.1.yaml
   

   What it does: Controller, webhook, cainjector.

   deployment.apps/cert-manager created
   

2. Wait

   kubectl wait -n cert-manager --for=condition=available deploy --all --timeout=180s
   

   What it does: Ready.

   condition met
   

3. CA

   kubectl apply -f local-ca.yaml
   

   What it does: Issuers and certs.

   clusterissuer.cert-manager.io/lab-ca created
   

4. Certs

   kubectl get certificate -A
   

   What it does: Ready True.

   shop   shop-tls   True   shop-tls
   

5. Ingress

   kubectl apply -f tls-ingress.yaml
   

   What it does: tls block.

   ingress.networking.k8s.io/storefront configured
   

6. HTTPS

   curl -s --cacert lab-ca.crt --resolve shop.example.com:443:127.0.0.1 https://shop.example.com/
   

   What it does: Trusted by our CA.

   web says hi
   

7. Redirect

   curl -sI --resolve shop.example.com:80:127.0.0.1 http://shop.example.com/
   

   What it does: 308.

   HTTP/1.1 308 Permanent Redirect
   

8. Dates

   openssl x509 -noout -dates
   

   What it does: 90 days.

   notAfter=Jan 23 02:20:11 2027 GMT
   

9. Issuers

   kubectl apply -f letsencrypt.yaml
   

   What it does: EKS only.

   clusterissuer.cert-manager.io/letsencrypt-prod created
   

10. Status

    cmctl status certificate stores-wildcard -n shop
    

    What it does: Renewal time.

    Ready: True
    

When it is red 1

Certificate stuck with READY False and an order pending

Cause. Challenge cannot complete: DNS not pointing at the controller, or port 80 blocked.

Fix, step by step

1. kubectl get challenges -A and describe.
2. curl the /.well-known path from outside.

When it is red 2

429 urn:ietf:params:acme:error:rateLimited

Cause. Too many production requests for one domain.

Fix, step by step

1. Use the staging server while testing.
2. Wait for the limit window to pass.

When it is red 3

x509: certificate signed by unknown authority

Cause. Staging or lab CA certificate used where a public one is expected.

Fix, step by step

1. Switch issuerRef to letsencrypt-prod.

When it is red 4

DNS-01 AccessDenied on route53:ChangeResourceRecordSets

Cause. cert-manager service account lacks the IAM role.

Fix, step by step

1. Attach a scoped role through IRSA or Pod Identity.

When it is red 5

Ingress serves the controller's fake certificate

Cause. Secret name in tls block differs from the Certificate's secretName, or Secret not yet created.

Fix, step by step

1. kubectl get secret -n shop.
2. Match the names.

Practice tasks

kubectl get pods -n cert-manager

kubectl get clusterissuer

kubectl get certificate -n shop

curl -s --cacert lab-ca.crt --resolve shop.example.com:443:127.0.0.1 https://shop.example.com/

openssl s_client -connect 127.0.0.1:443 -servername shop.example.com

cmctl renew shop-tls -n shop

kubectl get clusterissuer

kubectl get certificate -A

Recap

• cert-manager turns certificates into Kubernetes objects that renew themselves.
• Issuer or ClusterIssuer says where; Certificate says what names and which Secret.
• HTTP-01 needs port 80 reachable; DNS-01 needs DNS rights and allows wildcards.
• Test against Let's Encrypt staging before production to avoid rate limits.
• An annotation on the Ingress is enough for ingress-shim to create the Certificate.

FAQ

Q. Do I still need cert-manager with AWS ACM?

Not for certificates terminated on an ALB or NLB, since ACM renews those itself. You still need cert-manager for TLS inside the cluster, for controllers like NGINX or Traefik, and for internal mTLS.

Q. Where is the private key stored?

In the Kubernetes Secret named by secretName. Encrypt Secrets at rest and limit who can read them with RBAC.

Q. What happens if renewal fails?

The old certificate keeps working until expiry. cert-manager retries with backoff, and Let's Encrypt emails the account address. Alert on certmanager_certificate_expiration_timestamp_seconds.

Q. Can cert-manager issue certificates from a private CA such as AWS Private CA or Vault?

Yes, through external issuers or the built-in Vault issuer.

Interview questions

1. Explain HTTP-01 versus DNS-01.

   Model answer: HTTP-01 serves a token on port 80 and cannot do wildcards; DNS-01 writes a TXT record, works for wildcards and private names, but needs DNS API rights.

2. How does cert-manager know when to renew?

   Model answer: From renewBefore, or by default at two thirds of the certificate's life.

3. Issuer or ClusterIssuer?

   Model answer: ClusterIssuer for a shared Let's Encrypt account across namespaces; Issuer when a team must own its own CA.

4. How would you debug a stuck certificate?

   Model answer: Follow the chain: Certificate, CertificateRequest, Order, Challenge, and describe each.

Samajla ka?