Build a CI/CD Pipeline for Kubernetes with Immutable Deployments

A CI CD pipeline contract for cloud native architecture

A reliable ci cd pipeline promotes an image digest, not a Git SHA tag such as api:4f91c2a. A tag is a mutable registry pointer; a digest identifies the manifest bytes that were tested. Make the promotion contract explicit: build once, record registry/repository@sha256:..., sign that exact digest, and let deployment jobs consume only that recorded value. This is a practical devops training exercise because it exposes a common production failure: rebuilding the same commit in a later stage can produce different dependency resolution, base-image, or build-context results.

name: build-image
on: [push]
permissions:
  contents: read
  packages: write
  id-token: write
jobs:
  build:
    runs-on: ubuntu-latest
    outputs:
      digest: ${{ steps.image.outputs.digest }}
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - id: image
        env:
          IMAGE: ghcr.io/acme/payments
        run: |
          docker buildx build             --platform linux/amd64             --push --provenance=true --sbom=true             --metadata-file metadata.json             -t $IMAGE:${GITHUB_SHA} .
          DIGEST=$(jq -r '."containerimage.digest"' metadata.json)
          echo digest=$DIGEST >> $GITHUB_OUTPUT
          cosign sign --yes $IMAGE@$DIGEST

Use cosign verify --certificate-identity-regexp=... in the deploy job, or enforce the same rule with a Kyverno verifyImages policy in the cluster. The signature must be checked against the digest rather than the tag: signing api:stable only proves what the tag referenced at signing time, while a registry user can retarget that tag afterward. For a cloud native architecture, also retain Buildx provenance and SBOM artifacts so a CVE finding can be joined to the exact digest deployed to each environment.

Docker training: make builds fast without hiding dependency drift

In docker training, measure the build before adding cache directives. Run time docker buildx build --progress=plain --load -t payments:bench . twice, inspect cache storage with docker buildx du --verbose, and compare the duration of the dependency-download step. A well-ordered Dockerfile copies dependency manifests before application source, so a change to internal/handler.go reuses the module-download layer instead of invalidating it.

# syntax=docker/dockerfile:1.7
FROM --platform=$BUILDPLATFORM golang:1.22 AS build
ARG TARGETOS
ARG TARGETARCH
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod     --mount=type=cache,target=/root/.cache/go-build     go mod download
COPY . .
RUN --mount=type=cache,target=/go/pkg/mod     --mount=type=cache,target=/root/.cache/go-build     CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH     go build -trimpath -ldflags='-s -w' -o /out/payments ./cmd/payments

FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/payments /payments
USER nonroot:nonroot
ENTRYPOINT ["/payments"]

The two cache mounts speed up compilation but do not become image layers, which prevents the Go module cache from inflating the runtime image. Pin the builder and runtime base images to registry digests in the production Dockerfile, then use docker scout cves payments:bench or Trivy against the resulting digest. The subtle mistake is treating a BuildKit cache as trusted source input: a shared remote cache can restore compiled objects, but it must not replace dependency checksum verification from go.sum.

Kubernetes training for safe container orchestration rollouts

For Kubernetes training, deploy the digest produced by the build job and distinguish liveness from readiness. A liveness probe should answer “is this process irrecoverably stuck?”, while /readyz should fail during startup, migration, or unavailable mandatory dependencies. If both probes call the same deep database check, a short database incident can make kubelet kill healthy processes, creating a restart storm during container orchestration.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments
spec:
  replicas: 3
  progressDeadlineSeconds: 180
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    metadata:
      labels:
        app: payments
    spec:
      containers:
        - name: payments
          image: ghcr.io/acme/payments@sha256:REPLACE_WITH_SIGNED_DIGEST
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet: { path: /readyz, port: 8080 }
            periodSeconds: 3
            failureThreshold: 2
          livenessProbe:
            httpGet: { path: /healthz, port: 8080 }
            initialDelaySeconds: 20
            periodSeconds: 10
          startupProbe:
            httpGet: { path: /healthz, port: 8080 }
            failureThreshold: 30
            periodSeconds: 2

Before changing the cluster, run kubectl apply --server-side --dry-run=server -f deployment.yaml; server-side validation catches admission-policy and CRD validation failures that client-side dry runs miss. Then use kubectl rollout status deployment/payments --timeout=180s and query a concrete service-level metric, such as a five-minute 5xx ratio in Prometheus, before promotion. A successful rollout only means the Deployment controller observed available pods; it does not prove the new version preserves latency or business correctness. Keep kubectl rollout undo deployment/payments available, but remember that it restores the previous ReplicaSet template, not external schema changes.

Use minikube to test deployment assumptions before a shared cluster

Use minikube as a disposable integration target for manifests, probes, Services, and admission policies. Start it with the same Kubernetes minor-version policy used by your organization, enable the metrics server, load the locally built image, and wait on the Deployment instead of relying on a fixed sleep. This catches invalid selectors, missing ConfigMaps, and probe paths before a pull request reaches a shared environment.

minikube start --driver=docker
minikube addons enable metrics-server
docker build -t payments:dev .
minikube image load payments:dev
kubectl apply -f k8s/
kubectl wait --for=condition=Available deployment/payments --timeout=180s
kubectl port-forward service/payments 18080:80 &
PID=$!
trap 'kill $PID' EXIT
curl --fail --retry 20 --retry-connrefused http://127.0.0.1:18080/readyz
kubectl top pods -l app=payments

A local image load is intentionally different from a registry pull: it can conceal missing image-pull credentials and architecture mismatches. Run a second CI check that deploys the immutable registry digest to a clean namespace, and inspect events with kubectl get events --sort-by=.lastTimestamp. If a manifest uses a tag with imagePullPolicy: IfNotPresent, minikube may retain an old local tag; digest references avoid that misleading result.

Terraform and infrastructure as code for cloud promotion

Treat Terraform state as a production dependency of infrastructure as code, not as a CI workspace artifact. The same repository pattern is useful in aws training and google cloud training labs: isolate state by environment, give CI a workload-identity role instead of static keys, and let the pipeline apply an approved plan from protected branches only. On Google Cloud, a versioned GCS bucket is a straightforward remote backend; on AWS, use an S3 backend with IAM restricted to the environment prefix and enable bucket versioning for state recovery.

terraform {
  required_version = ">= 1.6.0"
  backend "gcs" {}
}

resource "google_artifact_registry_repository" "images" {
  location      = var.region
  repository_id = "payments"
  format        = "DOCKER"
}

resource "google_service_account" "deployer" {
  account_id   = "payments-deployer"
  display_name = "CI deployment identity"
}

Initialize backend coordinates outside source control, then preserve the binary plan only for the short approval-to-apply window: terraform init -backend-config=bucket=$TF_STATE_BUCKET -backend-config=prefix=payments/prod, terraform plan -detailed-exitcode -out=tfplan, and terraform apply tfplan. Exit code 2 from -detailed-exitcode means a change exists and should create a review gate. Do not publish tfplan as a broadly readable CI artifact: plans often contain rendered secret values. Re-plan if the approval window is long or state changes, because applying a stale plan is correctly rejected when Terraform detects state serial drift.

Frequently Asked Questions

How should a CI CD pipeline pass a Docker image to Kubernetes?

Have the build job extract the Buildx manifest digest from its metadata file, store it as a job output, and render image: registry.example/app@sha256:... into the deployment manifest. Verify that digest with cosign verify before kubectl apply; do not reconstruct the image from a Git tag in the deploy job.

Can minikube test a Kubernetes deployment that uses a private registry?

Yes. Create a pull secret with kubectl create secret docker-registry regcred --docker-server=REGISTRY --docker-username=USER --docker-password=TOKEN, reference it through imagePullSecrets, and deploy a registry digest. Avoid only using minikube image load for this test, because that bypasses the authentication path used by a remote cluster.

What Terraform workflow is safest for infrastructure as code in a CI CD pipeline?

Run terraform fmt -check, terraform validate, and terraform plan -detailed-exitcode -out=tfplan in the review stage. Restrict terraform apply tfplan to a protected environment with short-lived OIDC credentials, encrypt the plan artifact, and re-plan when state has changed since approval.

Why does container orchestration need readiness probes if a process is running?

A running process can still be unable to serve traffic while it warms caches, loads configuration, or waits for a migration lock. Configure readinessProbe against a shallow endpoint such as /readyz; Kubernetes removes unready pods from Service endpoints without restarting them, unlike a liveness-probe failure.