Kubernetes Failure Modeling: Memory Kills, Health Checks, Configuration Objects, and Logical Partitions

DevOps learning notes.

Following the previous two posts (K8s core architecture, Service/HPA/Debug), this one digs into actual behavior under K8s failure scenarios, plus how to use ConfigMap, Secret, and Namespace. The goal is being able to predict and diagnose problems — not just apply YAML.

Environment: k3s v1.34 single-node cluster + go-api (Go HTTP server, scratch image)

Memory Exhaustion: When the OS Terminates Your Workload

How It Happens

Container memory usage exceeds the ceiling set by limits.memory, and the Linux kernel’s OOM killer terminates the process directly.

It is not K8s doing the killing — it is the kernel. K8s merely observes that the container died and records the reason.

Lab Setup

Add a /oom endpoint to go-api that aggressively allocates memory:

1
2
3
4
5
6
7
8
9

func OOM(w http.ResponseWriter, r *http.Request) {
    log.Println("oom endpoint hit, allocating memory until killed")
    var data [][]byte
    for {
        block := make([]byte, 10*1024*1024) // 10MB per iteration
        data = append(data, block)
        log.Printf("allocated %d MB", len(data)*10)
    }
}

Set the memory limit to 64Mi in the Deployment:

1
2
3
4
5

resources:
  requests:
    memory: 32Mi
  limits:
    memory: 64Mi

Send a request with curl to /oom — the connection drops immediately because the container was killed, so the connection breaks.

What the Output Shows

1

kubectl describe pod <pod-name>

1
2
3

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137

After restart, because /oom was not hit again, the Pod stabilizes. But if the container OOMs on startup (for example, consuming too much memory during init), it will keep restarting and eventually enter CrashLoopBackOff.

Expiration Versus Restart Loop

These two are often confused, but they operate at different levels:

• OOMKilled is a cause — the process was killed for exceeding memory.
• CrashLoopBackOff is a state — the container keeps crashing, and K8s increases the restart interval (exponential backoff: 10s → 20s → 40s → … up to 5 minutes).

A container can enter CrashLoopBackOff because of OOMKilled, or for entirely different reasons (code bugs, misconfiguration).

Guaranteed Minimum Versus Hard Ceiling: Two Different Concepts

1
2
3
4
5

resources:
  requests:
    memory: 32Mi    # used by Scheduler
  limits:
    memory: 64Mi    # runtime hard cap

Guaranteed Minimum: ‘I Need at Least This Much’

The Scheduler uses requests to decide which Node a Pod lands on. If a Node’s remaining allocatable resources are less than the Pod’s requests, the Scheduler will not place the Pod there, and the Pod’s status becomes Pending.

Hard Ceiling: ‘You Cannot Exceed This’

The hard cap enforced at runtime for a container. Exceed it and you get OOM-killed.

Core Distinction

Lab: Simulating Scarcity With Quotas

My local k3s node has 39 GB of memory, so tiny requests never fill it up. Instead, I used a ResourceQuota to constrain things at the namespace level:

1
2
3
4
5
6
7
8
9

apiVersion: v1
kind: ResourceQuota
metadata:
  name: small-quota
  namespace: resource-test
spec:
  hard:
    requests.memory: 100Mi
    limits.memory: 200Mi

Deploy 4 Pods, each with requests of 32Mi (4 × 32Mi = 128Mi > quota 100Mi):

1
2
3
4
5
6

Warning  FailedCreate  replicaset/go-api-hungry-xxx
  Error creating: pods "..." is forbidden:
  exceeded quota: small-quota,
  requested: requests.memory=32Mi,
  used: requests.memory=96Mi,
  limited: requests.memory=100Mi

3 Pods succeeded, the 4th was blocked. Note this differs from genuine Node resource shortage:

Health Checks: How the Orchestration Layer Knows Your Workload Is Alive

A running container does not mean a healthy app — it could be deadlocked, or the database connection might have dropped. K8s needs a way to verify.

A health check is K8s periodically hitting an endpoint you specify and checking the HTTP status code:

• 200 → healthy
• Non-200 → something is wrong

Two Flavors of Health Checks

Restaurant analogy:

• Readiness = “Is the kitchen ready?” → If not, stop seating guests, but the kitchen stays.
• Liveness = “Is the chef breathing?” → If not, get a new chef (restart).

Configuration Syntax

Use different key names in the container spec:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

containers:
  - name: go-api
    image: go-api:0.0.5
    readinessProbe:         # ← this key makes it a readiness probe
      httpGet:
        path: /ready
        port: 8080
      initialDelaySeconds: 2
      periodSeconds: 5
    livenessProbe:          # ← this key makes it a liveness probe
      httpGet:
        path: /alive
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 3
      failureThreshold: 3   # 3 consecutive failures = dead

You can set both at the same time, and even point them at the same endpoint. In production they are usually separate: readiness checks more things (database, cache), liveness only checks whether the app itself is stuck.

Traffic-Acceptance Check Lab

Set the readiness probe to point at a non-existent /ready endpoint; the probe hits it every 5 seconds and gets 404 every time:

1

kubectl get pods -l app=go-api-probe-test

1
2
3

NAME                                 READY   STATUS    RESTARTS   AGE
go-api-probe-test-86cd4c5787-7lmrj   0/1     Running   0          39s
go-api-probe-test-86cd4c5787-mtxl2   0/1     Running   0          39s

READY is 0/1 — the container is running, but K8s considers it unready.

1

kubectl describe pod <pod-name>

1
2

Conditions:
  Ready       False

Check the EndpointSlice:

1

kubectl get endpointslice -l kubernetes.io/service-name=go-api-probe-test

The EndpointSlice lists all addresses, but each is marked ready: false. The Service will not send traffic to a ready: false Pod.

Note: Starting with K8s v1.33+, kubectl get endpoints shows a deprecation warning. Use kubectl get endpointslice -l kubernetes.io/service-name=<svc> instead.

Process-Alive Check Lab

Set the liveness probe to point at a non-existent /alive endpoint. It fails every 3 seconds; after 3 consecutive failures, K8s kills the container:

1

kubectl get pods -l app=go-api-liveness-test -w

1
2
3
4
5

NAME                                   READY   STATUS             RESTARTS     AGE
go-api-liveness-test-95bb8f89c-7qn85   1/1     Running            3 (8s ago)   45s
go-api-liveness-test-95bb8f89c-7qn85   0/1     CrashLoopBackOff   3 (0s ago)   49s
go-api-liveness-test-95bb8f89c-7qn85   1/1     Running            4 (29s ago)  78s
go-api-liveness-test-95bb8f89c-7qn85   0/1     CrashLoopBackOff   4 (0s ago)   88s

Completely different from readiness — RESTARTS keeps climbing, the container is killed and restarted over and over, eventually entering CrashLoopBackOff.

When to Use Which: Frequent Mistakes

A common mistake: using a liveness probe to check the database connection. If the database goes down, liveness fails → K8s restarts your app → the database is still down after the app comes back up → liveness fails again → infinite restart loop.

Correct pattern: put the database connection check on the readiness probe. When the database drops, readiness fails and the Pod merely stops receiving traffic temporarily. When the database recovers, the Pod automatically comes back — no restart needed.

Configuration Objects and Sensitive Data

What Problem They Solve

Apps need configuration values (environment, log level, DB password). Hard-coding them means rebuilding the image whenever a value changes, and different environments (dev/staging/prod) require different images.

ConfigMap and Secret extract configuration from the image so the same image works across environments with only the configuration swapped.

Non-Sensitive Configuration

A key-value configuration object stored in K8s:

1
2
3
4
5
6
7

apiVersion: v1
kind: ConfigMap
metadata:
  name: go-api-config
data:
  APP_ENV: "staging"
  LOG_LEVEL: "debug"

Sensitive Data

Almost identical to ConfigMap, but used for passwords and API keys:

1
2
3
4
5
6
7

apiVersion: v1
kind: Secret
metadata:
  name: go-api-secret
type: Opaque
stringData:
  DB_PASSWORD: "my-secret-password-123"

type: Opaque is the Secret type meaning “general-purpose secret.” Use Opaque in 90% of cases.

Other built-in types:

Injection: Passing Values Into the Workload

Containers only understand two things: environment variables and files. You need to turn ConfigMap / Secret values into a format the container can read.

Option one — environment variables:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

containers:
  - name: go-api
    envFrom:
      - configMapRef:
          name: go-api-config    # all keys become env vars
    env:
      - name: DB_PASSWORD
        valueFrom:
          secretKeyRef:
            name: go-api-secret
            key: DB_PASSWORD     # pick a specific key from Secret

envFrom flattens the entire ConfigMap, valueFrom picks a single key from the Secret.

Option two — mount as files (volume mount):

1
2
3
4
5
6
7

volumes:
  - name: config
    configMap:
      name: go-api-config
volumeMounts:
  - name: config
    mountPath: /etc/config

Inside the container, /etc/config/APP_ENV will be a file whose content is staging. Works well for larger configuration files (nginx.conf, for example).

Lab Outcome

Go code reads the values with os.Getenv:

1

kubectl logs -l app=go-api-config-test

1

2026/04/02 23:17:11 env=staging log_level=debug db_configured=true

All three values injected successfully.

Are Sensitive Data Objects Safe?

K8s Secrets by default are only base64-encoded, not encrypted. Anyone with kubectl permissions can:

1
2

kubectl get secret go-api-secret -o yaml
# base64 encoded, decode = plaintext

In production you never write passwords directly in a Secret YAML and commit it to git. The actual workflow:

1
2
3
4
5

External Secret Manager (GCP Secret Manager / Vault / AWS Secrets Manager)
         ↓ (synced by tooling, not manual)
    K8s Secret object
         ↓ (envFrom / valueFrom)
    Container env vars

The “tooling” that handles the sync is typically:

• External Secrets Operator — installed in K8s, periodically syncs from the external secret manager into K8s Secrets
• CI/CD pipeline — creates Secrets at deploy time via kubectl create secret
• Terraform — creates them with the kubernetes_secret resource

K8s Secrets act as a bridge between the external secret manager and the container — the container does not call GCP Secret Manager API directly; it only reads environment variables from the K8s Secret.

If you use GCP Cloud Run (not K8s), this bridge layer is unnecessary — Cloud Run can inject environment variables directly from GCP Secret Manager.

Logical Partitions: Isolation Zones Within the Orchestrated Environment

What a Logical Partition Is

Think of a cluster as an office building:

• Logical partition = a floor — each floor has its own rooms (Pods), services (Service), configuration (ConfigMap)
• Different floors can have rooms with the same name without conflict
• But the building’s elevators (Nodes) and parking garage (Storage) are shared
• By default, the hallways between floors are open (network connectivity)

This is not the .NET code-organization namespace, nor the Linux kernel namespace (process isolation). K8s Namespaces are logical groupings of resources.

Common Use Cases

1
2
3
4
5

cluster
├── namespace: dev        ← development
├── namespace: staging    ← testing
├── namespace: prod       ← production
└── namespace: default    ← default, used when no namespace specified

What Gets Isolated, What Stays Shared

Per-partition independent resources:

• Pod, Deployment, Service, ConfigMap, Secret — names can repeat across partitions without interference
• ResourceQuota — different resource quotas can be set per partition
• RBAC permissions — you can restrict a person to operate only within a specific partition (RBAC = Role-Based Access Control, using roles to control who can do what; follow least privilege when assigning roles)

Shared across the entire cluster:

• Node — all partitions’ Pods run on the same set of machines
• Storage (PersistentVolume)
• Network — cross-partition connectivity is enabled by default; blocking it requires a NetworkPolicy

Cross-Partition Network Reachability Lab

Create dev and staging partitions, each with its own go-api deployment and Service (both named go-api, no conflict).

From a Pod in dev, call the Service in staging:

1
2
3
4
5

# run a Pod with curl in dev namespace
kubectl run test-curl -n dev --image=curlimages/curl -- sleep 3600

# from dev, call staging service
kubectl exec -n dev test-curl -- curl -s http://go-api.staging.svc.cluster.local/healthz

Got a successful response, proving that cross-partition network connectivity is on by default.

K8s internal DNS rule: <service-name>.<namespace>.svc.cluster.local. Within the same partition you can omit the suffix and use just the service name.

Note: A scratch image contains no tools at all (not even wget, curl, or sh), so you cannot kubectl exec into it for debugging. You need to run a separate Pod with tools to test.

Key Takeaways

Failure Diagnosis Cheat Sheet

Health Check Selection Rules

• Checking if the app itself is stuck → liveness (a restart can fix it)
• Checking external dependencies (DB, cache) → readiness (restarting won’t help; wait for the dependency to recover)
• Not sure? → Start with readiness; it is safer than liveness

Guaranteed Minimum Versus Hard Ceiling in One Sentence

• requests = used by the Scheduler for placement — “I need at least this much to land on a Node”
• limits = runtime ceiling — “exceed this and the process gets killed”

References

• Kubernetes Documentation — Container Resources — complete explanation of requests and limits
• Kubernetes Documentation — Pod Lifecycle — Pod status and restart policy
• Kubernetes Documentation — Liveness, Readiness and Startup Probes — configuration and differences of the three probe types
• Kubernetes Documentation — ConfigMaps — creating and using ConfigMaps
• Kubernetes Documentation — Secrets — Secret types, creation, and injection
• Kubernetes Documentation — Namespaces — Namespace use cases and isolation boundaries
• Kubernetes Documentation — Resource Quotas — limiting namespace resource consumption with ResourceQuota
• Kubernetes Documentation — DNS for Services and Pods — cross-namespace DNS rules
• Kubernetes Documentation — EndpointSlice — EndpointSlice replacing the legacy Endpoints object