Lou Stack Base
Lou Stack Base

Kubernetes 實戰操作: Service、Rolling Update、HPA 與 Debug 流程

Lou Chang Lou Chang 收錄於 DevOps  Kubernetes
 約 3400 字 預計閱讀 7 分鐘 

DevOps 學習筆記

承接上篇 (Cluster、Pod、Deployment), 這篇涵蓋 Service 網路存取、Rolling Update 零停機更新、CrashLoopBackOff 行為、HPA 自動擴縮, 以及遇到問題時的 debug 流程

前置條件: 已有 kind cluster + Deployment 跑著 go-api (3 replicas)

Service — 穩定的網路入口

為什麼需要 Service

Pod IP 是短暫的 — 每次重建都會變:

1
2
3
4
5
6

kubectl get pod go-api-xxx -o wide
# IP: 10.244.0.5

kubectl delete pod go-api-xxx
kubectl get pod go-api-yyy -o wide
# IP: 10.244.0.8 — changed

現在有 3 個 Pod, 每個 IP 都不同, 而且隨時可能變 其他服務要怎麼連?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

without Service:
  Pod 1: 10.244.0.5  ┐
  Pod 2: 10.244.0.6  ├── which IP? what if it changes?
  Pod 3: 10.244.0.7  ┘

with Service:
  Service: go-api (stable DNS + stable IP)
    ├── routes to Pod 1
    ├── routes to Pod 2
    └── routes to Pod 3

Service 怎麼找到 Pod

靠 label selector — Service 用 label 匹配 Pod:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# service.yaml
spec:
  selector:
    app: go-api        # find all Pods with this label

# deployment.yaml (Pod template)
template:
  metadata:
    labels:
      app: go-api      # matches the selector above

label 就是 Pod 上的標籤, Service 靠它認出「哪些 Pod 是我的」

Service YAML 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

apiVersion: v1
kind: Service
metadata:
  name: go-api
spec:
  type: ClusterIP
  selector:
    app: go-api
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP

三種 Service type

Type 用途 存取方式
ClusterIP cluster 內部存取 (預設) 只有 cluster 裡面能連
NodePort 透過 Node 的 port 從外部存取 <NodeIP>:<NodePort>
LoadBalancer 雲端環境拿外部 IP GKE/EKS 自動分配外部 IP

Endpoints — Service 的後端清單 

1

kubectl describe svc go-api

重點看 Endpoints:

1
2
3

Selector:   app=go-api
ClusterIP:  10.96.252.198
Endpoints:  10.244.0.5:8080, 10.244.0.6:8080, 10.244.0.7:8080

Endpoints 就是目前所有匹配 label 的 Pod IP 列表

刪掉一個 Pod, Deployment 自動重建, Endpoints 自動更新:

1
2
3

before: 10.244.0.5, 10.244.0.6, 10.244.0.7
after:  10.244.0.8, 10.244.0.6, 10.244.0.7
        ^^^^^^^^^^^ new Pod

Service 的 ClusterIP 10.96.252.198 完全沒變 — 這就是穩定入口的意思

Service Discovery — DNS

cluster 裡的其他 Pod 可以用 DNS name 連到 Service:

1
2
3
4
5
6

go-api.default.svc.cluster.local
  │      │       │      │
  │      │       │      └── cluster domain
  │      │       └── fixed suffix
  │      └── namespace
  └── service name

同一個 namespace 裡可以簡寫成 go-api:8080

port-forward — 從筆電連進 cluster

ClusterIP 只有 cluster 內部能連 從你的筆電要用 port-forward 開一條隧道:

1
2

your laptop                          K8s cluster
localhost:8080 ──── tunnel ────→ Service go-api:8080 ──→ Pod
1
2
3
4

kubectl port-forward svc/go-api 8080:8080

# in another terminal
curl http://localhost:8080/healthz

port-forward 只是開發/除錯用的 production 不會用它, 會用 LoadBalancer 或 Ingress

Rolling Update — 零停機更新

觸發 rolling update 

1
2
3
4
5
6

# build new image
docker build -t go-api:0.0.3 -f Dockerfile .
kind load docker-image go-api:0.0.3 --name devops-lab

# trigger update
kubectl set image deployment/go-api go-api=go-api:0.0.3

觀察更新過程 

1

kubectl get pods --watch
1
2
3
4
5
6
7

old-pod-1   Running                          ← 3 old Pods running
new-pod-1   Pending → ContainerCreating → Running   ← new Pod 1 ready
old-pod-1   Terminating                      ← THEN old Pod 1 killed
new-pod-2   Pending → Running               ← new Pod 2 ready
old-pod-2   Terminating                      ← old Pod 2 killed
new-pod-3   Pending → Running               ← new Pod 3 ready
old-pod-3   Terminating                      ← old Pod 3 killed

關鍵: 先確認新 Pod Running, 再殺舊 Pod 這就是零停機

由 Deployment 的 strategy 控制:

1
2
3
4
5

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1        # at most 1 extra Pod during update
    maxUnavailable: 0  # no downtime allowed

兩個 ReplicaSet 同時存在 

1

kubectl get rs
1
2

go-api-65577fc4f9   3   3   3   2m     ← new ReplicaSet (0.0.3), 3 Pods
go-api-668dcc5dd    0   0   0   4d     ← old ReplicaSet (0.0.2), scaled to 0

舊的 ReplicaSet 不會被刪, Pod 數量縮到 0 K8s 故意保留它, 讓你可以 rollback

Rollout History 和 Rollback 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# see revision history
kubectl rollout history deployment/go-api
# REVISION 1 ← go-api:0.0.2
# REVISION 2 ← go-api:0.0.3

# rollback to previous version
kubectl rollout undo deployment/go-api

# check — old ReplicaSet scales back up
kubectl get rs

rollout undo 把舊的 ReplicaSet 從 0 拉回 3, 新的從 3 縮到 0 同樣走 rolling update 流程, 一樣零停機

CrashLoopBackOff — Pod 持續 crash 的行為

什麼是 CrashLoopBackOff

當 container 一直 crash, kubelet 不會無限制地馬上重啟 它用指數退避 (exponential backoff):

1
2
3
4
5

crash #1 → restart immediately
crash #2 → wait ~10s, then restart
crash #3 → wait ~20s, then restart
crash #4 → wait ~40s, then restart
...keeps doubling, up to 5 minutes max

kubelet 不是放棄, 是越等越久 因為如果程式本身有 bug, 馬上重啟也只會馬上再 crash, 浪費資源

觀察 CrashLoopBackOff 

1

kubectl get pods --watch
1
2
3
4
5
6
7
8

my-pod   Running                        ← container starts
my-pod   Error                          ← container crashed (exit code != 0)
my-pod   Running   1                    ← kubelet restarted it
my-pod   Error                          ← crashed again
my-pod   CrashLoopBackOff              ← kubelet: "crashing too often, waiting..."
my-pod   Running   2 (14s ago)          ← restarted after 14s delay
my-pod   Error                          ← crashed again
my-pod   CrashLoopBackOff              ← waiting even longer (27s)

三個狀態的意思

STATUS 意思
Error container 剛死
CrashLoopBackOff 死太多次, kubelet 在等, 還沒重啟
Running 重啟成功

遇到 CrashLoopBackOff 怎麼辦 

1
2

# see the logs from the PREVIOUS crashed container
kubectl logs <pod-name> --previous

--previous 是關鍵 — 看上一次 crash 前的 log, 找出程式碼哪裡掛的

內部原理觀察

kube-system 裡的元件 

1

kubectl get pods -n kube-system
1
2
3
4
5
6

etcd-control-plane               ← database, stores all cluster state
kube-apiserver-control-plane      ← front door, all requests go through here
kube-scheduler-control-plane      ← decides which Node runs each Pod
kube-controller-manager-control-plane  ← runs reconciliation loops
kube-proxy-xxxxx                  ← network rules (iptables/ipvs)
coredns-xxxxx (x2)               ← DNS for service discovery

-n = --namespace K8s 自己的元件全部跑在 kube-system namespace

Events — 看元件怎麼協作 

1

kubectl describe pod <name>

Events 區塊記錄了完整的 Pod 誕生過程:

1
2
3
4

Scheduled  → default-scheduler  → assigned to devops-lab-control-plane
Pulled     → kubelet             → image already present
Created    → kubelet             → container created
Started    → kubelet             → container started

注意: Events 只保留 1 小時 舊的 Pod 看不到 Events, 只有新建的才有

get events — 看全局 

1

kubectl get events --sort-by='.lastTimestamp'

不指定 Pod name, 列出所有資源的事件 可以看到 ReplicaSet 建 Pod 的紀錄:

1

Normal  SuccessfulCreate  replicaset/go-api-668dcc5dd  Created pod: go-api-668dcc5dd-h5ttf

證明是 ReplicaSet 在負責建 Pod, 不是 Deployment 直接建

HPA — Horizontal Pod Autoscaler

為什麼需要 HPA

Deployment 的 replicas 是固定的 — 你設 3 就永遠是 3 但實際流量有高有低:

1
2
3
4
5
6

without HPA:
  replicas: 3 → always 3 Pods, even at 3am with zero traffic

with HPA:
  traffic high → scale up to 7 Pods
  traffic low  → scale down to 2 Pods

前置條件: metrics-server

HPA 需要知道每個 Pod 的 CPU 使用率 metrics-server 負責收集這些資料:

1
2
3
4
5
6
7

HPA: "CPU usage is how much?"
 ↓
metrics-server: "let me ask kubelet on each Node"
 ↓
kubelet: "Pod A uses 30m CPU, Pod B uses 45m CPU"
 ↓
HPA: "over threshold, scale up"

沒有 metrics-server, HPA 是瞎的

安裝 metrics-server (kind 環境) 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# install
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# kind uses self-signed certs, need to skip TLS verification
kubectl -n kube-system patch deployment metrics-server \
  --type='json' \
  -p='[{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "--kubelet-insecure-tls"}]'

# wait for ready
kubectl -n kube-system rollout status deployment/metrics-server

--kubelet-insecure-tls 只有 kind 需要 production (GKE/EKS) 有正式憑證, 不需要

kubectl patch 是什麼

patch 是就地修改 K8s 資源, 不需要重新寫整份 YAML

1
2
3
4
5

{
  "op": "add",
  "path": "/spec/template/spec/containers/0/args/-",
  "value": "--kubelet-insecure-tls"
}

path 對應 YAML 結構:

1
2
3
4
5
6
7
8

spec:                          # /spec
  template:                    # /spec/template
    spec:                      # /spec/template/spec
      containers:              # /spec/template/spec/containers
        - name: metrics-server # /spec/template/spec/containers/0
          args:                # /spec/template/spec/containers/0/args
            - --cert-dir=/tmp
            - --kubelet-insecure-tls  # ← /args/- means append here

怎麼知道 path? 先用 -o yaml 看結構:

1

kubectl -n kube-system get deployment metrics-server -o yaml

驗證 metrics-server 

1

kubectl top pods

有數字就代表 metrics-server 在運作

HPA YAML 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: go-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: go-api              # which Deployment to scale
  minReplicas: 2              # minimum Pods
  maxReplicas: 10             # maximum Pods
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50  # scale up when avg CPU > 50%

HPA 不直接管 Pod — 它修改 Deployment 的 replicas 數字, Deployment 再去調整 Pod 數量

averageUtilization: 50 的基準是 Deployment 裡的 resources.requests.cpu (50m) 所以 50% = 25m 如果 Pod 平均 CPU 超過 25m, 就開始 scale up

觀察 HPA 運作 

1
2
3
4
5

# apply HPA
kubectl apply -f k8s/hpa.yaml

# watch HPA
kubectl get hpa --watch
1
2
3
4
5
6

NAME     TARGETS        MINPODS  MAXPODS  REPLICAS
go-api   cpu: 0%/50%    2        10       3          ← idle
go-api   cpu: 27%/50%   2        10       3          ← load starting
go-api   cpu: 64%/50%   2        10       3          ← over threshold!
go-api   cpu: 65%/50%   2        10       4          ← scaled up: 3 → 4
go-api   cpu: 42%/50%   2        10       4          ← 4 Pods share load, CPU drops

產生負載的方式:

1
2

kubectl run load-test --rm -it --image=busybox --restart=Never -- sh -c \
  "while true; do wget -q -O- http://go-api.default.svc.cluster.local:8080/healthz; done"

停止負載後, 等約 5 分鐘 (cooldown), HPA 會自動 scale down 到 minReplicas: 2

Debug 流程

遇到問題永遠走這個順序, 從大到小:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

kubectl get pods                          ← what's the STATUS?
 ↓
status tells you the next step:

  ImagePullBackOff
   → image name wrong or forgot kind load

  Pending
   → kubectl describe pod <name>
   → Events: usually resource or scheduling issue

  CrashLoopBackOff
   → kubectl logs <name> --previous
   → application error in the logs

  Running but not working
   → kubectl logs <name>
   → check application logic

  don't know where to start
   → kubectl get events --sort-by='.lastTimestamp'
   → see everything that happened recently

常用 debug 指令

情境 指令
Pod 狀態不對 kubectl get pods
為什麼起不來 kubectl describe pod <name>
程式碼出錯 kubectl logs <name>
crash 後的 log kubectl logs <name> --previous
即時觀察 kubectl get pods --watch
全局事件 kubectl get events --sort-by='.lastTimestamp'
進 container debug kubectl exec -it <name> -- sh
看 Service 後端 kubectl describe svc <name>
看完整 YAML kubectl get <resource> <name> -o yaml

kubectl exec 可以進 container 裡面看 但如果 image 是 scratch, 沒有 shell, 進不去 這也是 scratch 安全的原因之一

kubectl logs 的注意事項 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# wrong — "go-api" is Deployment name, not Pod name
kubectl logs go-api

# correct — use actual Pod name
kubectl logs go-api-65577fc4f9-k9f9p

# shortcut — pick a Pod from Deployment automatically
kubectl logs deployment/go-api

# follow mode (like tail -f)
kubectl logs deployment/go-api -f

常見疑問

Docker Compose 跟 K8s 是什麼關係?

完全不同的工具, 解決不同階段的問題:

1
2

Docker Compose → single machine, multiple containers (dev/test)
Kubernetes     → multiple machines, managed containers (production)

它們是互相取代的, 不是一起用的:

1
2

dev (your laptop):  docker-compose.yml → docker compose up
production (cloud): k8s/*.yaml         → kubectl apply

Docker Compose 不會出現在 K8s 的世界裡 進了 K8s, 就用 K8s 自己的 YAML

Terraform、K8s YAML、Docker Compose 的分工 

1
2
3

Terraform        → infrastructure: EC2, VPC, EKS cluster, RDS
K8s YAML         → application deployment: Pods, Services, HPA
Docker Compose   → local dev only: quick multi-container setup

分層關係:

1
2
3
4
5

Terraform runs first: creates EKS cluster + RDS + VPC
 ↓
K8s YAML runs next:   deploys your app inside the cluster
 ↓
Docker Compose:        unrelated, only used on your laptop

kind 的 YAML 能直接搬到 GKE/EKS 嗎?

YAML 可以直接搬, cluster 不能搬 K8s 是標準化的, 不管底下是 kind、GKE、EKS, kubectl apply 的 YAML 格式完全一樣

需要調整的部分:

項目 kind GKE/EKS (production)
Image 來源 kind load 本機搬 Container Registry (GCR/ECR)
Service type ClusterIP + port-forward LoadBalancer
Resource requests 隨便設 根據實際負載調整
Ingress 不需要 域名、HTTPS
Secrets 寫死或不用 Secret Manager

核心的 Deployment、ReplicaSet、HPA 邏輯不用改

K8s 裡面用 Docker 嗎?

K8s 內部用的是 containerd (透過 CRI 介面), 不是 Docker

1
2
3
4
5
6
7
8

your laptop
├── Docker daemon
│   └── devops-lab-control-plane ← this is a Docker container (kind)
│       └── K8s cluster
│           └── containerd       ← K8s uses this, not Docker
│               ├── Pod 1
│               ├── Pod 2
│               └── Pod 3

docker stop devops-lab-control-plane 停的是整個 kind cluster 那個 container, 不是停某個 Pod K8s 裡面的操作全部用 kubectl

production (GKE/EKS) 根本不會碰到 docker 指令

清除方式

刪你的應用 (保留 cluster) 

1
2
3

kubectl delete -f k8s/hpa.yaml
kubectl delete -f k8s/service.yaml
kubectl delete -f k8s/deployment.yaml

順序: HPA → Service → Deployment (從外到內)

砍掉整個 cluster 

1

kind delete cluster --name devops-lab

一行搞定, 全部消失

暫停, 下次還要用 

1
2
3

docker stop devops-lab-control-plane
# next time:
docker start devops-lab-control-plane

建立的完整流程 

1
2
3
4
5
6

1. write YAML (deployment.yaml, service.yaml, hpa.yaml)
2. build image → load into kind (or push to registry)
3. kubectl apply -f deployment.yaml   ← app runs
4. kubectl apply -f service.yaml      ← app is accessible
5. install metrics-server              ← once per cluster
6. kubectl apply -f hpa.yaml          ← auto-scaling enabled

更新程式碼:

1

edit code → docker build new tag → kind load → kubectl set image → done

必須記住的東西

每天都用的指令 

1
2
3
4
5
6
7
8

kubectl get pods
kubectl get pods -o wide
kubectl describe pod <name>
kubectl logs <name>
kubectl logs <name> --previous
kubectl apply -f <file>
kubectl delete -f <file>
kubectl get events --sort-by='.lastTimestamp'

必須記住的概念 

1
2
3
4
5
6

Deployment → ReplicaSet → Pod → Container
desired state vs actual state → reconciliation loop
Pod is ephemeral → IP changes → need Service
Service finds Pods by label selector
Rolling update: new Pod ready → then kill old Pod
CrashLoopBackOff: exponential backoff (10s → 20s → 40s → ... → 5min max)

不用背, 查就好 

1
2
3

kubectl explain deployment.spec     # YAML format reference
kubectl get deploy <name> -o yaml   # see full YAML of any resource
kubectl api-resources               # all resource types and abbreviations

References

更新於 2026-02-25 
CC BY-NC-SA 4.0
閱讀原始文檔
LINE
KubernetesK8sDevOpsServiceHPARolling UpdateDebug
返回 | 主頁
0%