Cilium Study [1기] 7주차 K8S Performance & Tunning

Cilium Study [1기] 7주차 K8S Performance & Tunning

안녕하세요 이번 게시물에서는 K8S Performance & Tunning 라는 주제에 대한 정보를 전달 드리고자 합니다.

1. 실습 환경 구성

실습 환경 구성은 호스트 환경에서 가장 높은 퍼모먼스를 사용하기 위해 wsl를 이용하여 환경을 구성합니다.

제 환경은 vCPU 8, Mem 12GB로 Kind를 구성 했습니다.

kind create cluster --name myk8s --image kindest/node:v1.33.2 --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30000
    hostPort: 30000
  - containerPort: 30001
    hostPort: 30001
  - containerPort: 30002
    hostPort: 30002
  - containerPort: 30003
    hostPort: 30003
  kubeadmConfigPatches: # Prometheus Target connection refused bind-address 설정
  - |
    kind: ClusterConfiguration
    controllerManager:
      extraArgs:
        bind-address: 0.0.0.0
    etcd:
      local:
        extraArgs:
          listen-metrics-urls: http://0.0.0.0:2381
    scheduler:
      extraArgs:
        bind-address: 0.0.0.0
  - |
    kind: KubeProxyConfiguration
    metricsBindAddress: 0.0.0.0
EOF

# kube-ops-view
helm repo add geek-cookbook https://geek-cookbook.github.io/charts/
helm install kube-ops-view geek-cookbook/kube-ops-view --version 1.2.2 --set service.main.type=NodePort,service.main.ports.http.nodePort=30003 --set env.TZ="Asia/Seoul" --namespace kube-system

# metrics-server
helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
helm upgrade --install metrics-server metrics-server/metrics-server --set 'args[0]=--kubelet-insecure-tls' -n kube-system

# 확인
kubectl top node
kubectl top pod -A --sort-by='cpu'
kubectl top pod -A --sort-by='memory'

cat <<EOT > monitor-values.yaml
prometheus:
  prometheusSpec:
    scrapeInterval: "15s"
    evaluationInterval: "15s"
  service:
    type: NodePort
    nodePort: 30001

grafana:
  defaultDashboardsTimezone: Asia/Seoul
  adminPassword: prom-operator
  service:
    type: NodePort
    nodePort: 30002

alertmanager:
  enabled: false
defaultRules:
  create: false
prometheus-windows-exporter:
  prometheus:
    monitor:
      enabled: false
EOT

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# 배포
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack --version 75.15.1 \
-f monitor-values.yaml --create-namespace --namespace monitoring

# 기본 비밀번호는 admin , prom-operator
wslview http://127.0.0.1:30002

http://127.0.0.1:30002/dashboard/import
Grafana에서 15661, 12006 DashBoard를 다운로드 해서 업로드 합니다.

Figure 1.1 DashBoard Import

2. Kube-burner

✅ Kube-burner 개념

Kube-burner는 Kubernetes 클러스터에 대량의 오브젝트를 생성·삭제하여 Control Plane과 DataPlane에 의도적으로 부하를 가하고, 성능·스케일 특성을 계측·검증하는 오케스트레이션 도구입니다.

주로 대규모 리소스(Deployments, Pods 등)를 빠른 속도로 생성·삭제함으로써 API 서버, etcd, 스케줄러 등 제어면의 병목을 드러내고 성능 지표를 수집·분석할때 사용 하며, 실행 중/후 Prometheus로부터 메트릭을 수집하고, 자체 표현식으로 경보를 평가(check-alerts)하거나 메트릭을 인덱싱(index)할 수 있습니다.

더 자세한 내용은 공식문서를 참고 하시면 됩니다.

Kube-burner 설치

pwru를 이용한 간단한 실습을 진행해보겠습니다.

시나리오는 1.1.1.1 목적지 차단을 설정하고, curl을 호출한 상태에서 pwru를 이용하여 모니터링해서 차단 이유를 확인 과정 입니다.

git clone https://github.com/kube-burner/kube-burner.git
cd kube-burner

# 바이너리 설치
curl -LO https://github.com/kube-burner/kube-burner/releases/download/v1.17.3/kube-burner-V1.17.3-linux-x86_64.tar.gz

tar -xvf kube-burner-V1.17.3-linux-x86_64.tar.gz

sudo cp kube-burner /usr/local/bin

시나리오 1

Deployment 1개(pod 1개) 생성 → 삭제 작업을 통해 jobIterations qps burst 의미를 알아보겠습니다.

cat << EOF > s1-config.yaml
global:
  measurements:
    - name: none

jobs:
  - name: create-deployments
    jobType: create
    jobIterations: 1  # How many times to execute the job , 해당 job을 5번 반복 실행
    qps: 1            # Limit object creation queries per second ,     초당 최대 요청 수 (평균 속도 제한) - qps: 10이면 초당 10개 요청
    burst: 1          # Maximum burst for throttle , 순간적으로 처리 가능한 요청 최대치 (버퍼) - burst: 20이면 한순간에 최대 20개까지 처리 가능
    namespace: kube-burner-test
    namespaceLabels: {kube-burner-job: delete-me}
    waitWhenFinished: true # false
    verifyObjects: false
    preLoadImages: true # false
    preLoadPeriod: 30s # default 1m
    objects:
      - objectTemplate: s1-deployment.yaml
        replicas: 1 # JobIterations * Replicas의 갯수 만큼 pod 발생
EOF

cat << EOF > s1-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-{{ .Iteration}}-{{.Replica}}
  labels:
    app: test-{{ .Iteration }}-{{.Replica}}
    kube-burner-job: delete-me
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-{{ .Iteration}}-{{.Replica}}
  template:
    metadata:
      labels:
        app: test-{{ .Iteration}}-{{.Replica}}
    spec:
      containers:
        - name: nginx
          image: nginx:alpine
          ports:
            - containerPort: 80
EOF

cat << EOF > s1-config-delete.yaml
jobs:
  - name: delete-deployments-namespace
    qps: 500
    burst: 500
    namespace: kube-burner-test
    jobType: delete
    waitWhenFinished: true
    objects:
    - kind: Deployment
      labelSelector: {kube-burner-job: delete-me}
      apiVersion: apps/v1
    - kind: Namespace
      labelSelector: {kube-burner-job: delete-me}
EOF

# kube-ops-view로 모니터링 & watch
wslview http://127.0.0.1:30002
watch -d kubectl get ns,pod -A
# 부하 발생 실행 Launch benchmark
kube-burner init -c s1-config.yaml --log-level debug

Figure 2.1 kube-ops-view

테스트 수행 시 preload-kube-burner Pod는 Node에 테스트 시 사용하는 container역할을 수행 합니다.

테스트 이후 테스트때 수행한 리소스는 다음과 같이 삭제할 수 있습니다.

cat << EOF > s1-config-delete.yaml

jobs:
  - name: delete-deployments-namespace
    qps: 500
    burst: 500
    namespace: kube-burner-test
    jobType: delete
    waitWhenFinished: true
    objects:
    - kind: Deployment
      labelSelector: {kube-burner-job: delete-me}
      apiVersion: apps/v1
    - kind: Namespace
      labelSelector: {kube-burner-job: delete-me}
EOF

kube-burner init -c s1-config-delete.yaml --log-level debug

시나리오 1 기준 옵션 분석

# 명령어 반복 수행
kube-burner init -c s1-config.yaml --log-level  debug
kube-burner init -c s1-config-delete.yaml --log-level debug

preLoadImages: false 옵션을 적용하면 초기 이미지 다운로드 과정을 생략합니다.

# preLoadImages : true 상태
time="2025-08-29 23:50:38" level=info msg="Pre-load: Creating DaemonSet using images [nginx:alpine] in namespace preload-kube-burner" file="pre_load.go:195"
time="2025-08-29 23:50:38" level=info msg="Pre-load: Sleeping for 30s" file="pre_load.go:86"
time="2025-08-29 23:51:09" level=info msg="Deleting 1 namespaces with label: kube-burner-preload=true" file="namespaces.go:67"
time="2025-08-29 23:51:09" level=debug msg="Waiting for 1 namespaces labeled with kube-burner-preload=true to be deleted" file="namespaces.go:90"
time="2025-08-29 23:51:10" level=debug msg="Waiting for 1 namespaces labeled with kube-burner-preload=true to be deleted" file="namespaces.go:90"
time="2025-08-29 23:51:11" level=debug msg="Waiting for 1 namespaces labeled with kube-burner-preload=true to be deleted" file="namespaces.go:90"
time="2025-08-29 23:51:12" level=debug msg="Waiting for 1 namespaces labeled with kube-burner-preload=true to be deleted" file="namespaces.go:90"
time="2025-08-29 23:51:15" level=info msg="Triggering job: create-deployments" file="job.go:122"
time="2025-08-29 23:51:15" level=info msg="0/1 iterations completed" file="create.go:119"
time="2025-08-29 23:51:15" level=debug msg="Creating object replicas from iteration 0" file="create.go:122"
time="2025-08-29 23:51:16" level=info msg="Namespace kube-burner-test-0 already exists" file="namespaces.go:44"
time="2025-08-29 23:51:17" level=error msg="Deployment/deployment-0-1 in namespace kube-burner-test-0 already exists" file="create.go:269"
time="2025-08-29 23:51:17" level=info msg="Waiting up to 4h0m0s for actions to be completed" file="create.go:169"
time="2025-08-29 23:51:18" level=info msg="Actions in namespace kube-burner-test-0 completed" file="waiters.go:74"
time="2025-08-29 23:51:18" level=info msg="Job create-deployments took 3s" file="job.go:191"
time="2025-08-29 23:51:18" level=info msg="Finished execution with UUID: 63b8fd12-ecb5-42fe-b1ca-5d904f5fa670" file="job.go:264"
time="2025-08-29 23:51:18" level=info msg="👋 Exiting kube-burner 63b8fd12-ecb5-42fe-b1ca-5d904f5fa670" file="kube-burner.go:90"

# preLoadImages : false 상태
time="2025-08-29 23:52:20" level=info msg="🔥 Starting kube-burner (1.17.3@917540ff45a89386bb25de45af9b96c9fc360e93) with UUID bda9f688-5538-4d24-ad05-783d3ace9031" file="job.go:91"
time="2025-08-29 23:52:20" level=warning msg="Measurement [none] is not supported" file="factory.go:101"
time="2025-08-29 23:52:20" level=debug msg="job.MaxWaitTimeout is zero in create-deployments, override by timeout: 4h0m0s" file="job.go:361"
time="2025-08-29 23:52:20" level=info msg="QPS: 1" file="job.go:371"
time="2025-08-29 23:52:20" level=info msg="Burst: 1" file="job.go:378"
time="2025-08-29 23:52:20" level=debug msg="Preparing create job: create-deployments" file="create.go:46"
time="2025-08-29 23:52:20" level=debug msg="Rendering template: s1-deployment.yaml" file="create.go:52"
time="2025-08-29 23:52:20" level=info msg="Job create-deployments: 1 iterations with 1 Deployment replicas" file="create.go:84"
time="2025-08-29 23:52:20" level=info msg="Triggering job: create-deployments" file="job.go:122"
time="2025-08-29 23:52:20" level=info msg="0/1 iterations completed" file="create.go:119"
time="2025-08-29 23:52:20" level=debug msg="Creating object replicas from iteration 0" file="create.go:122"
time="2025-08-29 23:52:21" level=debug msg="Created namespace: kube-burner-test-0" file="namespaces.go:55"
time="2025-08-29 23:52:21" level=debug msg="Created Deployment/deployment-0-1 in namespace kube-burner-test-0" file="create.go:288"
time="2025-08-29 23:52:21" level=info msg="Waiting up to 4h0m0s for actions to be completed" file="create.go:169"
time="2025-08-29 23:52:22" level=debug msg="Waiting for replicas from Deployment in ns kube-burner-test-0 to be ready" file="waiters.go:152"
time="2025-08-29 23:52:23" level=info msg="Actions in namespace kube-burner-test-0 completed" file="waiters.go:74"
time="2025-08-29 23:52:23" level=info msg="Job create-deployments took 3s" file="job.go:191"
time="2025-08-29 23:52:23" level=info msg="Finished execution with UUID: bda9f688-5538-4d24-ad05-783d3ace9031" file="job.go:264"
time="2025-08-29 23:52:23" level=info msg="👋 Exiting kube-burner bda9f688-5538-4d24-ad05-783d3ace9031" file="kube-burner.go:90"

두개의 설정값 적용 차이를 확인해보면, preload 부분의 생략 여부로 나눠져있다는 것을 알 수 있습니다.

waitWhenFinished: false 옵션은 테스트 시 필요한 리소스가 정상적으로 생성됐는지 확인하는 옵션입니다.

false로 지정을 하게되면 확인 여부를 생략하게됩니다.

# waitWhenFinished: false 옵션적용
time="2025-08-29 23:57:21" level=debug msg="Creating object replicas from iteration 0" file="create.go:122"
time="2025-08-29 23:57:23" level=debug msg="Created namespace: kube-burner-test-0" file="namespaces.go:55"
time="2025-08-29 23:57:23" level=debug msg="Created Deployment/deployment-0-1 in namespace kube-burner-test-0" file="create.go:288"
time="2025-08-29 23:57:23" level=info msg="Job create-deployments took 3s" file="job.go:191"
time="2025-08-29 23:57:23" level=info msg="Finished execution with UUID: 498e888a-6648-4bc2-8b07-2473f3dfc46d" file="job.go:264"
time="2025-08-29 23:57:23" level=info msg="👋 Exiting kube-burner 498e888a-6648-4bc2-8b07-2473f3dfc46d" file="kube-burner.go:90"

jobIterations: 10, qps: 1, burst:10 objects.replicas: 1 조건으로 테스트를 진행해보겠습니다.

로그를 확인 해보면 qps가 1이 때문에, 요청 시 한건으로만 보내 지만 최총 테스트 갯수가 burst의 숫자와 동일하기 때문에 빠르게 생성을 하는 것을 확인 할 수 있습니다.

time="2025-08-30 09:52:16" level=info msg="Triggering job: create-deployments" file="job.go:122"
time="2025-08-30 09:52:17" level=debug msg="Creating object replicas from iteration 0" file="create.go:122"
time="2025-08-30 09:52:17" level=debug msg="Created namespace: kube-burner-test-0" file="namespaces.go:55"
time="2025-08-30 09:52:17" level=info msg="1/10 iterations completed" file="create.go:119"
time="2025-08-30 09:52:17" level=debug msg="Creating object replicas from iteration 1" file="create.go:122"
time="2025-08-30 09:52:17" level=debug msg="Created Deployment/deployment-0-1 in namespace kube-burner-test-0" file="create.go:288"
time="2025-08-30 09:52:17" level=debug msg="Created namespace: kube-burner-test-1" file="namespaces.go:55"
time="2025-08-30 09:52:17" level=info msg="2/10 iterations completed" file="create.go:119"
time="2025-08-30 09:52:17" level=debug msg="Creating object replicas from iteration 2" file="create.go:122"
time="2025-08-30 09:52:17" level=debug msg="Created namespace: kube-burner-test-2" file="namespaces.go:55"
time="2025-08-30 09:52:17" level=info msg="3/10 iterations completed" file="create.go:119"
time="2025-08-30 09:52:17" level=debug msg="Creating object replicas from iteration 3" file="create.go:122"
time="2025-08-30 09:52:17" level=debug msg="Created Deployment/deployment-1-1 in namespace kube-burner-test-1" file="create.go:288"

jobIterations: 20, qps: 2, burst:20 objects.replicas: 2 조건으로 테스트를 진행해보겠습니다.

최총테스트가 40개 여서, burst값이 20이여서 최초 20개 생성까지는 빠르게 생성하지만 그 이후로는 qps의 값대로 생성하는 것을 확인할 수 있습니다.

time="2025-08-30 10:22:48" level=debug msg="Created Deployment/deployment-0-1 in namespace kube-burner-test-0" file="create.go:288"
time="2025-08-30 10:22:48" level=debug msg="Created Deployment/deployment-0-2 in namespace kube-burner-test-0" file="create.go:288"
....
time="2025-08-30 10:22:52" level=debug msg="Created Deployment/deployment-10-2 in namespace kube-burner-test-10" file="create.go:288"
time="2025-08-30 10:22:53" level=debug msg="Creating object replicas from iteration 11" file="create.go:122"
time="2025-08-30 10:22:53" level=debug msg="Created Deployment/deployment-10-1 in namespace kube-burner-test-10" file="create.go:288"
time="2025-08-30 10:22:53" level=debug msg="Created namespace: kube-burner-test-11" file="namespaces.go:55"
time="2025-08-30 10:22:53" level=debug msg="Created Deployment/deployment-11-2 in namespace kube-burner-test-11" file="create.go:288"
time="2025-08-30 10:22:54" level=info msg="12/20 iterations completed" file="create.go:119"
time="2025-08-30 10:22:54" level=debug msg="Creating object replicas from iteration 12" file="create.go:122"
time="2025-08-30 10:22:54" level=debug msg="Created Deployment/deployment-11-1 in namespace kube-burner-test-11" file="create.go:288"
time="2025-08-30 10:22:54" level=debug msg="Created namespace: kube-burner-test-12" file="namespaces.go:55"
time="2025-08-30 10:22:54" level=debug msg="Created Deployment/deployment-12-2 in namespace kube-burner-test-12" file="create.go:288"
time="2025-08-30 10:22:55" level=debug msg="Creating object replicas from iteration 13" file="create.go:122"
time="2025-08-30 10:22:55" level=debug msg="Created namespace: kube-burner-test-13" file="namespaces.go:55"
time="2025-08-30 10:22:55" level=debug msg="Created Deployment/deployment-12-1 in namespace kube-burner-test-12" file="create.go:288"

시나리오 2

이번에는 노드 1대에 최대 파드(150개) 배포하는 시나리오를 수행해보겠습니다

jobIterations: 100, qps: 300, burst: 300, objects.replicas: 1 값을 적용해서 테스트합니다.

node에 pod생성 갯수가 110으로 설정 되어있어, 다음과 같은 에러가 발생합니다.

kube-burner init -c s1-config.yaml --log-level debug


kubectl get pod -A | grep -v '1/1     Running'
# NAMESPACE             NAME                                                        READY   STATUS    RESTARTS      AGE
# kube-burner-test-94   deployment-94-1-6c676c4c6f-xgpfl                            0/1     Pending   0             87s
# kube-burner-test-95   deployment-95-1-c6bf8559c-9vnjp                             0/1     Pending   0             86s

kubectl describe pod -n kube-burner-test-99 | grep Events: -A5
# Events:
#   Type     Reason            Age   From               Message
#   ----     ------            ----  ----               -------
#   Warning  FailedScheduling  119s  default-scheduler  0/1 nodes are available: 1 Too many pods. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.

kubectl describe node

# Capacity:
#   cpu:                6
#   ephemeral-storage:  1055762868Ki
#   hugepages-1Gi:      0
#   hugepages-2Mi:      0
#   memory:             12249832Ki
#   pods:               110
# Allocatable:
#   cpu:                6
#   ephemeral-storage:  1055762868Ki
#   hugepages-1Gi:      0
#   hugepages-2Mi:      0
#   memory:             12249832Ki
#   pods:               110

Figure 2.2 Pod Number and nodes DashBoard

조치를 하기위해 pod 배포 갯수를 늘려보도록 하겠습니다.

docker exec -it myk8s-control-plane bash
apt update && apt install vim -y
vim /var/lib/kubelet/config.yaml
# maxPods: 150 추가
systemctl restart kubelet

kubectl describe node
# Capacity:
#   cpu:                6
#   ephemeral-storage:  1055762868Ki
#   hugepages-1Gi:      0
#   hugepages-2Mi:      0
#   memory:             12249832Ki
#   pods:               150
# Allocatable:
#   cpu:                6
#   ephemeral-storage:  1055762868Ki
#   hugepages-1Gi:      0
#   hugepages-2Mi:      0
#   memory:             12249832Ki
#   pods:               150

Grafana Dash Board를 통해 증가된 maxPod와 Pending 상태에있던 pod들이 정상적으로 배포된 것을 확인할 수 있습니다.

Figure 2.3 Pod Number and nodes DashBoard

# 리소스 제거
kube-burner init -c s1-config-delete.yaml --log-level debug

3. K8S Performance & Tuning

k8s 공식문서에 따르면 v1.34 버전 기준으로 기본적으로 다음과 같은 규모를 사용할 수 있다고 나와있습니다.

• 5000대 노드 이하
• 노드 당 110개 파드 이하
• 총 파드 수 150,000개 이하
• 총 컨테이너 수 300,000개 이하

시나리오 3

이번에는 kubernetes api 서버에 많은 부하를 부여하는 시나리오 입니다.

Deployment, Service, Secret, configmap 을 생성 하고, Deployment에 Label을 추가하며 리소스를 삭제하는 내용입니다.

cd examples/workloads/api-intensive

cat << EOF > api-intensive-100.yml
jobs:
  - name: api-intensive
    jobIterations: 100
    qps: 100
    burst: 100
    namespacedIterations: true
    namespace: api-intensive
    podWait: false
    cleanup: true
    waitWhenFinished: true
    preLoadImages: false # true
    objects:
      - objectTemplate: templates/deployment.yaml
        replicas: 1
      - objectTemplate: templates/configmap.yaml
        replicas: 1
      - objectTemplate: templates/secret.yaml
        replicas: 1
      - objectTemplate: templates/service.yaml
        replicas: 1

  - name: api-intensive-patch
    jobType: patch
    jobIterations: 10
    qps: 100
    burst: 100
    objects:
      - kind: Deployment
        objectTemplate: templates/deployment_patch_add_label.json
        labelSelector: {kube-burner-job: api-intensive}
        patchType: "application/json-patch+json"
        apiVersion: apps/v1
      - kind: Deployment
        objectTemplate: templates/deployment_patch_add_pod_2.yaml
        labelSelector: {kube-burner-job: api-intensive}
        patchType: "application/apply-patch+yaml"
        apiVersion: apps/v1
      - kind: Deployment
        objectTemplate: templates/deployment_patch_add_label.yaml
        labelSelector: {kube-burner-job: api-intensive}
        patchType: "application/strategic-merge-patch+json"
        apiVersion: apps/v1

  - name: api-intensive-remove
    qps: 500
    burst: 500
    jobType: delete
    waitForDeletion: true
    objects:
      - kind: Deployment
        labelSelector: {kube-burner-job: api-intensive}
        apiVersion: apps/v1

  - name: ensure-pods-removal
    qps: 100
    burst: 100
    jobType: delete
    waitForDeletion: true
    objects:
      - kind: Pod
        labelSelector: {kube-burner-job: api-intensive}

  - name: remove-services
    qps: 100
    burst: 100
    jobType: delete
    waitForDeletion: true
    objects:
      - kind: Service
        labelSelector: {kube-burner-job: api-intensive}

  - name: remove-configmaps-secrets
    qps: 100
    burst: 100
    jobType: delete
    objects:
      - kind: ConfigMap
        labelSelector: {kube-burner-job: api-intensive}
      - kind: Secret
        labelSelector: {kube-burner-job: api-intensive}

  - name: remove-namespace
    qps: 100
    burst: 100
    jobType: delete
    waitForDeletion: true
    objects:
      - kind: Namespace
        labelSelector: {kube-burner-job: api-intensive}
EOF

# 수행
kube-burner init -c api-intensive-100.yml --log-level debug

Figure 3.1 API Request DashBoard

Kubernetes API Performance Metrics

Figure 3.2 Kubernetes에서 일반적인 애플리케이션 워크로드의 배포

Kubernetes API가 최근 5분간 처리한 요청을 PromQL로 확인해보면 다음과 같습니다.

sum by(resource, code, verb) (rate(apiserver_request_total{resource=~".+"}[5m]))
or
sum by(resource, code, verb) (irate(apiserver_request_total{resource=~".+"}[5m]))

Figure 3.3 Prometheus에서 확인된 Metric

Prometheus에서 Kubernetes API서버가 Read Request Success Rate and Write Request Success Rate 정보도 다음과 같은 PromQL로 알아낼 수 있습니다.

# Read Request Success Rate : 읽기 요청(GET, LIST)에 대한 성공률 
sum(irate(apiserver_request_total{code=~"20.*",verb=~"GET|LIST"}[5m]))/sum(irate(apiserver_request_total{verb=~"GET|LIST"}[5m]))

# Write Request Success Rate : 쓰기 요청(GET, LIST, WATCH, CONNECT 제외)에 대한 성공률 
sum(irate(apiserver_request_total{code=~"20.*",verb!~"GET|LIST|WATCH|CONNECT"}[5m]))/sum(irate(apiserver_request_total{verb!~"GET|LIST|WATCH|CONNECT"}[5m]))

Figure 3.4 Prometheus에서 확인된 읽기, 쓰기 요청 Metric

4. Tuning

API Server

kube-apiserver의 flag값 max-requests-inflight(기본값 400), max-mutating-requests-inflight(기본값 200)으로 한 API서버에서 총 동시성 풀과 PriorityAndFairness를 이용하여 600개의 요청을 처리 할 수 있어, 요청량이 많은 경우 해당 flag값을 증가시켜 문제를 해결할 수 있습니다.

Figure 4.1 PriorityAndFairness

Core DNS

MultiSocket 기능을 사용하면 동일한 포트에서 청취할 서버의 수를 정의할 수 있습니다.

SO_REUSEPORT 소켓 옵션을 사용하면 동일한 주소와 포트에서 여러 청취 소켓을 열 수 있습니다. 이 경우 커널은 소켓 간에 들어오는 연결을 분산합니다.

이 옵션을 활성화하면 여러 서버를 시작할 수 있어 CPU 코어가 많은 환경에서 CoreDNS의 처리량이 증가합니다.

MultiSocket 기능을 사용하지 않는다면 CPU가 많아져도 QPS가 늘어나지 않습니다.