Cilium Study [1기] 5주차 ClusterMesh
안녕하세요 이번 게시물에서는 ClusterMesh 라는 주제에 대한 정보를 전달 드리고자 합니다.
1. 실습 환경 구성
Kind를 이용해서 실습 환경을 구성합니다.
kind create cluster --name west --image kindest/node:v1.33.2 --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
extraPortMappings:
- containerPort: 30000 # sample apps
hostPort: 30000
- containerPort: 30001 # hubble ui
hostPort: 30001
- role: worker
extraPortMappings:
- containerPort: 30002 # sample apps
hostPort: 30002
networking:
podSubnet: "10.0.0.0/16"
serviceSubnet: "10.2.0.0/16"
disableDefaultCNI: true
kubeProxyMode: none
EOF
docker exec -it west-control-plane sh -c 'apt update && apt install tree psmisc lsof wget net-tools dnsutils tcpdump ngrep iputils-ping git -y'
docker exec -it west-worker sh -c 'apt update && apt install tree psmisc lsof wget net-tools dnsutils tcpdump ngrep iputils-ping git -y'
kind create cluster --name east --image kindest/node:v1.33.2 --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
extraPortMappings:
- containerPort: 31000 # sample apps
hostPort: 31000
- containerPort: 31001 # hubble ui
hostPort: 31001
- role: worker
extraPortMappings:
- containerPort: 31002 # sample apps
hostPort: 31002
networking:
podSubnet: "10.1.0.0/16"
serviceSubnet: "10.3.0.0/16"
disableDefaultCNI: true
kubeProxyMode: none
EOF
docker exec -it east-control-plane sh -c 'apt update && apt install tree psmisc lsof wget net-tools dnsutils tcpdump ngrep iputils-ping git -y'
docker exec -it east-worker sh -c 'apt update && apt install tree psmisc lsof wget net-tools dnsutils tcpdump ngrep iputils-ping git -y'
# alias 설정
alias kwest='kubectl --context kind-west'
alias keast='kubectl --context kind-east'# cilium CLI 설치
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
# cilium cli 로 cilium cni 설치
cilium install --version 1.17.6 --set ipam.mode=kubernetes \
--set kubeProxyReplacement=true --set bpf.masquerade=true \
--set endpointHealthChecking.enabled=false --set healthChecking=false \
--set operator.replicas=1 --set debug.enabled=true \
--set routingMode=native --set autoDirectNodeRoutes=true --set ipv4NativeRoutingCIDR=10.0.0.0/16 \
--set ipMasqAgent.enabled=true --set ipMasqAgent.config.nonMasqueradeCIDRs='{10.1.0.0/16}' \
--set cluster.name=west --set cluster.id=1 \
--context kind-west
cilium install --version 1.17.6 --set ipam.mode=kubernetes \
--set kubeProxyReplacement=true --set bpf.masquerade=true \
--set endpointHealthChecking.enabled=false --set healthChecking=false \
--set operator.replicas=1 --set debug.enabled=true \
--set routingMode=native --set autoDirectNodeRoutes=true --set ipv4NativeRoutingCIDR=10.1.0.0/16 \
--set ipMasqAgent.enabled=true --set ipMasqAgent.config.nonMasqueradeCIDRs='{10.0.0.0/16}' \
--set cluster.name=east --set cluster.id=2 \
--context kind-east
2. Cluster Mesh 구성
Native Routing + 같은 네트워크 내에서 ClusterMesh 설정 시 자동으로 라우팅이 주입이 됩니다.
상대방의 k8s cluster와 통신을 하기 위해서는 Node별 PodCIDR에 대한 정보를 알고 있어야 하지만, 같은 네트워크 내에서 BGP를 사용하지 않고 Cluster Mesh를 사용하는 경우 자동으로 주입이 됩니다.
Cluster Mesh 연결
# East CA를 West CA와 동기화 작업 진행
keast get secret -n kube-system cilium-ca
keast delete secret -n kube-system cilium-ca
kubectl --context kind-west get secret -n kube-system cilium-ca -o yaml | \
kubectl --context kind-east create -f -
# 모니터링 : 신규 터미널 2개
cilium clustermesh status --context kind-west --wait
cilium clustermesh status --context kind-east --wait
# Enable Cluster Mesh : 간단한 실습 환경으로 NodePort 로 진행
cilium clustermesh enable --service-type NodePort --enable-kvstoremesh=false --context kind-west
cilium clustermesh enable --service-type NodePort --enable-kvstoremesh=false --context kind-east
kwest get svc,ep -n kube-system clustermesh-apiserver --context kind-west
# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
# service/clustermesh-apiserver NodePort 10.2.29.72 <none> 2379:32379/TCP 2m3s
# NAME ENDPOINTS AGE
# endpoints/clustermesh-apiserver 10.0.1.216:2379 2m3s
keast get svc,ep -n kube-system clustermesh-apiserver --context kind-east
# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
# service/clustermesh-apiserver NodePort 10.3.11.145 <none> 2379:32379/TCP 3m36s
# NAME ENDPOINTS AGE
# endpoints/clustermesh-apiserver 10.1.1.122:2379 3m36s
# 클러스터 간 연결
cilium clustermesh connect --context kind-west --destination-context kind-east
# ✨ Extracting access information of cluster west...
# 🔑 Extracting secrets from cluster west...
# ⚠️ Service type NodePort detected! Service may fail when nodes are removed from the cluster!
# ℹ️ Found ClusterMesh service IPs: [172.18.0.3]
# ✨ Extracting access information of cluster east...
# 🔑 Extracting secrets from cluster east...
# ⚠️ Service type NodePort detected! Service may fail when nodes are removed from the cluster!
# ℹ️ Found ClusterMesh service IPs: [172.18.0.5]
# ℹ️ Configuring Cilium in cluster kind-west to connect to cluster kind-east
# ℹ️ Configuring Cilium in cluster kind-east to connect to cluster kind-west
# ✅ Connected cluster kind-west <=> kind-east!
# 라우팅 정보 확인 : 클러스터간 PodCIDR 라우팅 주입 확인!
docker exec -it west-control-plane ip -c route
docker exec -it west-worker ip -c route
docker exec -it east-control-plane ip -c route
docker exec -it east-worker ip -c route
# 10.0.1.0/24 via 172.18.0.2 dev eth0 proto kernel
# 10.1.0.0/24 via 172.18.0.5 dev eth0 proto kernel
# 10.1.1.0/24 via 172.18.0.4 dev eth0 proto kernel
hubble 설치
helm upgrade cilium cilium/cilium --version 1.17.6 --namespace kube-system --reuse-values \
--set hubble.enabled=true --set hubble.relay.enabled=true --set hubble.ui.enabled=true \
--set hubble.ui.service.type=NodePort --set hubble.ui.service.nodePort=30001 --kube-context kind-west
kwest -n kube-system rollout restart ds/cilium
helm upgrade cilium cilium/cilium --version 1.17.6 --namespace kube-system --reuse-values \
--set hubble.enabled=true --set hubble.relay.enabled=true --set hubble.ui.enabled=true \
--set hubble.ui.service.type=NodePort --set hubble.ui.service.nodePort=31001 --kube-context kind-east
keast -n kube-system rollout restart ds/cilium
# hubble-ui 접속
wslview http://localhost:30001
wslview http://localhost:31001
West <-> East Pod 통신 확인
cat << EOF | kubectl apply --context kind-west -f -
apiVersion: v1
kind: Pod
metadata:
name: curl-pod
labels:
app: curl
spec:
containers:
- name: curl
image: nicolaka/netshoot
command: ["tail"]
args: ["-f", "/dev/null"]
terminationGracePeriodSeconds: 0
EOF
cat << EOF | kubectl apply --context kind-east -f -
apiVersion: v1
kind: Pod
metadata:
name: curl-pod
labels:
app: curl
spec:
containers:
- name: curl
image: nicolaka/netshoot
command: ["tail"]
args: ["-f", "/dev/null"]
terminationGracePeriodSeconds: 0
EOF
kwest get pod -owide && keast get pod -owide
# NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
# curl-pod 1/1 Running 0 116s 10.0.1.252 west-worker <none> <none>
# NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
# curl-pod 1/1 Running 0 115s 10.1.1.11 east-worker <none> <none>
# 통신
kubectl exec -it curl-pod --context kind-west -- ping 10.1.1.11
Load-balancing & Service Discovery
West, East Cluster에 샘플 Application 을 배포하여 확인해보겠습니다.
cat << EOF | kubectl apply --context kind-west -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: webpod
spec:
replicas: 2
selector:
matchLabels:
app: webpod
template:
metadata:
labels:
app: webpod
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- sample-app
topologyKey: "kubernetes.io/hostname"
containers:
- name: webpod
image: traefik/whoami
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: webpod
labels:
app: webpod
annotations:
service.cilium.io/global: "true"
spec:
selector:
app: webpod
ports:
- protocol: TCP
port: 80
targetPort: 80
type: ClusterIP
EOF
#
cat << EOF | kubectl apply --context kind-east -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: webpod
spec:
replicas: 2
selector:
matchLabels:
app: webpod
template:
metadata:
labels:
app: webpod
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- sample-app
topologyKey: "kubernetes.io/hostname"
containers:
- name: webpod
image: traefik/whoami
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: webpod
labels:
app: webpod
annotations:
service.cilium.io/global: "true"
spec:
selector:
app: webpod
ports:
- protocol: TCP
port: 80
targetPort: 80
type: ClusterIP
EOF
# West to East service list 확인
kwest exec -it -n kube-system ds/cilium -c cilium-agent -- cilium service list --clustermesh-affinity
# 13 10.2.157.165:80/TCP ClusterIP 1 => 10.0.1.31:80/TCP (active)
# 2 => 10.0.1.69:80/TCP (active)
# 3 => 10.1.1.241:80/TCP (active)
# 4 => 10.1.1.178:80/TCP (active)
# East to West service list 확인
keast exec -it -n kube-system ds/cilium -c cilium-agent -- cilium service list --clustermesh-affinity
# 13 10.3.83.188:80/TCP ClusterIP 1 => 10.0.1.69:80/TCP (active)
# 2 => 10.0.1.31:80/TCP (active)
# 3 => 10.1.1.241:80/TCP (active)
# 4 => 10.1.1.178:80/TCP (active)
Service Affinity
트래픽 인입 시 service.cilium.io/affinity=loca Annotation을 사용해서 인입된 트래픽을 Cluster(Local)로 우선 인입 시킬 수 있습니다.
인입된 Cluster에 Replica 가 0인 경우에는 Preferred 옵션이기 때문에, Replica가 존재하는 Cluster로 트래픽이 인입됩니다.
반대로 service.cilium.io/affinity=remote Annotaion을 이용해서 인입된 트래픽을 Non Local로 우선 인입 시킬 수 있습니다.
service.cilium.io/shared Annotation을 false로 지정을 하게 되면 통신이 차단이 되며, 클러스터 간 통신이 되지 않습니다.
# Session Affinity Local 설정
kwest annotate service webpod service.cilium.io/affinity=local --overwrite
keast annotate service webpod service.cilium.io/affinity=local --overwrite
kwest exec -it -n kube-system ds/cilium -c cilium-agent -- cilium service list --clustermesh-affinity
# 13 10.2.157.165:80/TCP ClusterIP 1 => 10.0.1.31:80/TCP (active) (preferred)
# 2 => 10.0.1.69:80/TCP (active) (preferred)
keast exec -it -n kube-system ds/cilium -c cilium-agent -- cilium service list --clustermesh-affinity
# 3 => 10.1.1.241:80/TCP (active) (preferred)
# 4 => 10.1.1.178:80/TCP (active) (preferred)'DevOps > Study' 카테고리의 다른 글
| Cilium Study [1기] 6주차 2/2 ServiceMesh (0) | 2025.08.23 |
|---|---|
| Cilium Study [1기] 6주차 1/2 ServiceMesh (0) | 2025.08.22 |
| Cilium Study [1기] 5주차 BGP Control Plane (3) | 2025.08.16 |
| Cilium Study [1기] 4주차 Service LB-IPAM, Cilium L2 Announement (1) | 2025.08.09 |
| Cilium Study [1기] 4주차 Native Routing Mode, Encapsulation Node (3) | 2025.08.09 |