kubeadm

 

목차

 

모니터링 툴 실습을 위한 바닐라 쿠버네티스 클러스터 구성

모든 설치는 root 가 아닌 user(ubuntu) 로서 sudo 를 사용함

 

구성 : docker engine, containerd, calico, kubeadm, kubectl, kubelet

 

 

master1, worker1

config	master1, worker1
os	ubuntu 22.04 LTS
type	t3a.large
storage	16gib
SG	allow all traffic
etc	default

(sample app 을 빠르게 배포 후 테스트 하기 위해 large 선택, 쿠버네티스 최소 스펙은 2cpu, 2GB ram 이상)

 

user data

#!/bin/bash
apt-get upgrade -y 
apt-get dist-upgrade -y 
apt-get update -y
apt update && apt upgrade -y
apt-get install vim -y

 

1. 컨테이너 런타임 설치

node : master1, worker1

이전 버전 제거

sudo apt-get remove docker docker-engine docker.io containerd runc

sudo swapoff /swap.img
sudo sed -i -e '/swap.img/d' /etc/fstab

 

k8s 1.24(2022/05) 이후 k8s 에서 기본적으로 내부 연결을 지원해주던 dockershim 이 제거되어

cri-docker 를 추가 설치하여 docker를 k8s 에 연결하는 작업이 필요

1.1 도커 설치

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo systemctl enable --now docker && sudo systemctl status docker --no-pager
sudo usermod -aG docker worker

sudo docker container ls

 

1.2 cri-docker 설치

VER=$(curl -s https://api.github.com/repos/Mirantis/cri-dockerd/releases/latest|grep tag_name | cut -d '"' -f 4|sed 's/v//g')
echo $VER
wget https://github.com/Mirantis/cri-dockerd/releases/download/v${VER}/cri-dockerd-${VER}.amd64.tgz
tar xvf cri-dockerd-${VER}.amd64.tgz
sudo mv cri-dockerd/cri-dockerd /usr/local/bin/

cri-docker --version

인스턴스 타입에 따라 url 경로 변경 필요

 

1.3 cri-docker.service

wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/master/packaging/systemd/cri-docker.service
wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/master/packaging/systemd/cri-docker.socket
sudo mv cri-docker.socket cri-docker.service /etc/systemd/system/
sudo sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service

sudo systemctl daemon-reload
sudo systemctl enable cri-docker.service
sudo systemctl enable --now cri-docker.socket

# cri-docker Active Check
sudo systemctl restart docker && sudo systemctl restart cri-docker
sudo systemctl status cri-docker.socket --no-pager

 

1.4 docker cgroup 변경

systemd 사용

sudo mkdir /etc/docker
cat <<EOF | sudo tee /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

sudo systemctl restart docker && sudo systemctl restart cri-docker
sudo docker info | grep Cgroup

 

1.5 kernel forwarding

# Kernel Forwarding 
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

sudo sysctl --system

 

2. k8s 설치

node : master1, worker1

2.1 설치 관련 다운로드

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update

 

2.2 k8s 설치

# k8s 설치
sudo apt-get install -y kubelet kubeadm kubectl

# 버전 확인하기
kubectl version --short

# 버전 고정하기
sudo apt-mark hold kubelet kubeadm kubectl

3. Control-plane 초기화

node : master1

3.1 kubeadm init

# Controller Node
sudo kubeadm config images pull --cri-socket unix:///run/cri-dockerd.sock

sudo kubeadm init --ignore-preflight-errors=all --pod-network-cidr=192.168.0.0/16 --apiserver-advertise-address=172.31.2.139 --cri-socket /var/run/cri-dockerd.sock
## --apiserver-advertise-address=203.248.23.161 -> Controller 서버 IP.

sudo kubeadm join 172.31.2.139:6443 --token hg6dtl.zby5lq1z3zcn70ah \
        --discovery-token-ca-cert-hash sha256:da0298416cbb5fe069541e4e08673d3cb5240db5037c2c2024164285f19b49ca --cri-socket /var/run/cri-dockerd.sock

# kubeadm 을 root 처럼 사용하기 위한 추가 설정
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

 

확인

# Token 기억이 안 날 경우
kubeadm token create --print-join-command

# 서비스 확인 ( Ready , Running ) 
kubectl get nodes -o wide
kubectl get pod -A

# Worker Node에서 kubectl 을 사용하려면 admin.conf 를 복사한다
sudo scp /etc/kubernetes/admin.conf worker@node02:/home/worker/admin.conf
sudo scp /etc/kubernetes/admin.conf worker@node03:/home/worker/admin.conf

 

4. Node Join

node : worker1

# Worker Node Join. --cri-docker sorket 수동 지정해서 가입해야 한다. 
init 시 나온 토큰값을 참고한다. 기억이 안 날 경우 커맨드로 확인한다

sudo kubeadm join --token <token> <controlplane-host>:<controlplane-port> --ignore-preflight-errors=all --cri-socket unix:///var/run/cri-dockerd.sock

워커노드가 Ready 상태로 바뀔때까지 5분정도 소요된다고 함

5. CNI 설치

node : master1

5.1 calico 설치

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.1/manifests/tigera-operator.yaml

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.1/manifests/custom-resources.yaml

5.2 calicoctl 설치

cd /usr/local/bin
sudo curl -L https://github.com/projectcalico/calico/releases/latest/download/calicoctl-linux-amd64 -o calicoctl
sudo chmod +x ./calicoctl

## CNI Type Check 
calicoctl get ippool -o wide

## Block Check
sudo calicoctl ipam show --show-blocks

## BGP Protocol Check
sudo calicoctl node status

## Node Endpoint Check
calicoctl get workloadendpoint -A

6. Rejoin or Reset

node : any

#cleanup
sudo systemctl stop kubelet
sudo kubeadm reset -f --cri-socket /var/run/cri-dockerd.sock

sudo rm -rf ~/.kube
sudo rm -rf /root/.kube
sudo rm -rf /var/lib/etcd
sudo rm -rf /etc/kubernetes

 

node: master1

kubectl delete -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.1/manifests/tigera-operator.yaml
kubectl delete -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.1/manifests/custom-resources.yaml
sudo rm -rf /var/run/calico/
sudo rm -rf /var/lib/calico/
sudo rm -rf /etc/cni/net.d/
sudo rm -rf /var/lib/cni/
sudo rm -rf /opt/cni
sudo reboot

 

참고

https://docs.docker.com/engine/install/ubuntu/

Install Docker Engine on Ubuntu

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

Installing kubeadm

https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/

Configuring a cgroup driver

https://kubernetes.io/ko/docs/tasks/administer-cluster/network-policy-provider/weave-network-policy/

네트워크 폴리시로 위브넷(Weave Net) 사용하기

https://tech.hostway.co.kr/2022/08/30/1374/

Kubernetes 1.24 + cri-docker Installation ( kubeadm ) - HOSTWAY Tech Blog