로컬에서 minikube로 k8s 맛보기

학습 배경

• 이미 회사 개발 환경은 k8s로 돌아가고 있지만 사용만 하고 자세하게는 몰라서 로컬에서 간단한 모델을 구현해보기로함
• 로컬에서 url path로 로드밸런싱 하는 ingress 구조

미리 준비할 것

1. linux 운영체제 환경

현재 회사에서 Azure 사용 중이라 azure vm 을 사용했다

• Ubuntu Server 24.04 LTS x64
• 머신은 Standard B2s 사용 : 2 vcpu, 4GiB 메모리
  • minikube 사용을 위해 2 pcu, 2GB 를 선택했다 (minikube 공식문서)

2. 간단한 Spring Boot application

1@RestController
2@Slf4j
3public class KubeController {
4
5    @GetMapping("/domain")
6    public String domain(){
7        log.info("===== domain service =====");
8        log.info("ABCDEFGHIJKLMNOPQRSTUVWXYZ");
9        log.info("ABCDEFGHIJKLMNOPQRSTUVWXYZ");
10        log.info("==========================");
11        return "domain";
12    }
13
14    @GetMapping("/non-domain")
15    public String nonDomain(){
16        log.info("===== non domain =====");
17        log.info("        BLANK");
18        log.info("======================");
19        return "non domain";
20    }
21}

bootJar 실행해서 jar 파일 생성해둠

3. Docker image 준비

• Dockerfile

1FROM openjdk:17-jdk-alpine
2
3WORKDIR /app
4
5COPY target/spring-app.jar /app/spring-app.jar
6# target/ 디렉토리에 있는 Spring Boot JAR 파일을 Docker 컨테이너의 /app 디렉토리에 복사
7
8EXPOSE 8080
9
10ENTRYPOINT ["java", "-jar", "spring-app.jar"]

• image build

1C:\> docker build -t spring-app .
2# 현재 디렉토리 (.)에 있는 Dockerfile로 spring-app 이미지 생성
3# docker images로 생성된 이미지 확인 가능

• docker hub로 업로드

1docker login -u username
2
3docker tag spring-app <username>/spring-app
4docker push <username>/spring-app

필요한 프로그램 설치

1. docker

1# 필요한 패키지 설치
2azureuser@m3rri17:~$ sudo apt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common
3# docker 공식 gpg 키 추가
4azureuser@m3rri17:~$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
5# docker 공식 apt 저장소 추가
6azureuser@m3rri17:~$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
7# docker 설치
8azureuser@m3rri17:~$ sudo apt-get install docker-ce docker-ce-cli containerd.io

minikube driver를 docker로 사용하기 위해서는 아래 명령어를 추가로 실행해줘야 한다

1azureuser@m3rri17:~$ sudo usermod -aG docker $USER && newgrp docker

⚠위 명령을 실행하지 않고 minikube를 시작하려고 할 때 발생하는 에러

1azureuser@m3rri17:~$ minikube start --driver=docker --memory 200
2😄  minikube v1.34.0 on Ubuntu 24.04
3✨  Using the docker driver based on user configuration
4
5💣  Exiting due to PROVIDER_DOCKER_NEWGRP: "docker version --format <no value>-<no value>:<no value>" exit status 1: permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.47/version": dial unix /var/run/docker.sock: connect: permission denied
6💡  Suggestion: Add your user to the 'docker' group: 'sudo usermod -aG docker $USER && newgrp docker'
7📘  Documentation: https://docs.docker.com/engine/install/linux-postinstall/

💡 Suggestion 부분의 명령어를 실행

2. minikube

1azureuser@m3rri17:~$ curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
2azureuser@m3rri17:~$ sudo install minikube-linux-amd64 /usr/local/bin/minikube && rm minikube-linux-amd64

출처 : minikube 공식문서

3. kubectl

1# download the lastes release
2azureuser@m3rri17:~$ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl

1# install kubectl
2azureuser@m3rri17:~$ sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
3# version check
4azureuser@m3rri17:~$ kubectl version --client

출처 : kubernetes 공식문서

4. ingress-controller

1azureuser@m3rri17:~$ minikube addons enable ingress
2💡  ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
3You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
4    ▪ Using image registry.k8s.io/ingress-nginx/controller:v1.11.2
5    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.3
6    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.3
7🔎  Verifying ingress addon...
8🌟  The 'ingress' addon is enabled
9
10azureuser@m3rri17:~$ kubectl get all -n ingress-nginx
11NAME                                           READY   STATUS      RESTARTS   AGE
12pod/ingress-nginx-admission-create-p7mcn       0/1     Completed   0          89s
13pod/ingress-nginx-admission-patch-p5bf4        0/1     Completed   1          89s
14pod/ingress-nginx-controller-bc57996ff-p9488   1/1     Running     0          89s
15
16NAME                                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
17service/ingress-nginx-controller             NodePort    10.107.25.163    <none>        80:31473/TCP,443:30846/TCP   89s
18service/ingress-nginx-controller-admission   ClusterIP   10.101.131.201   <none>        443/TCP                      89s
19
20NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
21deployment.apps/ingress-nginx-controller   1/1     1            1           89s
22
23NAME                                                 DESIRED   CURRENT   READY   AGE
24replicaset.apps/ingress-nginx-controller-bc57996ff   1         1         1       89s
25
26NAME                                       STATUS     COMPLETIONS   DURATION   AGE
27job.batch/ingress-nginx-admission-create   Complete   1/1           8s         89s
28job.batch/ingress-nginx-admission-patch    Complete   1/1           9s         89s

k8s 구동

1. minikube 시작

1azureuser@m3rri17:~$ minikube start --driver=docker --memory 2048
2# --driver=docker : driver 겹쳐서 에러발생하는 경우 사용
3# --memory 2048 : ram 에 여유가 없어서 따로 지정함
4
5azureuser@m3rri17:~$ minikube status
6# 아래와 같이 나오면 정상
7minikube
8type: Control Plane
9host: Running
10kubelet: Running
11apiserver: Running
12kubeconfig: Configured
13
14azureuser@m3rri17:~$ minikube ip
15# ip 확인하여 이후 Ingress 파일 작성 단계에서 사용

2. Deployment

• file 생성
  • spring-deployment-1.yaml

    1apiVersion: apps/v1
    2kind: Deployment
    3metadata:
    4    name: spring-boot-deployment-1 # deployment의 이름
    5spec:
    6    replicas: 1 # 생성할 pod 수
    7    selector:
    8        matchLabels:
    9            app: spring-boot-app-1 # app 이름 service 파일의 spec.selector.app 에서 동일한 이름 사용하면 매핑됨
    10    template:
    11        metadata:
    12            labels:
    13                app: spring-boot-app-1
    14        spec:
    15            containers:
    16                - name: spring-boot-container-1
    17                  image: <dockerhub-username>/spring-app # Docker Hub에서 가져올 이미지
    18                  ports:
    19                      - containerPort: 8080 # Spring Boot 기본 포트

  • spring-deployment-2.yaml

    1apiVersion: apps/v1
    2kind: Deployment
    3metadata:
    4    name: spring-boot-app-2
    5spec:
    6    replicas: 1
    7    selector:
    8        matchLabels:
    9            app: spring-boot-app-2
    10    template:
    11        metadata:
    12            labels:
    13                app: spring-boot-app-2
    14        spec:
    15            containers:
    16                - name: spring-boot-container-2
    17                  image: <dockerhub-username>/spring-app
    18                  ports:
    19                      - containerPort: 8080

• 클러스터에 적용

1azureuser@m3rri17:~$ kubectl apply -f spring-deployment-1.yaml
2azureuser@m3rri17:~$ kubectl apply -f spring-deployment-2.yaml
3azureuser@m3rri17:~$ kubectl get pods # pod name 확인
4NAME                                        READY   STATUS    RESTARTS   AGE
5spring-boot-deployment-1-5b4b9d89f8-2sjcq   1/1     Running   0          21s
6spring-boot-deployment-2-cd877558-jqw4x     1/1     Running   0          7s
7
8azureuser@m3rri17:~$ kubectl logs <pod-name> # -f는 실시간 로그 확인 결과는 아래 다시 정리

3. Service

• file 생성
  • spring-service-1.yaml

    1apiVersion: v1
    2kind: Service
    3metadata:
    4    name: spring-boot-service-1 # service 이름. ingress 파일의 spec.rules.http.paths.backend.service.name에 동일하게 사용하여 매핑
    5spec:
    6    selector:
    7        app: spring-boot-app-1
    8    ports:
    9        - protocol: TCP
    10          port: 80
    11          targetPort: 8080 # 80포트로 들어온 요청을 8080 포트로 연결

  • spring-service-2.yaml

    1apiVersion: v1
    2kind: Service
    3metadata:
    4    name: spring-boot-service-2
    5spec:
    6    selector:
    7        app: spring-boot-app-2
    8    ports:
    9        - protocol: TCP
    10          port: 80
    11          targetPort: 8080

• 클러스터에 적용

1azureuser@m3rri17:~$ kubectl apply -f spring-service-1.yaml
2azureuser@m3rri17:~$ kubectl apply -f spring-service-2.yaml
3azureuser@m3rri17:~$ kubectl get svc
4NAME                    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
5kubernetes              ClusterIP   10.96.0.1      <none>        443/TCP        6m14s
6spring-boot-service-1   NodePort    10.98.26.57    <none>        80:32309/TCP   7s
7spring-boot-service-2   NodePort    10.99.23.101   <none>        80:32206/TCP   3s

4. Ingress

• file 생성 ingress.yaml

  1apiVersion: networking.k8s.io/v1
  2kind: Ingress
  3metadata:
  4    name: spring-boot-ingress
  5    annotations:
  6        nginx.ingress.kubernetes.io/rewrite-target: /
  7spec:
  8    rules:
  9        - host: "<minikube-ip>.nip.io" # 또는 도메인 이름
  10          http:
  11              paths:
  12                  - path: /domain
  13                    pathType: Prefix
  14                    backend:
  15                        service:
  16                            name: spring-boot-service-1
  17                            port:
  18                                number: 80
  19                  - path: /non-domain
  20                    pathType: Prefix
  21                    backend:
  22                        service:
  23                            name: spring-boot-service-2
  24                            port:
  25                                number: 80

• 클러스터에 적용

1azureuser@m3rri17:~$ kubectl apply -f ingress.yaml
2azureuser@m3rri17:~$ kubectl get all -n ingress-nginx
3NAME                                           READY   STATUS      RESTARTS   AGE
4pod/ingress-nginx-admission-create-p7mcn       0/1     Completed   0          9m51s
5pod/ingress-nginx-admission-patch-p5bf4        0/1     Completed   1          9m51s
6pod/ingress-nginx-controller-bc57996ff-p9488   1/1     Running     0          9m51s
7
8NAME                                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
9service/ingress-nginx-controller             NodePort    10.107.25.163    <none>        80:31473/TCP,443:30846/TCP   9m51s
10service/ingress-nginx-controller-admission   ClusterIP   10.101.131.201   <none>        443/TCP                      9m51s
11
12NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
13deployment.apps/ingress-nginx-controller   1/1     1            1           9m51s
14
15NAME                                                 DESIRED   CURRENT   READY   AGE
16replicaset.apps/ingress-nginx-controller-bc57996ff   1         1         1       9m51s
17
18NAME                                       STATUS     COMPLETIONS   DURATION   AGE
19job.batch/ingress-nginx-admission-create   Complete   1/1           8s         9m51s
20job.batch/ingress-nginx-admission-patch    Complete   1/1           9s         9m51s

5. 테스트 및 로그 확인

• /domain

1azureuser@m3rri17:~$ curl http://192.168.49.2.nip.io:80/domain
2domain
3
4azureuser@m3rri17:~$ kubectl logs spring-boot-deployment-1-5b4b9d89f8-2sjcq
5
6  .   ____          _            __ _ _
7 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
8( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
9 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
10  '  |____| .__|_| |_|_| |_\__, | / / / /
11 =========|_|==============|___/=/_/_/_/
12
13 :: Spring Boot ::                (v3.3.4)
14
152024-10-22T07:15:39.011Z  INFO 1 --- [kubetarget] [           main] c.e.kubetarget.KubetargetApplication     : Starting KubetargetApplication v0.0.1-SNAPSHOT using Java 17-ea with PID 1 (/app/spring-app.jar started by root in /app)
162024-10-22T07:15:39.016Z  INFO 1 --- [kubetarget] [           main] c.e.kubetarget.KubetargetApplication     : No active profile set, falling back to 1 default profile: "default"
172024-10-22T07:15:41.258Z  INFO 1 --- [kubetarget] [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port 8080 (http)
182024-10-22T07:15:41.297Z  INFO 1 --- [kubetarget] [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
192024-10-22T07:15:41.297Z  INFO 1 --- [kubetarget] [           main] o.apache.catalina.core.StandardEngine    : Starting Servlet engine: [Apache Tomcat/10.1.30]
202024-10-22T07:15:41.608Z  INFO 1 --- [kubetarget] [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
212024-10-22T07:15:41.614Z  INFO 1 --- [kubetarget] [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2529 ms
222024-10-22T07:15:43.224Z  INFO 1 --- [kubetarget] [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port 8080 (http) with context path '/'
232024-10-22T07:15:43.262Z  INFO 1 --- [kubetarget] [           main] c.e.kubetarget.KubetargetApplication     : Started KubetargetApplication in 4.891 seconds (process running for 5.76)
242024-10-22T07:23:43.510Z  INFO 1 --- [kubetarget] [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
252024-10-22T07:23:43.510Z  INFO 1 --- [kubetarget] [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
262024-10-22T07:23:43.512Z  INFO 1 --- [kubetarget] [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 2 ms
272024-10-22T07:25:43.823Z  INFO 1 --- [kubetarget] [nio-8080-exec-5] c.e.k.d.controller.DomainController      : ===== domain service =====
282024-10-22T07:25:43.823Z  INFO 1 --- [kubetarget] [nio-8080-exec-5] c.e.k.d.controller.DomainController      : ABCDEFGHIJKLMNOPQRSTUVWXYZ
292024-10-22T07:25:43.823Z  INFO 1 --- [kubetarget] [nio-8080-exec-5] c.e.k.d.controller.DomainController      : ABCDEFGHIJKLMNOPQRSTUVWXYZ
302024-10-22T07:25:43.823Z  INFO 1 --- [kubetarget] [nio-8080-exec-5] c.e.k.d.controller.DomainController      : ==========================

• /non-domain

1azureuser@m3rri17:~$ curl http://192.168.49.2.nip.io:80/non-domain
2non domain
3
4azureuser@m3rri17:~$ kubectl logs spring-boot-deployment-2-cd877558-jqw4x
5
6  .   ____          _            __ _ _
7 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
8( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
9 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
10  '  |____| .__|_| |_|_| |_\__, | / / / /
11 =========|_|==============|___/=/_/_/_/
12
13 :: Spring Boot ::                (v3.3.4)
14
152024-10-22T07:15:42.639Z  INFO 1 --- [kubetarget] [           main] c.e.kubetarget.KubetargetApplication     : Starting KubetargetApplication v0.0.1-SNAPSHOT using Java 17-ea with PID 1 (/app/spring-app.jar started by root in /app)
162024-10-22T07:15:42.657Z  INFO 1 --- [kubetarget] [           main] c.e.kubetarget.KubetargetApplication     : No active profile set, falling back to 1 default profile: "default"
172024-10-22T07:15:44.533Z  INFO 1 --- [kubetarget] [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port 8080 (http)
182024-10-22T07:15:44.551Z  INFO 1 --- [kubetarget] [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
192024-10-22T07:15:44.552Z  INFO 1 --- [kubetarget] [           main] o.apache.catalina.core.StandardEngine    : Starting Servlet engine: [Apache Tomcat/10.1.30]
202024-10-22T07:15:44.697Z  INFO 1 --- [kubetarget] [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
212024-10-22T07:15:44.699Z  INFO 1 --- [kubetarget] [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1821 ms
222024-10-22T07:15:45.653Z  INFO 1 --- [kubetarget] [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port 8080 (http) with context path '/'
232024-10-22T07:15:45.706Z  INFO 1 --- [kubetarget] [           main] c.e.kubetarget.KubetargetApplication     : Started KubetargetApplication in 4.273 seconds (process running for 5.877)
242024-10-22T07:25:51.778Z  INFO 1 --- [kubetarget] [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
252024-10-22T07:25:51.778Z  INFO 1 --- [kubetarget] [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
262024-10-22T07:25:51.780Z  INFO 1 --- [kubetarget] [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 1 ms
272024-10-22T07:25:51.822Z  INFO 1 --- [kubetarget] [nio-8080-exec-1] c.e.k.d.controller.DomainController      : ===== non domain =====
282024-10-22T07:25:51.822Z  INFO 1 --- [kubetarget] [nio-8080-exec-1] c.e.k.d.controller.DomainController      :         BLANK
292024-10-22T07:25:51.822Z  INFO 1 --- [kubetarget] [nio-8080-exec-1] c.e.k.d.controller.DomainController      : ======================

path에 따라 deployment1과 deployment2로 요청이 잘 나뉘어 들어갔다