리피아드의 잡동사니

모든 커맨드는 루트로 작업

1. 호스트 추가
   #ceph ssh-key 배포(클러스터 구성한 노드에서 각 노드들에게)
   

   $ ssh-copy-id -f -i /etc/ceph/ceph.pub root@"host ip"

   
   #클러스터에 호스트 추가
   

   $ ceph orch host add "호스트명" "호스트 ip" "tag(생략 가능)"

   
   #클러스터에 호스트 추가 결과 확인
   

   $ ceph orch host ls

   
   결과 예시:
   

   $ ceph orch host ls
   HOST    ADDR          LABELS  STATUS
   ceph-1  172.30.0.193  _admin
   ceph-2  172.30.3.170  OSD
   ceph-3  172.30.1.200  OSD
   ceph-4  172.30.2.96   OSD

   
   
   
2. OSD 추가
   

   $ ceph orch daemon add osd "호스트명":"드라이브명"

   
   #확인
   

   $ ceph -s
   $ ceph orch device ls

   
   결과 예시:
   

   $ ceph -s
     cluster:
       id:     af39f080-af03-11ec-9050-fa163e37df68
       health: HEALTH_OK
     services:
       mon: 4 daemons, quorum ceph-1,ceph-2,ceph-3,ceph-4 (age 2d)
       mgr: ceph-1.ppytcz(active, since 25h), standbys: ceph-2.dedeoe
       mds: 1/1 daemons up, 3 standby
       osd: 4 osds: 4 up (since 2d), 4 in (since 2d)
   
   
   $ ceph orch device ls
   HOST    PATH      TYPE  DEVICE ID              SIZE  AVAILABLE  REJECT REASONS
   ceph-1  /dev/vdb  hdd   0e8c4f4b-ca72-48c3-8  1073G             Insufficient space (<10 extents) on vgs, LVM detected, locked
   ceph-2  /dev/vdb  hdd   382bb362-d64e-4041-9  1073G             Insufficient space (<10 extents) on vgs, LVM detected, locked
   ceph-3  /dev/vdb  hdd   3e5cec61-0c30-4d61-a  1073G             Insufficient space (<10 extents) on vgs, LVM detected, locked
   ceph-4  /dev/vdb  hdd   c63d1d1f-6a74-4c3a-9  1073G             Insufficient space (<10 extents) on vgs, LVM detected, locked

   
   
   
3. 결과 확인
   

   $ ceph orch status
   $ ceph orch ps

   
   결과 예시:
   

   $ ceph orch status
   Backend: cephadm
   Available: Yes
   Paused: No
   
   $ ceph orch ps
   NAME                          HOST    PORTS        STATUS         REFRESHED  AGE  MEM USE  MEM LIM  VERSION  IMAGE ID      CONTAINER ID
   alertmanager.ceph-1           ceph-1  *:9093,9094  running (2d)      9m ago   2d    12.7M        -  0.20.0   0881eb8f169f  169d759e6ebb
   crash.ceph-1                  ceph-1               running (2d)      9m ago   2d    7436k        -  16.2.7   c92aec2cd894  cf8d4667fc0e
   crash.ceph-2                  ceph-2               running (2d)      9m ago   2d    7304k        -  16.2.7   c92aec2cd894  220f004b583c
   crash.ceph-3                  ceph-3               running (2d)     49s ago   2d    10.9M        -  16.2.7   c92aec2cd894  efa886f81ef9
   crash.ceph-4                  ceph-4               running (2d)     49s ago   2d    7256k        -  16.2.7   c92aec2cd894  276eaf7238a4
   grafana.ceph-1                ceph-1  *:3000       running (2d)      9m ago   2d    35.8M        -  6.7.4    557c83e11646  2684c2c21a43
   mgr.ceph-1.ppytcz             ceph-1  *:9283       running (1h)      9m ago   2d     506M        -  16.2.7   c92aec2cd894  654bc9d468db
   mgr.ceph-2.dedeoe             ceph-2  *:8443,9283  running (2d)      9m ago   2d     380M        -  16.2.7   c92aec2cd894  730a9e27d05f
   mon.ceph-1                    ceph-1               running (2d)      9m ago   2d     881M    2048M  16.2.7   c92aec2cd894  c2f75db158da
   mon.ceph-2                    ceph-2               running (2d)      9m ago   2d     888M    2048M  16.2.7   c92aec2cd894  05f31cf6a2d3
   mon.ceph-3                    ceph-3               running (2d)     49s ago   2d     883M    2048M  16.2.7   c92aec2cd894  d31c6d4115c4
   mon.ceph-4                    ceph-4               running (2d)     49s ago   2d     891M    2048M  16.2.7   c92aec2cd894  8bade1f43df6
   node-exporter.ceph-1          ceph-1  *:9100       running (2d)      9m ago   2d    11.8M        -  0.18.1   e5a616e4b9cf  3debf7ae68eb
   node-exporter.ceph-2          ceph-2  *:9100       running (2d)      9m ago   2d    11.8M        -  0.18.1   e5a616e4b9cf  7fe3fbc71085
   node-exporter.ceph-3          ceph-3  *:9100       running (2d)     49s ago   2d    12.0M        -  0.18.1   e5a616e4b9cf  37e0338834bb
   node-exporter.ceph-4          ceph-4  *:9100       running (2d)     49s ago   2d    11.0M        -  0.18.1   e5a616e4b9cf  4ba70a679bf2
   osd.0                         ceph-2               running (2d)      9m ago   2d     212M    4096M  16.2.7   c92aec2cd894  20bf30027ca5
   osd.1                         ceph-3               running (2d)     49s ago   2d     226M    4096M  16.2.7   c92aec2cd894  36607cbb6458
   osd.2                         ceph-4               running (2d)     49s ago   2d     222M    4096M  16.2.7   c92aec2cd894  c90cf1973629
   osd.3                         ceph-1               running (2d)      9m ago   2d     216M    4096M  16.2.7   c92aec2cd894  0fc6bbac67eb
   prometheus.ceph-1             ceph-1  *:9095       running (2d)      9m ago   2d    36.5M        -  2.18.1   de242295e225  71d62fcef51e

   
   
4. Dashboard 접속
   Ceph mgr 노드 ip에서 8. Ceph 클러스터 구성의 접속정보로 로그인 후 비밀번호 변경하면 Dashboard로 관리 가능
   

   

   
   
   
   참고 : https://yjwang.tistory.com/119
   

모든 커맨드는 루트로 작업

1. 필요 패키지 설치(모든 서버)
   

   $ apt update; apt install -y ansible

   
   
2. 첫번째 서버에 Cephadm-ansible git repo 복제
   

   $ git clone https://github.com/YoungjuWang/ubuntu-cephadm-ansible.git

3. ubuntu-cephadm-ansible 폴더로 들어가서 vars.yml 수정
   

   $ cd ubuntu-cephadm-ansible/
   vi vars.yml
   ---
   container_engine: "docker"         수정
   ---

4. ansible inventory 수정
   

   $ vi ceph.inventory
   ---
    bootstrap ansible_connection=local
   mon2 ansible_host="host2 ip"
   mon3 ansible_host=""host3 ip"
   mon4 ansible_host=""host4 ip"
   (host 개수만큼 추가)
   
   all:vars] ansible_ssh_common_args='-o StrictHostKeyChecking=no'
   ---

5. ssh-key를 생성하고 copy
   

   $ ssh-keygen -N "" -f ~/.ssh/id_rsa
   $ ssh-copy-id "host ip"
   (host 전부 등록)

6. ssh connection 확인
   

   $ ansible -i ceph.inventory -m ping all

   
   
   결과 예시:
   

   $ ansible -i ceph.inventory -m ping all
   ceph-1 | SUCCESS => {
       "ansible_facts": {
           "discovered_interpreter_python": "/usr/bin/python3"
       },
       "changed": false,
       "ping": "pong"
   }
   ceph-2 | SUCCESS => {
       "ansible_facts": {
           "discovered_interpreter_python": "/usr/bin/python3"
       },
       "changed": false,
       "ping": "pong"
   }
   ceph-3 | SUCCESS => {
       "ansible_facts": {
           "discovered_interpreter_python": "/usr/bin/python3"
       },
       "changed": false,
       "ping": "pong"
   }
   ceph-4 | SUCCESS => {
        "ansible_facts": {
            "discovered_interpreter_python": "/usr/bin/python3" 
        }, 
        "changed": false, 
        "ping": "pong" 
   }

   
   
7. Ceph 패키지 설치
   

   $ ansible-playbook -i ceph.inventory preflight.yml

   
   
8. Ceph 클러스터 구성
   

   $ cephadm bootstrap --mon-ip "첫번째 host ip"

   
   # 아래 항목이 구성됨
   - Create a monitor and manager daemon for the new cluster on the local host.
   - Generate a new SSH key for the Ceph cluster and add it to the root user’s /root/.ssh/authorized_keys file.
   - Write a minimal configuration file to /etc/ceph/ceph.conf. This file is needed to communicate with the new cluster.
   - Write a copy of the client.admin administrative (privileged!) secret key to /etc/ceph/ceph.client.admin.keyring.
   - Write a copy of the public key to /etc/ceph/ceph.pub.
   
   결과 예시:

$ cephadm bootstrap --mon-ip "첫번째 host ip"

Fetching dashboard port number...
Ceph Dashboard is now available at:

             URL: https://ceph-1:8443/
            User: admin
        Password: kxnjyyu66x

Enabling client.admin keyring and conf on hosts with "admin" label
You can access the Ceph CLI with:

        sudo /usr/sbin/cephadm shell --fsid af39f080-af03-11ec-9050-fa163e37df68 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring

Please consider enabling telemetry to help improve Ceph:

        ceph telemetry on

For more information see:

        https://docs.ceph.com/docs/pacific/mgr/telemetry/

참고 : https://yjwang.tistory.com/119

$ sudo vi /etc/docker/daemon.json
---
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "1"
  },
  "storage-driver": "overlay2"
}
---
$ sudo systemctl daemon-reload
$ sudo systemctl restart docker

18.06.2ce에서는 로그 크기 이슈가 없었으나.. 19.03ce에서는 발생해서 쓰고 있는 옵션입니다.

cgroupdriver의 경우 기본값인 cgroupfs를 사용하게 되면 램누수가 발생할 수 있다고 하네요.

(저는 겪어본적 없습니다.)

필요시 sudo usermod -aG docker $USER

출처 : https://kubernetes.io/ko/docs/setup/production-environment/container-runtimes/

1. 테스트환경

   4Core 32GB RAM, Ubuntu 18.04

2. Topology

- 사용 포트 

  1) docker-proxy 자동 지정(mysql57 mysql80 mysql57-exporter mysql80-exporter)

  2) Prometheus : 9090

     * mysql의 3306포트는 host와 매핑하지 않음

3. 절차

0) docker 설정

$ vi /etc/docker/daemon.json

---

{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "1"
  },
  "storage-driver": "overlay2",
  "metrics-addr" : "[hostIP]:9323",
  "experimental" : true

}

---

$ sudo systemctl daemon-reload

$ sudo systemctl restart docker

1) mysql57 배포

$ sudo docker network create db_network

$ docker run -d --name mysql57 --publish 3306 --network db_network --restart unless-stopped \
--env MYSQL_ROOT_PASSWORD=mypassword --volume mysql57-datadir:/var/lib/mysql \
mysql:5.7

2) mysql80 배포

$ docker run -d --name mysql80 --publish 3306 --network db_network --restart unless-stopped \

--env MYSQL_ROOT_PASSWORD=mypassword --volume mysql80-datadir:/var/lib/mysql \

--default-authentication-plugin=mysql_native_password \

mysql:8

3) mysql에 exporter 계정생성

$ docker exec -it mysql80 mysql -uroot -pmypassword

mysql> CREATE USER 'exporter'@'%' IDENTIFIED BY 'exporterpassword' WITH MAX_USER_CONNECTIONS 3;
mysql> GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'%';

$ docker exec -it mysql57 mysql -uroot -pmypassword

mysql> CREATE USER 'exporter'@'%' IDENTIFIED BY 'exporterpassword' WITH MAX_USER_CONNECTIONS 3;
mysql> GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'%';

4) Node_exporter 생성

$ docker run -d --name mysql80-exporter --publish 9104 --network db_network --restart always \
--env DATA_SOURCE_NAME="exporter:exporterpassword@(mysql80:3306)/" \
--collect.info_schema.processlist --collect.info_schema.innodb_metrics --collect.info_schema.tablestats \
--collect.info_schema.tables --collect.info_schema.userstats --collect.engine_innodb_status \
prom/mysqld-exporter:latest

$ docker run -d --name mysql57-exporter --publish 9104 --network db_network --restart always \
-e DATA_SOURCE_NAME="exporter:exporterpassword@(mysql57:3306)/" \
--collect.info_schema.processlist --collect.info_schema.innodb_metrics --collect.info_schema.tablestats \
--collect.info_schema.tables --collect.info_schema.userstats --collect.engine_innodb_status \
prom/mysqld-exporter:latest

5) 프로메테우스 생성

$ vim ~/prometheus.yml

---

global:
  scrape_interval:     5s
  scrape_timeout:      3s
  evaluation_interval: 5s

# Our alerting rule files
rule_files:
  - "alert.rules"

# Scrape endpoints
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'mysql'
    static_configs:
      - targets: ['mysql57-exporter:9104','mysql80-exporter:9104']

  - job_name: 'docker'
    static_configs:
      - targets: ['hostIP:9323']

---

$ docker run -d --name prometheus-server --publish 9090:9090 --network db_network --restart unless-stopped \

--mount type=volume,src=prometheus-data,target=/prometheus --mount type=bind,src=$(pwd)/prometheus.yml,target=/etc/prometheus/prometheus.yml \

prom/prometheus

6) 확인

hostip:9090 접속 후 status > target에서 연결 확인

참고 :

https://severalnines.com/database-blog/how-monitor-mysql-containers-prometheus-deployment-standalone-and-swarm-part-one

재택 근무중 PC에서 kubectl을 사용해야하는 경우가 발생해 설정법을 정리합니다.

1. kubectl 다운 후 압축 해제(필요하다면 HELMv3도)

2. kubectl.exe 가 있는 폴더에 Path설정

3. 이후 powershell에서 kubectl 테스트