sst:xtrabackup(galera) 사용 시 노드가 클러스터에 가입하는 문제

sst:xtrabackup(galera) 사용 시 노드가 클러스터에 가입하는 문제

노드가 클러스터에 가입하고 있는 것 같으며, 그 후 장애가 발생합니다.rsync와 xtrabackup을 모두 시도했지만 상태 전송 중에 실패합니다.제가 보기엔 정말 간단한 걸 놓치고 있는 것 같아서 정확히 짚어낼 수가 없어요.어떤 도움이라도 주시면 감사하겠습니다.

노드에 대한 자세한 정보

마스터 - 10.XXX.XX.161 node 1 - 10 。XXX.XX.160

설치된 패키지:MariaDB 호환 MariaDB-common MariaDB-devel MariaDB-devel MariaDB-client MariaDB-test MariaDB-Galera-server(v5.5.29-1) galera(v23.2.4-1.rhel6) per-cona-xtrackup(v2.1.6-r706r)

노드 1의 설정

[mysqld]
wsrep_cluster_address = gcomm://10.XXX.XXX.161
wsrep_provider = /usr/lib64/galera/libgalera_smm.so
wsrep_provider_options = gcache.size=4G; gcache.page_size=1G
wsrep_cluster_name = galera_cluster
default_storage_engine = InnoDB
innodb_autoinc_lock_mode = 2
innodb_locks_unsafe_for_binlog = 1
wsrep_sst_method = xtrabackup
wsrep_sst_auth = root:rootpassword
wsrep_node_name=1

마스터 설정

[mysqld]
wsrep_cluster_address = gcomm://
wsrep_provider = /usr/lib64/galera/libgalera_smm.so
wsrep_provider_options = gcache.size=4G; gcache.page_size=1G
wsrep_cluster_name = galera_cluster
default_storage_engine = InnoDB
innodb_autoinc_lock_mode = 2
innodb_locks_unsafe_for_binlog = 1
wsrep_sst_method = rsync
wsrep_slave_threads = 4
wsrep_sst_auth = root:rootpassword
wsrep_node_name = 2

node1 로그 파일

131203 16:09:03 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
131203 16:09:03 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.f2EedjRjbQ
131203 16:09:08 mysqld_safe WSREP: Recovered position 359350ee-5c63-11e3-0800-6673d15135cd:2188
131203 16:09:08 [Note] WSREP: wsrep_start_position var submitted: '359350ee-5c63-11e3-0800-6673d15135cd:2188'
131203 16:09:08 [Note] WSREP: Read nil XID from storage engines, skipping position init
131203 16:09:08 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so'
131203 16:09:08 [Note] WSREP: wsrep_load(): Galera 23.2.4(r147) by Codership Oy <info@codership.com]]> loaded succesfully.
131203 16:09:08 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
131203 16:09:08 [Note] WSREP: Reusing existing '/var/lib/mysql//galera.cache'.
131203 16:09:08 [Note] WSREP: Passing config to GCS: base_host = 10.XXX.XXX.160; base_port = 4567; cert.log_conflicts = no; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 1G; gcache.size = 4G; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
131203 16:09:08 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
131203 16:09:08 [Note] WSREP: wsrep_sst_grab()
131203 16:09:08 [Note] WSREP: Start replication
131203 16:09:08 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
131203 16:09:08 [Note] WSREP: protonet asio version 0
131203 16:09:08 [Note] WSREP: backend: asio
131203 16:09:08 [Note] WSREP: GMCast version 0
131203 16:09:08 [Note] WSREP: (8814b4ba-5c67-11e3-0800-91035d554a96, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
131203 16:09:08 [Note] WSREP: (8814b4ba-5c67-11e3-0800-91035d554a96, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
131203 16:09:08 [Note] WSREP: EVS version 0
131203 16:09:08 [Note] WSREP: PC version 0
131203 16:09:08 [Note] WSREP: gcomm: connecting to group 'galera_cluster', peer '10.XXX.XXX.161:'
131203 16:09:09 [Note] WSREP: declaring 7a9a87e8-5c67-11e3-0800-8cb6cba8f65a stable
131203 16:09:09 [Note] WSREP: Node 7a9a87e8-5c67-11e3-0800-8cb6cba8f65a state prim
131203 16:09:09 [Note] WSREP: view(view_id(PRIM,7a9a87e8-5c67-11e3-0800-8cb6cba8f65a,2) memb {
     7a9a87e8-5c67-11e3-0800-8cb6cba8f65a,
     8814b4ba-5c67-11e3-0800-91035d554a96,
} joined {
} left {
} partitioned {
})
131203 16:09:09 [Note] WSREP: gcomm: connected
131203 16:09:09 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
131203 16:09:09 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
131203 16:09:09 [Note] WSREP: Opened channel 'galera_cluster'
131203 16:09:09 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
131203 16:09:09 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
131203 16:09:09 [Note] WSREP: Waiting for SST to complete.
131203 16:09:09 [Note] WSREP: STATE EXCHANGE: sent state msg: 8861cdd5-5c67-11e3-0800-cc70fcc5f515
131203 16:09:09 [Note] WSREP: STATE EXCHANGE: got state msg: 8861cdd5-5c67-11e3-0800-cc70fcc5f515 from 0 (2)
131203 16:09:09 [Note] WSREP: STATE EXCHANGE: got state msg: 8861cdd5-5c67-11e3-0800-cc70fcc5f515 from 1 (1)
131203 16:09:09 [Note] WSREP: Quorum results:
     version    = 2,
     component  = PRIMARY,
     conf_id    = 1,
     members    = 1/2 (joined/total),
     act_id     = 2521,
     last_appl. = -1,
     protocols  = 0/4/2 (gcs/repl/appl),
     group UUID = 359350ee-5c63-11e3-0800-6673d15135cd
131203 16:09:09 [Note] WSREP: Flow-control interval: [23, 23]
131203 16:09:09 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 2521)
131203 16:09:09 [Note] WSREP: State transfer required:
     Group state: 359350ee-5c63-11e3-0800-6673d15135cd:2521
     Local state: 00000000-0000-0000-0000-000000000000:-1
131203 16:09:09 [Note] WSREP: New cluster view: global state: 359350ee-5c63-11e3-0800-6673d15135cd:2521, view# 2: Primary, number of nodes: 2, my index: 1, protocol version 2
131203 16:09:09 [Warning] WSREP: Gap in state sequence. Need state transfer.
131203 16:09:11 [Note] WSREP: Running: 'wsrep_sst_xtrabackup --role 'joiner' --address '10.XXX.XXX.160' --auth 'root:rootpassword' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '13175''
131203 16:09:11 [Note] WSREP: Prepared SST request: xtrabackup|10.162.143.160:4444/xtrabackup_sst
131203 16:09:11 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
131203 16:09:11 [Note] WSREP: Assign initial position for certification: 2521, protocol version: 2
131203 16:09:11 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (359350ee-5c63-11e3-0800-6673d15135cd): 1 (Operation not permitted)
      at galera/src/replicator_str.cpp:prepare_for_IST():442. IST will be unavailable.
131203 16:09:11 [Note] WSREP: Node 1 (1) requested state transfer from '*any*'. Selected 0 (2)(SYNCED) as donor.
131203 16:09:11 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 2525)
131203 16:09:11 [Note] WSREP: Requesting state transfer: success, donor: 0
tar: dbexport/db.opt: Cannot open: Permission denied
tar: Exiting with failure status due to previous errors
131203 16:10:22 [Note] WSREP: 0 (2): State transfer to 1 (1) complete.
131203 16:10:22 [Note] WSREP: Member 0 (2) synced with group.
WSREP_SST: [ERROR] Error while getting st data from donor node:  0, 2 (20131203 16:10:22.379)
131203 16:10:22 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup --role 'joiner' --address '10.XXX.XXX.160' --auth 'root:rootpassword' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '13175': 32 (Broken pipe)
131203 16:10:22 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
131203 16:10:22 [ERROR] WSREP: SST failed: 32 (Broken pipe)
131203 16:10:22 [ERROR] Aborting
131203 16:10:24 [Note] WSREP: Closing send monitor...
131203 16:10:24 [Note] WSREP: Closed send monitor.
131203 16:10:24 [Note] WSREP: gcomm: terminating thread
131203 16:10:24 [Note] WSREP: gcomm: joining thread
131203 16:10:24 [Note] WSREP: gcomm: closing backend
131203 16:10:25 [Note] WSREP: view(view_id(NON_PRIM,7a9a87e8-5c67-11e3-0800-8cb6cba8f65a,2) memb {
     8814b4ba-5c67-11e3-0800-91035d554a96,
} joined {
} left {
} partitioned {
     7a9a87e8-5c67-11e3-0800-8cb6cba8f65a,
})
131203 16:10:25 [Note] WSREP: view((empty))
131203 16:10:25 [Note] WSREP: gcomm: closed
131203 16:10:25 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
131203 16:10:25 [Note] WSREP: Flow-control interval: [16, 16]
131203 16:10:25 [Note] WSREP: Received NON-PRIMARY.
131203 16:10:25 [Note] WSREP: Shifting JOINER -> OPEN (TO: 2607)
131203 16:10:25 [Note] WSREP: Received self-leave message.
131203 16:10:25 [Note] WSREP: Flow-control interval: [0, 0]
131203 16:10:25 [Note] WSREP: Received SELF-LEAVE. Closing connection.
131203 16:10:25 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 2607)
131203 16:10:25 [Note] WSREP: RECV thread exiting 0: Success
131203 16:10:25 [Note] WSREP: recv_thread() joined.
131203 16:10:25 [Note] WSREP: Closing slave action queue.
131203 16:10:25 [Note] WSREP: Service disconnected.
131203 16:10:25 [Note] WSREP: rollbacker thread exiting
131203 16:10:26 [Note] WSREP: Some threads may fail to exit.
131203 16:10:26 [Note] /usr/sbin/mysqld: Shutdown complete
Error in my_thread_global_end(): 2 threads didn't exit
131203 16:10:31 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

마스터 로그 파일

131203 16:08:47 [Warning] Recovery from master pos 103358630 and file mysql-bin.000131.
131203 16:08:47 [Note] Event Scheduler: Loaded 0 events
131203 16:08:47 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
131203 16:08:47 [Note] WSREP: Assign initial position for certification: 2497, protocol version: 2
131203 16:08:47 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.29-MariaDB-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server, wsrep_23.7.3.rXXXX
131203 16:08:47 [Note] WSREP: Synchronized with group, ready for connections
131203 16:08:47 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
131203 16:09:09 [Note] WSREP: declaring 8814b4ba-5c67-11e3-0800-91035d554a96 stable
131203 16:09:09 [Note] WSREP: Node 7a9a87e8-5c67-11e3-0800-8cb6cba8f65a state prim
131203 16:09:09 [Note] WSREP: view(view_id(PRIM,7a9a87e8-5c67-11e3-0800-8cb6cba8f65a,2) memb {
     7a9a87e8-5c67-11e3-0800-8cb6cba8f65a,
     8814b4ba-5c67-11e3-0800-91035d554a96,
} joined {
} left {
} partitioned {
})
131203 16:09:09 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
131203 16:09:09 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 8861cdd5-5c67-11e3-0800-cc70fcc5f515
131203 16:09:09 [Note] WSREP: STATE EXCHANGE: sent state msg: 8861cdd5-5c67-11e3-0800-cc70fcc5f515
131203 16:09:09 [Note] WSREP: STATE EXCHANGE: got state msg: 8861cdd5-5c67-11e3-0800-cc70fcc5f515 from 0 (2)
131203 16:09:09 [Note] WSREP: STATE EXCHANGE: got state msg: 8861cdd5-5c67-11e3-0800-cc70fcc5f515 from 1 (1)
131203 16:09:09 [Note] WSREP: Quorum results:
     version    = 2,
     component  = PRIMARY,
     conf_id    = 1,
     members    = 1/2 (joined/total),
     act_id     = 2521,
     last_appl. = 2517,
     protocols  = 0/4/2 (gcs/repl/appl),
     group UUID = 359350ee-5c63-11e3-0800-6673d15135cd
131203 16:09:09 [Note] WSREP: Flow-control interval: [23, 23]
131203 16:09:09 [Note] WSREP: New cluster view: global state: 359350ee-5c63-11e3-0800-6673d15135cd:2521, view# 2: Primary, number of nodes: 2, my index: 0, protocol version 2
131203 16:09:09 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
131203 16:09:09 [Note] WSREP: Assign initial position for certification: 2521, protocol version: 2
131203 16:09:11 [Note] WSREP: Node 1 (1) requested state transfer from '*any*'. Selected 0 (2)(SYNCED) as donor.
131203 16:09:11 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 2525)
131203 16:09:11 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
131203 16:09:11 [Note] WSREP: Running: 'wsrep_sst_xtrabackup --role 'donor' --address '10.XXX.XXX.160:4444/xtrabackup_sst' --auth 'root:rootpassword' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --gtid '359350ee-5c63-11e3-0800-6673d15135cd:2525''
131203 16:09:11 [Note] WSREP: sst_donor_thread signaled with 0
131203 16:10:20 [Note] WSREP: Provider paused at 359350ee-5c63-11e3-0800-6673d15135cd:2604
131203 16:10:22 [Note] WSREP: Provider resumed.
131203 16:10:22 [Note] WSREP: 0 (2): State transfer to 1 (1) complete.
131203 16:10:22 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 2606)
131203 16:10:22 [Note] WSREP: Member 0 (2) synced with group.
131203 16:10:22 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 2606)
131203 16:10:22 [Note] WSREP: Synchronized with group, ready for connections
131203 16:10:22 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
131203 16:10:25 [Note] WSREP: Node 7a9a87e8-5c67-11e3-0800-8cb6cba8f65a state prim
131203 16:10:25 [Note] WSREP: view(view_id(PRIM,7a9a87e8-5c67-11e3-0800-8cb6cba8f65a,3) memb {
     7a9a87e8-5c67-11e3-0800-8cb6cba8f65a,
} joined {
} left {
} partitioned {
     8814b4ba-5c67-11e3-0800-91035d554a96,
})
131203 16:10:25 [Note] WSREP: forgetting 8814b4ba-5c67-11e3-0800-91035d554a96 (tcp://10.XXX.XXX.160:4567)
131203 16:10:25 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
131203 16:10:25 [Note] WSREP: STATE_EXCHANGE: sent state UUID: b5dda52e-5c67-11e3-0800-4b2360dd84f9
131203 16:10:25 [Note] WSREP: STATE EXCHANGE: sent state msg: b5dda52e-5c67-11e3-0800-4b2360dd84f9
131203 16:10:25 [Note] WSREP: STATE EXCHANGE: got state msg: b5dda52e-5c67-11e3-0800-4b2360dd84f9 from 0 (2)
131203 16:10:25 [Note] WSREP: Quorum results:
     version    = 2,
     component  = PRIMARY,
     conf_id    = 2,
     members    = 1/1 (joined/total),
     act_id     = 2607,
     last_appl. = 2597,
     protocols  = 0/4/2 (gcs/repl/appl),
     group UUID = 359350ee-5c63-11e3-0800-6673d15135cd
131203 16:10:25 [Note] WSREP: Flow-control interval: [16, 16]
131203 16:10:25 [Note] WSREP: New cluster view: global state: 359350ee-5c63-11e3-0800-6673d15135cd:2607, view# 3: Primary, number of nodes: 1, my index: 0, protocol version 2
131203 16:10:25 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
131203 16:10:25 [Note] WSREP: Assign initial position for certification: 2607, protocol version: 2
131203 16:10:30 [Note] WSREP:  cleaning up 8814b4ba-5c67-11e3-0800-91035d554a96 (tcp://10..XXX.XXX.160:4567)

문제는 데이터베이스 백업 디렉토리가 있다는 것이었습니다.dbexportMariaDB의 데이터 디렉토리에 (아마도)/var/lib/mysql/SST 를 실행할 때, 프로바이더는 데이터 디렉토리를 스캔 해 송신할 파일을 검색합니다.이 디렉토리는 데이터 디렉토리의 디렉토리가 목적이기 때문에 데이터베이스용이라고 가정합니다.백업 디렉토리를 삭제하면 문제가 해결.베스트 프랙티스로서는, 에서는 아무것도 변경하지 말아 주세요./var/lib/; 프로그램들은 보통 그들의 데이터 파일을 그곳에 보관하고 그것들을 만지는 것은 이와 같은 문제를 일으킬 수 있다.

주요 문제가 해결된 후 로그에 다음과 같은 새로운 메시지가 표시되었습니다.

[Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (359350ee-5c63-11e3-0800-6673d15135cd): 1 (Operation not permitted) at galera/src/replicator_str.cpp:prepare_for_IST():442. IST will be unavailable.

이 메시지는 정상입니다.노드가 갤러라 클러스터에 가입하면 전체 SST(State Snapshot Transfer) 대신 IST(Incremental State Transfer)를 수행하려고 합니다.노드가 이전에 클러스터의 일부였고 떠날 때의 상태와 클러스터의 현재 상태의 차이가 충분히 작으면 노드의 현재 상태와 클러스터 상태 간의 차이만 전송하는 IST를 사용할 수 있습니다.이것은 모든 데이터를 전송하는 것보다 훨씬 빠릅니다.노드가 이전에 클러스터의 일부였지만 오래 전에 떠난 경우 SST를 수행해야 합니다.이 경우 노드의 상태 UUID에 가입하는 것은00000000-0000-0000-0000-000000000000이는 기본적으로 클러스터가 새로워졌다는 것을 의미합니다.MariaDB/galera 클러스터를 실행하고 있는데 IST를 사용할 수 없을 때마다 이 메시지가 거슬립니다.경고가 아니라 다른 말로 바꾸면 좋을 텐데.왜 그랬는지 모르겠다Operation not permitted저 안에 있지만 걱정할 것 없어요

또한 분할된 뇌 상태를 방지하기 위해 홀수 개수의 노드를 실행하는 것이 좋습니다.가능하면 클러스터에 다른 MariaDB 서버를 추가하거나, 그렇지 않으면 garbd를 실행해야 합니다. garbd는 데이터베이스 서버가 아닌 클러스터 내의 노드로 동작합니다.이를 통해 다른 데이터베이스 서버를 보유할 필요 없이 홀수 개수의 노드를 보유할 수 있습니다.

이 경우 프라이머리 클러스터를 세컨더리 클러스터로 교체하여 문제를 해결합니다.db1 이전

[mysqld]
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
query_cache_size=0
query_cache_type=0
bind-address=0.0.0.0

# Galera Provider Configuration
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_name="name"
wsrep_cluster_address="gcomm://37.x.x.104,37.x.x.117"
wsrep_sst_method=rsync
wsrep_node_address="37.x.x.104"
wsrep_node_name="db1"

pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
log_error = /var/log/mysql/error.log
expire_logs_days = 10
max_binlog_size = 100M

db2의

[mysqld]
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
query_cache_size=0
query_cache_type=0
bind-address=0.0.0.0

# Galera Provider Configuration
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_name="name"
wsrep_cluster_address="gcomm://37.x.x.104,37.x.x.117"
wsrep_sst_method=rsync
wsrep_node_address="37.x.x.104"
wsrep_node_name="db1"

pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
log_error = /var/log/mysql/error.log
expire_logs_days = 10
max_binlog_size = 100M

나는 변한다

wsrep_cluster_address="gcomm://37.x.x.104,37.x.x.117"

로.

wsrep_cluster_address="gcomm://37.x.x.117,37.x.x.104"

그리고.

wsrep_node_address="37.x.x.**104**" to **117**

클러스터가 시작되었습니다!

언급URL : https://stackoverflow.com/questions/20363540/problems-with-node-joining-the-cluster-when-using-sstxtrabackup-galera