Second pod start hangs on Running wsrep_sst_rsync

[solsson]

With a fresh microk8s 1.14.4 cluster kubectl apply -k . gets mariadb-0 to Ready state, but mariadb-1 blocks during start.

2019-07-27 14:25:40 139726685889856 [Note] mysqld (mysqld 10.2.25-MariaDB-1:10.2.25+maria~bionic) starting as process 1 ...
2019-07-27 14:25:40 139726685889856 [Note] WSREP: Read nil XID from storage engines, skipping position init
2019-07-27 14:25:40 139726685889856 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
2019-07-27 14:25:40 139726685889856 [Note] WSREP: wsrep_load(): Galera 25.3.26(r3857) by Codership Oy <info@codership.com> loaded successfully.
2019-07-27 14:25:40 139726685889856 [Note] WSREP: CRC-32C: using hardware acceleration.
2019-07-27 14:25:40 139726685889856 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1
2019-07-27 14:25:40 139726685889856 [Note] WSREP: Passing config to GCS: base_dir = /data/db/; base_host = 10.1.1.49; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /data/db/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /data/db//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false;
2019-07-27 14:25:40 139726685889856 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> 00000000-0000-0000-0000-000000000000:-1
2019-07-27 14:25:40 139726685889856 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
2019-07-27 14:25:40 139726685889856 [Note] WSREP: wsrep_sst_grab()
2019-07-27 14:25:40 139726685889856 [Note] WSREP: Start replication
2019-07-27 14:25:40 139726685889856 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
2019-07-27 14:25:40 139726685889856 [Note] WSREP: protonet asio version 0
2019-07-27 14:25:40 139726685889856 [Note] WSREP: Using CRC-32C for message checksums.
2019-07-27 14:25:40 139726685889856 [Note] WSREP: backend: asio
2019-07-27 14:25:40 139726685889856 [Note] WSREP: gcomm thread scheduling priority set to other:0 
2019-07-27 14:25:40 139726685889856 [Note] WSREP: restore pc from disk successfully
2019-07-27 14:25:40 139726685889856 [Note] WSREP: GMCast version 0
2019-07-27 14:25:40 139726685889856 [Warning] WSREP: Failed to resolve tcp://mariadb-1.mariadb:4567
2019-07-27 14:25:40 139726685889856 [Warning] WSREP: Failed to resolve tcp://mariadb-2.mariadb:4567
2019-07-27 14:25:40 139726685889856 [Note] WSREP: (5559eb88, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
2019-07-27 14:25:40 139726685889856 [Note] WSREP: (5559eb88, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
2019-07-27 14:25:40 139726685889856 [Note] WSREP: EVS version 0
2019-07-27 14:25:40 139726685889856 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer 'mariadb-0.mariadb:,mariadb-1.mariadb:,mariadb-2.mariadb:'
2019-07-27 14:25:40 139726685889856 [Note] WSREP: (5559eb88, 'tcp://0.0.0.0:4567') connection established to 4188e6e8 tcp://10.1.1.48:4567
2019-07-27 14:25:40 139726685889856 [Note] WSREP: (5559eb88, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 
2019-07-27 14:25:41 139726685889856 [Note] WSREP: gcomm: connected
2019-07-27 14:25:41 139726685889856 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
2019-07-27 14:25:41 139726685889856 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
2019-07-27 14:25:41 139726685889856 [Note] WSREP: Opened channel 'my_wsrep_cluster'
2019-07-27 14:25:41 139726685889856 [Note] WSREP: Waiting for SST to complete.
2019-07-27 14:25:43 139726519949056 [Warning] WSREP: no nodes coming from prim view, prim not possible
2019-07-27 14:25:43 139726519949056 [Note] WSREP: view(view_id(NON_PRIM,5559eb88,4) memb {
	5559eb88,0
} joined {
} left {
} partitioned {
})
2019-07-27 14:25:43 139726505051904 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2019-07-27 14:25:43 139726505051904 [Note] WSREP: Flow-control interval: [16, 16]
2019-07-27 14:25:43 139726505051904 [Note] WSREP: Trying to continue unpaused monitor
2019-07-27 14:25:43 139726505051904 [Note] WSREP: Received NON-PRIMARY.
2019-07-27 14:25:43 139726684378880 [Note] WSREP: New cluster view: global state: :-1, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version -1
2019-07-27 14:25:43 139726684378880 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2019-07-27 14:25:43 139726519949056 [Note] WSREP: (5559eb88, 'tcp://0.0.0.0:4567') turning message relay requesting off
2019-07-27 14:25:44 139726519949056 [Note] WSREP: declaring 4188e6e8 at tcp://10.1.1.48:4567 stable
2019-07-27 14:25:44 139726519949056 [Note] WSREP: re-bootstrapping prim from partitioned components
2019-07-27 14:25:44 139726519949056 [Note] WSREP: view(view_id(PRIM,4188e6e8,5) memb {
	4188e6e8,0
	5559eb88,0
} joined {
} left {
} partitioned {
})
2019-07-27 14:25:44 139726519949056 [Note] WSREP: save pc into disk
2019-07-27 14:25:44 139726505051904 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
2019-07-27 14:25:44 139726505051904 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
2019-07-27 14:25:44 139726519949056 [Note] WSREP: clear restored view
2019-07-27 14:25:44 139726505051904 [Note] WSREP: STATE EXCHANGE: sent state msg: 6b41ad2f-b07a-11e9-b313-aefa479322f6
2019-07-27 14:25:44 139726505051904 [Note] WSREP: STATE EXCHANGE: got state msg: 6b41ad2f-b07a-11e9-b313-aefa479322f6 from 0 (mariadb-0)
2019-07-27 14:25:44 139726505051904 [Note] WSREP: STATE EXCHANGE: got state msg: 6b41ad2f-b07a-11e9-b313-aefa479322f6 from 1 (mariadb-1)
2019-07-27 14:25:44 139726505051904 [Warning] WSREP: Quorum: No node with complete state:

	Version      : 4
	Flags        : 0x3
	Protocols    : 0 / 9 / 3
	State        : NON-PRIMARY
	Desync count : 0
	Prim state   : SYNCED
	Prim UUID    : 55b40e71-b07a-11e9-8369-fb3b92b43c93
	Prim  seqno  : 2
	First seqno  : -1
	Last  seqno  : 4
	Prim JOINED  : 1
	State UUID   : 6b41ad2f-b07a-11e9-b313-aefa479322f6
	Group UUID   : 3f1e1add-b07a-11e9-b142-86a198db1bc6
	Name         : 'mariadb-0'
	Incoming addr: '10.1.1.48:3306'

	Version      : 4
	Flags        : 00
	Protocols    : 0 / 9 / 3
	State        : NON-PRIMARY
	Desync count : 0
	Prim state   : NON-PRIMARY
	Prim UUID    : 00000000-0000-0000-0000-000000000000
	Prim  seqno  : -1
	First seqno  : -1
	Last  seqno  : -1
	Prim JOINED  : 0
	State UUID   : 6b41ad2f-b07a-11e9-b313-aefa479322f6
	Group UUID   : 00000000-0000-0000-0000-000000000000
	Name         : 'mariadb-1'
	Incoming addr: '10.1.1.49:3306'

2019-07-27 14:25:44 139726505051904 [Note] WSREP: Full re-merge of primary 55b40e71-b07a-11e9-8369-fb3b92b43c93 found: 1 of 1.
2019-07-27 14:25:44 139726505051904 [Note] WSREP: Quorum results:
	version    = 4,
	component  = PRIMARY,
	conf_id    = 2,
	members    = 1/2 (joined/total),
	act_id     = 4,
	last_appl. = -1,
	protocols  = 0/9/3 (gcs/repl/appl),
	group UUID = 3f1e1add-b07a-11e9-b142-86a198db1bc6
2019-07-27 14:25:44 139726505051904 [Note] WSREP: Flow-control interval: [23, 23]
2019-07-27 14:25:44 139726505051904 [Note] WSREP: Trying to continue unpaused monitor
2019-07-27 14:25:44 139726505051904 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4)
2019-07-27 14:25:44 139726684378880 [Note] WSREP: State transfer required: 
	Group state: 3f1e1add-b07a-11e9-b142-86a198db1bc6:4
	Local state: 00000000-0000-0000-0000-000000000000:-1
2019-07-27 14:25:44 139726684378880 [Note] WSREP: New cluster view: global state: 3f1e1add-b07a-11e9-b142-86a198db1bc6:4, view# 3: Primary, number of nodes: 2, my index: 1, protocol version 3
2019-07-27 14:25:44 139726684378880 [Warning] WSREP: Gap in state sequence. Need state transfer.
2019-07-27 14:25:44 139726291986176 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '10.1.1.49' --datadir '/data/db/'   --parent '1'  ''  '''

Nothing happens after that last line. The state is minimal because all volumes were empty. The --address is the pod IP of mariadb-1.

[solsson]

No such issue on Minikube, both default (v1.15.0 + docker) and with --kubernetes-version=v1.14.4 --container-runtime=containerd which corresponds to my microk8s.

2019-07-27 14:49:43 140000855160576 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4)
2019-07-27 14:49:43 140001237112576 [Note] WSREP: State transfer required: 
	Group state: ad9fe8df-b07d-11e9-8e0f-aaec1cd4e104:4
	Local state: 00000000-0000-0000-0000-000000000000:-1
2019-07-27 14:49:43 140001237112576 [Note] WSREP: New cluster view: global state: ad9fe8df-b07d-11e9-8e0f-aaec1cd4e104:4, view# 2: Primary, number of nodes: 2, my index: 1, protocol version 3
2019-07-27 14:49:43 140001237112576 [Warning] WSREP: Gap in state sequence. Need state transfer.
2019-07-27 14:49:43 140000846767872 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '10.1.0.7' --datadir '/data/db/'   --parent '106'  ''  '''
2019-07-27 14:49:43 140001237112576 [Note] WSREP: Prepared SST request: rsync|10.1.0.7:4444/rsync_sst
2019-07-27 14:49:43 140001237112576 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2019-07-27 14:49:43 140001237112576 [Note] WSREP: REPL Protocols: 9 (4, 2)
2019-07-27 14:49:43 140001237112576 [Note] WSREP: Assign initial position for certification: 4, protocol version: 4
2019-07-27 14:49:43 140000901457664 [Note] WSREP: Service thread queue flushed.
2019-07-27 14:49:43 140001237112576 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (ad9fe8df-b07d-11e9-8e0f-aaec1cd4e104): 1 (Operation not permitted)
	 at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
2019-07-27 14:49:43 140000855160576 [Note] WSREP: Member 1.0 (mariadb-1) requested state transfer from '*any*'. Selected 0.0 (mariadb-0)(SYNCED) as donor.
2019-07-27 14:49:43 140000855160576 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 4)
2019-07-27 14:49:43 140001237112576 [Note] WSREP: Requesting state transfer: success, donor: 0
2019-07-27 14:49:43 140001237112576 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> ad9fe8df-b07d-11e9-8e0f-aaec1cd4e104:4
MySQL init process in progress...
2019-07-27 14:49:45 140000855160576 [Note] WSREP: 0.0 (mariadb-0): State transfer to 1.0 (mariadb-1) complete.
2019-07-27 14:49:45 140000855160576 [Note] WSREP: Member 0.0 (mariadb-0) synced with group.

The IP is again the pods' own.

[solsson]

If I edit the configmap to set wsrep-sst-method=mariabackup the outcome is the same but the log ends with:

2019-07-27 15:08:19 139821674493696 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4)
2019-07-27 15:08:19 139822055872256 [Note] WSREP: State transfer required: 
	Group state: 6c6be05c-b07f-11e9-9619-b2a920ba13e1:4
	Local state: 00000000-0000-0000-0000-000000000000:-1
2019-07-27 15:08:19 139822055872256 [Note] WSREP: New cluster view: global state: 6c6be05c-b07f-11e9-9619-b2a920ba13e1:4, view# 12: Primary, number of nodes: 2, my index: 0, protocol version 3
2019-07-27 15:08:19 139822055872256 [Warning] WSREP: Gap in state sequence. Need state transfer.
2019-07-27 15:08:19 139821666100992 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.1.1.53' --datadir '/data/db/'   --parent '106'  ''  '''
WSREP_SST: [INFO] Streaming with xbstream (20190727 15:08:19.412)
WSREP_SST: [INFO] Using socat as streamer (20190727 15:08:19.415)
WSREP_SST: [INFO] Stale sst_in_progress file: /data/db//sst_in_progress (20190727 15:08:19.418)
WSREP_SST: [INFO] Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20190727 15:08:19.453)
MySQL init process in progress...
MySQL init process in progress...
2019-07-27 15:08:21 139821687219968 [Note] WSREP: (5d726e91, 'tcp://0.0.0.0:4567') turning message relay requesting off
MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...
2019-07-27 15:08:29 139822055872256 [Note] WSREP: Prepared SST request: mariabackup|10.1.1.53:4444/xtrabackup_sst//1
2019-07-27 15:08:29 139822055872256 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2019-07-27 15:08:29 139822055872256 [Note] WSREP: REPL Protocols: 9 (4, 2)
2019-07-27 15:08:29 139822055872256 [Note] WSREP: Assign initial position for certification: 4, protocol version: 4
2019-07-27 15:08:29 139821720790784 [Note] WSREP: Service thread queue flushed.
2019-07-27 15:08:29 139822055872256 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (6c6be05c-b07f-11e9-9619-b2a920ba13e1): 1 (Operation not permitted)
	 at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
2019-07-27 15:08:29 139821674493696 [Note] WSREP: Member 0.0 (mariadb-1) requested state transfer from '*any*'. Selected 1.0 (mariadb-0)(SYNCED) as donor.
2019-07-27 15:08:29 139821674493696 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 4)
2019-07-27 15:08:29 139822055872256 [Note] WSREP: Requesting state transfer: success, donor: 1
2019-07-27 15:08:29 139822055872256 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> 6c6be05c-b07f-11e9-9619-b2a920ba13e1:4
MySQL init process in progress...
WSREP_SST: [INFO] WARNING: Stale temporary SST directory: /data/db//.sst from previous state transfer. Removing (20190727 15:08:30.195)
WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20190727 15:08:30.206)
WSREP_SST: [INFO] Proceeding with SST (20190727 15:08:30.209)
WSREP_SST: [INFO] Cleaning the existing datadir and innodb-data/log directories (20190727 15:08:30.214)
removed '/data/db/aria_log_control'
removed '/data/db/ibdata1'
removed '/data/db/ib_buffer_pool'
removed '/data/db/aria_log.00000001'
removed '/data/db/ib_logfile1'
removed '/data/db/mysql/roles_mapping.MYI'
removed '/data/db/mysql/time_zone_transition_type.MYD'
removed '/data/db/mysql/time_zone_leap_second.MYD'
removed '/data/db/mysql/tables_priv.MYD'
removed '/data/db/mysql/plugin.frm'
removed '/data/db/mysql/help_category.MYD'
removed '/data/db/mysql/columns_priv.MYI'
removed '/data/db/mysql/servers.MYI'
removed '/data/db/mysql/func.frm'
removed '/data/db/mysql/event.MYD'
removed '/data/db/mysql/time_zone_name.MYI'
removed '/data/db/mysql/help_topic.MYD'
removed '/data/db/mysql/innodb_index_stats.frm'
removed '/data/db/mysql/index_stats.MYI'
removed '/data/db/mysql/columns_priv.MYD'
removed '/data/db/mysql/time_zone_transition.MYD'
removed '/data/db/mysql/proxies_priv.MYI'
removed '/data/db/mysql/procs_priv.frm'
removed '/data/db/mysql/help_relation.MYD'
removed '/data/db/mysql/slow_log.CSV'
removed '/data/db/mysql/servers.frm'
removed '/data/db/mysql/table_stats.MYD'
removed '/data/db/mysql/index_stats.MYD'
removed '/data/db/mysql/proc.MYI'
removed '/data/db/mysql/proxies_priv.frm'
removed '/data/db/mysql/slow_log.CSM'
removed '/data/db/mysql/proc.frm'
removed '/data/db/mysql/db.MYI'
removed '/data/db/mysql/time_zone_leap_second.frm'
removed '/data/db/mysql/servers.MYD'
removed '/data/db/mysql/plugin.MYD'
removed '/data/db/mysql/time_zone_transition_type.MYI'
removed '/data/db/mysql/column_stats.frm'
removed '/data/db/mysql/help_topic.frm'
removed '/data/db/mysql/time_zone.frm'
removed '/data/db/mysql/procs_priv.MYD'
removed '/data/db/mysql/user.MYD'
removed '/data/db/mysql/time_zone_name.MYD'
removed '/data/db/mysql/db.frm'
removed '/data/db/mysql/help_category.MYI'
removed '/data/db/mysql/general_log.CSM'
removed '/data/db/mysql/time_zone.MYD'
removed '/data/db/mysql/innodb_table_stats.ibd'
removed '/data/db/mysql/proc.MYD'
removed '/data/db/mysql/procs_priv.MYI'
removed '/data/db/mysql/column_stats.MYD'
removed '/data/db/mysql/gtid_slave_pos.frm'
removed '/data/db/mysql/time_zone_transition_type.frm'
removed '/data/db/mysql/user.frm'
removed '/data/db/mysql/host.frm'
removed '/data/db/mysql/roles_mapping.MYD'
removed '/data/db/mysql/time_zone.MYI'
removed '/data/db/mysql/db.opt'
removed '/data/db/mysql/tables_priv.frm'
removed '/data/db/mysql/time_zone_transition.frm'
removed '/data/db/mysql/general_log.frm'
removed '/data/db/mysql/gtid_slave_pos.ibd'
removed '/data/db/mysql/help_topic.MYI'
removed '/data/db/mysql/help_keyword.frm'
removed '/data/db/mysql/event.frm'
removed '/data/db/mysql/func.MYD'
removed '/data/db/mysql/innodb_table_stats.frm'
removed '/data/db/mysql/slow_log.frm'
removed '/data/db/mysql/table_stats.frm'
removed '/data/db/mysql/tables_priv.MYI'
removed '/data/db/mysql/time_zone_name.frm'
removed '/data/db/mysql/roles_mapping.frm'
removed '/data/db/mysql/innodb_index_stats.ibd'
removed '/data/db/mysql/host.MYD'
removed '/data/db/mysql/host.MYI'
removed '/data/db/mysql/table_stats.MYI'
removed '/data/db/mysql/event.MYI'
removed '/data/db/mysql/columns_priv.frm'
removed '/data/db/mysql/help_relation.frm'
removed '/data/db/mysql/func.MYI'
removed '/data/db/mysql/column_stats.MYI'
removed '/data/db/mysql/proxies_priv.MYD'
removed '/data/db/mysql/db.MYD'
removed '/data/db/mysql/index_stats.frm'
removed '/data/db/mysql/help_keyword.MYI'
removed '/data/db/mysql/time_zone_transition.MYI'
removed '/data/db/mysql/general_log.CSV'
removed '/data/db/mysql/time_zone_leap_second.MYI'
removed '/data/db/mysql/plugin.MYI'
removed '/data/db/mysql/help_category.frm'
removed '/data/db/mysql/help_keyword.MYD'
removed '/data/db/mysql/help_relation.MYI'
removed '/data/db/mysql/user.MYI'
removed directory '/data/db/mysql'
removed '/data/db/performance_schema/db.opt'
removed directory '/data/db/performance_schema'
WSREP_SST: [INFO] Waiting for SST streaming to complete! (20190727 15:08:30.232)
MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...
2019-07-27 15:08:40 139821674493696 [Warning] WSREP: 1.0 (mariadb-0): State transfer to 0.0 (mariadb-1) failed: -22 (Invalid argument)
2019-07-27 15:08:40 139821674493696 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():737: Will never receive state. Need to abort.
2019-07-27 15:08:40 139821674493696 [Note] WSREP: gcomm: terminating thread
2019-07-27 15:08:40 139821674493696 [Note] WSREP: gcomm: joining thread
2019-07-27 15:08:40 139821674493696 [Note] WSREP: gcomm: closing backend
WSREP_SST: [ERROR] xtrabackup_checkpoints missing, failed innobackupex/SST on donor (20190727 15:08:40.246)
WSREP_SST: [ERROR] Cleanup after exit with status:2 (20190727 15:08:40.249)
2019-07-27 15:08:40 139821666100992 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'joiner' --address '10.1.1.53' --datadir '/data/db/'   --parent '106'  ''  '': 2 (No such file or directory)
2019-07-27 15:08:40 139821666100992 [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script.
2019-07-27 15:08:40 139822057383232 [ERROR] WSREP: SST failed: 2 (No such file or directory)
2019-07-27 15:08:40 139822057383232 [ERROR] Aborting

MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...
/usr/local/bin/docker-entrypoint.sh: line 110:   106 Aborted                 "$@" --skip-networking --socket="${SOCKET}"
MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...
MySQL init process failed.

[solsson]

I wanted to try minikube with the kubenet network plugin that microk8s uses, but I fail to get an operational cluster with --network-plugin=kubenet.

Edit: According to https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/ kubeadm doesn't support kubenet. And it looks like kubeadm is the only bootstrapper in the current minikube version.