child node not joining the cluster

[anweshtupili]

After running compose file child nodes are not joining the cluster. child nodes are staying in starting stage for long time and restarting.

Log showing:

inno_mysql-2.1.u9eabv1l8sqm@node2 | info: attempting to join the C55153C1-1574-4972-BF06-7332D6AD46A7 group using mysql-1:6606,mysql-2:6606,mysql-3:6606 as seeds
inno_mysql-2.1.u9eabv1l8sqm@node2 | Initializing database
inno_mysql-2.1.u9eabv1l8sqm@node2 | Database initialized
inno_mysql-2.1.u9eabv1l8sqm@node2 | MySQL init process in progress...
inno_mysql-2.1.u9eabv1l8sqm@node2 | Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it.
inno_mysql-2.1.u9eabv1l8sqm@node2 | Warning: Unable to load '/usr/share/zoneinfo/leapseconds' as time zone. Skipping it.
inno_mysql-2.1.u9eabv1l8sqm@node2 | Warning: Unable to load '/usr/share/zoneinfo/tzdata.zi' as time zone. Skipping it.
inno_mysql-2.1.u9eabv1l8sqm@node2 | Warning: Unable to load '/usr/share/zoneinfo/zone.tab' as time zone. Skipping it.
inno_mysql-2.1.u9eabv1l8sqm@node2 | Warning: Unable to load '/usr/share/zoneinfo/zone1970.tab' as time zone. Skipping it.
inno_mysql-2.1.u9eabv1l8sqm@node2 | Note (Code 1008): Can't drop database 'test'; database doesn't exist
inno_mysql-2.1.u9eabv1l8sqm@node2 |
inno_mysql-2.1.u9eabv1l8sqm@node2 | /entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
inno_mysql-2.1.u9eabv1l8sqm@node2 |
inno_mysql-2.1.u9eabv1l8sqm@node2 | mysql: [Warning] Using a password on the command line interface can be insecure.
inno_mysql-2.1.u9eabv1l8sqm@node2 | mysql: [Warning] Using a password on the command line interface can be insecure.
inno_mysql-2.1.u9eabv1l8sqm@node2 | Note (Code 1759): Sending passwords in plain text without SSL/TLS is extremely insecure.
inno_mysql-2.1.u9eabv1l8sqm@node2 | Note (Code 1760): Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
inno_mysql-2.1.u9eabv1l8sqm@node2 |
inno_mysql-2.1.u9eabv1l8sqm@node2 | MySQL init process done. Ready for start up.
inno_mysql-2.1.u9eabv1l8sqm@node2 |
inno_mysql-2.1.vt7ebm1wknni@node2 | info: attempting to join the C55153C1-1574-4972-BF06-7332D6AD46A7 group using mysql-1:6606 as seeds
inno_mysql-2.1.vt7ebm1wknni@node2 | Initializing database
inno_mysql-2.1.vt7ebm1wknni@node2 | Database initialized
inno_mysql-2.1.vt7ebm1wknni@node2 | MySQL init process in progress...
inno_mysql-2.1.vt7ebm1wknni@node2 | Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it.
inno_mysql-2.1.vt7ebm1wknni@node2 | Warning: Unable to load '/usr/share/zoneinfo/leapseconds' as time zone. Skipping it.
inno_mysql-2.1.vt7ebm1wknni@node2 | Warning: Unable to load '/usr/share/zoneinfo/tzdata.zi' as time zone. Skipping it.
inno_mysql-2.1.vt7ebm1wknni@node2 | Warning: Unable to load '/usr/share/zoneinfo/zone.tab' as time zone. Skipping it.
inno_mysql-2.1.vt7ebm1wknni@node2 | Warning: Unable to load '/usr/share/zoneinfo/zone1970.tab' as time zone. Skipping it.
inno_mysql-2.1.vt7ebm1wknni@node2 | Note (Code 1008): Can't drop database 'test'; database doesn't exist
inno_mysql-2.1.vt7ebm1wknni@node2 |
inno_mysql-2.1.vt7ebm1wknni@node2 | /entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
inno_mysql-2.1.vt7ebm1wknni@node2 |
inno_mysql-2.1.vt7ebm1wknni@node2 | mysql: [Warning] Using a password on the command line interface can be insecure.
inno_mysql-2.1.vt7ebm1wknni@node2 | mysql: [Warning] Using a password on the command line interface can be insecure.
inno_mysql-2.1.vt7ebm1wknni@node2 | Note (Code 1759): Sending passwords in plain text without SSL/TLS is extremely insecure.
inno_mysql-2.1.vt7ebm1wknni@node2 | Note (Code 1760): Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
inno_mysql-2.1.vt7ebm1wknni@node2 |
inno_mysql-2.1.vt7ebm1wknni@node2 | MySQL init process done. Ready for start up.
inno_mysql-2.1.vt7ebm1wknni@node2 |
inno_mysql-2.1.3cpv7igk0vnk@node2 | info: attempting to join the C55153C1-1574-4972-BF06-7332D6AD46A7 group using mysql-1:6606,mysql-2:6606,mysql-3:6606 as seeds
inno_mysql-2.1.3cpv7igk0vnk@node2 | Initializing database
inno_mysql-2.1.3cpv7igk0vnk@node2 | Database initialized
inno_mysql-2.1.3cpv7igk0vnk@node2 | MySQL init process in progress...
inno_mysql-2.1.3cpv7igk0vnk@node2 | Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it.
inno_mysql-2.1.3cpv7igk0vnk@node2 | Warning: Unable to load '/usr/share/zoneinfo/leapseconds' as time zone. Skipping it.
inno_mysql-2.1.3cpv7igk0vnk@node2 | Warning: Unable to load '/usr/share/zoneinfo/tzdata.zi' as time zone. Skipping it.
inno_mysql-2.1.3cpv7igk0vnk@node2 | Warning: Unable to load '/usr/share/zoneinfo/zone.tab' as time zone. Skipping it.
inno_mysql-2.1.3cpv7igk0vnk@node2 | Warning: Unable to load '/usr/share/zoneinfo/zone1970.tab' as time zone. Skipping it.
inno_mysql-2.1.3cpv7igk0vnk@node2 | Note (Code 1008): Can't drop database 'test'; database doesn't exist
inno_mysql-2.1.3cpv7igk0vnk@node2 |
inno_mysql-2.1.3cpv7igk0vnk@node2 | /entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
inno_mysql-2.1.3cpv7igk0vnk@node2 |
inno_mysql-2.1.3cpv7igk0vnk@node2 | mysql: [Warning] Using a password on the command line interface can be insecure.
inno_mysql-2.1.3cpv7igk0vnk@node2 | mysql: [Warning] Using a password on the command line interface can be insecure.
inno_mysql-2.1.3cpv7igk0vnk@node2 | Note (Code 1759): Sending passwords in plain text without SSL/TLS is extremely insecure.
inno_mysql-2.1.3cpv7igk0vnk@node2 | Note (Code 1760): Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
inno_mysql-2.1.3cpv7igk0vnk@node2 |
inno_mysql-2.1.3cpv7igk0vnk@node2 | MySQL init process done. Ready for start up.
inno_mysql-2.1.3cpv7igk0vnk@node2 |
inno_mysql-2.1.ub63qcvutmo3@node2 | info: attempting to join the C55153C1-1574-4972-BF06-7332D6AD46A7 group using mysql-1:6606,mysql-2:6606,mysql-3:6606 as seeds
inno_mysql-2.1.ub63qcvutmo3@node2 | Initializing database
inno_mysql-2.1.7qd7vz1mrj8q@node2 | info: attempting to join the C55153C1-1574-4972-BF06-7332D6AD46A7 group using mysql-1:6606 as seeds
inno_mysql-2.1.7qd7vz1mrj8q@node2 | Initializing database
inno_mysql-2.1.7qd7vz1mrj8q@node2 | Database initialized
inno_mysql-2.1.7qd7vz1mrj8q@node2 | MySQL init process in progress...
inno_mysql-2.1.7qd7vz1mrj8q@node2 | Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it.
inno_mysql-2.1.7qd7vz1mrj8q@node2 | Warning: Unable to load '/usr/share/zoneinfo/leapseconds' as time zone. Skipping it.
inno_mysql-2.1.7qd7vz1mrj8q@node2 | Warning: Unable to load '/usr/share/zoneinfo/tzdata.zi' as time zone. Skipping it.
inno_mysql-2.1.7qd7vz1mrj8q@node2 | Warning: Unable to load '/usr/share/zoneinfo/zone.tab' as time zone. Skipping it.
inno_mysql-2.1.7qd7vz1mrj8q@node2 | Warning: Unable to load '/usr/share/zoneinfo/zone1970.tab' as time zone. Skipping it.
inno_mysql-2.1.7qd7vz1mrj8q@node2 | Note (Code 1008): Can't drop database 'test'; database doesn't exist
inno_mysql-2.1.7qd7vz1mrj8q@node2 |
inno_mysql-2.1.7qd7vz1mrj8q@node2 | /entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
inno_mysql-2.1.7qd7vz1mrj8q@node2 |
inno_mysql-2.1.7qd7vz1mrj8q@node2 | mysql: [Warning] Using a password on the command line interface can be insecure.
inno_mysql-2.1.7qd7vz1mrj8q@node2 | mysql: [Warning] Using a password on the command line interface can be insecure.
inno_mysql-2.1.7qd7vz1mrj8q@node2 | Note (Code 1759): Sending passwords in plain text without SSL/TLS is extremely insecure.
inno_mysql-2.1.7qd7vz1mrj8q@node2 | Note (Code 1760): Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
inno_mysql-2.1.7qd7vz1mrj8q@node2 |
inno_mysql-2.1.7qd7vz1mrj8q@node2 | MySQL init process done. Ready for start up.
inno_mysql-2.1.7qd7vz1mrj8q@node2 |

[anweshtupili]

My Compose File:

version: "3"
services:
mysql-1:
image: "mattalord/innodb-cluster"
restart: always
hostname: mysql-1
deploy:
mode: "replicated"
replicas: 1
placement:
constraints: [node.hostname == node1]
environment:
- "MYSQL_ROOT_PASSWORD=root"
- "GROUP_NAME=C55153C1-1574-4972-BF06-7332D6AD46A7"
- "BOOTSTRAP=1"
volumes:
- mysql-1db:/var/lib/mysqlrouter
networks:
- inno_ydel_net

mysql-2:
image: "mattalord/innodb-cluster"
restart: always
hostname: mysql-2
deploy:
mode: "replicated"
replicas: 1
placement:
constraints: [node.hostname == node2]
environment:
- "MYSQL_ROOT_PASSWORD=root"
- "GROUP_NAME=C55153C1-1574-4972-BF06-7332D6AD46A7"
- "GROUP_SEEDS=mysql-1:6606"
volumes:
- mysql-2db:/var/lib/mysqlrouter
networks:
- inno_ydel_net
depends_on:
- mysql-1

mysql-3:
image: "mattalord/innodb-cluster"
restart: always
hostname: mysql-3
deploy:
mode: "replicated"
replicas: 1
placement:
constraints: [node.hostname == node3]
environment:
- "MYSQL_ROOT_PASSWORD=root"
- "GROUP_NAME=C55153C1-1574-4972-BF06-7332D6AD46A7"
- "GROUP_SEEDS=mysql-1:6606"
volumes:
- mysql-3db:/var/lib/mysqlrouter
networks:
- inno_ydel_net
depends_on:
- mysql-1

router:
image: "mattalord/innodb-cluster"
restart: always
hostname: router
deploy:
mode: "replicated"
replicas: 1
placement:
constraints: [node.hostname == node4]
ports:
- "3306:3306"
- "6606:6606"
- "6446-6447:6446-6447"
volumes:
- mysql-routerdb:/var/lib/mysqlrouter
networks:
- inno_ydel_net
depends_on:
- mysql-1
- mysql-2
- mysql-3
environment:
- "NODE_TYPE=router"
- "MYSQL_ROOT_PASSWORD=root"
- "MYSQL_HOST=mysql-1"

volumes:
mysql-1db:
driver: local
mysql-2db:
driver: local
mysql-3db:
driver: local
mysql-routerdb:
driver: local

networks:
inno_ydel_net:

[mattlord]

I don't notice anything obvious in the log. What did the performance_schema.group* tables show?

This was my hobby project in the past and I'm not really working on it anymore (no longer at MySQL). I would recommend trying to use the more official MySQL work happening in this space:

• https://mysqlhighavailability.com/setting-up-mysql-group-replication-with-mysql-docker-images/
• https://mysqlrelease.com/2018/03/docker-compose-setup-for-innodb-cluster/

If you do find an issue though, I'd be more than happy with a pull request. :-)