[Bug]: Couchbase 7.6.2 does not start

[vforchi]

Module

Couchbase

Testcontainers version

1.20.1

Using the latest Testcontainers version?

Yes

Host OS

Linux

Host Arch

x86

Docker version

27-RC2

What happened?

When upgrading from couchbase 7.1.1 to 7.6.2 my tests stopped working on Jenkins.
Initially I thought it was the hardcoded timeout, but after some investigation it turns out that the container does not start up properly, and trying to access /pools/default gives a 500.
Note: if I start the container manually (mapping all the ports), I can access /pools/default without any problems.

Relevant log output

couchbase logs:

[ns_server:error,2024-08-09T14:35:49.172Z,ns_1@172.17.0.3:memcached_config_mgr<0.3586.0>:memcached_config_mgr:push_tls_config:240]Failed to push TLS config to memcached: couldnt_connect_to_memcached
 [ns_server:error,2024-08-09T14:35:00.547Z,n... [ns_server:error,2024-08-09T14:35:00.547Z,ns_1@172.17.0.3:<0.653.0>:menelaus_util:reply_server_error_before_close:233]Server error during processing: ["web request failed",
                                {path,"/pools/default"},
                                {method,'GET'},
                                {type,exit},
                                {what,
                                 {{{{badmatch,
                                     {error,couldnt_connect_to_memcached}},
                                    [{ns_audit_cfg,notify_memcached,1,
                                      [{file,"src/ns_audit_cfg.erl"},
                                       {line,153}]},
                                     {ns_audit_cfg,handle_info,2,
                                      [{file,"src/ns_audit_cfg.erl"},
                                       {line,128}]},
                                     {gen_server,try_dispatch,4,
                                      [{file,"gen_server.erl"},{line,1123}]},
                                     {gen_server,handle_msg,6,
                                      [{file,"gen_server.erl"},{line,1200}]},
                                     {proc_lib,init_p_do_apply,3,
                                      [{file,"proc_lib.erl"},{line,240}]}]},
                                   {gen_server,call,[ns_audit_cfg,get_uid]}},
                                  {gen_server,call,
                                   [menelaus_web_cache,
                                    #Fun<menelaus_web_cache.2.98503217>,
                                    infinity]}}},
                                {trace,
                                 [{gen_server,call,3,
                                   [{file,"gen_server.erl"},{line,385}]},
                                  {menelaus_web_pools,pool_info,6,
                                   [{file,"src/menelaus_web_pools.erl"},
                                    {line,115}]},
                                  {menelaus_web_pools,handle_pool_info,2,
                                   [{file,"src/menelaus_web_pools.erl"},
                                    {line,105}]},
                                  {request_tracker,request,2,
                                   [{file,"src/request_tracker.erl"},
                                    {line,40}]},
                                  {menelaus_util,handle_request,2,
                                   [{file,"src/menelaus_util.erl"},
                                    {line,244}]},
                                  {mochiweb_http,headers,6,
                                   [{file,
                                     "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/mochiweb/mochiweb_http.erl"},
                                    {line,153}]},
                                  {proc_lib,init_p_do_apply,3,
                                   [{file,"proc_lib.erl"},{line,240}]}]}]

In the memcache log I found this. 
 2024-08-09T14:38:10.256106+00:00 INFO -----16:38
 2024-08-09T14:38:10.256106+00:00 INFO ---------- Opening logfile:
 2024-08-09T14:38:10.257814+00:00 INFO Couchbase version 7.6.2-3721 starting.
 2024-08-09T14:38:10.257854+00:00 INFO Process identifier: 812
 2024-08-09T14:38:10.257903+00:00 WARNING max_connections is set higher than the available number of file descriptors available. Reduce max_connections to: 63468
 2024-08-09T14:38:10.257919+00:00 INFO recalculate_max_connections: {"engine_fds":1050,"max_connections":63468,"max_fds":65536,"system_connections":5000}
 2024-08-09T14:38:10.257993+00:00 INFO Breakpad enabled. Minidumps will be written to '/opt/couchbase/var/lib/couchbase/crash'
 2024-08-09T14:38:10.258428+00:00 INFO Fine clock resolution:444ns, overhead:412ns
 2024-08-09T14:38:10.258557+00:00 INFO Coarse clock resolution:1000008ns, overhead:10ns
 2024-08-09T14:38:10.258566+00:00 INFO (Clock measurement period: 1ns)
 2024-08-09T14:38:10.258930+00:00 INFO Using SLA configuration: {"COMPACT_DB":{"slow":"1800 s"},"CREATE_BUCKET":{"slow":"5 s"},"DELETE_BUCKET":{"slow":"10 s"},"SELECT_BUCKET":{"slow":"10 ms"},"SEQNO_PERSISTENCE":{"slow":"30 s"},"comment":"Current MCBP SLA configuration","default":{"slow":"500 ms"},"version":1}
 2024-08-09T14:38:10.258943+00:00 INFO Enable standard input listener
 2024-08-09T14:38:10.259014+00:00 INFO NUMA: NOT setting memory allocation policy to 'interleave' - request failed: Operation not permitted
 2024-08-09T14:38:10.259027+00:00 INFO Loading RBAC configuration from [/opt/couchbase/var/lib/couchbase/config/memcached.rbac]
 2024-08-09T14:38:10.259152+00:00 INFO Loading error maps from [/opt/couchbase/etc/couchbase/kv/error_maps]
 2024-08-09T14:38:10.259834+00:00 INFO Starting external authentication manager
 2024-08-09T14:38:10.263311+00:00 INFO Changing logging level to 0
 2024-08-09T14:38:10.263326+00:00 INFO recalculate_max_connections: {"engine_fds":1050,"max_connections":63468,"max_fds":65536,"system_connections":5000}
 2024-08-09T14:38:10.263328+00:00 INFO Initialize bucket manager
 2024-08-09T14:38:10.263329+00:00 INFO Initialize SASL
 2024-08-09T14:38:10.263922+00:00 INFO Starting network interface manager
 2024-08-09T14:38:10.264125+00:00 INFO Enable port(s)
 2024-08-09T14:38:10.264274+00:00 CRITICAL Failed to create required listening socket: "{"family":"inet","host":"*","port":11210,"system":false,"tag":"","tls":false,"type":"mcbp"}". Errors: getrandom. Terminating.
 2024-08-09T14:38:10.264384+00:00 INFO ---------- Closing logfile

If I start the container manually and I do the same thing, I get 200 and the error in the memcache is still there

note, we start the container with publish all and we get this:
6e474c024c80   couchbase/server:7.6.2   "/entrypoint.sh couc…"   About a minute ago   Up About a minute   0.0.0.0:32848->8091/tcp, :::32848->8091/tcp, 0.0.0.0:32847->8092/tcp, :::32847->8092/tcp, 0.0.0.0:32846->8093/tcp, :::32846->8093/tcp, 0.0.0.0:32845->8094/tcp, :::32845->8094/tcp, 0.0.0.0:32844->8095/tcp, :::32844->8095/tcp, 0.0.0.0:32843->8096/tcp, :::32843->8096/tcp, 0.0.0.0:32842->8097/tcp, :::32842->8097/tcp, 0.0.0.0:32841->9123/tcp, :::32841->9123/tcp, 0.0.0.0:32840->11207/tcp, :::32840->11207/tcp, 0.0.0.0:32839->11210/tcp, :::32839->11210/tcp, 0.0.0.0:32838->11280/tcp, :::32838->11280/tcp, 0.0.0.0:32837->18091/tcp, :::32837->18091/tcp, 0.0.0.0:32836->18092/tcp, :::32836->18092/tcp, 0.0.0.0:32835->18093/tcp, :::32835->18093/tcp, 0.0.0.0:32834->18094/tcp, :::32834->18094/tcp, 0.0.0.0:32833->18095/tcp, :::32833->18095/tcp, 0.0.0.0:32832->18096/tcp, :::32832->18096/tcp, 0.0.0.0:32831->18097/tcp, :::32831->18097/tcp   jolly_roentgen (edited) 

I did simply this:
curl -v -u Administrator:password http://hostname:32820/pools/default

Additional Information

No response

[vforchi]

Forgot to add the testcontainers logs:

09:20:43.101 [Test worker] INFO org.testcontainers.DockerClientFactory -- ✔︎ Docker server version should be at least 1.6.0
09:20:43.105 [Test worker] INFO tc.couchbase/server:7.6.2 -- Creating container for image: couchbase/server:7.6.2
09:20:43.145 [Test worker] INFO tc.couchbase/server:7.6.2 -- Container couchbase/server:7.6.2 is starting: d6f348fdee8cdcdbed35c55fce8c156a8992faca9a45d347b0f1af23a52ba9b7

[kiview]

Important to add, that this works locally for @vforchi with Docker Desktop. I also couldn't reproduce it, with either Docker Desktop or Testcontainers Cloud.

[vforchi]

Hi, is there any news on this?

[vforchi]

Hi, I just tried with the latest versions (testcontainers 1.20.4 and couchbase 7.6.4), but the error is still there. Do you need any additional input?

[vforchi]

I created a small project to recreate the bug:
https://github.com/vforchi/testcontainers-bug

It fails also with Docker version 24.0.6, build ed223bc

[vforchi]

This started working when we upgraded our Jenkins nodes to RedHat 9.
Not sure what the problem was, I guess either something with the docker version or the network configuration.