Log failure in internalSend

[DaveCTurner]

An exception here should be impossible, but we don't assert that, nor do
we emit a log message to prove it didn't happen in a production
environment. This commit adds the missing log and assert.

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[mhl-b]

I'm suspecting other place that might throw error that we dont catch. Async connect call that we dont check for exceptions. Looking at netty's code path I cannot tell 100% that channel can be null with exception as we assume in the code.

ChannelFuture connectFuture = bootstrapWithHandler.connect(); -> connect().await(timeout);

elasticsearch/modules/transport-netty4/src/main/java/org/elasticsearch/transport/netty4/Netty4Transport.java

Lines 268 to 274 in e713f7c

	ChannelFuture connectFuture = bootstrapWithHandler.connect();
	Channel channel = connectFuture.channel();
	if (channel == null) {
	ExceptionsHelper.maybeDieOnAnotherThread(connectFuture.cause());
	throw new IOException(connectFuture.cause());
	}

[DaveCTurner]

Possibly, but that wouldn't cause a long-established channel to close. We can address that separately, no need to do so here.

[DaveCTurner]

I opened #131520

[mhl-b]

LGTM