[error] ** System NOT running to use fully qualified hostnames ** Kubernetes DNSSRV Strategy

[paltaa]

So I've followed this tutorial:

https://tech.xing.com/creating-an-erlang-elixir-cluster-on-kubernetes-d53ef89758f6

On the logs getting this errors:

14:02:03.126 [error] ** System NOT running to use fully qualified hostnames ** ** Hostname 192-168-4-107.ueuropea-excalibur-headless-service.default.svc.cluster.local is illegal ** 14:02:03.325 [warn] [libcluster:k8s_excalibur] unable to connect to :"excalibur@192-168-4-107.ueuropea-excalibur-headless-service.default.svc.cluster.local"

`root@ueuropea-excalibur-74df5dddbc-kjfql:/excalibur# nslookup -q=srv ueuropea-excalibur-headless-service.default.svc.cluster.local
Server: 10.100.0.10
Address: 10.100.0.10#53

Non-authoritative answer:
ueuropea-excalibur-headless-service.default.svc.cluster.local service = 0 33 4000 192-168-4-107.ueuropea-excalibur-headless-service.default.svc.cluster.local.
ueuropea-excalibur-headless-service.default.svc.cluster.local service = 0 33 4000 192-168-53-92.ueuropea-excalibur-headless-service.default.svc.cluster.local.
ueuropea-excalibur-headless-service.default.svc.cluster.local service = 0 33 4000 192-168-65-161.ueuropea-excalibur-headless-service.default.svc.cluster.local.

Authoritative answers can be found from:
192-168-65-161.ueuropea-excalibur-headless-service.default.svc.cluster.local internet address = 192.168.65.161
192-168-53-92.ueuropea-excalibur-headless-service.default.svc.cluster.local internet address = 192.168.53.92
192-168-4-107.ueuropea-excalibur-headless-service.default.svc.cluster.local internet address = 192.168.4.107
`

It seems its missing a . at the end of the returned DNS right?

Is there something wrong with the tutorial? something could be missing?

Entrypoint:

`mix release
#elixir -S mix phx.server --name excalibur@${MY_POD_IP} --cookie "secret"

_build/prod-kubernetes/rel/excalibur/bin/excalibur start`

Deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
io.kompose.service: excalibur
name: ueuropea-excalibur
spec:
progressDeadlineSeconds: 90
replicas: 3
strategy:
type: Recreate
template:
metadata:
labels:
io.kompose.service: excalibur
spec:
containers:
- image: 975847796244.dkr.ecr.us-west-2.amazonaws.com/excalibur:dnsrv
name: excalibur
imagePullPolicy: Always
ports:
- containerPort: 4000
- containerPort: 4369
env:
- name: DATABASE_URL
value: *****
- name: MIX_ENV
value: prod-kubernetes
- name: SECRET_KEY_BASE
value: *****
- name: ERLANG_COOKIE
value: ******
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: ERLANG_CLUSTER_SERVICE_NAME
value: ueuropea-excalibur-headless-service
resources: {}
restartPolicy: Always
status: {}

Headless service:

apiVersion: v1
kind: Service
metadata:
labels:
io.kompose.service: excalibur
name: ueuropea-excalibur-headless-service
spec:
type: ClusterIP
clusterIP: None
ports:
- name: "http"
port: 80
targetPort: 4000
publishNotReadyAddresses: true
selector:
io.kompose.service: excalibur

Let me know if any more information is needed

Hi @paltaa - I'm the author of the Kubernetes DNSSRV Strategy - and I've just discovered the same problem in some training material I'm supposed to be teaching tomorrow 😂!!!!

This is highly annoying - looks like an arbitrary change in statefulset DNS names - I'll investigate - if only for my own selfish reasons and report back.

Thanks for the bug report

@bitwalker - assign this to me if you like

Reason for breakage:

Google Cloud have stopped using CoreDNS as DNS resolver as of GKE 1.1.3 - they are instead using kube-dns which doesn't provide service discovery via SRV record resolution. (obviously they want everyone using k8s API for everything).

Lame.

Release notes (1.1.3) - they switched :
https://cloud.google.com/kubernetes-engine/docs/release-notes#new_features_17

Someone trying to debug the issue :
kubernetes/kubernetes#85759

Stackoverflow thread :
https://stackoverflow.com/questions/55122234/installing-coredns-on-gke

The kubernetes/docker ecosystem is just like that - stuff arbitrarily breaks all the time - recommend you try using Paul's strategy/kubernetes instead .

I'm away but will set a reminder to create a docs PR - maybe changing the name to strategy/k8s-coredns-srv or something that makes it clear you need coredns.

Man - that Google - always causing problems ! Sorry !

[paltaa]

Thanks a lot for the reply!! Been trying to make this work for a couple of days, followed about 4 different tutorials hahaha, glad to know I helped by posting this issue! let me know if you manage to fix this, good luck tomorrow

[paltaa]

@bryanhuntesl Also, this is currently happening on AWS EKS, which still uses coreDNS, by making some tweaks the illegal hostname error and system not running fqdns too, but still it wont connect

[mrchypark]

I have the same problem.

My environment is aks (azure kube) v1.15.7.
I had the same problem using kubernetnes.DNS strategy.

[paltaa]

@mrchypark Hey, what worked for me is Elixir.Cluster.Strategy.Kubernetes

My example:
topologies = [
k8s_excalibur: [
strategy: Elixir.Cluster.Strategy.Kubernetes,
config: [
service: System.get_env("ERLANG_CLUSTER_SERVICE_NAME"),
application_name: "excalibur",
kubernetes_node_basename: "excalibur",
kubernetes_namespace: "default",
kubernetes_selector: "io.kompose.service=excalibur"
]
]
]

[mrchypark]

@paltaa Thank you for your reply!
I have question.

What is service means? It's service resource on kubernetes?

[paltaa]

Yes, the service for kubernetes deployments

[mrchypark]

@paltaa Thank you! I'll try this :)

[mrchypark]

@paltaa I have more question T.T

what is your node name now?

mine is like hydra@hydraapp-799457d75f-948qc.

I have no error but empty node list too.

[paltaa]

You need to use distillery, setup a pre hook before the erlang VM is up and setup the ENV VAR ERLANG_NODE with its local ip

[mrchypark]

local ip means pod ip?

[paltaa]

For example.

Deployment part:

        - name: MY_POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP

Then the pre hook:

#!/bin/sh

echo 'Setting ERLANG_NAME...'
export ERLANG_NAME=$MY_POD_IP
echo $ERLANG_NAME
export ERLANG_COOKIE=****
echo $ERLANG_COOKIE

vm.args:

-name excalibur@${ERLANG_NAME}

-setcookie ${ERLANG_COOKIE}

[mrchypark]

I maybe pass the set cookie.

I'll try this.