Node Local DNS keeps retrying timed-out request forever

[markkrj]

We have a DNS solution from Palo Alto which blocks domains based on categories.
When a request is made to a blocked category (Phishing, for example) the request is dropped and no response is returned.
Today, we found out that Node Local DNS keeps retrying those requests forever.

I tested it against CoreDNS and it is retried only once by it.

Here is logs for a domain blocked by Phishing:

❯ k logs daemonsets/nodelocaldns --max-log-requests 10 --all-pods --all-containers -f --tail=0
[pod/nodelocaldns-8k9rs/node-cache] [INFO] 169.254.25.10:45893 - 23889 "A IN nimek.com.br. udp 53 false 4096" - - 0 6.003017673s
[pod/nodelocaldns-8k9rs/node-cache] [ERROR] plugin/errors: 2 nimek.com.br. A: read udp 10.192.4.8:42252->10.180.214.1:53: i/o timeout
[pod/nodelocaldns-8k9rs/node-cache] [INFO] 169.254.25.10:46885 - 51611 "A IN nimek.com.br. udp 53 false 4096" - - 0 6.002141208s
[pod/nodelocaldns-8k9rs/node-cache] [ERROR] plugin/errors: 2 nimek.com.br. A: read udp 10.192.4.8:33377->10.180.214.2:53: i/o timeout
[pod/nodelocaldns-8k9rs/node-cache] [INFO] 169.254.25.10:33456 - 50803 "A IN nimek.com.br. udp 53 false 4096" - - 0 6.002448761s
[pod/nodelocaldns-8k9rs/node-cache] [ERROR] plugin/errors: 2 nimek.com.br. A: read udp 10.192.4.8:44141->10.180.214.2:53: i/o timeout
[pod/nodelocaldns-8k9rs/node-cache] [INFO] 169.254.25.10:45893 - 23889 "A IN nimek.com.br. udp 53 false 4096" - - 0 6.002577108s
[pod/nodelocaldns-8k9rs/node-cache] [ERROR] plugin/errors: 2 nimek.com.br. A: read udp 10.192.4.8:45622->10.180.214.1:53: i/o timeout
[pod/nodelocaldns-8k9rs/node-cache] [INFO] 169.254.25.10:40717 - 28149 "A IN nimek.com.br. udp 53 false 4096" - - 0 6.002913797s
[pod/nodelocaldns-8k9rs/node-cache] [ERROR] plugin/errors: 2 nimek.com.br. A: read udp 10.192.4.8:39078->10.180.214.1:53: i/o timeout
[pod/nodelocaldns-8k9rs/node-cache] [INFO] 169.254.25.10:37491 - 43464 "A IN nimek.com.br. udp 53 false 4096" - - 0 6.003396595s
[pod/nodelocaldns-8k9rs/node-cache] [ERROR] plugin/errors: 2 nimek.com.br. A: read udp 10.192.4.8:34878->10.180.214.1:53: i/o timeout
[pod/nodelocaldns-8k9rs/node-cache] [INFO] 169.254.25.10:33889 - 33196 "A IN nimek.com.br. udp 53 false 4096" - - 0 6.001887249s
[pod/nodelocaldns-8k9rs/node-cache] [ERROR] plugin/errors: 2 nimek.com.br. A: read udp 169.254.25.10:57744->169.254.25.10:53: i/o timeout
[pod/nodelocaldns-8k9rs/node-cache] [INFO] 169.254.25.10:47742 - 21714 "A IN nimek.com.br. udp 53 false 4096" - - 0 6.002801422s
[pod/nodelocaldns-8k9rs/node-cache] [ERROR] plugin/errors: 2 nimek.com.br. A: read udp 169.254.25.10:58427->169.254.25.10:53: i/o timeout
[pod/nodelocaldns-8k9rs/node-cache] [INFO] 169.254.25.10:57744 - 9859 "A IN nimek.com.br. udp 53 false 4096" - - 0 6.002010628s
[pod/nodelocaldns-8k9rs/node-cache] [ERROR] plugin/errors: 2 nimek.com.br. A: read udp 169.254.25.10:42774->169.254.25.10:53: i/o timeout
[pod/nodelocaldns-8k9rs/node-cache] [INFO] 169.254.25.10:43708 - 22557 "A IN nimek.com.br. udp 53 false 4096" - - 0 6.001404522s
[pod/nodelocaldns-8k9rs/node-cache] [ERROR] plugin/errors: 2 nimek.com.br. A: read udp 10.192.4.8:50276->10.180.214.2:53: i/o timeout
[pod/nodelocaldns-8k9rs/node-cache] [INFO] 169.254.25.10:60737 - 63817 "A IN nimek.com.br. udp 53 false 4096" - - 0 6.001317451s
[pod/nodelocaldns-8k9rs/node-cache] [ERROR] plugin/errors: 2 nimek.com.br. A: read udp 10.192.4.8:42442->10.180.214.1:53: i/o timeout
[pod/nodelocaldns-8k9rs/node-cache] [INFO] 169.254.25.10:58427 - 39440 "A IN nimek.com.br. udp 53 false 4096" - - 0 6.001854649s
[pod/nodelocaldns-8k9rs/node-cache] [ERROR] plugin/errors: 2 nimek.com.br. A: read udp 10.192.4.8:45970->10.180.214.1:53: i/o timeout
[pod/nodelocaldns-8k9rs/node-cache] [INFO] 169.254.25.10:60918 - 50389 "A IN nimek.com.br. udp 53 false 4096" - - 0 6.001841172s
[pod/nodelocaldns-8k9rs/node-cache] [ERROR] plugin/errors: 2 nimek.com.br. A: read udp 10.192.4.8:39476->10.180.214.2:53: i/o timeout
[pod/nodelocaldns-8k9rs/node-cache] [INFO] 169.254.25.10:40609 - 9036 "A IN nimek.com.br. udp 53 false 4096" - - 0 6.001963206s
[pod/nodelocaldns-8k9rs/node-cache] [ERROR] plugin/errors: 2 nimek.com.br. A: read udp 169.254.25.10:49844->169.254.25.10:53: i/o timeout
[pod/nodelocaldns-8k9rs/node-cache] [INFO] 169.254.25.10:42774 - 34309 "A IN nimek.com.br. udp 53 false 4096" - - 0 6.00220121s
[pod/nodelocaldns-8k9rs/node-cache] [ERROR] plugin/errors: 2 nimek.com.br. A: read udp 10.192.4.8:53370->10.180.214.1:53: i/o timeout
[pod/nodelocaldns-8k9rs/node-cache] [INFO] 169.254.25.10:44540 - 63637 "A IN nimek.com.br. udp 53 false 4096" - - 0 6.00115686s
[pod/nodelocaldns-8k9rs/node-cache] [INFO] 169.254.25.10:54663 - 51615 "A IN nimek.com.br. udp 53 false 4096" - - 0 6.001404803s
[pod/nodelocaldns-8k9rs/node-cache] [ERROR] plugin/errors: 2 nimek.com.br. A: read udp 10.192.4.8:55560->10.180.214.2:53: i/o timeout
[pod/nodelocaldns-8k9rs/node-cache] [ERROR] plugin/errors: 2 nimek.com.br. A: read udp 169.254.25.10:48279->169.254.25.10:53: i/o timeout

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[markkrj]

/remove-lifecycle stale