nodegroup ext_pillar handles nodegroups differently than targeting when using yaml list of minions

[jeff350]

Description of Issue

I was able to identify a bug in how the nodegroup ext_pillar handles parsing of nodegreoups so that there are inconsistencies in group membership between a pillar.get nodegroups and targeting with -N when the nodegroup is defined as a yaml list.

The core of this issue appears to be how the compound match in the nodegroup ext_pillar handles a minion that has not had it's key accepted on the master when the nodegroup is defined as a yaml list. In the case where a minion is defined in the nodegroup yaml but has not had it's key accepted an empty list is always returned from pillar.get nodegroups when it should return the list of hosts that it does know about.

Setup

1. set up a salt master and 2 salt minions named salt-minion-01 and salt-minion-02 and accept the kets
2. add this file to the /etc/salt/master.d/nodegroup.conf

ext_pillar:
  - nodegroups:
      pillar_name: 'nodegroups'
nodegroups:
  minions:
    - salt-minion-01
    - salt-minion-02
    - salt-minion-03
  group1: 'L@salt-minion-01'
  group2: 'L@salt-minion-02'

3. restart the salt-master for the config to be reloaded.
4. run sudo salt '*' saltutil.sync_all

At this point the system is fully setup to replicate the bug.

Steps to Reproduce Issue

1. run sudo salt -N minions test.ping this will return:

salt-minion-01:
    True
salt-minion-02:
    True

This shows that both salt minions are in the minions group as defined in the above config file.

2. run sudo salt '*' pillar.get nodegroups this will return:

salt-minion-01:
    - group1
salt-minion-02:
    - group2

See that the pillar.get nodegroups does not show that the minions are not in the minions nodegroup this is because in line https://github.com/saltstack/salt/blob/develop/salt/pillar/nodegroups.py#L68 match for the minions nodegroup is being evaluated to [] instead of ['salt-minion-01', 'salt-minion-02']

A workaround for this issue is to define the nodegroups as defined below but it makes the config files a lot more complex

nodegroups:
  minions:
    - salt-minion-01
    - 'or'
    - salt-minion-02
    - 'or'
    - salt-minion-03
  group1: 'L@salt-minion-01'
  group2: 'L@salt-minion-02'

another solution is to use the L@ targeting where it will also return correctly as seen below

nodegroups:
  minions:
    - 'L@salt-minion-01,salt-minion-02,salt-minion-03'
  group1: 'L@salt-minion-01'
  group2: 'L@salt-minion-02'

In this case sudo salt '*' pillar.get nodegroups will show all minions in the minions group as expected.

salt-minion-01:
    - minions
    - group1
salt-minion-02:
    - minions
    - group2

Related Docs

• https://docs.saltstack.com/en/latest/topics/targeting/nodegroups.html#defining-nodegroups-as-lists-of-minion-ids

Versions Report

Salt Version:
Salt: 2019.2.2

Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.5.3
docker-py: Not Installed
gitdb: 2.0.0
gitpython: 2.1.1
ioflo: Not Installed
Jinja2: 2.9.4
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.8
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.13 (default, Sep 26 2018, 18:42:22)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 16.0.2
RAET: Not Installed
smmap: 2.0.1
timelib: Not Installed
Tornado: 4.4.3
ZMQ: 4.2.1

System Versions:
dist: debian 9.11
locale: UTF-8
machine: x86_64
release: 4.9.0-11-amd64
system: Linux
version: debian 9.11

[frogunder]

@jeff350 Thank you for reporting this issues. I am seeing the same thing.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[stale]

Thank you for updating this issue. It is no longer marked as stale.

[rnickle]

I have what looks like a variant case, and Jeff pointed me to this case. I'm willing to open another issue, but I thought I'd add my findings.

My keys were all accepted before I created this nodegroup.

I did a full cache flush using the method Whytewolf suggested here: https://github.com/whytewolf/salt-phase0-orch/blob/master/orch/sys/salt/update.sls

My node group of the form:

group1: L@minion1, minion2, minion3

Works

My node group of the form:

group2:

• minion1
• minion2
• minion3

Does not work

My node group of the form with boolean or works:

group3:

• minion1
• 'or'
• minion2
• 'or'
• minion3

My node group in list form with L@ works:

group4:

• L@minion1,minion2,minion3

Salt Version:
Salt: 3003

Dependency Versions:
cffi: 1.13.2
cherrypy: 8.9.1
dateutil: 2.7.3
docker-py: Not Installed
gitdb: 2.0.5
gitpython: 2.1.11
Jinja2: 2.10
libgit2: 1.1.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 0.5.6
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.19
pycrypto: 3.9.4
pycryptodome: 3.6.1
pygit2: 1.6.1
Python: 3.7.3 (default, Apr 3 2019, 05:39:12)
python-gnupg: Not Installed
PyYAML: 3.13
PyZMQ: 17.1.2
smmap: 2.0.5
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.1

System Versions:
dist: debian 10 buster
locale: UTF-8
machine: x86_64
release: 4.19.0-6-amd64
system: Linux
version: Debian GNU/Linux 10 buster

[mpounsett]

I can confirm the same behaviour as @rnickle in 3005.1

Salt Version:
          Salt: 3005.1

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.0
       libgit2: 1.1.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: 1.6.1
        Python: 3.9.16 (main, Jan  6 2023, 22:49:58)
  python-gnupg: 0.4.8
        PyYAML: 5.4.1
         PyZMQ: 23.2.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: debian 11 bullseye
        locale: utf-8
       machine: x86_64
       release: 5.10.0-21-amd64
        system: Linux
       version: Debian GNU/Linux 11 bullseye

[oeko2002]

a fix could be in the file https://github.com/saltstack/salt/blob/develop/salt/pillar/nodegroups.py
add
from salt.utils.minions import nodegroup_comp

and after following line:
for nodegroup_name in six.iterkeys(all_nodegroups):

put in:
all_nodegroups[nodegroup_name] = nodegroup_comp(nodegroup_name, all_nodegroups)

The nodegroup_comp transforms a list of hosts to L@ syntax (and regex strings to 'E@')