Unable to install version > 3.6.1 on Ubuntu 18.04 using python 3.7 after module update

[svenboll]

We are using puppetboard for almost 2 years now and switched from vcsrepo to pip installation method a few weeks ago.
But after all i don't see a newer version installed than 3.6.1.
First i figured out that on Ubuntu 18.04 the python version installed is 3.6 by default and with puppetboard version 4 the support for python3.6 was removed.
So i decided to use parameter python_version => '3.7', but nothing happened. We are still on version 3.6.1 of puppetboard.

After all, the venv-python directory did not update to python3.7 as expected and is still on python3.6.

So i removed the entire directory /srv/puppetboard/virtenv-puppetboard
puppet sets up a new virtual env with python3.7 (now as expected), but the version of our visible puppetboard was still running on version 3.6.1 and not as seen in the logs with version 4.2.0

Affected Puppet, Ruby, OS and module versions/distributions

• Puppet: 7.18
• Ruby: 2.5.1
• Distribution: Ubuntu 18.04
• Module version: this=9.0.0 / puppet-python=6.3.0

How to reproduce (e.g Puppet code you use)

Code with vcsrepo installation method (using module version 8.2.0):

class { 'puppetboard':
    default_environment => '*',
    manage_git          => true,
    manage_virtualenv   => true,
    revision            => 'v3.6.1',
  }

  class { 'puppetboard::apache::vhost':
    vhost_name => $facts['fqdn'],
    port       => 80,
  }

After switching to pip installaton method our code looks like (using module version 9.0.0):

  class { 'puppetboard':
    default_environment => '*',
    manage_virtualenv   => true,
    python_version      => '3.7',
  }

  class { 'puppetboard::apache::vhost':
    vhost_name => $facts['fqdn'],
    port       => 80,
  }

What are you seeing

Puppetboard version 3.6.1 with old virtualenv directory including python3.6:

After creating a new virtualenv directory with python3.7 i see the following puppetboard. Still on version 3.6.1 but without the graph of failed, changed or unchanged puppet runs!

Output log

puppet run after removing directory /srv/puppetboard/virtenv-puppetboard :

Notice: /Stage[main]/Puppetboard/Python::Pyvenv[/srv/puppetboard/virtenv-puppetboard]/File[/srv/puppetboard/virtenv-puppetboard]/ensure: created (corrective)
Notice: /Stage[main]/Puppetboard/Python::Pyvenv[/srv/puppetboard/virtenv-puppetboard]/Exec[python_virtualenv_/srv/puppetboard/virtenv-puppetboard]/returns: executed successfully (corrective)
Notice: /Stage[main]/Puppetboard/Python::Pip[puppetboard]/Exec[pip_install_puppetboard]/returns: executed successfully (corrective)
Notice: Applied catalog in 26.99 seconds

root@puppet-dev:/srv/puppetboard/virtenv-puppetboard/lib# ls
python3.7
root@puppet-dev:/srv/puppetboard/virtenv-puppetboard/lib/python3.7/site-packages/puppetboard# cat version.py 
#
# Puppetboard version module
#

__version__ = '4.2.0'

Any additional information you'd like to impart

I don't look at other directories or files relating to apache, wsgi apache module or something else right now.

[gdubicki]

Hi @svenboll, thanks for reporting the issue.

Changing python_version value on an existing installation is not working because last time I checked the puppet-python module did not support changing this version on the fly...

Sorry for the stupid question but did you (or Puppet) restart Apache/Puppetboard after removing the virtualenv and letting Puppet recreate it?

[svenboll]

Hi @gdubicki ,
i've tested restarting apache2 after recreating the virtualenv, but then i get an "500 - Internal Server Error"

[gdubicki]

You should find the root cause of this 500 in apache2 logs. Can you share it please?

[svenboll]

could this be a wsgi update/plugin error?
so, these are the actions and error log outputs:

bash history:
1971 06.12.22 08:04:05 mv virtenv-puppetboard/ virtenv-puppetboard_old/
1976 06.12.22 08:06:14 puppet agent -t --logdest console
1977 06.12.22 08:08:43 systemctl restart apache2.service
1978 06.12.22 08:09:13 cd /srv/puppetboard/virtenv-puppetboard

error.log (from restart at 08:08:43 to 08:14):
[Tue Dec 06 08:14:11.595640 2022] [wsgi:error] ### mod_wsgi (pid=7855): Target WSGI script '/srv/puppetboard/puppetboard/wsgi.py' cannot be loaded as Python module.
[Tue Dec 06 08:14:11.595706 2022] [wsgi:error] ### mod_wsgi (pid=7855): Exception occurred processing WSGI script '/srv/puppetboard/puppetboard/wsgi.py'.
[Tue Dec 06 08:14:11.595791 2022] [wsgi:error] ### Traceback (most recent call last):
[Tue Dec 06 08:14:11.595827 2022] [wsgi:error] ### File "/srv/puppetboard/puppetboard/wsgi.py", line 14, in
[Tue Dec 06 08:14:11.595830 2022] [wsgi:error] ### from puppetboard.app import app as application
[Tue Dec 06 08:14:11.595844 2022] [wsgi:error] ### ModuleNotFoundError: No module named 'puppetboard'

bash history:
1989 06.12.22 08:17:31 rm -rf virtenv-puppetboard
1990 06.12.22 08:17:36 mv virtenv-puppetboard_old/ virtenv-puppetboard/
1991 06.12.22 08:17:46 systemctl restart apache2.service

error.log:
[Tue Dec 06 08:17:50.563835 2022] [wsgi:error] ### INFO:puppetboard.utils:PuppetDB Version 7.11.0

[gdubicki]

Thanks for that info!

I suspect that the puppetboard package did not install for you in the new virtualenv that used Python 3.7, but the error was suppressed and not shown.

You should check the pip.log at /srv/puppetboard/virtenv-puppetboard, of course for the new Python 3.7 virtualenv.

And while most probably it's the puppetboard package itself, you can also compare the whole list of packages with this good working list from one of my instances on a Centos 7 node using Python 3.8:

# . /srv/puppetboard/virtenv-puppetboard/bin/activate
(virtenv-puppetboard) # pip freeze
certifi==2022.6.15
charset-normalizer==2.1.0
click==8.1.3
commonmark==0.9.1
Flask==2.1.3
Flask-WTF==1.0.1
idna==3.3
importlib-metadata==4.12.0
itsdangerous==2.1.2
Jinja2==3.1.2
MarkupSafe==2.1.1
packaging==21.3
puppetboard==4.2.2
pyparsing==3.0.9
pypuppetdb==3.1.0
requests==2.28.1
urllib3==1.26.10
Werkzeug==2.1.2
WTForms==3.0.1
zipp==3.8.1

If any of the packages is missing, even if you don't see any errors in the log, you can try to install them manually while having the virtualenv activated and see what are the results of that.

Please share all the findings!

[svenboll]

So i tested now python 3.7 and python 3.8 venv with the same result of installed packages, and after all i still ended up in "500 - Internal Server Error":

(virtenv-puppetboard) root@puppet-dev:/srv/puppetboard/virtenv-puppetboard/bin# pip freeze
certifi==2022.12.7
charset-normalizer==2.1.1
click==8.1.3
commonmark==0.9.1
Flask==2.2.2
Flask-WTF==1.0.1
idna==3.4
importlib-metadata==4.13.0
itsdangerous==2.1.2
Jinja2==3.1.2
MarkupSafe==2.1.1
packaging==21.3
pkg_resources==0.0.0
puppetboard==4.2.3
pyparsing==3.0.9
pypuppetdb==3.1.0
requests==2.28.1
urllib3==1.26.13
Werkzeug==2.2.2
WTForms==3.0.1
zipp==3.11.0

with both python3.7 and 3.8 virtenv there are some suspicious errors in the pip.log:

2022-12-12T14:29:26,284 Skipping link: not a file: https://pypi.org/simple/puppetboard/
2022-12-12T14:29:26,286 Given no hashes to check 0 links for project 'puppetboard': discarding no candidates
2022-12-12T14:29:26,286 ERROR: Could not find a version that satisfies the requirement puppetboard==notreallyaversion (from versions: 0.0.3, 0.0.4, 0.0.5, 0.1.0, 0.1.1, 0.1.3, 0.2.0, 0.2.1, 0.2.2, 0.3.0, 1.0.0, 1.1.0, 2.0.0, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 3.0.0, 3.0.0.post2, 3.0.0.post3, 3.0.0.post4, 3.1.0rc1, 3.1.0, 3.1.0.post1, 3.1.0.post2, 3.1.0.post3, 3.1.0.post4, 3.2.0rc1, 3.2.0, 3.3.0rc1, 3.3.0, 3.4.0rc1, 3.4.0rc2, 3.4.0rc3, 3.4.0rc4, 3.4.0rc5, 3.4.0rc6, 3.4.0, 3.4.0.post1, 3.4.0.post2, 3.4.1, 3.4.2, 3.5.0rc1, 3.5.0, 3.5.1, 3.6.0, 3.6.1, 4.0.0a1, 4.0.0b1, 4.0.0, 4.0.1, 4.0.2, 4.0.3, 4.0.5, 4.1.0rc1, 4.1.0rc2, 4.1.0rc3, 4.1.0, 4.1.1, 4.1.2, 4.2.0, 4.2.1, 4.2.2, 4.2.3)
2022-12-12T14:29:26,287 ERROR: No matching distribution found for puppetboard==notreallyaversion
2022-12-12T14:29:26,287 Exception information:
2022-12-12T14:29:26,287 Traceback (most recent call last):
2022-12-12T14:29:26,287   File "/srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 348, in resolve
2022-12-12T14:29:26,287     self._add_to_criteria(self.state.criteria, r, parent=None)
2022-12-12T14:29:26,287   File "/srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
2022-12-12T14:29:26,287     raise RequirementsConflicted(criterion)
2022-12-12T14:29:26,287 pip._vendor.resolvelib.resolvers.RequirementsConflicted: Requirements conflict: SpecifierRequirement('puppetboard==notreallyaversion')
2022-12-12T14:29:26,287 
2022-12-12T14:29:26,287 During handling of the above exception, another exception occurred:
2022-12-12T14:29:26,287 
2022-12-12T14:29:26,287 Traceback (most recent call last):
2022-12-12T14:29:26,287   File "/srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
2022-12-12T14:29:26,287     result = self._result = resolver.resolve(
2022-12-12T14:29:26,287   File "/srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
2022-12-12T14:29:26,287     state = resolution.resolve(requirements, max_rounds=max_rounds)
2022-12-12T14:29:26,287   File "/srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 350, in resolve
2022-12-12T14:29:26,287     raise ResolutionImpossible(e.criterion.information)
2022-12-12T14:29:26,287 pip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('puppetboard==notreallyaversion'), parent=None)]
2022-12-12T14:29:26,287 
2022-12-12T14:29:26,287 The above exception was the direct cause of the following exception:
2022-12-12T14:29:26,287 
2022-12-12T14:29:26,287 Traceback (most recent call last):
2022-12-12T14:29:26,287   File "/srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
2022-12-12T14:29:26,287     status = run_func(*args)
2022-12-12T14:29:26,287   File "/srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
2022-12-12T14:29:26,287     return func(self, options, args)
2022-12-12T14:29:26,287   File "/srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 400, in run
2022-12-12T14:29:26,287     requirement_set = resolver.resolve(
2022-12-12T14:29:26,287   File "/srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 101, in resolve
2022-12-12T14:29:26,287     raise error from e
2022-12-12T14:29:26,287 pip._internal.exceptions.DistributionNotFound: No matching distribution found for puppetboard==notreallyaversion
2022-12-12T14:29:26,290 Remote version of pip: 21.3.1
2022-12-12T14:29:26,290 Local version of pip:  22.3.1
2022-12-12T14:29:26,290 Was pip installed by pip? True

then i tried to install puppetboard manually, but with no change:

(virtenv-puppetboard) root@puppet-dev:/srv/puppetboard/virtenv-puppetboard/bin# pip install puppetboard
Requirement already satisfied: puppetboard in /srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages (4.2.3)
Requirement already satisfied: WTForms<4,>=3 in /srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages (from puppetboard) (3.0.1)
Requirement already satisfied: itsdangerous<3,>=2 in /srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages (from puppetboard) (2.1.2)
Requirement already satisfied: MarkupSafe<3,>=2 in /srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages (from puppetboard) (2.1.1)
Requirement already satisfied: Werkzeug<3,>=2 in /srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages (from puppetboard) (2.2.2)
Requirement already satisfied: pypuppetdb<4,>=3.1.0 in /srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages (from puppetboard) (3.1.0)
Requirement already satisfied: packaging==21.3 in /srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages (from puppetboard) (21.3)
Requirement already satisfied: importlib-metadata<5 in /srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages (from puppetboard) (4.13.0)
Requirement already satisfied: Flask<3,>=2 in /srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages (from puppetboard) (2.2.2)
Requirement already satisfied: Flask-WTF<2,>=1 in /srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages (from puppetboard) (1.0.1)
Requirement already satisfied: CommonMark==0.9.1 in /srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages (from puppetboard) (0.9.1)
Requirement already satisfied: Jinja2<4,>=3 in /srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages (from puppetboard) (3.1.2)
Requirement already satisfied: requests<3,>=2.28 in /srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages (from puppetboard) (2.28.1)
Requirement already satisfied: pyparsing==3.0.9 in /srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages (from puppetboard) (3.0.9)
Requirement already satisfied: click>=8.0 in /srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages (from Flask<3,>=2->puppetboard) (8.1.3)
Requirement already satisfied: zipp>=0.5 in /srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages (from importlib-metadata<5->puppetboard) (3.11.0)
Requirement already satisfied: idna<4,>=2.5 in /srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages (from requests<3,>=2.28->puppetboard) (3.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages (from requests<3,>=2.28->puppetboard) (1.26.13)
Requirement already satisfied: certifi>=2017.4.17 in /srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages (from requests<3,>=2.28->puppetboard) (2022.12.7)
Requirement already satisfied: charset-normalizer<3,>=2 in /srv/puppetboard/virtenv-puppetboard/lib/python3.8/site-packages (from requests<3,>=2.28->puppetboard) (2.1.1)

[gdubicki]

The notreallyaversion string comes from the puppet-python module, it's not an issue here.

[gdubicki]

Can you please check the WSGI config in your Apache vhost, @svenboll ?

Maybe for some reason you don't have the virtualenv configured there.

For example I have something like this in my /opt/rh/httpd24/root/etc/httpd/conf.d/25-<redacted>.conf:

  ## WSGI configuration
  WSGIDaemonProcess puppetboard group=puppetboard python-home=/srv/puppetboard/virtenv-puppetboard request-timeout=180 threads=10 user=puppetboard
  WSGIProcessGroup puppetboard
  WSGIScriptAlias / "/srv/puppetboard/puppetboard/wsgi.py"

[svenboll]

it looks the same for me on my apache vhost conf.

i think it could be a similar issue to #369

in my apache2 module folder i can only find these entries related to mod_wsgi, no version for python 3.7 or 3.8 which are the versions i tested. could this be a problem?

/usr/lib/apache2/modules:
lrwxrwxrwx 1 root root        15 Jul 25 15:24 mod_wsgi.so -> mod_wsgi.so-3.6
-rw-r--r-- 1 root root 237168 Jul 25 15:24 mod_wsgi.so-3.6

[gdubicki]

You are totally right, it seems related to that... You should have mod_wsgi for Python 3.7 or 3.8, whatever you are using. I am not sure how can you get it on Ubuntu though.

We are using Centos 7 and Python 3.8 and HTTPD 2.4 from SCL. The HTTP 2.4 is mainly to have a mod_wsgi working with that Python 3.8. We install it using this code:

  class { 'apache::mod::wsgi':
    package_name => 'rh-python38-python-mod_wsgi',
    mod_path     => 'modules/mod_rh-python38-wsgi.so',
  }

I hope that it helps!