Skip to content

Commit 2151c10

Browse files
authored
tests: Cover the current grass.lib loading behavior (#8093)
Add tests which pin the current behavior of grass.lib without changing any behavior. grass.lib can be imported and used with a session set up by init in an environment without the dynamic library search path variable (on Windows, the C libraries do not see the session), a session does not load the C libraries into the process, and the search path variable set before the process starts makes grass.lib work without a session even for an installation moved away from its prefix. The import after init works because the ctypes loader reads the search path variable, which init sets for its subprocesses, from the environment of the running process to find a library, while the dynamic linker resolves the library dependencies through the search path recorded at build time. For a moved installation, the variable is needed before the process starts because the dynamic linker reads it only at startup. The tests construct the environment of their subprocesses themselves. New CI steps run them standalone on Linux, macOS, and Windows without the search path setup done by the other pytest steps, and against a relocated installation on Linux and macOS. The shared subprocess fixtures are in python/conftest.py because a conftest.py in python/grass would make pytest import the grass package from the source tree. Correct the init docstring, which claimed since 2015 that the libraries used through ctypes are not usable after init. It now says that both importing and using grass.lib depend on init, that grass.lib sees only a session in the global environment, and when the import works.
1 parent d913558 commit 2151c10

7 files changed

Lines changed: 359 additions & 4 deletions

File tree

‎.github/workflows/macos.yml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -129,6 +129,36 @@ jobs:
129129
pytest \
130130
@.github/workflows/pytest_args_gunittest.txt \
131131
--junitxml=pytest.gunittest.junit.xml
132+
- name: Run pytest for the grass.lib loading tests standalone
133+
shell: micromamba-shell {0}
134+
run: |
135+
# No library search path variable is exported here: the tests
136+
# provide their own environment, so this pins what works without
137+
# the setup done by the other steps.
138+
PYTHONPATH="$(grass --config python_path)"
139+
export PYTHONPATH
140+
pytest \
141+
python/grass/script/tests/grass_script_setup_lib_loading_test.py \
142+
python/grass/app/tests/grass_app_runtime_lib_loading_test.py \
143+
--junitxml=pytest.lib_loading.junit.xml
144+
- name: Run pytest for the grass.lib loading tests with a relocated installation
145+
shell: micromamba-shell {0}
146+
run: |
147+
# The installation is moved away from the prefix it was configured
148+
# with, so that the libraries cannot find each other through the
149+
# install names recorded at configure time. Only the tests which
150+
# provide the search path variables themselves can pass here. The
151+
# Python package moves with the installation, so its path is taken
152+
# beforehand.
153+
python_path="$(grass --config python_path)"
154+
mv "${HOME}/install" "${HOME}/install-relocated"
155+
trap 'mv "${HOME}/install-relocated" "${HOME}/install"' EXIT
156+
PYTHONPATH="${HOME}/install-relocated${python_path#"${HOME}/install"}"
157+
export PYTHONPATH
158+
pytest \
159+
python/grass/app/tests/grass_app_runtime_lib_loading_test.py \
160+
-k manual \
161+
--junitxml=pytest.lib_loading_relocated.junit.xml
132162
133163
- name: Upload test results to Codecov
134164
if: ${{ !cancelled() }}

‎.github/workflows/osgeo4w.yml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -171,6 +171,19 @@ jobs:
171171
--junitxml=pytest.gunittest.junit.xml
172172
# Obfuscated usage of GitHub Actions features, Windows CMD shell limits analysis
173173
shell: cmd /D /E:ON /V:OFF /S /C "CALL C:/OSGeo4W/OSGeo4W.bat "{0}"" # zizmor: ignore[misfeature]
174+
- name: Run pytest for the grass.lib loading tests standalone
175+
run: |
176+
call %OSGEO4W_ROOT%\opt\grass\etc\env.bat
177+
set PYTHONPATH=%GISBASE%\etc\python;%PYTHONPATH%
178+
REM The library directory is deliberately not added to PATH here:
179+
REM the tests provide their own environment, so this pins what
180+
REM works without the setup done by the other steps.
181+
pytest ^
182+
python/grass/script/tests/grass_script_setup_lib_loading_test.py ^
183+
python/grass/app/tests/grass_app_runtime_lib_loading_test.py ^
184+
--junitxml=pytest.lib_loading.junit.xml
185+
# Obfuscated usage of GitHub Actions features, Windows CMD shell limits analysis
186+
shell: cmd /D /E:ON /V:OFF /S /C "CALL C:/OSGeo4W/OSGeo4W.bat "{0}"" # zizmor: ignore[misfeature]
174187

175188
- name: Upload test results to Codecov
176189
if: ${{ !cancelled() }}

‎.github/workflows/pytest.yml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -139,6 +139,33 @@ jobs:
139139
@.github/workflows/pytest_args_cov.txt \
140140
@.github/workflows/pytest_args_gunittest.txt \
141141
--junitxml=pytest.gunittest.junit.xml
142+
- name: Run pytest for the grass.lib loading tests standalone
143+
run: |
144+
# No library search path variable is exported here: the tests
145+
# provide their own environment, so this pins what works without
146+
# the setup done by the other steps.
147+
PYTHONPATH="$(grass --config python_path)"
148+
export PYTHONPATH
149+
pytest \
150+
python/grass/script/tests/grass_script_setup_lib_loading_test.py \
151+
python/grass/app/tests/grass_app_runtime_lib_loading_test.py \
152+
--junitxml=pytest.lib_loading.junit.xml
153+
- name: Run pytest for the grass.lib loading tests with a relocated installation
154+
run: |
155+
# The installation is moved away from the prefix it was configured
156+
# with, so that the libraries cannot find each other through their
157+
# RUNPATH. Only the tests which provide the search path variable
158+
# themselves can pass here. The Python package moves with the
159+
# installation, so its path is taken beforehand.
160+
python_path="$(grass --config python_path)"
161+
mv "${HOME}/install" "${HOME}/install-relocated"
162+
trap 'mv "${HOME}/install-relocated" "${HOME}/install"' EXIT
163+
PYTHONPATH="${HOME}/install-relocated${python_path#"${HOME}/install"}"
164+
export PYTHONPATH
165+
pytest \
166+
python/grass/app/tests/grass_app_runtime_lib_loading_test.py \
167+
-k manual \
168+
--junitxml=pytest.lib_loading_relocated.junit.xml
142169
- name: Fix non-standard installed script paths in coverage data
143170
run: |
144171
PYTHONPATH="$(grass --config python_path):${PYTHONPATH}"

‎python/conftest.py‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
# SPDX-FileCopyrightText: 2026 Vaclav Petras
2+
# SPDX-FileCopyrightText: GRASS Development Team
3+
# SPDX-License-Identifier: GPL-2.0-or-later
4+
"""Fixtures shared by tests of the Python packages
5+
6+
This file is in the python directory rather than in python/grass because
7+
python/grass is a package: pytest would import a conftest.py there as
8+
grass.conftest, and with it the grass package from the source tree instead of
9+
the installed one.
10+
"""
11+
12+
import json
13+
import os
14+
import subprocess
15+
import sys
16+
from pathlib import Path
17+
from textwrap import dedent
18+
19+
import pytest
20+
21+
22+
def _contains_grass_core_library(directory):
23+
try:
24+
return any(
25+
path.name.startswith(("libgrass_gis.", "grass_gis."))
26+
for path in Path(directory).iterdir()
27+
)
28+
except OSError:
29+
return False
30+
31+
32+
@pytest.fixture
33+
def clean_subprocess_environment():
34+
"""Environment without a session and without the dynamic library search paths"""
35+
env = os.environ.copy()
36+
for variable in (
37+
"LD_LIBRARY_PATH",
38+
"DYLD_LIBRARY_PATH",
39+
"LD_RUN_PATH",
40+
"GISRC",
41+
"GIS_LOCK",
42+
"GISBASE",
43+
):
44+
env.pop(variable, None)
45+
if sys.platform.startswith("win"):
46+
# On Windows, the search path variable is PATH, so only the
47+
# directories with the GRASS libraries are removed.
48+
env["PATH"] = os.pathsep.join(
49+
directory
50+
for directory in env.get("PATH", "").split(os.pathsep)
51+
if not _contains_grass_core_library(directory)
52+
)
53+
return env
54+
55+
56+
@pytest.fixture
57+
def run_code_in_environment(tmp_path):
58+
"""Run Python code in a subprocess with the given environment
59+
60+
Asserts a zero exit code and returns the parsed JSON from stdout.
61+
"""
62+
63+
def run(code, env):
64+
source_file = tmp_path / "subprocess_code.py"
65+
source_file.write_text(dedent(code))
66+
result = subprocess.run(
67+
[sys.executable, os.fspath(source_file)],
68+
capture_output=True,
69+
text=True,
70+
check=False,
71+
env=env,
72+
)
73+
assert result.returncode == 0, (
74+
result.stderr
75+
or f"Exit code {result.returncode} without any output (a fatal"
76+
" error in the GRASS C libraries ends the process without a"
77+
" traceback)"
78+
)
79+
return json.loads(result.stdout)
80+
81+
return run
Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# SPDX-FileCopyrightText: 2026 Vaclav Petras
2+
# SPDX-FileCopyrightText: GRASS Development Team
3+
# SPDX-License-Identifier: GPL-2.0-or-later
4+
"""Tests of making the GRASS C libraries (grass.lib) work without a session
5+
6+
This file is meant to be executed also standalone (pytest with this file as
7+
the only argument), including against an installation moved away from the
8+
prefix it was configured with: the tests provide any needed environment
9+
themselves instead of relying on the environment of the test process.
10+
"""
11+
12+
import os
13+
import sys
14+
from pathlib import Path
15+
16+
from grass.app.runtime import RuntimePaths
17+
from grass.script.setup import get_install_path
18+
19+
20+
def test_grass_lib_importable_with_manual_search_path(
21+
run_code_in_environment, clean_subprocess_environment
22+
):
23+
"""Check that the search path variable set at process start is enough"""
24+
lib_path = os.fspath(Path(get_install_path()) / "lib")
25+
env = clean_subprocess_environment
26+
variables = [RuntimePaths().ld_library_path_var]
27+
if sys.platform == "darwin":
28+
# The variable GRASS knows on macOS (LD_RUN_PATH) is read by the
29+
# ctypes loader but ignored by dyld, which reads DYLD_LIBRARY_PATH.
30+
variables.append("DYLD_LIBRARY_PATH")
31+
for variable in variables:
32+
if env.get(variable):
33+
env[variable] = lib_path + os.pathsep + env[variable]
34+
else:
35+
env[variable] = lib_path
36+
code = """
37+
import json
38+
39+
import grass.lib.gis
40+
import grass.lib.raster
41+
42+
print(json.dumps({"imported": True}))
43+
"""
44+
assert run_code_in_environment(code, env=env)["imported"]

‎python/grass/script/setup.py‎

Lines changed: 17 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -313,10 +313,23 @@ def init(
313313
standard main executable grass. No GRASS modules shall be called before
314314
call of this function but any module or user script can be called
315315
afterwards because a GRASS session has been set up. GRASS Python
316-
libraries are usable as well in general but the ones using C
317-
libraries through ``ctypes`` are not (which is caused by library
318-
path not being updated for the current process which is a common
319-
operating system limitation).
316+
libraries are usable as well, including the ones using C libraries
317+
through ``ctypes`` (:mod:`grass.lib`). Importing :mod:`grass.lib`
318+
depends on this function, and so does using it: the C libraries read
319+
the session from the environment of the process, so :mod:`grass.lib`
320+
sees the session only when it is set up in the global environment,
321+
i.e., without *env*. The import works for an installation at the
322+
prefix it was configured for, where the GRASS libraries find the
323+
libraries they depend on through the search path recorded at build
324+
time. For an installation moved elsewhere, the dynamic library search
325+
path variable must be set before the process starts, because the
326+
operating system does not apply a modified variable to an already
327+
running process.
328+
329+
On Windows, the C libraries do not see environment variables set in
330+
the running process, so :mod:`grass.lib` functions see a session set
331+
up by this function only in processes started afterwards, such as
332+
subprocesses.
320333
321334
When the path or specified mapset does not exist, ValueError is raised.
322335
Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
# SPDX-FileCopyrightText: 2026 Vaclav Petras
2+
# SPDX-FileCopyrightText: GRASS Development Team
3+
# SPDX-License-Identifier: GPL-2.0-or-later
4+
"""Tests of using grass.lib in a session set up by grass.script.setup.init
5+
6+
This file is meant to be executed also standalone (pytest with this file as
7+
the only argument). The tests run their code in a subprocess without the
8+
dynamic library search path variable, so finding the libraries depends only
9+
on the variables init sets in the environment of the running process and on
10+
the search path recorded in the libraries at build time.
11+
"""
12+
13+
import sys
14+
from textwrap import dedent
15+
16+
import pytest
17+
18+
WINDOWS = sys.platform.startswith("win")
19+
20+
21+
@pytest.mark.usefixtures("mock_no_session")
22+
def test_grass_lib_importable_after_init(
23+
tmp_path, run_code_in_environment, clean_subprocess_environment
24+
):
25+
"""Check that grass.lib modules can be imported after init"""
26+
project = tmp_path / "test"
27+
code = f"""
28+
import json
29+
import grass.script as gs
30+
31+
gs.create_project(r"{project}")
32+
with gs.setup.init(r"{project}"):
33+
import grass.lib.gis
34+
import grass.lib.raster
35+
print(json.dumps({{"imported": True}}))
36+
"""
37+
assert run_code_in_environment(code, env=clean_subprocess_environment)["imported"]
38+
39+
40+
@pytest.mark.xfail(
41+
WINDOWS,
42+
strict=True,
43+
reason="On Windows, the C libraries (built with a different C runtime than"
44+
" Python) do not see the environment variables set by init in the running"
45+
" process, so they do not see the session",
46+
)
47+
@pytest.mark.usefixtures("mock_no_session")
48+
def test_grass_lib_usable_after_init(
49+
tmp_path, run_code_in_environment, clean_subprocess_environment
50+
):
51+
"""Check that grass.lib sees the session and works with rasters after init"""
52+
project = tmp_path / "test"
53+
code = f"""
54+
import json
55+
import grass.script as gs
56+
57+
gs.create_project(r"{project}")
58+
with gs.setup.init(r"{project}"):
59+
import grass.lib.gis as libgis
60+
import grass.lib.raster as libraster
61+
62+
libgis.G_gisinit(b"test")
63+
gs.run_command("g.region", rows=2, cols=2)
64+
gs.mapcalc("ones = 1")
65+
fd = libraster.Rast_open_old(b"ones", b"")
66+
libraster.Rast_close(fd)
67+
print(json.dumps({{"raster_opened": True}}))
68+
"""
69+
assert run_code_in_environment(code, env=clean_subprocess_environment)[
70+
"raster_opened"
71+
]
72+
73+
74+
@pytest.mark.usefixtures("mock_no_session")
75+
def test_grass_lib_usable_in_subprocess_of_init(
76+
tmp_path, run_code_in_environment, clean_subprocess_environment
77+
):
78+
"""Check that grass.lib works in a process started from a session of init
79+
80+
The session exists in the environment of the started process from its
81+
start, so the C libraries see it on all platforms, including Windows,
82+
where they do not see a session set up in their own process.
83+
"""
84+
project = tmp_path / "test"
85+
child_file = tmp_path / "child.py"
86+
child_file.write_text(
87+
dedent(
88+
"""
89+
import json
90+
91+
import grass.lib.gis as libgis
92+
import grass.lib.raster as libraster
93+
94+
libgis.G_gisinit(b"test")
95+
fd = libraster.Rast_open_old(b"ones", b"")
96+
libraster.Rast_close(fd)
97+
print(json.dumps({"raster_opened": True}))
98+
"""
99+
)
100+
)
101+
code = f"""
102+
import subprocess
103+
import sys
104+
105+
import grass.script as gs
106+
107+
gs.create_project(r"{project}")
108+
with gs.setup.init(r"{project}"):
109+
gs.run_command("g.region", rows=2, cols=2)
110+
gs.mapcalc("ones = 1")
111+
result = subprocess.run(
112+
[sys.executable, r"{child_file}"],
113+
capture_output=True,
114+
text=True,
115+
check=True,
116+
)
117+
print(result.stdout)
118+
"""
119+
assert run_code_in_environment(code, env=clean_subprocess_environment)[
120+
"raster_opened"
121+
]
122+
123+
124+
@pytest.mark.skipif(sys.platform != "linux", reason="Reads /proc/self/maps")
125+
@pytest.mark.usefixtures("mock_no_session")
126+
def test_session_does_not_load_libraries(
127+
tmp_path, run_code_in_environment, clean_subprocess_environment
128+
):
129+
"""Check that a session does not load the C libraries into the process
130+
131+
The check reads /proc/self/maps, hence it runs only on Linux.
132+
"""
133+
project = tmp_path / "test"
134+
code = f"""
135+
import json
136+
import grass.script as gs
137+
138+
gs.create_project(r"{project}")
139+
with gs.setup.init(r"{project}"):
140+
pass
141+
with open("/proc/self/maps") as maps:
142+
mapped = [line for line in maps if "libgrass_" in line]
143+
print(json.dumps({{"mapped": mapped}}))
144+
"""
145+
assert (
146+
run_code_in_environment(code, env=clean_subprocess_environment)["mapped"] == []
147+
)

0 commit comments

Comments
 (0)