Improve performance when loading a given module with numerous alternative versions or non-modulefile

[xdelaruelle]

As a follow-up to a discussion on the modules-interest mailing-list, bad performance is observed on module load when targeting a qualified name/version module if the modpath/name directory contains:

• numerous version files (400+)
• large amount of non-modulefile

The ancient 3.2 version of Modules was not affected by such performance issue as only the specified modulefile were analyzed. Newer version of modules evaluate all files within modpath/name directory to correctly fetch all symbols applying to the specified modulefile.

Introducing an option to avoid looking at other files next to specified modulefile is interesting to restore performance similar to version 3.2 on setup where modulefiles are mixed with a large amount of versions for same module and non-modulefiles.

[xdelaruelle]

Additional ideas come up on the mailing-list regarding this issue: new modulefile commands may be introduced to define the path pattern to ignore or to only take into account.

A new modulepath-ignore command may take a list of file path pattern as arguments (1..N). When walking down the content of modulepath directory, entries that match any of these patterns are ignored (files are not evaluated to check if they are modulefiles, directories are not searched down to find other modulefiles)

In the issue described on the mailing-list, modulefiles are a few files among large amount of other kind of files (installed software files). In this case another command may be useful, modulepath-only, to define a list of path pattern (1..N) to only take into account to check for modulefiles. When modulepath-only is used, patterns defined via modulepath-ignore are ignored as the logic is not to accept all ignore some but to ignore all accept some.

Preferably, such commands should be set in the top .modulerc of modulepath, as it determines how to walk through its content. If used in a .modulerc file found deeper, current directory will be added to the pattern. For instance, if /path/to/modpath/modname/.modulerc defines foo* pattern to ignore, it will be converted into /path/to/modpath/modname/foo*.

If user mentions a file that matches a modulepath-ignore pattern or does not match modulepath-only pattern when set:

• it should get a file path not found error
• such files should be excluded from generated .modulecache

[zerothi]

I really like where this is heading. My main hurdle of using Lmod is that it's so slow compared to env-modules. If the overhead gets the same in both its would be really annoying IMHO.

Another approach could be to build the names of the modules, only based on file-names. Then when loading, only those files that get loaded are actually sourced.

[xdelaruelle]

One use case for the ignore/only feature:

The use case I was hoping to address with the new feature is the following: For the sake of modularity we have some files in module directories that are e.g. read by modulefiles but which are not valid modulefiles themselves. Currently these are hidden files (i.e. their name starting with a dot, e.g. '.some_function.tcl') so that the module command does not 'see' them.

When managing the module directory tree these hidden files sometimes cause confusion. If we were able to define a modulefile name filter (e.g. ignore all *.tcl files) we could make such files regular files w/o any impact on the traversing of the modulepath. Hence the idea of using the gitignore syntax as this would enable ignoring both (classes of) files as well as directories. Or explicitly including them via negation.

Additional design information:

• multiple pattern may be specified on one command and command may be defined several times
• this will give a list of all pattern to ignore (or only include)
• as soon as a "modulepath-only" command is set, "modulepath-ignore" patterns are all ignored
• patterns are prefixed by the location of the modulerc where they are defined
  • for instance "modulepath-ignore 1*" in "modpath/foo/.modulerc" produces the "modpath/foo/1*" ignore pattern
• findModulesFromDirsAndFiles procedure will be updated to check each element found in a directory to determine if it should be taken into account or ignored

[xdelaruelle]

Another feature idea comes up on the mailing-list: defining the file extension of the Tcl modulefile to take into account. Like if .tcl is set, only *.tcl are treated as modulefiles. It may also replace the header file (i.e., #%Module) check.

[xdelaruelle]

At some point this issue may be refined/splited as 3 different feature ideas are expressed:

• optimize direct access to a version file
• new modulerc command to define files to ignore or only take into account
• set a file extension for Tcl modulefile to look at

[xdelaruelle]

Additional benchmark information:

$ ./timeit.sh
3.2.10 large: 0.080s
3.2.10 small: 0.064s
5.5.0 large, no cache: 0.583s (7.2x longer)
5.5.0 large, with cache: 1.985s (24x longer)
5.5.0 small, no cache: 0.348s (5.4x longer)
5.5.0 small, with cache: 1.895s (30x longer)

Test details:
~10000 modulefiles spanning 720 packages.
The .modulecache file is 7.3MB in size.
The "large" package has 432 modulefile versions.
The "small" package has 32 modulefile versions.
Module tree only contains valid modulefiles
"modulecmd load package/x.y.z" is run 10 times (where x.y.z is a fully resolved version)

Such use case may be improved by working on the optimize direct access to a version file part

[cdorbell2]

do you try your test on a nfs mount or a local disk ?
the cache access and depth cache can generate some problem.

[xdelaruelle]

@cdorbell2 This is an input received on the mailing-list. I am able to reproduce somehow this performance on a local SSD as described on #564 (but not the exactly the same magnitude).