Compress modulefile/modulerc content within modulecache

[xdelaruelle]

Based on the same situation than described on #XXX with modulepath containing 10 thousands of modulefiles, which produces a cache file of about 7MB.

It may be interesting to see if compressing the content of every files saved in the cache file could help. It could reduce the size of the overall cache file but requires to uncompress data when needing to read modulefile.

An option could be added to configure if modulefile and modulerc content should be compressed when saved to cache to improve cache efficiency.

Prior implementing such option, it should be validated that compressing content is benefical.

[xdelaruelle]

I have made a test setup composed of 12450 modulefiles in a single modulepath. Following modulefile template is used for all modulefiles to get 2Kb files:

#%Module

proc ModulesHelp {} {
    puts stderr {Lorem ipsum dolor sit amet, consectetur adipiscing elit.}
    puts stderr {Sed non risus. Suspendisse lectus tortor, dignissim sit amet,}
    puts stderr {adipiscing nec, ultricies sed, dolor. Cras elementum ultrices diam.}
    puts stderr {Maecenas ligula massa, varius a, semper congue, euismod non, mi.}
    puts stderr {Proin porttitor, orci nec nonummy molestie, enim est eleifend mi,}
    puts stderr {non fermentum diam nisl sit amet erat. Duis semper. Duis arcu massa,}
    puts stderr {scelerisque vitae, consequat in, }
    puts stderr {Lorem ipsum dolor sit amet, consectetur adipiscing elit.}
    puts stderr {Sed non risus. Suspendisse lectus tortor, dignissim sit amet,}
    puts stderr {adipiscing nec, ultricies sed, dolor. Cras elementum ultrices diam.}
    puts stderr {Maecenas ligula massa, varius a, semper congue, euismod non, mi.}
    puts stderr {Proin porttitor, orci nec nonummy molestie, enim est eleifend mi,}
    puts stderr {non fermentum diam nisl sit amet erat. Duis semper. Duis arcu massa,}
    puts stderr {scelerisque vitae, consequat in, }
    puts stderr {Lorem ipsum dolor sit amet, consectetur adipiscing elit.}
    puts stderr {Sed non risus. Suspendisse lectus tortor, dignissim sit amet,}
    puts stderr {adipiscing nec, ultricies sed, dolor. Cras elementum ultrices diam.}
    puts stderr {Maecenas ligula massa, varius a, semper congue, euismod non, mi.}
    puts stderr {Proin porttitor, orci nec nonummy molestie, enim est eleifend mi,}
    puts stderr {non fermentum diam nisl sit amet erat. Duis semper. Duis arcu massa,}
    puts stderr {scelerisque vitae, consequat in, }
}

# comment code of the modulefile
module-whatis {returns all various module-info values}

# comment code of the modulefile
setenv FOO value
append-path BAR /path/to/directory
setenv QUX {Lorem ipsum dolor sit amet, consectetur adipiscing elit.}
setenv FOO2 value
append-path BAR2 /path/to/directory
setenv QUX2 {Lorem ipsum dolor sit amet, consectetur adipiscing elit.}

The module tree (mp) is generated the following way:

mkdir mp
for i in {001..249}; do
    mkdir mp/m$i;
    for j in {00..49};  do
        cp template mp/m$i/$j
    done
done

With regular cache, we end up with a 25MB .modulecache file.

We obtain the following performances on a module --timer show m072/15:

• with cache: ~430ms
• cache ignored: ~90ms

Modulefile content compression in cache may be implemented using zlib compress/decompress Tcl commands.

With such compression enabled, we get a .modulecache file of 8MB.

We obtain the following performances on the same module show command:

• with cache: ~380ms

We can conclude that zipping modulefile content help a bit but it does not solve the performance issue.

[xdelaruelle]

For the record, following code has been used to implement the content compression in modulecache:

diff --git a/tcl/cache.tcl.in b/tcl/cache.tcl.in
index dee18df7..3ed90bab 100644
--- a/tcl/cache.tcl.in
+++ b/tcl/cache.tcl.in
@@ -174,7 +174,7 @@ proc formatModuleCacheContent {modpath} {
                fheader]} {
                set fheader {}
             }
-            lappend entry_list $fheader $fcontent
+            lappend entry_list $fheader [zlib compress $fcontent]
          } errMsg]} {
             # rethrow read error after parsing message
             knerror [parseAccessIssue $modpath/$elt]
@@ -328,7 +328,7 @@ proc modulefile-content {mod mtime header content} {
    set modfile [currentState modulepath]/$mod
 
    # record modulefile information in memory structures
-   set ::g_modfileContent($modfile) [list $header $content]
+   set ::g_modfileZContent($modfile) [list $header $content]
    set ::g_fileMtime($modfile) $mtime
    set ::g_modfileValid($modfile) [list true {}]
 
@@ -343,7 +343,7 @@ proc modulerc-content {modrc header content} {
    set modrcfile [currentState modulepath]/$modrc
 
    # record modulerc information in memory structures
-   set ::g_modfileContent($modrcfile) [list $header $content]
+   set ::g_modfileZContent($modrcfile) [list $header $content]
 
    # gather all modulefiles/modulercs of a modulepath in a global variable
    # with same information structure than findModules result
diff --git a/tcl/modfind.tcl.in b/tcl/modfind.tcl.in
index 8dcec01e..e8d03573 100644
--- a/tcl/modfind.tcl.in
+++ b/tcl/modfind.tcl.in
@@ -2546,6 +2546,10 @@ proc getModuleValidy {modfile} {
 proc getModuleHeaderAndContent {modfile {only_get_header 0}} {
    if {[info exists ::g_modfileContent($modfile)]} {
       set res $::g_modfileContent($modfile)
+   } elseif {[info exists ::g_modfileZContent($modfile)]} {
+      lassign $::g_modfileZContent($modfile) header zcontent
+      set res [list $header [zlib decompress $zcontent]]
+      set ::g_modfileContent($modfile) $res
    } else {
       # only read beginning of file if just checking validity and not asked to
       # always fully read files. when full file should be read cookie is