2DAlphabet test run fails

[AFJohan92]

Hi @lcorcodilos , on the test run that is written on the setup page, there are some errors.
I am continuing to investigate, but first it seems to work on cmslpc but not on lxplus

On the LPC it runs all the way through but those these errors when running:
Importing data_obs_fail(pass)_LOW(/HIGH/SIG)_lin100kv10k from data_obs, fail_LOW, nominal, RDH
[#1] INFO:ObjectHandling -- RooWorkspace::import(w_lin100kv10k) importing dataset data_obs_fail_LOW_lin100kv10k
[#0] ERROR:InputArguments -- RooWorkspace::import(w_lin100kv10k) ERROR: unrecognized command: RecycleConflictNodes
[#0] ERROR:InputArguments -- RooWorkspace::import(w_lin100kv10k) ERROR: unrecognized command: Silence
Then at the end it says this:
Exception AttributeError: "TwoDAlphabet instance has no attribute 'workspace'" in <bound method TwoDAlphabet.del of <TwoDAlphabetClass.TwoDAlphabet instance at 0x7f164417a710>> ignored

On lxplus, the first set of errors are also thrown, but also it crashes with this:
cd test
cd test
Executing combineCards.py --X-no-jmax lin100kv10k/card_lin100kv10k.txt > card_test.txt
cd test
Executing: combine -M FitDiagnostics -d card_test.txt --setParameters r=1 --saveWorkspace --cminDefaultMinimizerStrategy 0 --rMin 0 --rMax 5 -v 0
<<< Combine >>>

method used is FitDiagnostics
random number generator seed is 123456
Missing background ModelConfig 'ModelConfig_bonly' in workspace 'w' in file roostats-pkRYZU.root
Will make one from the signal ModelConfig 'ModelConfig' setting signal strenth 'r' to zero
Set Default Value of Parameter r To : 1
SimNLL created with 0 channels, 0 generic constraints, 3 fast gaussian constraints, 0 fast poisson constraints, 0 fast group constraints,
SimNLL created with 6 channels, 0 generic constraints, 3 fast gaussian constraints, 0 fast poisson constraints, 0 fast group constraints,

--- FitDiagnostics ---
Best fit r: 4.44961e-07 -4.44961e-07/+0.0720774 (68% CL)
Done in 0.04 min (cpu), 0.04 min (real)
Executing: python $CMSSW_BASE/src/HiggsAnalysis/CombinedLimit/test/diffNuisances.py fitDiagnostics.root --abs -g nuisance_pulls.root
diffNuisances run on fitDiagnostics.root, at 2020-09-22 13:31:03.338463 with the following options ... {'absolute_values': True, 'show_all_parameters': False, 'format': 'text', 'stol2': 0.5, 'pullDef': '', 'vtol': 0.3, 'stol': 0.1, 'plotfile': 'nuisance_pulls.root', 'skipFitB': False, 'vtol2': 2.0, 'poi': 'r', 'skipFitS': False}

name pre fit b-only fit s+b fit rho
shape 0.000000 +/- 1.000000 +0.00 +/- 0.99 (+0.00sig, 0.99) +0.36 +/- 1.00 (+0.36sig, 1.00) +0.08
Executing: python $CMSSW_BASE/src/HiggsAnalysis/CombinedLimit/test/systematicsAnalyzer.py card_test.txt --all -f html > systematics_table.html
Info in TCanvas::Print: pdf file nuisance_pulls.pdf has been created

*** Break *** segmentation violation

===========================================================
There was a crash.
This is the entire stack trace of all threads:

Thread 2 (Thread 0x7f8099279700 (LWP 17475)):
#0 0x00007f80ab92bb3b in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1 0x00007f80ab92bbcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2 0x00007f80ab92bc6b in sem_wait

GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3 0x00007f80abc82cf8 in PyThread_acquire_lock (lock=lock
entry=0x125aa30, waitflag=waitflag
entry=1) at Python/thread_pthread.h:324
#4 0x00007f80abc3f286 in PyEval_RestoreThread (tstate=tstate
entry=0x45ecce0) at Python/ceval.c:359
#5 0x00007f80abe7abd4 in floatsleep (secs=) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc700/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc700/external/python/2.7.14-pafccj/python-2.7.14/Modules/timemodule.c:1057
#6 time_sleep (self=, args=) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc700/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc700/external/python/2.7.14-pafccj/python-2.7.14/Modules/timemodule.c:206
#7 0x00007f80abc48e56 in call_function (oparg=, pp_stack=0x7f80992785d0) at Python/ceval.c:4357
#8 PyEval_EvalFrameEx (f=f
entry=0x7f809b69d9b0, throwflag=throwflag
entry=0) at Python/ceval.c:2994
#9 0x00007f80abc49b68 in PyEval_EvalCodeEx (co=, globals=, locals=locals
entry=0x0, args=args
entry=0x7f809abdc828, argcount=, kws=kws
entry=0x7f80abf00068, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3589
#10 0x00007f80abbbfdad in function_call (func=0x7f809b675b18, arg=0x7f809abdc810, kw=0x7f809b6aa5c8) at Objects/funcobject.c:523
#11 0x00007f80abb8ff73 in PyObject_Call (func=func
entry=0x7f809b675b18, arg=arg
entry=0x7f809abdc810, kw=kw
entry=0x7f809b6aa5c8) at Objects/abstract.c:2547
#12 0x00007f80abc42479 in ext_do_call (nk=, na=0, flags=, pp_stack=0x7f8099278848, func=0x7f809b675b18) at Python/ceval.c:4671
#13 PyEval_EvalFrameEx (f=f
entry=0x7f80aacd3770, throwflag=throwflag
entry=0) at Python/ceval.c:3033
#14 0x00007f80abc48437 in fast_function (nk=, na=, n=1, pp_stack=0x7f8099278950, func=) at Python/ceval.c:4442
#15 call_function (oparg=, pp_stack=0x7f8099278950) at Python/ceval.c:4377
#16 PyEval_EvalFrameEx (f=f
entry=0x7f8094000910, throwflag=throwflag
entry=0) at Python/ceval.c:2994
#17 0x00007f80abc48437 in fast_function (nk=, na=, n=1, pp_stack=0x7f8099278a60, func=) at Python/ceval.c:4442
#18 call_function (oparg=, pp_stack=0x7f8099278a60) at Python/ceval.c:4377
#19 PyEval_EvalFrameEx (f=f
entry=0x7f80aacaa750, throwflag=throwflag
entry=0) at Python/ceval.c:2994
#20 0x00007f80abc49b68 in PyEval_EvalCodeEx (co=, globals=, locals=locals
entry=0x0, args=args
entry=0x7f809abdc7a8, argcount=, kws=kws
entry=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3589
#21 0x00007f80abbbfcd9 in function_call (func=0x7f80abdc0398, arg=0x7f809abdc790, kw=0x0) at Objects/funcobject.c:523
#22 0x00007f80abb8ff73 in PyObject_Call (func=func
entry=0x7f80abdc0398, arg=arg
entry=0x7f809abdc790, kw=kw
entry=0x0) at Objects/abstract.c:2547
#23 0x00007f80abb9ed1c in instancemethod_call (func=0x7f80abdc0398, arg=0x7f809abdc790, kw=0x0) at Objects/classobject.c:2600
#24 0x00007f80abb8ff73 in PyObject_Call (func=func
entry=0x7f80aacc4fa0, arg=arg
entry=0x7f80abf00050, kw=) at Objects/abstract.c:2547
#25 0x00007f80abc3f937 in PyEval_CallObjectWithKeywords (func=0x7f80aacc4fa0, arg=0x7f80abf00050, kw=) at Python/ceval.c:4226
#26 0x00007f80abc88ec2 in t_bootstrap (boot_raw=0x46012b0) at ./Modules/threadmodule.c:620
#27 0x00007f80ab925ea5 in start_thread () from /lib64/libpthread.so.0
#28 0x00007f80aaf458dd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f80abf40740 (LWP 17440)):
#0 0x00007f80aaf0c4b9 in waitpid () from /lib64/libc.so.6
#1 0x00007f80aae89f62 in do_system () from /lib64/libc.so.6
#2 0x00007f80aae8a311 in system () from /lib64/libc.so.6
#3 0x00007f80a9684eed in TUnixSystem::StackTrace() () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_14/external/slc7_amd64_gcc700/lib/libCore.so
#4 0x00007f80a9687794 in TUnixSystem::DispatchSignals(ESignals) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_14/external/slc7_amd64_gcc700/lib/libCore.so
#5
#6 0x00007f80a95f62e0 in TObjArray::Expand(int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_14/external/slc7_amd64_gcc700/lib/libCore.so
#7 0x00007f8098675603 in RooAbsArg::ioStreamerPass2Finalize() () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/lcg/root/6.14.09-pafccj5/lib/libRooFitCore.so
#8 0x00007f8098772d33 in RooFitResult::Streamer(TBuffer&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/lcg/root/6.14.09-pafccj5/lib/libRooFitCore.so
#9 0x00007f80a9cc1962 in TKey::ReadObjectAny(TClass const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_14/external/slc7_amd64_gcc700/lib/libRIO.so
#10 0x00007f80a9c6688b in TDirectoryFile::GetObjectChecked(char const*, TClass const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_14/external/slc7_amd64_gcc700/lib/libRIO.so
#11 0x00007f80aac5336c in (anonymous namespace)::TDirectoryFileGet(PyROOT::ObjectProxy*, _object*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/lcg/root/6.14.09-pafccj5/lib/libPyROOT.so
#12 0x00007f80aac61466 in PyROOT::im_call(_object*, _object*, _object*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/lcg/root/6.14.09-pafccj5/lib/libPyROOT.so
#13 0x00007f80abb8ff73 in PyObject_Call (func=func
entry=0x7f8098a75eb0, arg=arg
entry=0x7f80921d8f50, kw=kw
entry=0x0) at Objects/abstract.c:2547
#14 0x00007f80abc4170a in do_call (nk=, na=, pp_stack=0x7ffee14d4e60, func=) at Python/ceval.c:4574
#15 call_function (oparg=, pp_stack=0x7ffee14d4e60) at Python/ceval.c:4379
#16 PyEval_EvalFrameEx (f=f
entry=0x5208980, throwflag=throwflag
entry=0) at Python/ceval.c:2994
#17 0x00007f80abc49b68 in PyEval_EvalCodeEx (co=, globals=, locals=locals
entry=0x0, args=, argcount=, kws=0x104ba58, kwcount=2, defs=0x7f80aad18218, defcount=2, closure=0x0) at Python/ceval.c:3589
#18 0x00007f80abc45e59 in fast_function (nk=2, na=, n=, pp_stack=0x7ffee14d5030, func=) at Python/ceval.c:4452
#19 call_function (oparg=, pp_stack=0x7ffee14d5030) at Python/ceval.c:4377
#20 PyEval_EvalFrameEx (f=f
entry=0x104b8c0, throwflag=throwflag
entry=0) at Python/ceval.c:2994
#21 0x00007f80abc49b68 in PyEval_EvalCodeEx (co=co
entry=0x7f80abe3cd30, globals=globals
entry=0x7f80abed7168, locals=locals
entry=0x7f80abed7168, args=args
entry=0x0, argcount=argcount
entry=0, kws=kws
entry=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3589
#22 0x00007f80abc49d79 in PyEval_EvalCode (co=co
entry=0x7f80abe3cd30, globals=globals
entry=0x7f80abed7168, locals=locals
entry=0x7f80abed7168) at Python/ceval.c:669
#23 0x00007f80abc6c32a in run_mod (arena=0x10c1650, flags=0x7ffee14d528c, locals=0x7f80abed7168, globals=0x7f80abed7168, filename=0x7f80abe3cd30 "\002", mod=) at Python/pythonrun.c:1385
#24 PyRun_FileExFlags (fp=fp
entry=0x104b8a0, filename=filename
entry=0x7ffee14d6f23 "run_MLfit.py", start=start
entry=257, globals=globals
entry=0x7f80abed7168, locals=locals
entry=0x7f80abed7168, closeit=closeit
entry=1, flags=0x7ffee14d528c) at Python/pythonrun.c:1371
#25 0x00007f80abc6e131 in PyRun_SimpleFileExFlags (fp=fp
entry=0x104b8a0, filename=0x7ffee14d6f23 "run_MLfit.py", closeit=closeit
entry=1, flags=flags
entry=0x7ffee14d528c) at Python/pythonrun.c:957
#26 0x00007f80abc6e7a3 in PyRun_AnyFileExFlags (fp=fp
entry=0x104b8a0, filename=, closeit=closeit
entry=1, flags=flags
entry=0x7ffee14d528c) at Python/pythonrun.c:761
#27 0x00007f80abc86d4e in Py_Main (argc=4, argv=) at Modules/main.c:645
#28 0x00007f80aae69555 in __libc_start_main () from /lib64/libc.so.6
#29 0x000000000040066e in _start ()

The lines below might hint at the cause of the crash.
You may get help by asking at the ROOT forum http://root.cern.ch/forum
Only if you are really convinced it is a bug in ROOT then please submit a
report at http://root.cern.ch/bugs Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.

#6 0x00007f80a95f62e0 in TObjArray::Expand(int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_14/external/slc7_amd64_gcc700/lib/libCore.so
#7 0x00007f8098675603 in RooAbsArg::ioStreamerPass2Finalize() () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/lcg/root/6.14.09-pafccj5/lib/libRooFitCore.so
#8 0x00007f8098772d33 in RooFitResult::Streamer(TBuffer&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/lcg/root/6.14.09-pafccj5/lib/libRooFitCore.so
#9 0x00007f80a9cc1962 in TKey::ReadObjectAny(TClass const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_14/external/slc7_amd64_gcc700/lib/libRIO.so
#10 0x00007f80a9c6688b in TDirectoryFile::GetObjectChecked(char const*, TClass const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_14/external/slc7_amd64_gcc700/lib/libRIO.so
#11 0x00007f80aac5336c in (anonymous namespace)::TDirectoryFileGet(PyROOT::ObjectProxy*, _object*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/lcg/root/6.14.09-pafccj5/lib/libPyROOT.so
#12 0x00007f80aac61466 in PyROOT::im_call(_object*, _object*, _object*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/lcg/root/6.14.09-pafccj5/lib/libPyROOT.so
#13 0x00007f80abb8ff73 in PyObject_Call (func=func
entry=0x7f8098a75eb0, arg=arg
entry=0x7f80921d8f50, kw=kw
entry=0x0) at Objects/abstract.c:2547
#14 0x00007f80abc4170a in do_call (nk=, na=, pp_stack=0x7ffee14d4e60, func=) at Python/ceval.c:4574
#15 call_function (oparg=, pp_stack=0x7ffee14d4e60) at Python/ceval.c:4379
#16 PyEval_EvalFrameEx (f=f
entry=0x5208980, throwflag=throwflag
entry=0) at Python/ceval.c:2994
#17 0x00007f80abc49b68 in PyEval_EvalCodeEx (co=, globals=, locals=locals
entry=0x0, args=, argcount=, kws=0x104ba58, kwcount=2, defs=0x7f80aad18218, defcount=2, closure=0x0) at Python/ceval.c:3589
#18 0x00007f80abc45e59 in fast_function (nk=2, na=, n=, pp_stack=0x7ffee14d5030, func=) at Python/ceval.c:4452
#19 call_function (oparg=, pp_stack=0x7ffee14d5030) at Python/ceval.c:4377
#20 PyEval_EvalFrameEx (f=f
entry=0x104b8c0, throwflag=throwflag
entry=0) at Python/ceval.c:2994
#21 0x00007f80abc49b68 in PyEval_EvalCodeEx (co=co
entry=0x7f80abe3cd30, globals=globals
entry=0x7f80abed7168, locals=locals
entry=0x7f80abed7168, args=args
entry=0x0, argcount=argcount
entry=0, kws=kws
entry=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3589
#22 0x00007f80abc49d79 in PyEval_EvalCode (co=co
entry=0x7f80abe3cd30, globals=globals
entry=0x7f80abed7168, locals=locals
entry=0x7f80abed7168) at Python/ceval.c:669
#23 0x00007f80abc6c32a in run_mod (arena=0x10c1650, flags=0x7ffee14d528c, locals=0x7f80abed7168, globals=0x7f80abed7168, filename=0x7f80abe3cd30 "002", mod=) at Python/pythonrun.c:1385
#24 PyRun_FileExFlags (fp=fp
entry=0x104b8a0, filename=filename
entry=0x7ffee14d6f23 "run_MLfit.py", start=start
entry=257, globals=globals
entry=0x7f80abed7168, locals=locals
entry=0x7f80abed7168, closeit=closeit
entry=1, flags=0x7ffee14d528c) at Python/pythonrun.c:1371
#25 0x00007f80abc6e131 in PyRun_SimpleFileExFlags (fp=fp
entry=0x104b8a0, filename=0x7ffee14d6f23 "run_MLfit.py", closeit=closeit
entry=1, flags=flags
entry=0x7ffee14d528c) at Python/pythonrun.c:957
#26 0x00007f80abc6e7a3 in PyRun_AnyFileExFlags (fp=fp
entry=0x104b8a0, filename=, closeit=closeit
entry=1, flags=flags
entry=0x7ffee14d528c) at Python/pythonrun.c:761
#27 0x00007f80abc86d4e in Py_Main (argc=4, argv=) at Modules/main.c:645
#28 0x00007f80aae69555 in __libc_start_main () from /lib64/libc.so.6
#29 0x000000000040066e in _start ()

[AFJohan92]

And then I tried running the exact same command on lxplus again and it worked...
I didn't log out or anything, just ran the same command. Weird...
Still, the first set of errors are there, but it does make plots and runs all the way through

[lcorcodilos]

This is an issue that occasionally pops up. I've actually spent a considerable amount of time in the past trying to debug it but as you found, it's not consistently reproducible so debugging requires one to wait for to it actually break. The good news at least is that if it breaks for students, they should be able to just run the command again for it to work. Since this code won't be used until next week, I can look into it a bit more later in the week or over the weekend if there's time.

[AFJohan92]

Thanks for confirming Lucas. It's good to know it's a known problem. I've had these kinds of bugs before, they're pesky and annoying! Good thing is that rerunning 'fixes' the issue. But it produces everything we need right? It's just getting rid of the error messages?