Segmentation fault (core dumped) and simulation stops executing

[jannissimo]

Hello together,

This is my first question in github, so I apologize if I am not following the usual guidelines for asking questions.
I want to simulate a directed energy deposition process and want to use the results for model order reduction for a scientific publication.

To become familiar with the program, I first wanted to simulate the hollow_cylinder example. I am interested in the evolving residual stress field, so I run the "master_app_mechanical.i" input file.

Command in the terminal:

~/moose_projects/malamute$ ./malamute-opt -i ./examples/directed_energy_deposition/hollow_cylinder/master_app_mechanical.i > logfile.log 2> logfile_err.log &

On two seperate hardware systems I get always after the 13th time step the following error error:

[1]+ Segmentation fault (core dumped) ./malamute-opt -i ./examples/...

Desktop-PC Information (Linux)

Model: Hewlett-Packard HP Z440 Workstation
Memory: 32.0 GiB
Processor: Intel Xeon E5-1630 v3 x 8
Graphics: AMD CAICOS
Disk Capacity: 512.1 GB (200GB free)
OS Name: Ubuntu 24.04.2 LTS
OS Type: 64-bit
Kernel Version: Linux 6.11.0-17-generic

Laptop-PC Information (WSL Subsystem for Windows)

Memory: 16 GiB
Processor: Intel Core i7-1370P

In my error file "logfile_err.log" there is no error.
If I run only the "sub_app_thermal.i" input file the simulation runs normal and the .e files have all been created.

Does anyone have any suggestions for solving the problem?

Thank you very much.

[jannissimo]

UPDATE: The following message comes from the gdb debugger after timestep 13

Assertion `find_side != find_elem->second.end()' failed
Material property does not have side entry
at /home/users/jku/moose_projects/malamute/moose/framework/build/header_symlinks/MaterialPropertyStorage.h, line 465
Stack frames: 16
0: libMesh::print_trace(std::ostream&)
1: /home/users/jku/moose_projects/malamute/moose/framework/libmoose-dbg.so.0(+0x2cac3e9) [0x7ffff61323e9]
2: MaterialPropertyStorage::prolongStatefulProps(unsigned int, std::vector<std::vector<QpMap, std::allocator >, std::allocator<std::vector<QpMap, std::allocator > > > const&, libMesh::QBase const&, libMesh::QBase const&, MaterialPropertyStorage&, unsigned int, libMesh::Elem const&, int, int, int)
3: ProjectMaterialProperties::onElement(libMesh::Elem const*)
4: ThreadedElementLoopBase<libMesh::StoredRange<__gnu_cxx::__normal_iterator<libMesh::Elem const**, std::vector<libMesh::Elem const*, std::allocator<libMesh::Elem const*> > >, libMesh::Elem const*> >::operator()(libMesh::StoredRange<__gnu_cxx::__normal_iterator<libMesh::Elem const**, std::vector<libMesh::Elem const*, std::allocator<libMesh::Elem const*> > >, libMesh::Elem const*> const&, bool)
5: /home/users/jku/moose_projects/malamute/moose/framework/libmoose-dbg.so.0(+0x1769d49) [0x7ffff4befd49]
6: FEProblemBase::meshChangedHelper(bool)
7: FEProblemBase::adaptMesh()
8: TransientBase::incrementStepOrReject()
9: TransientBase::execute()
10: MooseApp::executeExecutioner()
11: MooseApp::run()
12: /home/users/jku/moose_projects/malamute/malamute-dbg(+0x7d8c) [0x55555555bd8c]
13: main
14: __libc_start_main
15: /home/users/jku/moose_projects/malamute/malamute-dbg(+0x6079) [0x55555555a079]
[0] /home/users/jku/moose_projects/malamute/moose/framework/build/header_symlinks/MaterialPropertyStorage.h, line 465, compiled Mar 3 2025 at 10:48:50
Abort(1) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
[Thread 0x7fffda632700 (LWP 57684) exited]
[Thread 0x7fffdae95700 (LWP 57683) exited]
[Inferior 1 (process 57676) exited with code 01]

[cticenhour]

Hello @jannissimo - thank you for your question and for adding the extra debug output information.

I want to loop @dewenyushu in to this conversation, as she created this example, and might have a quick answer as to what is going on.

[benediktgh]

Hi, I am facing the same problem and would appreciate any help. @dewenyushu have you had a chance to look into this issue? Thanks in advance!

[whitmanjack]

I'm also facing the exact same issue. The solver stops with the same error as mentioned by @jannissimo after 13th time step.
Could you please provide any update to this please. @cticenhour @dewenyushu

[keassyguang]

This problem seems to be caused by moose, due to the stateful material projection when using adaptive mesh refinement at the moving boundaries. See idaholab/moose#30274. If anyone solves this problem, please let me know. Many thanks! @cticenhour @jannissimo @dewenyushu @whitmanjack

[KamalnathOSU]

Hi @cticenhour @dewenyushu
I am also facing the same issue on the hollow_cylinder example. Do you have any update ? I get segmentation fault at 13th timestep.

I am pasting the output of the last iteration for your reference.

Time Step 13, time = 260, dt = 20
 * Nonlinear |R| = 1.047077e+02 (Before preset BCs, predictors, correctors, and constraints)
 0 Nonlinear |R| = 1.047077e+02
      0 Linear |R| = 1.047077e+02
      1 Linear |R| = 1.074292e-13
 1 Nonlinear |R| = 2.022600e-01
      0 Linear |R| = 2.022600e-01
      1 Linear |R| = 3.173528e-16
 2 Nonlinear |R| = 1.042074e-05
  Finished Solving                                                                       [ 10.05 s] [  408 MB]
 Solve Converged!
thermo_mech0:
thermo_mech0: Time Step 13, time = 260, dt = 20
thermo_mech0:  * Nonlinear |R| = 4.699861e-02 (Before preset BCs, predictors, correctors, and constraints)
thermo_mech0:  0 Nonlinear |R| = 4.747034e+03
thermo_mech0:       0 Linear |R| = 4.747034e+03
thermo_mech0:       1 Linear |R| = 1.174723e-12
thermo_mech0:  1 Nonlinear |R| = 3.500660e-02
thermo_mech0:       0 Linear |R| = 3.500660e-02
thermo_mech0:       1 Linear |R| = 2.829620e-13
thermo_mech0:  2 Nonlinear |R| = 6.500066e-05
thermo_mech0:       0 Linear |R| = 6.500066e-05
thermo_mech0:       1 Linear |R| = 1.250532e-15
thermo_mech0:  3 Nonlinear |R| = 9.593832e-09
thermo_mech0:  Solve Converged!
thermo_mech0:
thermo_mech0: Postprocessor Values:
thermo_mech0: +----------------+----------------+------------------+------------------+------------------+------------------+
thermo_mech0: | time           | bead_volume    | bead_x_coord_max | bead_y_coord_max | bead_y_coord_min | bead_z_coord_max |
thermo_mech0: +----------------+----------------+------------------+------------------+------------------+------------------+
thermo_mech0: |   0.000000e+00 |   0.000000e+00 |     0.000000e+00 |     0.000000e+00 |     0.000000e+00 |     0.000000e+00 |
thermo_mech0: |   2.000000e+01 |   1.001000e-06 |     7.886751e-04 |     7.886751e-04 |     2.113249e-04 |     7.884638e-01 |
thermo_mech0: |   4.000000e+01 |   3.376001e-03 |     2.068301e+00 |     1.183013e-01 |     2.113249e-04 |     1.018301e+00 |
thermo_mech0: |   6.000000e+01 |   2.362600e-02 |     2.218301e+00 |     4.341506e-01 |     2.113249e-04 |     1.034151e+00 |
thermo_mech0: |   8.000000e+01 |   5.062600e-02 |     2.218301e+00 |     7.341506e-01 |     2.113249e-04 |     1.168301e+00 |
thermo_mech0: |   1.000000e+02 |   6.707913e-02 |     2.218301e+00 |     8.841506e-01 |     2.113249e-04 |     1.168301e+00 |
thermo_mech0: |   1.200000e+02 |   9.239163e-02 |     2.218301e+00 |     1.109151e+00 |     2.113249e-04 |     1.184151e+00 |
thermo_mech0: |   1.400000e+02 |   1.155948e-01 |     2.218301e+00 |     1.259151e+00 |     2.113249e-04 |     1.184151e+00 |
thermo_mech0: |   1.600000e+02 |   1.404854e-01 |     2.218301e+00 |     1.484151e+00 |     2.113249e-04 |     1.184151e+00 |
thermo_mech0: |   1.800000e+02 |   1.645323e-01 |     2.218301e+00 |     1.634151e+00 |     2.113249e-04 |     1.184151e+00 |
thermo_mech0: |   2.000000e+02 |   1.894229e-01 |     2.218301e+00 |     1.784151e+00 |     2.113249e-04 |     1.184151e+00 |
thermo_mech0: |   2.200000e+02 |   2.147354e-01 |     2.218301e+00 |     1.859151e+00 |     2.113249e-04 |     1.184151e+00 |
thermo_mech0: |   2.400000e+02 |   2.404698e-01 |     2.218301e+00 |     2.009151e+00 |     2.113249e-04 |     1.184151e+00 |
thermo_mech0: |   2.600000e+02 |   2.666260e-01 |     2.218301e+00 |     2.084151e+00 |     2.113249e-04 |     1.184151e+00 |
thermo_mech0: +----------------+----------------+------------------+------------------+------------------+------------------+
thermo_mech0:

Postprocessor Values:
+----------------+----------------------+----------------------+
| time           | max_von_mises_stress | min_von_mises_stress |
+----------------+----------------------+----------------------+
|   0.000000e+00 |         0.000000e+00 |         0.000000e+00 |
|   2.000000e+01 |         2.557708e+02 |         2.557708e+02 |
|   4.000000e+01 |         2.100804e+02 |         1.002525e+01 |
|   6.000000e+01 |         2.100804e+02 |         7.071657e+00 |
|   8.000000e+01 |         2.100804e+02 |         4.865289e+00 |
|   1.000000e+02 |         2.100804e+02 |         5.037859e+00 |
|   1.200000e+02 |         2.100804e+02 |         4.139185e+00 |
|   1.400000e+02 |         2.100804e+02 |         4.641678e+00 |
|   1.600000e+02 |         2.750409e+02 |         4.721216e+00 |
|   1.800000e+02 |         3.447571e+02 |         4.368057e+00 |
|   2.000000e+02 |         4.021468e+02 |         4.303420e+00 |
|   2.200000e+02 |         4.559579e+02 |         3.984803e+00 |
|   2.400000e+02 |         5.008031e+02 |         4.126624e+00 |
|   2.600000e+02 |         5.354695e+02 |         4.369218e+00 |
+----------------+----------------------+----------------------+


===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 3614147 RUNNING AT c0022
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions