els.spread_fire_with_phi_field() back-end memory based crash

[mengsig]

@lambdatronic
Currently, I can force the following issue:

corrupted size vs. prev_size
double free or corruption (out)
run.sh: line 67: 232732 Aborted

By setting the following parameters in the environment using the SpaceTimeCube:

    "fuel_moisture_dead_1hr": SpaceTimeCube(cube_shape, 0.15),
    "fuel_moisture_dead_10hr": SpaceTimeCube(cube_shape, 0.25),
    "fuel_moisture_dead_100hr": SpaceTimeCube(cube_shape, 0.50),

Whereas, with the exact same SpaceTimeCube, I do not get this crash for the following parameters:

    "fuel_moisture_dead_1hr": SpaceTimeCube(cube_shape, 0.10),
    "fuel_moisture_dead_10hr": SpaceTimeCube(cube_shape, 0.25),
    "fuel_moisture_dead_100hr": SpaceTimeCube(cube_shape, 0.50),

If this is intended - e.g. (fuel_moisture_dead_1hr <= 0.10), then it would be great to have better error-handling so that the user can identify the issue, and if not, well, then happy hunting!

[lambdatronic]

Hi @mengsig . My apologies for the slow response. I didn't see this issue until recently. I must have accidentally turned off Github issue notification emails on my account somehow.

Can you provide me with a complete Python script that throws this error? Also, which version of Pyretechnics were you using when this happened?

[mengsig]

Hello @lambdatronic ,

Hi @mengsig . My apologies for the slow response. I didn't see this issue until recently. I must have accidentally turned off Github issue notification emails on my account somehow.

Can you provide me with a complete Python script that throws this error? Also, which version of Pyretechnics were you using when this happened?

I am using the main branch. Unfortunately I lost the script that produced the error, but I have recreated a script hereunder:

# global imports
import os
import shutil
import sys
import time
from pprint import pprint

import numpy as np

import pyretechnics.eulerian_level_set as els

# pyretechnics imports
from pyretechnics.space_time_cube import SpaceTimeCube

script_dir = os.path.dirname(__file__)
project_root = os.path.abspath(os.path.join(script_dir, os.pardir, os.pardir))
sys.path.insert(0, project_root)
from src.utils.loadingUtils import (
    load_raster,
    normalize,
)
from src.utils.parsingUtils import (
    parse_args,
)
from src.utils.plottingUtils import (
    save_matrix_as_heatmap,
)

# -------------Tunable parameters------------- #
xlen, ylen, savename, centrality, fuel_break_fraction = parse_args()
time_steps = int(500 * np.sqrt(xlen * ylen) / (250))
# ----------end of tunable parameters----------#


savedir = f"results/{savename}"
fuel_breaks_file = f"{savedir}/{centrality}_{fuel_break_fraction}.txt"
fuel_breaks_img = f"{savedir}/{centrality}_{fuel_break_fraction}.png"
# copy the file into that directory (keeps the same basename)

# fuel_breaks_file = ""
if fuel_breaks_file != "":
    savedir = fuel_breaks_file.removesuffix(".txt")
os.makedirs(savedir, exist_ok=True)
# directory containing the already-cropped rasters
raster_dir = "cropped_rasters"

# load rasters
xoffset = 100
yoffset = 100
xsubset = (xoffset, int(xoffset + xlen))
ysubset = (yoffset, int(yoffset + ylen))

x = np.random.uniform(0, 1)

slope = load_raster("slp", xsubset, ysubset)
aspect = load_raster("asp", xsubset, ysubset)
dem = load_raster("dem", xsubset, ysubset)
cc = load_raster("cc", xsubset, ysubset)
cbd = load_raster("cbd", xsubset, ysubset)
cbh = load_raster("cbh", xsubset, ysubset)
ch = load_raster("ch", xsubset, ysubset)
fuel_model = load_raster("fbfm", xsubset, ysubset)


# implementing fuel-breaks
if fuel_breaks_file != "":
    fuel_breaks = np.loadtxt(fuel_breaks_file).astype(bool)
    fuel_model[fuel_breaks] = 91
    cc[fuel_breaks] = 0
    cbd[fuel_breaks] = 0
#    ch[fuel_breaks] = 0
#    cbh[fuel_breaks] = 0

rel_img_name = fuel_breaks_img.split("/")[-1]
# clean repo
shutil.move(fuel_breaks_img, savedir + f"/{rel_img_name}")
os.remove(fuel_breaks_file)

# convert to cubes
slope_cube = normalize(slope, datatype="slp")
aspect_cube = normalize(aspect, datatype="asp")
dem_cube = normalize(dem, datatype="dem")
cc_cube = normalize(cc, datatype="cc")
cbd_cube = normalize(cbd, datatype="cbd")
cbh_cube = normalize(cbh, datatype="cbh")
ch_cube = normalize(ch, datatype="ch")
fuel_model_cube = normalize(fuel_model, datatype="fbfm")

# Define cube shape (e.g. 24 hours)
rows, cols = slope.shape
cube_shape = (time_steps, rows, cols)

# build space-time cubes
space_time_cubes = {
    "slope": SpaceTimeCube(cube_shape, slope_cube),
    "aspect": SpaceTimeCube(cube_shape, aspect_cube),
    "fuel_model": SpaceTimeCube(cube_shape, fuel_model),
    "canopy_cover": SpaceTimeCube(cube_shape, cc_cube),
    "canopy_height": SpaceTimeCube(cube_shape, ch_cube),
    "canopy_base_height": SpaceTimeCube(cube_shape, cbh_cube),
    "canopy_bulk_density": SpaceTimeCube(cube_shape, cbd_cube),
    "wind_speed_10m": SpaceTimeCube(cube_shape, 0),
    "upwind_direction": SpaceTimeCube(cube_shape, 0),
    "fuel_moisture_dead_1hr": SpaceTimeCube(cube_shape, 0.25),
    "fuel_moisture_dead_10hr": SpaceTimeCube(cube_shape, 0.75),
    "fuel_moisture_dead_100hr": SpaceTimeCube(cube_shape, 0.99),
    "fuel_moisture_live_herbaceous": SpaceTimeCube(
        cube_shape, 0.90
    ),  # kg moisture/kg ovendry weight
    "fuel_moisture_live_woody": SpaceTimeCube(
        cube_shape, 0.60
    ),  # kg moisture/kg ovendry weight
    "foliar_moisture": SpaceTimeCube(cube_shape, 0.90),
    "fuel_spread_adjustment": SpaceTimeCube(cube_shape, 1.0),
    "weather_spread_adjustment": SpaceTimeCube(cube_shape, 1.0),
}

# Day 2 @ 10:30am
start_time = 0  # minutes

# 8 hours
max_duration = int(time_steps * 2 / 3) * 60  # minutes

# ignite in the middle
xcord, ycord = int(xlen / 8), int(ylen / 8)
num_burned_cells = 0
burned_cells_threshold = (xlen * ylen) / np.sqrt(xlen * ylen)

acres_burned = 0
simulation_runtime = 0
runtime_per_burned_cell = 0


t1 = time.time()
counter = 0
while num_burned_cells < burned_cells_threshold:
    if counter > 0:
        xcord += np.random.choice([-1, 1])
        ycord += np.random.choice([-1, 1])
    counter += 1
    spread_state = els.SpreadState(cube_shape).ignite_cell((xcord, ycord))
    cube_resolution = (
        60,  # band_duration: minutes
        30,  # cell_height:   meters
        30,  # cell_width:    meters
    )

    # ============================================================================================
    # Spread fire from the start time for the max duration
    # ============================================================================================

    runtime_start = time.perf_counter()
    fire_spread_results = els.spread_fire_with_phi_field(
        space_time_cubes,
        spread_state,
        cube_resolution,
        start_time,
        max_duration,
        surface_lw_ratio_model="behave",
    )
    runtime_stop = time.perf_counter()
    stop_time = fire_spread_results["stop_time"]  # minutes
    stop_condition = fire_spread_results[
        "stop_condition"
    ]  # "max duration reached" or "no burnable cells"
    spread_state = fire_spread_results[
        "spread_state"
    ]  # updated SpreadState object (mutated from inputs)
    output_matrices = spread_state.get_full_matrices()

    # ============================================================================================
    # Print out the acres burned, total runtime, and runtime per burned cell
    # ============================================================================================

    num_burned_cells = np.count_nonzero(output_matrices["fire_type"])  # cells
    if num_burned_cells == 0:
        continue
    acres_burned = num_burned_cells / 4.5  # acres
    simulation_runtime = runtime_stop - runtime_start  # seconds
    runtime_per_burned_cell = 1000.0 * simulation_runtime / num_burned_cells  # ms/cell

I know that there are some imports and hyperargs etc. however, I hope this helps. Here the crash happens when fuel_moisture_dead_1hr >= 0.25.

[lambdatronic]

Thanks for the testing script, @mengsig! I don't see anything obviously wrong in your invocation of the library on my first read through, but I am curious about your random ignition coordinate selection code within the fire simulation loop. Are you attempting to set the corners of the grid on fire, or is there a different outcome expected here? At any rate, I'll give it a try and see if I can debug the issue further on my end. Cheers!

[mengsig]

Thanks for the testing script, @mengsig! I don't see anything obviously wrong in your invocation of the library on my first read through, but I am curious about your random ignition coordinate selection code within the fire simulation loop. Are you attempting to set the corners of the grid on fire, or is there a different outcome expected here? At any rate, I'll give it a try and see if I can debug the issue further on my end. Cheers!

Oh you are absolutely right. It could certainly be that with the set parameters the fire cannot ignite to a point that allows it to exist the for loop, and the random walker ends up walking off the grid. Let me double check if this is the case on my end and I'll get back to you!