Move forcing script from fluxnet experiments to src #1238

kmdeck · 2025-07-17T00:34:27Z

Purpose

Attempts to clean up our fluxnet simulations. It mostly moves stuff into src but there is probably further cleanup to be had.

use the interpolating ability of TVI to handle gaps in data, rather than filling missing values by hand
remove metdrivers fluxnet script in favor of prescribed_forcing_fluxnet function in a new extension
set IC is a new function in the extension

It would be nice to pass dates or Itime to TVI but this does not currently work
It would be nice to be able to pass a preprocess function to TVI but this does not currently work, so I do unit changes by hand
Another cool thing would be if we could have TVI read a CSV and get the right column but Im not sure if that is worth it (rather than me reading in the csv and passing arrays to TVI)

To-do

Figure out how to handle to comparison to data in the output...
move plotting to LandSimulationVisualization extension

Content

I have read and checked the items on the review checklist.

juliasloan25 · 2025-07-17T22:00:18Z

experiments/integrated/fluxnet/run_fluxnet.jl

-site_ID = ARGS[1]
+site_ID = "US-MOz"


We should revert this before merging

juliasloan25 · 2025-07-17T22:10:54Z

Project.toml

@@ -53,8 +56,10 @@ ClimaTimeSteppers = "0.8"
 ClimaUtilities = "0.1.24"
 DataFrames = "1.4"
 Dates = "1"
+DelimitedFiles = "1"


I thought these packages would go under weakdeps rather than compat since they're used by the extension. I don't find very good Julia documentation about how to set up package extensions, but I did find this example repo someone made: https://github.com/pebeto/julia_extensions_example/tree/main

You should be able to add a weak dependency with Pkg.add(name_of_package, target=:weakdeps).

^^ This is what I did (then package name and uuid is listed under weak deps). However the compat is separate from that.

We are not required to list compats for weak dependencies, but one of the Aqua tests checks that (we can disable it if we want). If we do want to list compats, though, there is not weak_compats - all compats are in a single list.

Our extensions have added compat in the past (Neural Snow - Flux, LandSimulationVisualization (very recently) - Poppler, Printf, etc)

ext/FluxnetSimulations.jl

experiments/integrated/fluxnet/run_fluxnet.jl

ph-kev · 2025-07-17T23:56:16Z

.buildkite/Manifest-v1.11.toml


    [deps.ClimaLand.extensions]
+    FluxnetSimulations = ["DelimitedFiles", "Format"]


Would it be possible to remove Format as a dependency? It seems strange to me to include a package that just handle string formatting?

ph-kev · 2025-07-18T00:01:45Z

experiments/integrated/fluxnet/run_fluxnet.jl

+FluxnetSimulations =
+    Base.get_extension(ClimaLand, :FluxnetSimulations).FluxnetSimulations;
+


I am not sure if this work, but can you try

Suggested change

FluxnetSimulations =

Base.get_extension(ClimaLand, :FluxnetSimulations).FluxnetSimulations;

import ClimaLand.FluxnetSimulations as FluxnetSimulations

ph-kev · 2025-07-21T20:43:47Z

ext/fluxnet_sims/data_processing.jl

+function mask_data(t, v; val = -9999)
+    if ndims(v) == 1
+        not_missing_mask = .~var_missing.(v)


val isn't used in this function.

ph-kev · 2025-07-21T20:45:47Z

ext/fluxnet_sims/data_processing.jl

+var_missing(array; val = -9999) = array == val
+


Could documentation be added to all the functions in this file?

ph-kev · 2025-07-21T20:54:12Z

ext/fluxnet_sims/data_processing.jl

+        Float64.(t),
+        Float64.(preprocess_func.(eachcol(v)...)),


Is there a reason why this need to be Float64?

ph-kev · 2025-07-21T21:02:00Z

ext/fluxnet_sims/forcing.jl

+    if any(indices .== nothing)
+        nothing_id = findall(indices .== nothing)
+        @error("$(labels[nothing_id]) is missing in the data, but required.")
+    end


Suggested change

if any(indices .== nothing)

nothing_id = findall(indices .== nothing)

@error("$(labels[nothing_id]) is missing in the data, but required.")

end

nothing_id = findall(indices .== nothing)

if !isempty(nothing_id)

@error("$(labels[nothing_id]) is missing in the data, but required.")

end

ph-kev · 2025-07-21T23:45:32Z

experiments/integrated/fluxnet/run_fluxnet.jl

@@ -532,11 +500,11 @@ lines!(
    label = "Ice, 1.25cm",
 )

-if drivers.SWC.status != absent
+if comparison_data.SWC.absent == false


Suggested change

if comparison_data.SWC.absent == false

if !comparison_data.SWC.absent

ph-kev · 2025-07-21T23:45:55Z

experiments/integrated/fluxnet/run_fluxnet.jl

@@ -599,11 +562,11 @@ lines!(
    color = "blue",
 )

-if drivers.TS.status != absent
+if comparison_data.TS.absent == false


Suggested change

if comparison_data.TS.absent == false

if !comparison_data.TS.absent

ph-kev · 2025-07-21T23:46:31Z

ext/fluxnet_sims/data_processing.jl

+)
+    first_row = skip_header ? 2 : 1
+    idx = column_name_map[varname]
+    if idx isa Nothing


Suggested change

if idx isa Nothing

if isnothing(idx)

ph-kev · 2025-07-21T23:48:19Z

ext/fluxnet_sims/forcing.jl

+    varnames = ("TA_F", "VPD_F", "PA_F", "P_F", "WS_F", "LW_IN_F", "SW_IN_F")
+    indices = [findfirst(columns .== varname) for varname in varnames]
+    column_name_map = Dict(zip(varnames, indices))


Suggested change

varnames = ("TA_F", "VPD_F", "PA_F", "P_F", "WS_F", "LW_IN_F", "SW_IN_F")

indices = [findfirst(columns .== varname) for varname in varnames]

column_name_map = Dict(zip(varnames, indices))

varnames = ("TA_F", "VPD_F", "PA_F", "P_F", "WS_F", "LW_IN_F", "SW_IN_F")

colimn_name_map = Dict(varname => findfirst(columns .== varname) for varname in varnames)

kmdeck added 2 commits July 16, 2025 14:02

wip [skip ci]

f98f6ee

wip [skip ci]

bcaecc5

kmdeck requested review from ph-kev, juliasloan25 and imreddyTeja July 17, 2025 00:36

juliasloan25 reviewed Jul 17, 2025

View reviewed changes

kmdeck added 2 commits July 20, 2025 09:01

review comments wip [skip ci]

214a92f

still wip [skip ci]

c88100b

kmdeck commented Jul 21, 2025

View reviewed changes

experiments/integrated/fluxnet/run_fluxnet.jl Outdated Show resolved Hide resolved

kmdeck added 2 commits July 21, 2025 12:45

this runs for fluxnet sites

fd7b4e5

remove data tools

a581f57

juliasloan25 mentioned this pull request Jul 21, 2025

add default functions for fluxnet forcing #1247

Open

ph-kev reviewed Jul 21, 2025

View reviewed changes


		[deps.ClimaLand.extensions]
		FluxnetSimulations = ["DelimitedFiles", "Format"]

		FluxnetSimulations =
		Base.get_extension(ClimaLand, :FluxnetSimulations).FluxnetSimulations;

	FluxnetSimulations =
	Base.get_extension(ClimaLand, :FluxnetSimulations).FluxnetSimulations;
	import ClimaLand.FluxnetSimulations as FluxnetSimulations

	if comparison_data.SWC.absent == false
	if !comparison_data.SWC.absent

	if comparison_data.TS.absent == false
	if !comparison_data.TS.absent

		site_ID = ARGS[1]
		site_ID = "US-MOz"

Move forcing script from fluxnet experiments to src #1238

Are you sure you want to change the base?

Move forcing script from fluxnet experiments to src #1238

Uh oh!

Conversation

kmdeck commented Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

To-do

Content

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kmdeck commented Jul 17, 2025 •

edited

Loading