Serialization of `GPModel` objects

[SamuelBrand1]

Hi @fsaad ,

I've arrived with more annoying queries...

I'm hoping to use the incremental inference tools of AutoGP to refit with both new and/or revised data as it arrives in a weekly cadence.

However, the underlying Gen model doesn't appear to be serializing.

Small reprex: JLD2

using AutoGP, Dates, JLD2

# toy model
model = AutoGP.GPModel([Date("2025-01-01"), Date("2025-01-02")], [1.0, 2.0])
# fit
AutoGP.fit_smc!(model; schedule = AutoGP.Schedule.linear_schedule(2, 0.99))
# Save as jld2
save("model.jld2", "model", model)

# load back
deserialized_model = load("model.jld2", "model")

# New data arrives

AutoGP.add_data!(model, [Date("2025-01-03")], [3.0])
# This works

AutoGP.add_data!(deserialized_model, [Date("2025-01-03")], [3.0])
# this fails

With error message

ERROR: TaskFailedException

nested task error: Generative function changed at address: tree
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] gen_fn_changed_error(addr::Symbol)
    @ Gen ~/.julia/packages/Gen/mP0Sq/src/dynamic/dynamic.jl:175
  [3] traceat(state::Gen.GFUpdateState, gen_fn::Gen.DynamicDSLFunction{…}, args::Tuple{…}, key::Symbol)
    @ Gen ~/.julia/packages/Gen/mP0Sq/src/dynamic/update.jl:89
  [4] var"##model#231"(state#230::Gen.GFUpdateState, ts::Vector{Float64}, config::AutoGP.GP.GPConfig)
    @ AutoGP.Model ~/.julia/packages/AutoGP/6DM3P/src/Model.jl:112
  [5] exec(gen_fn::Gen.DynamicDSLFunction{Any}, state::Gen.GFUpdateState, args::Tuple{Vector{Float64}, AutoGP.GP.GPConfig})
    @ Gen ~/.julia/packages/Gen/mP0Sq/src/dynamic/dynamic.jl:58
  [6] update(trace::Gen.DynamicDSLTrace{…}, arg_values::Tuple{…}, arg_diffs::Tuple{…}, constraints::Gen.DynamicChoiceMap)
    @ Gen ~/.julia/packages/Gen/mP0Sq/src/dynamic/update.jl:192
  [7] macro expansion
    @ ~/.julia/packages/AutoGP/6DM3P/src/inference_smc_anneal_data.jl:134 [inlined]
  [8] (::AutoGP.Inference.var"#47#threadsfor_fun#19"{…})(tid::Int64; onethread::Bool)
    @ AutoGP.Inference ./threadingconstructs.jl:253
  [9] #47#threadsfor_fun
    @ ./threadingconstructs.jl:220 [inlined]
 [10] (::Base.Threads.var"#1#2"{AutoGP.Inference.var"#47#threadsfor_fun#19"{…}, Int64})()
    @ Base.Threads ./threadingconstructs.jl:154

small reprex: GenSerialization

using AutoGP, Dates, GenSerialization

# toy model
model = AutoGP.GPModel([Date("2025-01-01"), Date("2025-01-02")], [1.0, 2.0])
# fit
AutoGP.fit_smc!(model; schedule = AutoGP.Schedule.linear_schedule(2, 0.99))
# Save as jld2
serialize("model.gen", model)

# load back
deserialized_model = deserialize("model.gen")

# New data arrives

AutoGP.add_data!(model, [Date("2025-01-03")], [3.0])
# This works

AutoGP.add_data!(deserialized_model, [Date("2025-01-03")], [3.0])
# this fails

Same error message as the JLD2 based approach.

Fixes?

I realize that this could be an issue with Gen.jl anyway, but do you have any experience with either a direct solution or workaround to this problem? For example, saving model posteriors as priors and then only adding new data?

[SamuelBrand1]

This would seem to be involved with threading? Is there any chance of have an optionally unthreaded version of the inference?

[fsaad]

Hi @SamuelBrand1 If you install AutoGP in development mode (using Pkg; Pkg.develop("AutoGP")), you can remove all instances of Threads.@threads from the codebase. Does serialization then work?

You can find the location of the editable code via import AutoGP; pathof(AutoGP). For example, it could be at ~/.julia/dev/AutoGP/ or similar.

[SamuelBrand1]

Hi @fsaad ,

I was thinking about giving this a try. I'll let you know how I get along.

[SamuelBrand1]

Btw, you get the same error with deepcopy, e.g. in the reprex above you can replace the serialize-deserialize steps with just making a deep copied new GPModel instance and the same error appears.

[SamuelBrand1]

I suspect the problem will be a limitation of Gen.jl itself and if it turns out to be that I'll move this issue along to them.

[SamuelBrand1]

I'll go into this myself when I've got a minute but I set copilot on checking the Threads removal and its not that. That does suggest this is a higher problem with Gen.jl and I'll move on with my issue (unless you want to try and take this on @fsaad !).

https://github.com/CDCgov/NowcastAutoGP/blob/copilot/fix-32/INVESTIGATION_RESULTS.md

NB: I've not double checked these results yet

[fsaad]

@SamuelBrand1 Here is a new function in the API for serializing a GPModel to disk.

https://probsys.github.io/AutoGP.jl/stable/api.html#model_serialization

[SamuelBrand1]

@SamuelBrand1 Here is a new function in the API for serializing a GPModel to disk.

probsys.github.io/AutoGP.jl/stable/api.html#model_serialization

Brilliant work! Thanks for this... it really improves the use-cases for incremental inference.