Add fit_tracking_metrics option to configure_generation_strategy#5238
Open
saitcakmak wants to merge 2 commits into
Open
Add fit_tracking_metrics option to configure_generation_strategy#5238saitcakmak wants to merge 2 commits into
saitcakmak wants to merge 2 commits into
Conversation
Allows disabling modeling of tracking metrics via the high-level Client interface (client.configure_generation_strategy(fit_tracking_metrics=False)) instead of manually editing the generation strategy. The option is threaded through GenerationStrategyDispatchStruct and injected into the MBM node's generator_kwargs, where the registry routes it to the TorchAdapter constructor. When False, only metrics in the optimization config are modeled; tracking metrics are still recorded but not modeled, and model-dependent analyses are not produced for them. Closes facebook#4688
Move pyrefly-ignore comment to its own line (matching repo convention) to bring the line under 88 chars. Pre-existing error from facebook#5236, unrelated to this PR's feature but surfaced by pre-commit.
|
@saitcakmak has imported this pull request. If you are a Meta employee, you can view this in D109735998. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5238 +/- ##
=======================================
Coverage 96.56% 96.56%
=======================================
Files 619 619
Lines 70273 70296 +23
=======================================
+ Hits 67861 67884 +23
Misses 2412 2412 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the feature requested in #4688: a
fit_tracking_metricsoption on the high-levelClient.configure_generation_strategyAPI, so users can disable modeling of tracking metrics without manually editing the generation strategy.When
False, only metrics in the optimization config (objectives and outcome constraints) are modeled. Tracking metrics are still recorded but not modeled by the BoTorch model — useful when many tracking metrics are kept only for book-keeping.Changes
ax/api/client.py— addedfit_tracking_metrics: bool = Truetoconfigure_generation_strategyand_choose_generation_strategy, threading it into the dispatch struct. Docstrings flag that whenFalse, model-dependent analyses (cross-validation, sensitivity analysis, etc.) will not be produced for tracking metrics, that it requires an optimization config, and that it has no effect forrandom_search.ax/api/utils/structs.py— added thefit_tracking_metricsfield toGenerationStrategyDispatchStruct.ax/api/utils/generation_strategy_dispatch.py—_get_mbm_nodeinjectsfit_tracking_metricsinto the MBM node'sgenerator_kwargs; only the MBM node receives it (Sobol/random-search nodes don't model outcomes).test_gs_fit_tracking_metrics(dispatch) andtest_configure_generation_strategy_with_fit_tracking_metrics(client); updated existing tests that assert the full MBMgenerator_kwargsdict.Implementation note
fit_tracking_metricsis aTorchAdapter.__init__kwarg. The registry's kwarg-splitter (ax/adapter/registry.py) automatically routes any key matching an adapter constructor param into the adapter, so adding it to the node'sgenerator_kwargsis sufficient — no changes toregistry.py,torch.py, orbase.py.Test plan
m2:fit_tracking_metrics=True→ modeled outcomes['m1', 'm2'];False→['m1'](tracking metric excluded from modeling).Closes #4688