Adding support grid search like hydra for making it easier for user to run experiments

[aaishwarymishra]

currently we are using the omegaconf, for use case like training a single model by manually editing the parameter values before each run is fine.

But we want to do some serious experimenting , we literally need to write a separate script for it. Omega does not support it natively, but hydra does.

A simple example will look like this:

lr: 0.01
layers: 32
epochs: 10

hydra:
  mode: MULTIRUN  
  sweeper:
    params:
      lr: 0.1, 0.01, 0.001
      layers: 32, 64, 128

we can replace omegaconf with hydra if we want without changing much how current code works, but we do need to add the hydra decorator in our main function.

[aaishwarymishra]

@vfdev-5

[vfdev-5]

@aaishwarymishra can you provide a simple small runnable code + config to see in details how hydra works. You should also discuss pros/cons of omegaconf by hydra replacement

[aaishwarymishra]

@hydra.main(version_base=None, config_path=".",config_name="config")
def main(cfg: DictConfig) -> None:
    config_dict = OmegaConf.to_container(cfg, resolve=True)
    config =ProjectConfig.model_validate(config_dict)
    trainer = PreTrainer(config)
    trainer.run()

so before we were using argpase and manually parsing the yaml but with the hydra decorator it does it automatically.

pros:

• multi run for example we can set parameters both in cli and yaml python main.py -m optimizer.lr=1e-3,1e-4 dataset.dataloader.batch_size=32,64 it's the main purpose of using hydra, for trainite as its hard to do ourself.
• Optuna integration

cons:

• @hydra.main
• the multi run values will be validated at runtime one combination at time from pydantic not all together.
• Hydra also changes the working directory to output dir, so relative paths will break if used, we have explicitly add parameter in yaml or cli to disable it, or refactor code. I think we can disable it by default during code gen.

@vfdev-5