Skip to content

fix: disable default wandb cloud syncing to prevent local execution crashes#9

Open
Chetan0-0 wants to merge 1 commit intohumanai-foundation:mainfrom
Chetan0-0:fix-wandb-offline
Open

fix: disable default wandb cloud syncing to prevent local execution crashes#9
Chetan0-0 wants to merge 1 commit intohumanai-foundation:mainfrom
Chetan0-0:fix-wandb-offline

Conversation

@Chetan0-0
Copy link

Hey team,

While setting up the trainModel.ipynb environment locally, I noticed the execution hangs/throws an authentication error because wandb attempts to sync to the previous author's specific cloud entity.
To improve the out-of-the-box developer experience for new contributors, I added os.environ["WANDB_MODE"] = "offline" prior to the training loop. This allows the notebook to run successfully by saving logs locally. Users can simply comment out this line if they want to authenticate and push to their own W&B dashboards.

@Chetan0-0
Copy link
Author

image image

I've successfully verified the W&B offline fix with full local training run.

As shown in the attached images and in this all PSNR , SSIM and LPIPS metrics were recorded correctly in offline mode and the training loop completed all 5 epochs at the full 256x256 resolution without any crashes or issues.

As we can see the model does converges and thus we can say that the logging changes do not interfere with the core training logic (Loss=0.43 and Val PSNR=11.43)

@Chetan0-0
Copy link
Author

image update : finally the technical verification was successful After resolving several local resource constarints I have succesfully verified the full pipeline There are many things that changed key ones being implementing a new batching strategy ( making batch size 1) so that kernel doesnt crash on local hardware and debugged the mismatch of load_datasets recursion and train_test_model (personally it was very hard for me to solve sucked the soul of my body but yet here we are ) and finally a manual visualization wrapper to bypass view_extract problem.

And thus in the end successfully reconstructed 8 spectral channel from RGB input (screenshot attached).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant