OOM

[wen89-harsh]

作者好，我按照你的readme文件进行复现时，发现A100显卡在bf16精度下仍报错OOM，您是怎么处理train.sh那一块的

[TimeLovercc]

可以把具体的报错发过来吗

[wen89-harsh]

File "/data/shanjianping/.conda/envs/acw/lib/python3.11/site-packages/accelerate/commands/launch.py", line 993, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/data/shanjianping/.conda/envs/acw/bin/python3.11', 'src/train.py', '--task_name', 'gen_mbpp', '--model', 'deepseek-ai/deepseek-coder-1.3b-instruct', '--output_dir', './train', '--train_batch_size', '512', '--val_batch_size', '64', '--gradient_accumulation_steps', '1', '--num_epochs', '1000', '--precision', 'bf16', '--max_length', '2048', '--augmentation', 'false', '--limit', '0', '--alpha_distill', '0.1', '--alpha_ce', '0.1', '--alpha_switch', '8.0', '--context_width', '2', '--use_cache', 'false']' died with <Signals.SIGKILL: 9>.这个是按照您那边跑的

[TimeLovercc]

这个看不出有效信息，可以复制更多的错误堆栈吗？以及为什么您认为是OOM导致的？

[wen89-harsh]

我这边问了ai说是内存不够？‘2026-03-17 16:20:48,409 - src.datahelper - WARNING - Vocabulary size mismatch! Resizing model embeddings...
2026-03-17 16:20:48,498 - src.datahelper - INFO - Processing raw data...
2026-03-17 16:20:48,498 - src.datahelper - INFO - Loading generated data from outputs/dp_generation/mbpp_generated_data.pt
2026-03-17 16:20:48,998 - src.datahelper - INFO - Loaded 10000 generated examples
2026-03-17 16:20:48,998 - src.datahelper - INFO - Computing model outputs for all examples...
Computing model outputs: 39%|███████████▏ | 3860/10000 [20:47<29:31:53, 17.31s/it]
Computing model outputs: 39%|███████████▏ | 3861/10000 [21:04<29:30:27, 17.30s/itTraceback (most recent call last):
File "/data/shanjianping/.conda/envs/acw/bin/accelerate", line 6, in
sys.exit(main())
^^^^^^
File "/data/shanjianping/.conda/envs/acw/lib/python3.11/site-packages/accelerate/commands/accelerate_cli.py", line 50, in main
args.func(args)
File "/data/shanjianping/.conda/envs/acw/lib/python3.11/site-packages/accelerate/commands/launch.py", line 1405, in launch_command
simple_launcher(args)
File "/data/shanjianping/.conda/envs/acw/lib/python3.11/site-packages/accelerate/commands/launch.py", line 993, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/data/shanjianping/.conda/envs/acw/bin/python3.11', 'src/train.py', '--task_name', 'gen_mbpp', '--model', 'deepseek-ai/deepseek-coder-1.3b-instruct', '--output_dir', './train', '--train_batch_size', '512', '--val_batch_size', '64', '--gradient_accumulation_steps', '1', '--num_epochs', '1000', '--precision', 'bf16', '--max_length', '2048', '--augmentation', 'false', '--limit', '0', '--alpha_distill', '0.1', '--alpha_ce', '0.1', '--alpha_switch', '8.0', '--context_width', '2', '--use_cache', 'false']' died with <Signals.SIGKILL: 9>.’

[TimeLovercc]

这个sigkill确实常见原因是OOM，但是我的模型训练应该不会占用太多memory，如果按照1.3B*12大概会是20GB以内的VRAM或者RAM占用，我建议你在进程运行的时候看一下gpu占用，通过nvidia-smi这样的方式。