Why gpt-oss doesn’t support RL/GRPO Training?

[alex-ht]

I saw in the FAQ that it’s currently not possible to perform RL or GRPO training on gpt-oss. Could anyone explain why? Isn’t it possible to use transformers for inference and compute the GRPO loss?

[azizDentero]

For me the same and I am trying to work it around but I can't go around the runtime error: RuntimeError: Inference tensors cannot be saved for backward. To work around you can make a clone to get a normal tensor and use it in autograd.

But after some research I think the error is caused because how gpt oss is inferencing. maybe it's because of the mxfp 4 since it's already unique to use this on models as big as 20b or 120b.

[shimmyshimmer]

@alex-ht @azizDentero what a coincidence, we just supported with a new notebook too! Summary:

We’re introducing gpt-oss RL support and the fastest RL inference and lowest VRAM use vs. any implementation. Blog: https://docs.unsloth.ai/new/gpt-oss-reinforcement-learning

• Unsloth now offers the fastest inference (~3x faster), lowest VRAM (50% less) and most context (8x longer) for gpt-oss RL vs. any implementation - with no accuracy loss.
• Since RL on gpt-oss isn't yet vLLM compatible, we rewrote Transformers inference code to enable faster inference
• gpt-oss-20b GSPO free Colab notebook
• This notebook automatically creates faster matrix multiplication kernels and uses a new Unsloth reward function. We also show how to counteract reward-hacking which is one of RL's biggest challenges.

[azizDentero]

Thank you ! I actually managed to start the GPRO training by coincidence when I forced scoping on the temporarypatch/gpt_oss.py with a debugging print XD.

But I wasn't sure if the model's weights are actually adjusting in the right way.
anyway thanks for the lucky update! I really needed to use RGPO soon.