Inquires about custom reward and tool use

[altria-zewei-wang]

Dear ROLL team,

Thanks for your great work! I was trying to implement a near Search-R1 setup with custom format-based reward function with my custom ToolUse.
Per https://alibaba.github.io/ROLL/docs/English/UserGuide/agentic/Tool_Use, I have registered my tool.
Per https://alibaba.github.io/ROLL/docs/%E7%AE%80%E4%BD%93%E4%B8%AD%E6%96%87/%E6%89%A9%E5%B1%95%E5%BC%80%E5%8F%91%E6%89%8B%E5%86%8C/custom_reward_cn, I have registered my reward worker as well.

However, it has been a bit hard for me to find a good example yaml that would help me with my requirements. gem_math_hotpotqa_search does not seem to have the reward function configs, and traj_envs_gem_math appear to be a bit unclear on how it's used. Is it the right way to register everything in config/tra_envs_blabla, and cite them in custom_envs in gem_math_hotpotqa_search?

[PanAndy]

RewardWorker is specifically defined for the RLVR Pipeline:
https://alibaba.github.io/ROLL/docs/%E7%AE%80%E4%BD%93%E4%B8%AD%E6%96%87/%E6%89%A9%E5%B1%95%E5%BC%80%E5%8F%91%E6%89%8B%E5%86%8C/custom_reward_cn/

The page https://alibaba.github.io/ROLL/docs/English/UserGuide/agentic/Tool_Use corresponds to the Agentic Pipeline, where reward computation occurs within the environment's step (env.step). Env registrations are defined in https://github.com/alibaba/ROLL/blob/main/roll/pipeline/agentic/env/__init__.py, which contains the concrete Env class definitions.

math env like this: https://github.com/alibaba/ROLL/blob/main/roll/pipeline/agentic/env/gem/math_env.py
and you can install gem-llm to view gem MathEnv code.

[altria-zewei-wang]

Thank you for your rapid reply! Just to clarify, the RewardWorker are used only for RLVR, which in my understanding, is different from the tool-use like the one in Search R1 and it's referred as GEM in ROLL settings. Thus, I should instead create a custom env with reward function, register it in env/init.py, instead of modifying the RewardWorker?

[PanAndy]

Yes.
In the Agentic RL abstraction, the concrete form of a business application is the environment definition.
The environment needs to be carefully designed, including various dimensions of state and the integration/interpretation of tools.

[altria-zewei-wang]

Does Agentic RL support LoRA? I have only seen an example from RLVR on training LoRA

[PanAndy]

The backend supports LoRA, but you need to improve the Agentic Pipeline by following the LoRA usage pattern used in the RLVR pipeline.
You can give it a try.

Does Agentic RL support LoRA? I have only seen an example from RLVR on training LoRA

[altria-zewei-wang]

By LoRA usage pattern, do you mean the same setup as in the rlvr example? Or do you mean that there's more modification to be made( if so, would you kindly point the files I would need to modify)?