Rate limiting and delays between LLM calls

[hd24-jspoerer]

Is it possible to add delays or rate limits between LLM calls?

I would like to limit LLM call frequency by either introducing rate limits or delays (preferably delays).

If this is not yet possible, should I implement this and make a PR?

[evalstate]

Not built in, would happily accept a PR for this feature. There was a recent addition from @wreed4 for a "slow" LLM for testing.

[janspoerer]

Interesting!

I changed the delay behavior slightly. While the augmented_llm_slow sets a fixed delay, this delay is dependent on when the last LLM call happened. It will make sure that at least the delay time passes between calls, independent of how long some intermediate processes run (such as tool calls).

Created a PR for this. #222