Add cleanup for waiting tasks and reject adding more tasks #1670
Description
What's the problem? (required)
When there is an issue with Catalyst getting back to update the task status, the task is forever in the waiting phase, which in turn causes the rate limited error for the users.
What's the current behavior? (required)
There is no timeout/cleanup in Livepeer Studio. The only timeout is in Catalyst.
Also, I think that Livepeer Studio should reject and not add to DB any additional tasks if there are already too many of them. Currently, new tasks are added to DB even if rate limited is returned.
Describe a possible solution you've considered (optional)
Add a timeout to the task's lifecycle.
Additional context (optional)
Related Discord thread with the issue which caused record-tester to stop working