Adopt request-ratelimiter

[jbothma]

Sometimes we slow crawlers down with sleep to be extra nice to the website, e.g. in opensanctions/opensanctions#2664.

There's no point in sleeping if we're going to use a cached result.

[pudo]

I like that a lot, it could speed up a lot of the low and slow crawlers. So we do a request_sleep= argument on all context.fetch_ methods?

[leonhandreke]

From Slack a couple days ago:

One thing I've considered for Zyte, where the fetch methods have grown so many arguments, is to have a custom Response object that could be used to implement behavior like this.
assert media type, invalidate the cache for broken responses, do things differently for cache hits/misses...

[jbothma]

I've been wondering if this fits under the http config in dataset metadata.

@leonhandreke you've had some reservations about the http config - is this a good place to note that down or wanna make a new issue so we can consider that?

[leonhandreke]

Meh, mainly I don't really see a reason why some specific intricacies of how we do HTTP requests should be in the YAML. All the other nitty-gritty of how to access a source (Zyte or no, geolocation, magic headers...) we keep in the code, and I like that, since I can read it and see what's going on.

But honestly, even though I have an opinion, I don't think it matters all that much. It's very much at the surface and very easy to refactor, should we change our mind. So please just do what you feel is best :)

[leonhandreke]

I just did something like this for LLM responses opensanctions/opensanctions#3421

[leonhandreke]

Aliens from the future called me and let me know that they aren't using sleep anymore, they have something called a rate limiter and it's 🔥.

Turns out requests supports this quite well, see https://pypi.org/project/requests-ratelimiter/. You can even tell it to only use it for some requests. Maybe the crawlers in question could just do context.http.mount("https://unstable.website.net", adapter) (with adapter either being from requests-ratelimiter or a five-liner we put somewhere in zavod.meta.http)? Or is leaking the abstraction of context.http not something we want to do?