Save crawl file in a cloud bucket

[antoineeripret]

Hello @eliasdabbas,

It's me again with a new idea.

Context

I'm trying to create a crawl logic (including your library) through Airflow. It works great, **the issue is the amount of data created by the library in the local storage. ** I'm trying to rely as little as possible because the amount of data could be quite important in the use case I'm working on at the moment.

Idea

What I attempt to implement is the following:

• Launch the crawl
• Every 1,000 URLs (for instance), stop the crawl, upload the JL file to a cloud bucket and empty it
• Resume the crawl
• ....

By doing so, the file would be kept to a reduced size, while the output of Advertools won't be affected. The issue is that I'm not sure if there is a way of "stopping" the crawl without losing the queue.

Do you happen to have an idea?

Thank you!

[eliasdabbas]

Thanks again @antoineeripret !

• Cloud storage: this is already supported by scrapy, and you should be able to use it when crawling with advertools. Under custom_settings you can use AWC and GCS, and these have dedicated settings or them. Please check out the scrapy setting page for the details.
• To start, pause and resume crawling you can use the JOBDIR custom setting as well. This keeps track of the pages that were crawled already, and avoids crawling them. All you have to do is set this once, and then crawl again using this same setting. There is a tip here if you want. If the first option works, I don't think you'll need to use the second one.

Let me know if that works, and how it goes.

Here's another interesting case study :)

[antoineeripret]

Hey @eliasdabbas,

Awesome, great news ! I'll have a look next week and keep you posted !

I didn't forget the article I need to write btw, but I want to finalize what I'm doing with your library first to cover all possible cases :)

Thank you !

[eliasdabbas]

Great!

Let me know how it goes (on all fronts :)

Have a great weekend!

[antoineeripret]

Sure, thank you for your help !

[antoineeripret]

Hey @eliasdabbas,

I've just tested the streaming to S3, and it works like a charm. Ridiculous how easy it is to set up.

Thanks again for your help !

[eliasdabbas]

Wonderful to hear this!

Thanks for letting me know.