Skip to content

My scrapy program cannot keep pulling data from redis queue #148

Open
@Cehae

Description

@Cehae

When I open the scrapy program, the program executes normally. But after tens of minutes my program log keet printing:

2019-06-25 17:42:12 [scrapy.extensions.logstats] INFO: Crawled 64 pages (at 0 pages/min), scraped 64 items (at 0 items/min)
2019-06-25 17:43:12 [scrapy.extensions.logstats] INFO: Crawled 64 pages (at 0 pages/min), scraped 64 items (at 0 items/min)
2019-06-25 17:44:12 [scrapy.extensions.logstats] INFO: Crawled 64 pages (at 0 pages/min), scraped 64 items (at 0 items/min)
2019-06-25 17:45:12 [scrapy.extensions.logstats] INFO: Crawled 64 pages (at 0 pages/min), scraped 64 items (at 0 items/min)

But at this point my redis queue still has data, so I can only close the program and restart it. What should I do to solve this problem? Thank you!

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions