🌐 CC-Downloader: High-Performance Common Crawl Extraction

Python pipeline optimized for high-performance computing (HPC) environments to search and download WARC files from the CC-MAIN files of Common Crawl repository.

Designed for large-scale projects, it utilizes S3 byte-range requests (boto3) and automatically switches to http requests via a dynamically rotated API Gateway (requests_ip_rotator), if needed, ensuring uninterrupted data extraction at scale.

✨ Key Features

• Dynamic Protocol Switching: Automatically toggles between boto3 (AWS S3) and standard HTTPS via requests to circumvent rate limits without terminating the job.
• Built-in IP Rotation: Integrates with requests_ip_rotator to deploy a temporary AWS API Gateway, distributing requests across multiple regions.
• Parallel Processing: Utilizes Python's multiprocessing.Pool to chunk index files and extract WARC records concurrently across available CPU cores.

---

🛠 Prerequisites

This pipeline requires an active AWS Account to access the Common Crawl S3 buckets via boto3 and to deploy the temporary API Gateway for IP rotation.

You must set your AWS credentials in your environment before running the scripts:

export AWS_ACCESS_KEY_ID="your_access_key_here"
export AWS_SECRET_ACCESS_KEY="your_secret_key_here"

🚀 Example

run_downloader.py is pre-configured to run a demonstration. It will fetch index files for BBC News from the CC-MAIN-2019-09 crawl and download a limited sample of 300 WARC records to allow you to immediately test the pipeline. example directory contains the output of run_downloader.py

1. Clone & Install

git clone https://github.com/zfrb/CC-Downloader.git
cd CC-Downloader
pip install -r requirements.txt

2. Run the Pipeline

Run the main execution script. The script automatically executes sequentially, first downloading the CDX indices and then downloading the WARC files.

python run_downloader.py

🏗 Repository Structure

• run_downloader.py: The example execution script. Demonstrates configuration mapping, directory management, and multiprocessing chunking.

• functions.py: The main downloader functions fetching WARC files.

• cc_index_search.py: The function for downloading the Common Crawl CDX Index server.

• example/: The root directory automatically generated during runtime to house all scraped data.

  • data_cc_index/: Sub-directory for storing fetched CDX .csv files.
  • data_cc/: Sub-directory for storing the final HTML records extracted from WARCs. Failed path logs are also generated here.

⚙️ Custom Configuration

To scrape your own targets, edit the configuration variables at the top of run_downloader.py. You can adjust the NUM_DOWNLOAD variable to extract all records by changing it to "all".

🛡️ Costs

Even though the Common Crawl repository is free and open, deploying the API Gateway via requests_ip_rotator may incur trivial costs on your AWS account depending on the volume of fallback requests.