Web_Scrapper

This is a small Python web scraper.

It can:

• Download a webpage (HTML)
• Extract items (default: Hacker News headlines)
• Save results to:
  • CSV (easy to open)
  • SQLite database (easy to store history)

1) Setup (install requirements)

Open a terminal and run:

cd Web_Scrapper

# Create a virtual environment (recommended)
python3 -m venv .venv

# Activate it (macOS / Linux)
source .venv/bin/activate

# Install packages
python -m pip install -r requirements.txt

If you get an error like pip install -r (exit code 2), you probably forgot the file name. It must be:

python -m pip install -r requirements.txt

2) Run (default: Hacker News)

cd Web_Scrapper
source .venv/bin/activate
python main.py

3) Output files

After a successful run, you should get:

• Web_Scrapper/out/results.csv
• Web_Scrapper/out/results.sqlite3 (table name: scraped_items)

4) Scrape another website (CSS selector mode)

Use this if you want to grab headlines from another page.

Example:

python main.py --mode selector --url "https://example.com" --selector "h2 a"

Tip: If nothing is found, try a different selector.

5) Help

Show all options:

python main.py -h