web-crawler-project

Creating a web-crawler project using Nodejs

Features

• Crawls web pages for links.

• Outputs a report with URLs and their frequency on the page.

Technologies used

• Nodejs
• Jest (unit testing)

Installation

1. Clone Repository git clone https://github.com/AhmedMagedC/web-crawler-project.git
2. Navigate to the project directory cd web-crawler-project
3. Install dependencies npm install

Usage

npm start website-url

Example Output

======================
REPORT
======================
URL (www.example.com) appeared 2 times
URL (www.example.com/page) appeared 3 times
URL (www.example.com/404) appeared 1 time (Status: 404)
...
======================
END REPORT
======================