atymri / WebCrawler Public

Notifications You must be signed in to change notification settings
Fork 2
Star 7

WebCrawler is a C# console application that recursively scans a website starting from a given URL, collects all discovered links, and saves them to a file. It’s useful for site mapping, link analysis, and content discovery.

7 stars 2 forks Branches Tags Activity

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 3 Commits
WebCrawler		WebCrawler
.gitattributes		.gitattributes
.gitignore		.gitignore
README.md		README.md
WebCrawler.sln		WebCrawler.sln

Repository files navigation

WebCrawler

A simple C# console application that crawls a website starting from a URL, collects all links, and saves them to a file.

Features

Recursively visits links on a website
Saves all discovered links to a text file

Usage

Run the program.
Enter the starting URL.
The program will crawl and save links to crawled_links.txt.

Requirements

.NET 6 or newer
Internet connection

License

MIT License

About

WebCrawler is a C# console application that recursively scans a website starting from a given URL, collects all discovered links, and saves them to a file. It’s useful for site mapping, link analysis, and content discovery.

github git url cli commandline website crawler opensource web grabber web-crawler web-crawling commandline-interface commandline-tool crawler-engine crawling-sites crawler-csharp

Report repository

Languages

C# 100.0%