Support locally hosted website crawler/scraper during search.

[lukolszewski]

Hi, While a simple web retrieval mechanism is already supported by MCP it would be nice if a user had an option of using a local running scraper instead of a paid service like firecrawl and serper during the search as well.

I've implemented such a feature locally using the /md endpoint of Crawl4AI and was going to send a PR, but I saw the contribution guidelines about creating a discussion first. So here I'm creating a discussion about it.

The intention is for the functionality to be exactly the same for the user as serper/firecrawl, the only difference being setting the scraper type to crawl4ai, setting the URL and optionally a key (crawl4ai can be used without auth locally).

A potential enhancement could be to use the /llm endpoint of Crawl4AI which can receive a query that gets processed by an LLM before returning.

The feature requires modification of both agents and LibreChat. I've implemented it in my fork here: https://github.com/lukolszewski/LibreChat/tree/feature/add-crawl4ai-local-scraping and here: https://github.com/lukolszewski/agents/tree/feature/add-crawl4ai-local-scraping

Click to expand - suggestions how to test

Here is how I suggest one could test it (the feature works, but the specific way to run it will depend on environment): - please note you have to build agents locally and I pack it into a tarball (we assume both agents and LibreChat are in subfolders) for example:

cd agents
git checkout feature/add-crawl4ai-scraper
npm install
npm run build
npm pack

this will create the tarball which you can then copy to LibreChat

cp librechat-agents-3.0.13.tgz ../LibreChat/

Then you update the reference in package.json in LibreChat
Change: "@librechat/agents": "^3.0.5"
To: "@librechat/agents": "file:librechat-agents-3.0.13.tgz"

and install from tarball:

npm install file:librechat-agents-3.0.13.tgz

Then you build normally (If you're rebuilding you may need to do more, we keep the same version and replace the file so hashes may need to be replaced)

To configure this add the following to your .env file (assuming you're running on same docker host, with host network):

CRAWL4AI_API_KEY=
CRAWL4AI_API_URL=http://host.docker.internal:11235

Add to librechat.yaml for example(here we use searxng for search):

version: 1.1.9
  webSearch:
    searchProvider: 'searxng'
    scraperProvider: 'crawl4ai'
    crawl4aiApiUrl: '${CRAWL4AI_API_URL}'
    # Optional settings:
    crawl4aiOptions:
      timeout: 10000

Then use LibreChat normally

This assumes your Crawl4AI is running locally.