feat: AsyncPlasmateCrawlerStrategy — lightweight alternative to Playwright (no Chrome)

[dbhurley]

Summary

Adds AsyncPlasmateCrawlerStrategy — a drop-in alternative to AsyncPlaywrightCrawlerStrategy using Plasmate instead of Chrome.

Directly addresses:

• [Bug]: Memory Leak on Repeated /md Requests via Docker (MacOS) — Container Crashes Randomly Over Time #1256 — Memory leak / crash in Docker from Chrome processes. Plasmate uses ~64MB RAM per session vs ~300MB and leaves no persistent browser process.
• feat: expose token usage in CrawlResult (#1745) #1874 — Token usage tracking. Plasmate returns pre-processed content (text / markdown / SOM) instead of raw HTML, cutting upstream token counts by 10-100× before any LLM call.

What Plasmate is

Open-source Rust browser engine (Apache 2.0). Fetches pages and returns them as Structured Object Model (SOM) — a compact, semantically clean representation with nav, ads, cookie banners, and boilerplate stripped. Install: pip install plasmate.

Compression measured across 45 real sites: 17.7× average, 77× peak. Every token saved before the LLM is a direct cost reduction.

Drop-in usage

from crawl4ai import AsyncWebCrawler
from crawl4ai.async_plasmate_strategy import AsyncPlasmateCrawlerStrategy

strategy = AsyncPlasmateCrawlerStrategy(
    output_format="markdown",   # text | markdown | som | links
    timeout=30,
    fallback_to_playwright=True,  # retry with Playwright for JS-heavy SPAs
)

async with AsyncWebCrawler(crawler_strategy=strategy) as crawler:
    result = await crawler.arun("https://docs.python.org/3/")
    print(result.markdown[:500])

What changed

File	Change
crawl4ai/async_plasmate_strategy.py	New AsyncPlasmateCrawlerStrategy implementing AsyncCrawlerStrategy ABC
crawl4ai/__init__.py	Export AsyncPlasmateCrawlerStrategy
tests/general/test_plasmate_strategy.py	20 unit tests (init, cmd building, crawl, fallback, concurrency)

Comparison

	AsyncPlaywrightCrawlerStrategy	AsyncPlasmateCrawlerStrategy
RAM per session	~300MB	~64MB
Chrome required	Yes	No
Tokens per page (avg)	~75,000 (raw HTML)	~4,200 (SOM/text)
JS rendering	Yes	No (use fallback_to_playwright=True)
Install	playwright install (~300MB browser)	pip install plasmate
Persistent process	Yes (browser stays alive)	No (subprocess per fetch)

Notes

• No breaking changes — existing AsyncPlaywrightCrawlerStrategy usage is untouched
• fallback_to_playwright=True makes it safe for mixed static/SPA crawls
• Subprocess runs in asyncio executor — fully non-blocking, safe for concurrent gather() calls
• Tested with Python 3.9+

[mshi-hacks]

A Chrome-free crawling strategy is a big deal for containerized environments where Playwright's Chromium binary is a pain to manage — image sizes, security patches, and arm64 compatibility all improve significantly. The drop-in interface matching AsyncPlaywrightCrawlerStrategy is the right design choice. One thing I'm curious about: how does Plasmate handle JavaScript-heavy pages that render content via client-side frameworks? Is there a JS execution mode, or is it primarily for static/SSR content?

[dbhurley]

Hi @unclecode — bumping this gently three weeks in. Picking up on @mshi-hacks's note above: the chrome-free angle is the bigger story than the integration itself. Container size, ARM64 support, and the security-patch treadmill on Chromium binaries are real pain points for anyone running Crawl4AI on Cloud Run / Lambda / Fly / Pi-class hardware.

Happy to:

• Rebase against latest main
• Add anything you'd like to see in the test matrix (specific URLs, behaviors)
• Move bits to a separate crawl4ai-plasmate adapter package if keeping the core lean is preferred — the API surface is small enough to ship either way

No rush — just want to make sure it's not stuck behind something I can clear. Thanks for the project.