🕷 Crawl4AI

Crawl4AI is an async web crawler optimised for feeding content into LLMs and RAG pipelines. It returns clean Markdown, structured JSON, or raw HTML from any URL — including JavaScript-rendered SPAs — with built-in filtering to strip noise before it reaches your model.

• Returns clean Markdown ready for LLM ingestion
• LLMExtractionStrategy for structured JSON output
• Handles JS-rendered pages and SPAs via Playwright
• PruningContentFilter and BM25ContentFilter for noise removal
• arun_many() for fast parallel crawling with rate limiting