Crawl4AI

🚀🤖 Crawl4AI, Open-source LLM-Friendly Web Crawler & Scraper

Crawl4AI is the #1 trending GitHub repository, actively maintained by a vibrant community. It delivers blazing-fast, AI-ready web crawling tailored for large language models, AI agents, and data pipelines. Fully open source, flexible, and built for real-time performance, Crawl4AI empowers developers with unmatched speed, precision, and deployment ease. Supercharge your AI coding assistant with complete Crawl4AI knowledge! Download our comprehensive skill package that includes: Works with Claude, Cursor, Windsurf, and other AI coding assistants. Import the .zip file into your AI assistant's skill/knowledge system. Crawl4AI now features intelligent adaptive crawling that knows when to stop! Using advanced information foraging algorithms, it determines when sufficient information has been gathered to answer your query. Here's a quick example to show you how easy it is to use Crawl4AI with its asynchronous capabilities: Crawl4AI is a feature-rich crawler and scraper that aims to: To help you get started, we’ve organized our docs into clear sections: Throughout these sections, you’ll find code samples you can copy-paste into your environment. If something is missing or unclear, raise an issue or PR. Thank you for joining me on this journey. Let’s keep building an open, democratic approach to data extraction and AI together.