Crawlee for Python is an open-source library designed to simplify web scraping and crawling tasks. It provides a unified interface for both HTTP and headless browser crawling, making it easier to extract data from websites efficiently. With Crawlee, you can build scrapers that fly under the radar of modern bot protections, thanks to its advanced features like automatic proxy rotation, session management, and retries on errors.
The library supports multiple crawling strategies, including BeautifulSoupCrawler for efficient HTML parsing and PlaywrightCrawler for handling JavaScript-heavy websites. It also offers persistent storage for URLs and scraped data, allowing you to resume interrupted scraping sessions without starting from scratch.
Crawlee is built with modern Python features like type hints and asyncio, ensuring better performance and developer experience. It integrates seamlessly with other Python libraries and can be deployed anywhere, including the Apify platform for cloud-based scraping.
Key features include:
- Automatic parallel crawling: Optimizes performance based on system resources.
- Configurable request routing: Directs URLs to appropriate handlers.
- Pluggable storage: Supports both tabular data and files.
- State persistence: Saves progress during interruptions.
- Rich configuration options: Customize almost any aspect of Crawlee to fit your project's needs.
Crawlee is available as a PyPI package (crawlee) and can be installed with optional extras for additional functionality. The project is actively maintained and welcomes contributions from the community.