web scraping/crawling library
An adaptive Python web scraping framework that provides static/dynamic fetchers, stealth, and flexible selection methods for building scrapers from simple requests to large-scale crawls.
Same problem, same approach. Swapping one for another is a config change, not a rewrite.
Choose if you prefer Crawlee's active Python API for similar scraping tasks.
CrawleeβA web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
Choose the mature, battle-tested Scrapy when you need an extensible crawling framework with a large ecosystem.
Scrapy, a fast high-level web crawling & scraping framework for Python.
Solves the same problem with a different architecture or at a different layer. Expect to rewrite the integration.
Choose when you want LLM-based extraction instead of writing selectors and stealth logic.
Python scraper based on AI
Choose for a Go codebase needing a lightweight scraping/crawling framework.
Elegant Scraper and Crawler Framework for Golang
Choose when building TypeScript/Node scrapers that output LLM-friendly data and SERP results.
AnyCrawl π: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
Choose when your project is in PHP and you need a progressive scraping framework.
:spider: The progressive PHP crawler framework! δΌι ηζΈθΏεΌPHPιιζ‘ζΆγ
Choose for Rust projects needing high-performance crawling and LLM-focused data extraction.
Get web data for AI agents and LLMs - fast, efficient, and reliable with Rust
Choose for Node.js/TypeScript development requiring a robust crawling library.
CrawleeβA web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Choose for Java/JVM projects needing a scalable crawling framework.
A scalable web crawler framework for Java.
Choose when you want to scrape by example without writing explicit selectors.
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
Choose for a lightweight Python library that uses LLMs for web extraction.
Lightweight library for scraping web-sites with LLMs
An earlier or more limited way to do the job. Still the right call when you need something small, proven, or CPU-only.
Choose when you only need article text/metadata extraction rather than full-scale scraping.
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
An older ML-based example-driven scraper; useful for quick extraction but less maintained.
π€ Scrape data from HTML websites automatically by just providing examples
Does the same job, but ships as an app. Useful to a person, not swappable into a codebase.
Choose when you want a deployable scraping API endpoint rather than embedding a library.
The context API to search, scrape, and interact with the web at scale. π₯
Choose when you want to self-host a scraping API instead of coding within Python.
Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up
Choose when you want a Rust CLI/API tool for local content extraction.
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data β all from Rust. CLI, REST API, and MCP server.
Choose when you want a no-code GUI to build scraping APIs rather than write code.
π₯ The open-source no-code platform for web scraping, crawling, search and AI data extraction β’ Turn websites into structured APIs in minutes π₯
Choose when you want a self-hosted scraper web app as an out-of-the-box service.
Self-hosted webscraper.
These projects were analysed and named Scrapling among their alternatives. The relationship is not symmetric β how Scrapling rates them is a separate judgement, made when Scrapling is analysed in its own right.
calls Scrapling βChoose when you want a Python scraping framework with a simpler, adaptive API and less boilerplate.β
Scrapy, a fast high-level web crawling & scraping framework for Python.
calls Scrapling βChoose for a lightweight Python scraping framework with a smaller dependency footprint than Crawlee and similar request/browser automation coverage.β
CrawleeβA web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
calls Scrapling βChoose when you want a Python-native adaptive scraping library with anti-bot handling, rather than calling an external API.β
The context API to search, scrape, and interact with the web at scale. π₯
calls Scrapling βChoose when you need an adaptive Python scraper with strong anti-bot and full-scale crawling, no LLM graph setup.β
Python scraper based on AI
calls Scrapling βChoose when you want a Python adaptive scraping framework with anti-detection features.β
Elegant Scraper and Crawler Framework for Golang
calls Scrapling βChoose for adaptive Python scraping; different language and retrieval strategy.β
CrawleeβA web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
calls Scrapling βChoose when you want an adaptive Python scraping framework with low-level control instead of a no-code platform.β
π₯ The open-source no-code platform for web scraping, crawling, search and AI data extraction β’ Turn websites into structured APIs in minutes π₯
calls Scrapling βChoose over chromedp for Python web-scraping workflows that need adaptive anti-blocking and crawling behavior rather than raw Go CDP control.β
A faster, simpler way to drive browsers supporting the Chrome DevTools Protocol.
calls Scrapling βAdaptive Python scraping framework; choose for large-scale crawling needs with built-in parsing.β
π APIs for web automation, testing, and bypassing bot-detection.
calls Scrapling βChoose it when you prefer a Python scraping framework with adaptive parsing.β
A scalable web crawler framework for Java.
calls Scrapling βChoose it for Python scraping pipelines that need adaptive fetching and crawling strategies rather than Helium's label-based scripts.β
Lighter web automation with Python
calls Scrapling βChoose it for a maintainable Python framework with anti-detection and crawling controls, but write selectors instead of sample-based learning.β
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
calls Scrapling βChoose it over Rod when building scrapers in Python and wanting adaptive anti-bot handling instead of writing low-level Go CDP code.β
A Chrome DevTools Protocol driver for web automation and scraping.
calls Scrapling βWhen you want a higher-level Python scraping framework with adaptive HTTP/browser fetchers rather than writing low-level CDP code.β
Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.
calls Scrapling βUse when you need Python stealth/anti-bot fetching and site-specific selectors rather than automatic main-text extraction.β
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML