web scraping / content extraction
Python package and CLI for crawling web pages and extracting main text/metadata from raw HTML into clean formats such as Markdown, JSON, XML, CSV and plain text.
Solves the same problem with a different architecture or at a different layer. Expect to rewrite the integration.
Use when you need Python stealth/anti-bot fetching and site-specific selectors rather than automatic main-text extraction.
🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!
Use when you are building crawlers in Go and want a framework with middleware and collector control instead of a Python extraction library.
Elegant Scraper and Crawler Framework for Golang
Use when you need a Java crawler framework with pipeline/downloader abstractions in a JVM codebase.
A scalable web crawler framework for Java.
Use when you can provide sample pages and want learned field-level data extraction rather than generic boilerplate-free text.
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
Use when you are in a Node/TypeScript pipeline and need LLM-ready crawl output and SERP extraction.
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
Use when you need a PHP scraper framework with jQuery-style DOM traversal.
:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
Use when you want an asynchronous Rust crawler for native LLM-data or agent pipelines.
Get web data for AI agents and LLMs - fast, efficient, and reliable with Rust
Use when you need a full Python scraping framework with custom spiders, middlewares and pipelines instead of out-of-the-box article extraction.
Scrapy, a fast high-level web crawling & scraping framework for Python.
Use when you want LLM-driven extraction to arbitrary structured schemas and can tolerate heavier inference cost.
Python scraper based on AI
Use when you need browser automation and configurable crawling in Node/TypeScript rather than an embeddable Python extractor.
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Use when you need browser-automated Python crawling with queues and storage primitives instead of single-call text extraction.
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
Use when you prefer a self-hosted HTTP scraping API returning markdown/JSON and handling anti-detection over embedding a Python library.
Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up
Use when you want a local Rust CLI/REST/MCP extraction service rather than a Python package and CLI.
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
Use when you want simpler LLM-based scraping in Python and can accept GPL-2.0 licensing.
Lightweight library for scraping web-sites with LLMs
Use when you want a scalable self-hosted/cloud API for scraping and converting websites to markdown/structured data.
The context API to search, scrape, and interact with the web at scale. 🔥
Use when you need a Go crawling/scraping framework with JS rendering support.
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
Use when you are in Elixir/Erlang and want a BEAM-native high-level crawling/scraping framework.
Crawly, a high-level web crawling & scraping framework for Elixir.
Use when you want a lightweight Rust self-hosted scrape/crawl/search API with MCP support rather than a Python/CLI toolkit.
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
Use when you want a self-hosted web scraping service that returns clean markdown and browser sessions for agents.
Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents.
Use when you want an LLM-oriented crawler API that emits clean Markdown, rather than a Python library you import.
High-performance web crawler API optimized for LLMs. Turn any search or website into clean Markdown using remote browsers. Firecrawl alternative
An earlier or more limited way to do the job. Still the right call when you need something small, proven, or CPU-only.
Use when you already have HTML and need SLM-based clean-body extraction only, since it lacks crawl/fetch capabilities.
MinerU-HTML: An SLM-powered HTML main content extractor that outputs clean HTML bodies. Perfect for Deep Research Agents, RAG applications, and training data generation.
These projects were analysed and named trafilatura among their alternatives. The relationship is not symmetric — how trafilatura rates them is a separate judgement, made when trafilatura is analysed in its own right.
calls trafilatura “Choose when you need a Python/CLI text and metadata extraction tool for static pages, without JavaScript rendering or an API.”
The context API to search, scrape, and interact with the web at scale. 🔥
calls trafilatura “Choose when you need lightweight text and metadata extraction to build clean datasets, no LLM or crawling pipelines.”
Python scraper based on AI
calls trafilatura “Choose for text-centric scraping and extraction; also offers a CLI tool.”
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
calls trafilatura “Choose when you only need article text/metadata extraction rather than full-scale scraping.”
🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!
calls trafilatura “Choose for extracting clean text and metadata at scale without building full spiders.”
Scrapy, a fast high-level web crawling & scraping framework for Python.
calls trafilatura “Choose when you only need clean text/metadata extraction from pages, not a full general-purpose crawler.”
Elegant Scraper and Crawler Framework for Golang
calls trafilatura “Choose it when you focus on extracting clean text/metadata rather than running a general crawler.”
A scalable web crawler framework for Java.
calls trafilatura “Choose for extracting clean text/metadata from pages without building a full crawler; it is fast and minimal.”
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
calls trafilatura “Choose it when you only need main text/metadata from articles, not arbitrary element extraction based on sample data.”
A Smart, Automatic, Fast and Lightweight Web Scraper for Python