D4Vinci/Scrapling

What it does

Scrapling bundles the usual scraping stages into one Python library: fast HTTP requests with browser TLS impersonation, a stealth browser for JavaScript-heavy or Cloudflare-protected pages, CSS/XPath/text/regex parsing, and a concurrent spider framework with retries, proxy rotation, and export to JSON, JSONL, CSV, or XML. The same library also exposes a CLI, an interactive shell, and an MCP server so AI agents can browse and extract data. It suits developers who start with a single page and later need a scheduled crawl of a whole site.

A parsing detail worth knowing: elements found through selectors can be relocated after a site changes its markup by matching similar elements, instead of failing outright.

Quick start

Scrapling requires Python 3.10 or later. The base install only contains the parser; fetching and crawling need the fetchers extra and browser binaries:

pip install "scrapling[fetchers]"
scrapling install

Then fetch one page and extract text:

from scrapling.fetchers import Fetcher

page = Fetcher.get("https://quotes.toscrape.com/")
quotes = page.css(".quote .text::text").getall()
print(quotes)

To crawl and save results, define a subclass of Spider with start_urls and an async parse callback, then call .start() and export via result.items.to_json("quotes.json"). The README example against the same quotes site prints 10 and writes the authors and texts to the file (README examples). Headless browser fetchers need the assets installed by scrapling install; plain HTTP requests do not.

If you prefer not to write code, the extract CLI command saves a page as .txt, .md, or .html, optionally filtered by --css-selector (CLI docs). Install scrapling[shell] first and use scrapling shell for the interactive version.

Constraints

• The default pip install scrapling deliberately omits fetchers and CLI dependencies; importing scrapling.fetchers or scrapling.spiders from that install raises ModuleNotFoundError.
• Browser-based fetching (StealthyFetcher, stealth sessions) needs the browser and fingerprint dependencies from scrapling install, or the Docker image pyd4vinci/scrapling. HTTP fetching does not.
• robots_txt_obey is optional and off by default; respecting Disallow, Crawl-delay, and Request-rate is the caller's choice.
• The upstream disclaimer states the library is for educational and research purposes than blank; users are responsible for complying with local laws and site terms of service.
• License is BSD-3-Clause.

HotRepo's take

On the 2026-11-01 Python daily list, Scrapling recorded 844 new stars and a best position of 6 in that day's language ranking. That is a single snapshot, so it shows one strong day in the Python listing, not a sustained trend; there is no recorded all-language daily rank to compare against. The project has also been discussed on Hacker News twice, once at launch in October 2024 and once for a v0.2.99 release note in April 2025, but both threads attracted minimal discussion (4 and 1 points), so there is no substantive external commentary to weigh here. The growth reads more as a category signal — a scraping library that also ships an MCP server and agent skill — than as evidence of adoption quality.