anansi

The spider that learns.

A self-healing web scraper for hostile sites — and it's driveable by any LLM.

Every scraper starts working. The question is how long before it breaks. Anansi is built on a different assumption: the web is adversarial and unstable, and your scraper should handle that without your involvement.

When a site changes its layout, Anansi finds the data anyway and remembers the fix. When a page needs a browser to render, it switches to one silently. When bot detection gets in the way, it presents a coherent identity — matched TLS fingerprint, persona, and headers — that works to slip past detection instead of tripping it. And when you re-crawl, unchanged pages are skipped before a request is even made. The result is a crawler that survives redesigns, handles hostile sites, and gets better the longer it runs.

Ships with an MCP server so any LLM can drive a full crawl through conversation.

Highlights

• Selectors that repair themselves — CSS selectors carry confidence scores; when one breaks, four healing strategies compete and the winner is persisted. How it works →
• A browser only when you need one — every response is checked for JS shells and silently retried in a stealth Playwright browser, cached per domain. How it works →
• A coherent anti-bot identity — matched TLS/HTTP-2 fingerprint, persona, and headers, with vendor-aware handling of Cloudflare, Akamai, and DataDome. Anti-bot & identity →
• Re-crawls that skip the unchanged — ETag, Last-Modified, content hashing, and sitemap <lastmod> filtering skip pages before the request goes out. Capabilities →
• Data you can trust — attach a Pydantic item_schema and every scraped item is validated and coerced before it hits your database. Getting started →
• Driveable by any LLM — a FastMCP server exposes 17 tools so an agent can fetch, extract, crawl, and screenshot through conversation. MCP server →

See the full capability reference for everything else — proxy rotation, adaptive rate limiting, URL canonicalization, JS interaction, network capture, and more.

Install

# Core install
pip install "git+https://github.com/mdowis/anansi"

# For browser-based fetching (Cloudflare bypass, JS rendering):
playwright install chromium

# For TLS-fingerprint mimicry (curl-cffi impersonation):
pip install "anansi-scraper[tls] @ git+https://github.com/mdowis/anansi"

Full install matrix, extras, and Windows notes are in Getting started.

Quickstart

from pydantic import BaseModel
from anansi import Crawler
from anansi.core import Item, Request, Response
from anansi.spider.spider import Spider

class ProductItem(BaseModel):
    title: str
    price: float        # "49.99" strings are auto-coerced
    sku: str | None = None

class ShopSpider(Spider):
    name = "shop"
    start_urls = ["https://shop.example.com/products"]
    item_schema = ProductItem   # validate every yielded item against this model

    async def parse(self, response: Response):
        for link in response.css("a.product-link"):
            yield Request(response.urljoin(link["href"]), callback="parse_product")

    async def parse_product(self, response: Response):
        yield Item({"title": response.css("h1")[0].get_text(), "url": response.url})

# Self-healing, browser auto-upgrade, adaptive rate limiting, and incremental
# re-crawls are all on by default.
crawler = Crawler(ShopSpider, concurrency=10, max_pages=1000)

async for item in crawler.run():
    print(item.data)

More examples — structured-data extraction, pausing and resuming, proxies, exporting — in Getting started.

Drive it from any LLM

Anansi ships a FastMCP server exposing 17 scraping tools over stdio or SSE, so an LLM agent can run a full crawl through conversation:

# Register with Claude Code
claude mcp add anansi -- anansi-mcp

Works with Claude Code, Claude Desktop, Cursor, Windsurf, ChatGPT, LangChain, and the OpenAI Agents SDK. Setup for each is in the MCP server guide.

Documentation

• Getting started — install, first fetch and crawl, the CLI.
• Capabilities — the full feature reference.
• How it works — self-healing extraction, browser auto-upgrade, and adaptive rate limiting internals.
• Anti-bot & identity — TLS fingerprint mimicry, coherent personas, crawler impersonation, vendor-aware escalation, sticky sessions, proxy scoring, CAPTCHA, operator controls.
• MCP server — run the server and drive Anansi from any LLM (all client configs).
• Architecture — package layout at a glance.

Legal / Acceptable Use

Anansi is a powerful scraping tool. You are solely responsible for how you use it. Before scraping any site, ensure you have the right to access and use the data and that you comply with the site's Terms of Service, its robots.txt, applicable rate limits, and all relevant laws (including computer-misuse statutes such as the CFAA and data-protection law such as GDPR/CCPA).

The anti-bot, TLS-fingerprint-impersonation, and Cloudflare-handling features are intended for authorized testing, research, and scraping of content you have the right to access — not for circumventing access controls without permission. See DISCLAIMER.md for the full statement and Anti-bot & identity for operator controls.

License

Licensed under the Apache License, Version 2.0 — see LICENSE and NOTICE. Use of this software is additionally subject to the acceptable-use terms in DISCLAIMER.md.