Skip to content

Deepcrawl

Active
lumpinif/deepcrawl

An open-source Firecrawl alternative that turns public web pages into clean Markdown and link trees for AI agents, with API and auth Workers.

View on GitHub
Stars
702
Forks
80
Watchers
702
Open issues
2

Overview

Deepcrawl is an agent-oriented platform for extracting website context. It returns the cleaned Markdown of a page’s content, an agent-friendly hierarchical links tree and metadata that LLMs can digest with minimal tokens — reducing context switching and hallucination.

It is positioned as a free, open-source alternative to Firecrawl. The whole platform is open: a Next.js dashboard, API Workers, auth Workers and the database.

Scope

Deepcrawl is built for high-frequency agent workloads that read public pages. Its README states that it does not target anti-scraping or anti-bot use.

Status

The project is under rapid development, and its authors warn against using it in production yet. Documentation lives at deepcrawl.dev/docs.

Similar open-source projects, ranked by shared categories and tags.

All AI projects →