Crawler For Llm

Name: Crawler For Llm
Author: KonghaYao

@KonghaYao

About Crawler For Llm

高等级的免费爬虫MCP, 完全 Markdown 化支持，支持微信公众号等一大批国内网站。A powerful web crawler designed specifically for LLM applications, capable of extracting clean, readable content from various web pages and converting it to Markdown format. This tool is essential for building knowledge bases, training

#web-crawler #markdown-converter #content-extraction #llm-data-preparation #knowledge-base-building #model-context-protocol-mcp

Config

Add this server to your MCP-compatible client using the configuration below.

{
  "mcpServers": {
    "langgraph-crawler": {
      "command": "npx",
      "args": [
        "-y",
        "@langgraph-js/crawler-mcp@latest"
      ]
    }
  }
}

Tools

A powerful web content extraction tool that retrieves and processes raw content from specified URLs, ideal for data collection, content analysis, and research tasks.

A powerful web search tool that provides comprehensive, real-time results using search engine. Returns relevant web content with customizable parameters for result count, content type, and domain filtering. Ideal for gathering current information, news, and detailed web content analysis.

Overview

What is Crawler For Llm?

Crawler For Llm is a universal web crawler designed for LLM applications, extracting clean, readable content from web pages and converting it to Markdown. It includes built-in support for the Model Context Protocol (MCP) for enhanced context management, and is intended for building knowledge bases, training data collection, and content aggregation.

How to use Crawler For Llm?

Add the provided MCP server configuration to your project's settings. Use the command npx -y @langgraph-js/crawler-mcp@latest with the server name langgraph-crawler.

Key features of Crawler For Llm

Universal web crawler with MCP integration
Extracts clean content and converts to Markdown
Supports documentation, development, and Chinese platforms
Continuously expanding list of supported websites
Built for LLM knowledge bases and training data

Use cases of Crawler For Llm

Building knowledge bases for LLM applications
Collecting training data from diverse web sources
Aggregating content from multiple platforms

FAQ from Crawler For Llm

—

Comments

Basic information

More Browser Automation MCP servers

Fastcrw

Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary.

286