WebScraping.AI MCP Server

A Model Context Protocol (MCP) server implementation that integrates with WebScraping.AI for web data extraction capabilities.

Features

• Question answering about web page content
• Structured data extraction from web pages
• HTML content retrieval with JavaScript rendering
• Plain text extraction from web pages
• CSS selector-based content extraction
• Multiple proxy types (datacenter, residential) with country selection
• JavaScript rendering using headless Chrome/Chromium
• Concurrent request management with rate limiting
• Custom JavaScript execution on target pages
• Device emulation (desktop, mobile, tablet)
• Account usage monitoring

Installation# Running with npx

env WEBSCRAPING_AI_API_KEY=your_api_key npx -y webscraping-ai-mcp
```\n\n### Manual Installation
```bash\n\n# Clone the repository
git clone https://github.com/webscraping-ai/webscraping-ai-mcp-server.git
cd webscraping-ai-mcp-server\n\n# Install dependencies
npm install\n\n# Run
npm start
```\n\n### Configuring in Cursor
Note: Requires Cursor version 0.45.6+

The WebScraping.AI MCP server can be configured in two ways in Cursor:
1. **Project-specific Configuration** (recommended for team projects): Create a `.cursor/mcp.json` file in your project directory:
```json
{
  "servers": {
    "webscraping-ai": {
      "type": "command",
      "command": "npx -y webscraping-ai-mcp",
      "env": {
        "WEBSCRAPING_AI_API_KEY": "your-api-key",
        "WEBSCRAPING_AI_CONCURRENCY_LIMIT": "5"
      }
    }
  }
}

2. Global Configuration (for personal use across all projects): Create a ~/.cursor/mcp.json file in your home directory with the same configuration format as above.

If you are using Windows and are running into issues, try using cmd /c "set WEBSCRAPING_AI_API_KEY=your-api-key && npx -y webscraping-ai-mcp" as the command.

This configuration will make the WebScraping.AI tools available to Cursor's AI agent automatically when relevant for web scraping tasks.

License

MIT License - see LICENSE file for details