TM

The missing link between AI models and the real‑world web: one API that delivers clean, structured data from any site.

Created 3 months ago

The missing link between AI models and the real‑world web: one API that delivers clean, structured data from any site.

development location documentation public web scraping AI

What is The missing link between AI models and the real‑world web: one API that delivers clean, structured data from any site.?

The Oxylabs MCP server provides a bridge between AI models and the web. It enables them to scrape any URL, render JavaScript-heavy pages, extract and format content for AI use, bypass anti-scraping measures, and access geo-restricted web data from 195+ countries. This implementation leverages the Model Context Protocol (MCP) to create a secure, standardized way for AI assistants to interact with web content.

Documentation

smithery badge pypi package Licence Verified on MseeP Coverage badge

📖 Overview

The Oxylabs MCP server provides a bridge between AI models and the web. It enables them to scrape any URL, render JavaScript-heavy pages, extract and format content for AI use, bypass anti-scraping measures, and access geo-restricted web data from 195+ countries.

This implementation leverages the Model Context Protocol (MCP) to create a secure, standardized way for AI assistants to interact with web content.


Why Oxylabs MCP?  🕸️ ➜ 📦 ➜ 🤖

Imagine telling your LLM "Summarise the latest Hacker News discussion about GPT‑7" – and it simply answers.
MCP (Multi‑Client Proxy) makes that happen by doing the boring parts for you:

What Oxylabs MCP does Why it matters to you
Bypasses anti‑bot walls with the Oxylabs global proxy network Keeps you unblocked and anonymous
Renders JavaScript in headless Chrome Single‑page apps, sorted
Cleans HTML → JSON Drop straight into vector DBs or prompts
Optional structured parsers (Google, Amazon, etc.) One‑line access to popular targets

✨ Key Features

🛠️ MCP Tools

Oxylabs MCP provides two sets of tools that can be used together or independently:

Oxylabs Web Scraper API Tools

  1. universal_scraper: Uses Oxylabs Web Scraper API for general website scraping.
  2. google_search_scraper: Uses Oxylabs Web Scraper API to extract results from Google Search.
  3. amazon_search_scraper: Uses Oxylabs Web Scraper API to scrape Amazon search result pages.
  4. amazon_product_scraper: Uses Oxylabs Web Scraper API to extract data from individual Amazon product pages.

Oxylabs AI Studio Tools

The Oxylabs AI Studio MCP server provides various AI tools for your agents:

  1. ai_scraper: Scrape content from any URL in JSON or Markdown format with AI-powered data extraction.
  2. ai_crawler: Based on a prompt, crawls a website and collects data in Markdown or JSON format across multiple pages.
  3. ai_browser_agent: Given a task, the agent controls a browser to achieve the given objective and returns data in Markdown, JSON, HTML, or screenshot formats.
  4. ai_search: Search the web for URLs and their contents with AI-powered content extraction.

💡 Example Queries

When you've set up the MCP server with Claude, you can make requests like:

Web Scraper API Examples

  • Could you scrape https://www.google.com/search?q=ai page?
  • Scrape https://www.amazon.de/-/en/Smartphone-Contract-Function-Manufacturer-Exclusive/dp/B0CNKD651V with parse enabled
  • Scrape https://www.amazon.de/-/en/gp/bestsellers/beauty/ref=zg_bs_nav_beauty_0 with parse and render enabled
  • Use web unblocker with render to scrape https://www.bestbuy.com/site/top-deals/all-electronics-on-sale/pcmcat1674241939957.c

AI Studio Examples

  • Use AI scraper to get top news headlines from https://news-site.com in JSON format.
  • Use AI crawler with prompt "extract all product information" to crawl https://example-store.com
  • Use browser agent with task "log in and extract dashboard data" on https://complex-app.com
  • Use AI search to find 5 "latest AI developments" and return URLs with their content

✅ Prerequisites

Before you begin, make sure you have:

  • Oxylabs Web Scraper API Account: Obtain your username and password from Oxylabs (1-week free trial available)
  • Oxylabs AI Studio API Key (Optional): For AI-powered tools, obtain your API key from Oxylabs AI Studio (separate service)

Basic Usage

Via Smithery CLI:

  • Node.js (v16+)
  • npx command-line tool

Via uv:

  • uv package manager – install it using this guide

Local/Dev Setup

  • Python 3.12+
  • uv package manager – install it using this guide

🧩 API Parameters

The Oxylabs MCP Universal Scraper accepts these parameters:

Parameter Description Values
url The URL to scrape Any valid URL
render Use headless browser rendering html or None
geo_location Sets the proxy's geo location to retrieve data. Brasil, Canada, etc.
user_agent_type Device type and browser desktop, tablet, etc.
output_format The format of the output links, md, html

🔧 Configuration

Install MCP Server

  1. Go to https://smithery.ai/server/@oxylabs/oxylabs-mcp
  2. Login with GitHub
  3. Find the Install section
  4. Follow the instructions to generate the config

Auto install with Smithery CLI

npx -y @smithery/cli@latest install @upstash/context7-mcp --client claude --key <smithery_key>
  1. Install the uv
curl -LsSf https://astral.sh/uv/install.sh | sh

# Windows
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
  1. Use the following config
{
  "mcpServers": {
    "oxylabs": {
      "command": "uvx",
      "args": ["oxylabs-mcp"],
      "env": {
        "OXYLABS_USERNAME": "OXYLABS_USERNAME",
        "OXYLABS_PASSWORD": "OXYLABS_PASSWORD",
        "OXYLABS_AI_STUDIO_API_KEY": "OXYLABS_AI_STUDIO_API_KEY"
      }
    }
  }
}
  1. Install the uvx
curl -LsSf https://astral.sh/uv/install.sh | sh

# Windows
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
  1. Use the following config
{
  "mcpServers": {
    "oxylabs": {
      "command": "uv",
      "args": [
        "--directory",
        "/<Absolute-path-to-folder>/oxylabs-mcp",
        "run",
        "oxylabs-mcp"
      ],
      "env": {
        "OXYLABS_USERNAME": "OXYLABS_USERNAME",
        "OXYLABS_PASSWORD": "OXYLABS_PASSWORD",
        "OXYLABS_AI_STUDIO_API_KEY": "OXYLABS_AI_STUDIO_API_KEY"
      }
    }
  }
}

Manual Setup with Claude Desktop

Navigate to Claude → Settings → Developer → Edit Config and add one of the configurations above to the claude_desktop_config.json file.

Manual Setup with Cursor AI

Navigate to Cursor → Settings → Cursor Settings → MCP. Click Add new global MCP server and add one of the configurations above.


⚙️ Environment variables

Oxylabs MCP server supports the following environment variables

Name Description Default
OXYLABS_USERNAME Your Oxylabs Web Scraper API username
OXYLABS_PASSWORD Your Oxylabs Web Scraper API password
OXYLABS_AI_STUDIO_API_KEY Your Oxylabs AI Studio API key
LOG_LEVEL Log level for the logs returned to the client INFO

*At least one set of credentials (Web Scraper API or AI Studio) is required to use the MCP server.

Credential Requirements

The Oxylabs MCP server supports two independent services:

  • Oxylabs Web Scraper API: Requires OXYLABS_USERNAME and OXYLABS_PASSWORD
  • Oxylabs AI Studio: Requires OXYLABS_AI_STUDIO_API_KEY

You can use either service independently or both together. The server will automatically detect which credentials are available and enable the corresponding tools.


📝 Logging

Server provides additional information about the tool calls in notification/message events

{
  "method": "notifications/message",
  "params": {
    "level": "info",
    "data": "Create job with params: {\"url\": \"https://ip.oxylabs.io\"}"
  }
}
{
  "method": "notifications/message",
  "params": {
    "level": "info",
    "data": "Job info: job_id=7333113830223918081 job_status=done"
  }
}
{
  "method": "notifications/message",
  "params": {
    "level": "error",
    "data": "Error: request to Oxylabs API failed"
  }
}

🛡️ License

Distributed under the MIT License – see LICENSE for details.


About Oxylabs

Established in 2015, Oxylabs is a market-leading web intelligence collection platform, driven by the highest business, ethics, and compliance standards, enabling companies worldwide to unlock data-driven insights.

image

Server Config

{
  "mcpServers": {
    "the-missing-link-between-ai-models-and-the-real‑world-web:-one-api-that-delivers-clean,-structured-data-from-any-site.-server": {
      "command": "npx",
      "args": [
        "the-missing-link-between-ai-models-and-the-real‑world-web:-one-api-that-delivers-clean,-structured-data-from-any-site."
      ]
    }
  }
}

Links & Status

Repository: github.com
Hosted: Yes
Global: Yes
Official: Yes

Project Info

Hosted Featured
Created At: Aug 07, 2025
Updated At: Aug 07, 2025
Author: Oxylabs
Category: Web Scraping
License: MIT License
Tags:
development location documentation