Integration

Spider integration

Spider is a web crawling, scraping and search API built for AI pipelines. Scrape one page or a list of pages, crawl a whole site with page and depth caps, collect the links of a site without downloading content, run a web search, or capture a screenshot, with pages returned as raw HTML, markdown, text or XML, optional metadata, CSS-selector extraction, a headless browser when a page needs JavaScript, and residential proxies. The Unlimited plan variants of scrape, crawl and links are included for accounts on that plan. Not covered: the prompt-driven AI routes (they need an AI plan and their responses are not documented field by field), the HTML transform route, the per-domain fetch route, streaming and background crawls with webhooks, the credit balance read, and the deprecated unblocker. Authentication is an API key from the Spider dashboard, sent as Authorization: Bearer <key>; usage is paid from prepaid credits.

Connect Spider How integrations work

Ask it in chat

Say what should happen.
The workflow builds itself.

When this happens, do that in Spider.Built from the 8 Spider tools below, on the connection you add once.

What you can do with Spider

Each of these is one operation on one endpoint, available both as a workflow step and as a tool an AI agent can call on its own. Nothing bundles several operations into one action, so what a run did stays readable afterwards.

Most endpoints run on the free plan. A few, mainly the ones that publish on your behalf, need a paid plan.

Page, 4 operations

crawl_websiteCrawl websitePOSTCrawls a website from a starting URL, following its links within the site, and returns the content of every page found, one result per page.
scrape_pageScrape pagePOSTFetches one page (or each page of a comma-separated list of URLs) without following links and returns its content, by default as raw HTML.
unlimited_crawl_websiteUnlimited crawl websitePOSTCrawls a website from a starting URL, following its links within the site, and returns every page's content, one result per page. Needs an active Unlimited plan (403 without; else use crawl_website). No credits; 429 + Retry-After if all seats busy.
unlimited_scrape_pageUnlimited scrape pagePOSTFetches one page (or each of a comma-separated list of URLs) without following links and returns its content, raw HTML by default. Needs an active Unlimited plan (403 without; else use scrape_page). No credits; 429 with Retry-After if all seats busy.

Link, 2 operations

list_linksList linksPOSTCrawls a website from a starting URL and returns the URLs found on each page, without the page content.
unlimited_list_linksUnlimited list linksPOSTCrawls a website from a starting URL and returns the URLs found on each page, without the page content. Needs an active Unlimited plan (403 without it; then use list_links). Uses no credits; 429 with Retry-After when all seats are busy, never queued.

Screenshot, 1 operation

take_screenshotTake screenshotPOSTRenders a page in a browser and captures it as an image, returned as a base64 string in content (PNG unless cdp_params sets another format). One result per URL.

Search result, 1 operation

search_webSearch webPOSTRuns a web search and returns the ranked results, each with its title, URL and description. To read the pages themselves, pass the result URLs to scrape_page.

Use Spider from Claude Code, Cursor or Codex

Connect Spider in LiveContext, create an API key in Settings, then add the LiveContext MCP server to your client. Its Spider operations become tools the client can call, next to every other integration you connected.

Claude Code

claude mcp add --transport http livecontext https://livecontext.ai/mcp --header "X-API-Key: YOUR_API_KEY"

Codex (~/.codex/config.toml)

[mcp_servers.livecontext]
url = "https://livecontext.ai/mcp"

[mcp_servers.livecontext.http_headers]
"X-API-Key" = "YOUR_API_KEY"

Cursor, Claude Desktop and other JSON configs

{
  "mcpServers": {
    "livecontext": {
      "type": "http",
      "url": "https://livecontext.ai/mcp",
      "headers": {
        "X-API-Key": "YOUR_API_KEY"
      }
    }
  }
}

Set up the MCP server step by step

How to connect Spider

Create your workspace
Sign up free. Nothing to install.
Connect Spider once
Add your Spider connection in Settings. Every tool from this integration reuses it, and secrets are encrypted at rest and visible only to you.
Describe the job in chat
The workflow builds itself in front of you and picks the endpoints it needs. Drag the steps yourself on the canvas whenever you would rather be explicit.