Integration

ScrapFly integration

Web scraping, data extraction, screenshot capture, and web crawling API. Provides proxy rotation, anti-scraping protection bypass, JavaScript rendering, structured data extraction with AI/LLM prompts, and full-site crawling. Handles anti-bot measures, caching, sessions, and delivers content in multiple formats including HTML, markdown, JSON, and plain text.

Connect ScrapFly How integrations work

Ask it in chat

Say what should happen.
The workflow builds itself.

When this happens, do that in ScrapFly.Built from the 9 ScrapFly tools below, on the connection you add once.

Common ScrapFly use cases

data pipeline ingestion
competitive analysis
bulk data export
crawl monitoring
price monitoring
sitemap building
usage monitoring
visual monitoring
site-wide scraping
product data extraction

What you can do with ScrapFly

Each of these is one operation on one endpoint, available both as a workflow step and as a tool an AI agent can call on its own. Nothing bundles several operations into one action, so what a run did stays readable afterwards.

Most endpoints run on the free plan. A few, mainly the ones that publish on your behalf, need a paid plan.

Crawl, 2 operations

create_crawlCreate crawlPOSTStarts a web crawl job from a seed URL. Follows links with configurable depth, page limits, and path filtering. Returns a crawl UUID for status monitoring.
get_crawl_statusGet crawl statusGETGets the current status and progress of a crawl job. Returns status (PENDING, RUNNING, DONE, CANCELLED), page counts, and stop reason.

Account, 1 operation

get_account_infoGet account infoGETReturns account details: subscription plan, usage metrics, remaining API credits, concurrency limits, and project configuration.

Crawl artifact, 1 operation

get_crawl_artifactGet crawl artifactGETDownloads bulk result files from a completed crawl in WARC or HAR format. Returns binary archive data.

Crawl content, 1 operation

get_crawl_contentGet crawl contentGETRetrieves the extracted content from crawled pages in the formats specified during crawl creation (HTML, markdown, text, etc.).

Crawl urls, 1 operation

get_crawl_urlsGet crawl urlsGETLists all URLs discovered and crawled during a crawl job. Returns URL metadata with pagination support.

Extraction, 1 operation

extract_dataExtract dataPOSTExtracts structured data from HTML/text content using AI prompts, templates, or predefined models. Supports product, article, review, and real estate auto-extraction.

Screenshot, 1 operation

take_screenshotTake screenshotGETCaptures a screenshot of any webpage in JPG, PNG, WebP, or GIF format. Supports full page, viewport, or element-specific capture with configurable resolution.

Webpage, 1 operation

scrape_webpageScrape webpageGETScrapes a webpage by URL with proxy rotation, anti-bot bypass, JS rendering, and multiple output formats. Returns page content, metadata, and optional extracted data.

ScrapFly API rate limits

Concurrency and rate limits are defined by subscription tier. Default read timeout is 155 seconds.

These are the limits ScrapFly documents, not ones LiveContext measured. They can change and often depend on your plan, so check the ScrapFly documentation for current values.

Use ScrapFly from Claude Code, Cursor or Codex

Connect ScrapFly in LiveContext, create an API key in Settings, then add the LiveContext MCP server to your client. Its ScrapFly operations become tools the client can call, next to every other integration you connected.

Claude Code

claude mcp add --transport http livecontext https://livecontext.ai/mcp --header "X-API-Key: YOUR_API_KEY"

Codex (~/.codex/config.toml)

[mcp_servers.livecontext]
url = "https://livecontext.ai/mcp"

[mcp_servers.livecontext.http_headers]
"X-API-Key" = "YOUR_API_KEY"

Cursor, Claude Desktop and other JSON configs

{
  "mcpServers": {
    "livecontext": {
      "type": "http",
      "url": "https://livecontext.ai/mcp",
      "headers": {
        "X-API-Key": "YOUR_API_KEY"
      }
    }
  }
}

Set up the MCP server step by step

How to connect ScrapFly

Create your workspace
Sign up free. Nothing to install.
Connect ScrapFly once
Add your ScrapFly connection in Settings. Every tool from this integration reuses it, and secrets are encrypted at rest and visible only to you.
Describe the job in chat
The workflow builds itself in front of you and picks the endpoints it needs. Drag the steps yourself on the canvas whenever you would rather be explicit.

ScrapFly integration FAQ

What can you do with the ScrapFly integration?

LiveContext exposes 9 operations on ScrapFly, covering crawl, account, crawl artifact and crawl content. Common uses include data pipeline ingestion, competitive analysis and bulk data export.

What are the ScrapFly API rate limits?

Concurrency and rate limits are defined by subscription tier. Default read timeout is 155 seconds. Limits are set by ScrapFly and can depend on your plan.