API reference

Quickstart

Create a free key, then:

curl -X POST https://tomsindex.com/v1/extract \
  -H "X-API-Key: srch_your_key" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

import requests

res = requests.post(
    "https://tomsindex.com/v1/extract",
    headers={"X-API-Key": "srch_your_key"},
    json={"url": "https://example.com"},
)
print(res.json()["results"][0]["markdown"])

const res = await fetch("https://tomsindex.com/v1/extract", {
  method: "POST",
  headers: { "X-API-Key": "srch_your_key", "Content-Type": "application/json" },
  body: JSON.stringify({ url: "https://example.com" }),
});
console.log((await res.json()).results[0].markdown);

Authentication

Send your key in the X-API-Key header. Keys start with srch_ and are created in the dashboard. Browser requests from a signed-in session also work.

Extract

Fetches a page and returns markdown, metadata and links. Successful extracts are saved to your private index. Plain-text and markdown URLs pass through as-is; GitHub files, issues, pull requests and folders, npm and PyPI packages and Stack Exchange questions are read through their official APIs. Pages that need JavaScript are rendered in a headless browser (Crawl4AI); those results have metadata.rendered: true.

Body

Response

When the page is saved, the result includes its index id. A result may also carry warnings: when css_selector matched nothing, when the markdown seems to be missing an intro, sections, code blocks or tables the page has, or when the page is very large. It's a heuristic — a cue to try css_selector or another tool, not a guarantee.

{
  "results": [{
    "url": "https://example.com/",
    "markdown": "# Example Domain\n\nThis domain is for use in…",
    "raw_content": "…",
    "metadata": { "title": "Example Domain", "description": "", "language": "en" },
    "links": [{ "href": "https://iana.org/domains/example", "text": "Learn more", "type": "external" }]
  }],
  "failed_results": [],
  "response_time": 0.42
}

Index

Lists pages in your index, newest first. q runs a full-text search over titles and markdown, ranked by relevance, and adds a highlighted snippet to each hit. limit max is 100.

{ "results": [{ "id": "9b2c…", "url": "https://example.com/", "title": "Example Domain", "fetched_at": "2026-10-19T…" }] }

Returns one saved page with its full markdown. Your index is private: other accounts can't list or read it.

Removes a page from your index. Returns { "deleted": true }.

Your whole index as JSON Lines (id, url, title, markdown, fetched_at) — take your data with you any time.

Data & privacy

Extracted markdown is stored in your account's private index until you delete it (API or dashboard). Demo extracts from the homepage are never stored. TomsIndex fetches pages as TomsIndexBot/1.0; don't send URLs containing secrets or tokens.

MCP server

A remote MCP server (Streamable HTTP). Authenticate with X-API-Key or Authorization: Bearer srch_… (both work on every endpoint). Tools: extract, index_search, index_get. Without a key it runs as a trial: extract (plus index_get to page long results), 20 calls/day per IP, nothing saved.

# Claude Code
claude mcp add --transport http tomsindex https://tomsindex.com/mcp --header "X-API-Key: srch_your_key"

# Cursor — ~/.cursor/mcp.json
{ "mcpServers": { "tomsindex": { "url": "https://tomsindex.com/mcp", "headers": { "X-API-Key": "srch_your_key" } } } }

# Codex — ~/.codex/config.toml
[mcp_servers.tomsindex]
url = "https://tomsindex.com/mcp"
http_headers = { "X-API-Key" = "srch_your_key" }

Or run curl -fsSL https://tomsindex.com/install.sh | sh (read it first) to write these for every client it finds. Long pages come back in 25,000-character pages. extract returns the saved page's id, so the agent reads on with index_get and an offset — no re-fetch, no rate-limit cost.

Errors

Any URL that can't be extracted — invalid, blocked, HTTP 4xx/5xx from the target, non-HTML, timeout, or a JavaScript-only page with no readable HTML — returns 200 with the reason in failed_results[0].error. Nothing is saved.

{ "results": [], "failed_results": [{ "url": "…", "error": "Target responded with HTTP 403." }] }

Request problems use HTTP status with { "error": "…" }: 400 missing url, 401 bad or missing key, 402 monthly plan limit reached, 429 rate limited.

Plans and rate limits

Each account gets a monthly extract allowance: 100 on Free, 5,000 on Pro ($9/month). It resets on the 1st, UTC. Extracts that fail don't count. Over the allowance, POST /v1/extract returns 402 with code: "quota_exceeded"; reading your index is never metered. Upgrade from the dashboard.

Every response carries X-RateLimit-Limit, X-RateLimit-Remaining and X-RateLimit-Reset; a 429 adds Retry-After.