oerli.aiSEO crawler for server and browser

Explore any website page by page

Run the crawl on our servers or directly in your browser via a Chrome extension. A Screaming Frog–style grid, built for the web.

Scans the seed pageYou pick which links to crawl50 URLs free
OerliBot, a small robot with a lantern, walks at night through a landscape of browser windows while a sitemap glows above like a constellation

Two ways to crawl

Pick where the requests come from

OerliBot next to a server tower with green lights, throwing a paper airplane

Quick API Crawl

Server-side, OerliBot fetches

Type a URL above and hit Discover. Our bot fetches the site, parses SEO fields, and returns results in seconds. No install required.

  • 50 URLsfree tier
  • 500 URLswith API key
  • Title, H1, Meta, canonicalon-page SEO
  • OG / Twitter, JSON-LDmeta & schema
  • REST API + MCP serverdeveloper access
API docs
OerliBot leaning out of a browser window with a plug connected to its side

Browser Crawler

Beta
Chrome extension, your machine fetches

A Chrome extension turns your browser into the crawler. HTTP requests go from your machine: no CORS, no server bottleneck, your real IP.

  • Up to 2000 URLsconfigurable
  • Cross-origin fetchno CORS restrictions
  • Screaming Frog–style gridvirtualized, fast
  • robots.txt respectedin-browser parser
  • Screenshotsvia extension capture
Open Browser Crawler

Under the hood

How browser crawl works

The extension acts as a privileged fetch bridge. Your browser does the crawling, oerli.ai only caches the results.

Your BrowserChrome + oerli.ai tab
FETCH_URL message
Crawl ExtensionMV3 service worker
HTTP request
Target WebsiteAny http/https URL
oerli.ai cache
Session + URL rows + thumbnails

Results sync in batches after every 10 URLs. Thumbnails upload separately; the server only resizes, never re-fetches.

  1. A puzzle piece clicking into a browser frame
    01One-time setup

    Install the extension

    Load it unpacked from extensions/oerli-crawl/ in chrome://extensions (Developer mode). Takes about 30 seconds.

  2. A green map pin on an address bar with a dotted trail leading to a stack of pages
    02Per crawl

    Enter a URL and start

    The extension requests <all_urls> permission (Chrome prompt). The Web Worker starts BFS, fetching robots.txt first, then pages in parallel.

  3. A magnifying glass over a table with green, amber and coral status chips
    03Live as you crawl

    Explore the results

    A Screaming Frog–style virtualized grid shows status, title, meta, H1, canonical, links, and size for every URL. Click any row for a detail panel.

A shield with a keyhole in front of a laptop

HTTP requests run from your machine, not oerli.ai servers. The extension's externally_connectable manifest key restricts messaging to oerli.ai and localhost only, so no other site can use it as an open proxy.

Crawl data is synced to oerli.ai as a cache (URLs, SEO fields, screenshots). This is intentional and documented: it enables sharing, reports, and thumbnail delivery.

Per URL

What you get for every URL

Parsed from raw HTML, no JavaScript execution in the first release.

Response

  • HTTP status
  • Redirect chain
  • Content-Type
  • Body size
  • Fetch duration

On-page SEO

  • Title
  • Meta description
  • H1 (all)
  • Canonical URL
  • robots meta
  • Language (lang=)

Links

  • Internal links
  • External links
  • Depth in BFS

Social

  • OG title / image
  • Twitter Card
  • Favicon URL
OerliBot sits contentedly on a hill of browser windows with its lantern and a checked-off clipboard, next to a fully green sitemap

Every node green, every page checked

Start with one URL, OerliBot maps the rest.