Web crawling & sitemaps
Crawling lets you ingest an entire website (or a section of it) into the knowledge base in one job, instead of pasting pages one at a time. Open Dashboard → Knowledge → Add Item → URL and pick a sub-mode.
The URL tab has three sub-modes: Single URL, Crawl from URL, and Import from Sitemap. Single URL scrapes exactly one page — useful for a one-off. This page covers the two bulk modes.
Crawl from URL
Choose Crawl from URL and configure:
| Setting | Default | What it controls |
|---|---|---|
| Start URL | — | Where the crawl begins (required). |
| Category | — | Optional category applied to every page. |
| Max Depth | 2 | How many link-hops deep the crawler follows (1–5). |
| Max Pages | 50 | Ceiling on pages fetched for this job; your subscription caps the effective maximum (see below). |
| Same domain only | On | Stay on the starting domain. |
| Include subdomains | Off | Also crawl docs.example.com, blog.example.com, etc. Requires Same domain only. |
| Enable SPA rendering | Off | Use a headless browser to render JavaScript-heavy pages (slower). |
SPA rendering is included with the Chat module.
Your subscription also caps how far a crawl can go: with a subscription, 100 pages and depth 3 per job, and 25 crawl jobs a month; during the trial, 20 pages, depth 2, and 5 jobs a month. A job that would push your knowledge base past its size cap is clamped to what still fits — it stops early and keeps everything it already ingested instead of failing. See Pricing & limits.
The crawler respects robots.txt and rate-limits requests per domain. Click Start Crawl — the job runs in the background and you are taken to its status page.
🖼️ [Image] — The Crawl from URL form showing depth/max-pages sliders and the domain toggles.
Import from Sitemap
Choose Import from Sitemap to pull a curated list of URLs straight from a site's sitemap.xml. You can either:
- Sitemap URL — paste
https://example.com/sitemap.xml, or - Upload XML — drag in a
sitemap.xmlfile (max 10 MB).
Set Max Pages (default 200) and an optional Category. Use Preview to see how many URLs were found before committing, then click Import Sitemap. Sitemap import is usually cleaner than a depth crawl because it only fetches pages the site itself lists.
The crawl job status page
Every bulk job opens at /dashboard/knowledge/crawl/[jobId], and recent jobs are listed in the Recent Crawl Jobs panel on the Knowledge page. The status page updates live and shows:
- A progress bar splitting pages into ingested / failed / skipped.
- Stat cards: Discovered, Ingested, Failed, Skipped.
- A per-page table you can filter by All / Ingested / Failed / Skipped / Pending, with each page's URL, status, depth, render mode, and size.
While a job is running you can Cancel it (already-ingested pages are kept). After it finishes you can Delete KB Items (remove the content but keep the log) or Delete Job (remove everything).
🎬 [Video] — Starting a crawl and watching the progress page fill in live.
Clean content only
The crawler strips boilerplate before indexing — <script>, <style>, navigation, headers, footers, sidebars, and elements matching noise patterns like cookie banners, popups, breadcrumbs, and share/ad widgets. Only the real page content is chunked and embedded, so navigation menus and cookie notices don't pollute your bot's answers. Duplicate pages (identical content) are detected by hash and skipped.