Web crawling & sitemaps

Crawling lets you ingest an entire website (or a section of it) into the knowledge base in one job, instead of pasting pages one at a time. Open Dashboard → Knowledge → Add Item → URL and pick a sub-mode.

The URL tab has three sub-modes: Single URL, Crawl from URL, and Import from Sitemap. Single URL scrapes exactly one page — useful for a one-off. This page covers the two bulk modes.

Crawl from URL

Choose Crawl from URL and configure:

Setting	Default	What it controls
Start URL	—	Where the crawl begins (required).
Category	—	Optional category applied to every page.
Max Depth	2	How many link-hops deep the crawler follows (1–5).
Max Pages	50	Hard ceiling on pages fetched (1–1000).
Same domain only	On	Stay on the starting domain.
Include subdomains	Off	Also crawl text `docs.example.com` , text `blog.example.com` , etc. Requires Same domain only.
Enable SPA rendering	Off	Use a headless browser to render JavaScript-heavy pages (slower).

Requires the Grow plan or higher for SPA rendering.

The crawler respects

text

robots.txt

and rate-limits requests per domain. Click Start Crawl — the job runs in the background and you are taken to its status page.

🖼️ [Image] — The Crawl from URL form showing depth/max-pages sliders and the domain toggles.

Import from Sitemap

Choose Import from Sitemap to pull a curated list of URLs straight from a site's

text

sitemap.xml

. You can either:

Sitemap URL — paste
text
```
https://example.com/sitemap.xml
```
, or
Upload XML — drag in a
text
```
sitemap.xml
```
file (max 10 MB).

Set Max Pages (default 200) and an optional Category. Use Preview to see how many URLs were found before committing, then click Import Sitemap. Sitemap import is usually cleaner than a depth crawl because it only fetches pages the site itself lists.

The crawl job status page

Every bulk job opens at

text

/dashboard/knowledge/crawl/[jobId]

, and recent jobs are listed in the Recent Crawl Jobs panel on the Knowledge page. The status page updates live and shows:

A progress bar splitting pages into ingested / failed / skipped.
Stat cards: Discovered, Ingested, Failed, Skipped.
A per-page table you can filter by All / Ingested / Failed / Skipped / Pending, with each page's URL, status, depth, render mode, and size.

While a job is running you can Cancel it (already-ingested pages are kept). After it finishes you can Delete KB Items (remove the content but keep the log) or Delete Job (remove everything).

🎬 [Video] — Starting a crawl and watching the progress page fill in live.

Clean content only

The crawler strips boilerplate before indexing —

text

<script>

text

<style>

, navigation, headers, footers, sidebars, and elements matching noise patterns like cookie banners, popups, breadcrumbs, and share/ad widgets. Only the real page content is chunked and embedded, so navigation menus and cookie notices don't pollute your bot's answers. Duplicate pages (identical content) are detected by hash and skipped.

Web crawling & sitemaps

Web crawling & sitemaps

Crawl from URL

Import from Sitemap

The crawl job status page

Clean content only

Related