Skip to main content
Use this guide to add web pages to your chatbot’s content. There are three ways to do it: To add YouTube videos, see Add YouTube videos. To learn what SiteGPT does with a page after it adds it, see How answers work.

​
Before you start

  • An active plan. Each plan has a page limit. See Plans and limits.
  • The Super Admin, Admin, or Manager role on the chatbot. Agents cannot add content.
  • The pages must be public, unless you add them with custom headers. See Pages behind a login.
One page of quota is 2,500 characters of cleaned text. A long web page can use several pages of quota. The Max Pages to Scrape field counts URLs, not quota pages.

​
Crawl a website

1

Open the form

In the chatbot sidebar, go to Website Links > Scrape Website.
2

Enter the start URL

In Website URL, enter the page to start from, for example https://example.com.
3

Set the depth

In Recursion Depth, enter a number from 1 to 5. Depth 1 adds the start page, the pages it links to, and the pages that those pages link to.
4

Set the page cap

In Max Pages to Scrape, enter the most URLs to add. The field cannot go above the pages left in your quota.
5

Limit the crawl (optional)

Fill in the filter fields you need. See Crawl options.
6

Start the crawl

Select Add Links. The message “Added website to the queue” appears.

​
Import a sitemap

1

Open the form

In the chatbot sidebar, go to Website Links > Add from Sitemap.
2

Enter the sitemap URL

In Sitemap URL, enter the full address, for example https://example.com/sitemap.xml.
3

Set the page cap

In Max Pages to Scrape, enter the most URLs to add. The field cannot go above the pages left in your quota.
4

Set options (optional)

Fill in the filter fields you need. See Crawl options.
5

Start the import

Select Add Links. The message “Added sitemap to the queue” appears.
SiteGPT only adds sitemap URLs that have the same hostname as the sitemap. For example, a sitemap at https://example.com/sitemap.xml that lists https://www.example.com/... pages adds no pages. Use the sitemap address with the same hostname as the page URLs.
A sitemap index that points to other sitemaps is supported. SiteGPT follows the nested sitemaps.
1

Open the form

In the chatbot sidebar, go to Website Links > Add Multiple Links.
2

Paste the URLs

In Links, enter one URL per line.
3

Set options (optional)

Fill in the scrape options you need. See Crawl options.
4

Add the links

Select Add Links. The message “Added links to the queue” appears.
You cannot add more URLs than the pages left in your quota. The form shows how many you can add. Lists of links do not refresh on a schedule, even if you pick an Auto Sync Frequency in the form. To refresh them, resync them.

​
Crawl options

The forms share most options. Some options exist only on some forms. In the table, All means the Website, Sitemap, and Multiple Links forms. The auto-sync and auto-scan fields show only if your plan includes them.

​
Hostnames must match exactly

A crawl follows links only on the exact hostname of the start URL. www.example.com and example.com are different hostnames. So are subdomains such as docs.example.com.
  • Start the crawl on the hostname your pages use.
  • To also add pages from another hostname, list it in Allowed Domains.

​
Pages behind a login

SiteGPT sends your custom headers each time it fetches a page. You can use this to add pages that need a login, such as pages for signed-in customers:
  1. Sign in to your website in your browser, and copy the session cookie or access token that your site uses.
  2. Choose Add Multiple Links and paste the exact URLs of the pages you want.
  3. In Custom Headers (Optional), add the header, for example Cookie: session=YOUR_SESSION_VALUE or Authorization: Bearer YOUR_TOKEN.
Keep these points in mind:
  • Add the pages as a list of links. Custom headers are not sent while SiteGPT looks for links to follow, so a crawl or a sitemap cannot find pages that are linked only behind a login.
  • Session cookies and tokens expire. When they expire, a resync of these pages fails. Add a fresh value to update the pages.
  • Everything you add can appear in answers to anyone who can reach the chatbot. Only add private pages to a chatbot whose visitors may see that content.
  • Custom headers are available in the dashboard only. The API and CLI accept only the User-Agent, Accept, and Accept-Language headers.
You can also upload the content as files or add a text snippet.

​
What a crawl does not do

  • It does not read robots.txt, nofollow, or canonical tags.
  • It does not add image or video files as pages. Image alt text in a page stays in the text.
  • URLs with different query strings, such as ?utm_source=..., count as different pages. Exclude them with URL paths to exclude.
If your site links to PDF files that the chatbot must learn from, upload the PDFs. PDF files that the crawl finds on your site are added too. SiteGPT reads their text, and each PDF counts toward your pages like any other document. Scanned PDFs that contain only images give no text. Tables and complex layouts in a PDF can lose their formatting. PDFs on other domains are added only if you list that domain in Allowed Domains. To leave PDFs out, add .pdf to URL paths to exclude.

​
Check that it works

  1. Go to Website Links > Links List.
  2. Wait for the new links to change from Backlog or Processing to Success.
  3. Ask the chatbot a question that one of the new pages answers.
A row with a Failed badge could not be added. Point to the Failed badge to see the reason. Common reasons:
The site blocks automated visitors, or the page needs a login. If you control a bot-protection service, ask your site admin to allow SiteGPT’s requests. For pages behind a login, see Pages behind a login.
The URL is wrong or the page no longer exists. Check that the URL opens in a private browser window. Delete the link if the page is gone.
Resync the link later.
Your plan’s page limit was reached. Delete content you do not need, or upgrade. See Plans and limits.
Delete the unwanted links. Then add the site again with URL paths to exclude or URL paths to include, and turn on Skip pages that are already trained.
To try failed links again, select them in Links List and select Resync N Selected. To try all failed links, use the resync button on the Failed status card.
  1. In Links List, select the links.
  2. Select Update Config.
  3. Change the scrape options.
  4. Select Update Configuration. SiteGPT adds those URLs again with the new settings.