> ## Documentation Index
> Fetch the complete documentation index at: https://sitegpt.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Add your website

> Add pages from your website to your chatbot by crawling the site, importing a sitemap, or pasting a list of links.

Use this guide to add web pages to your chatbot's content. There are three ways to do it:

| Method                 | Use it when                                                          |
| ---------------------- | -------------------------------------------------------------------- |
| **Scrape Website**     | You want SiteGPT to start at one URL and follow links to find pages. |
| **Add from Sitemap**   | Your site has a `sitemap.xml` that lists the pages you want.         |
| **Add Multiple Links** | You know the exact pages you want.                                   |

To add YouTube videos, see [Add YouTube videos](/docs/guides/content/add-youtube-videos). To learn what SiteGPT does with a page after it adds it, see [How answers work](/docs/concepts/how-answers-work).

## Before you start

* An active plan. Each plan has a page limit. See [Plans and limits](/docs/reference/plans-and-limits).
* The Super Admin, Admin, or Manager role on the chatbot. Agents cannot add content.
* The pages must be public, unless you add them with custom headers. See [Pages behind a login](#pages-behind-a-login).

<Note>
  One page of quota is 2,500 characters of cleaned text. A long web page can use several pages of quota. The **Max Pages to Scrape** field counts URLs, not quota pages.
</Note>

## Crawl a website

<Steps>
  <Step title="Open the form">
    In the chatbot sidebar, go to **Website Links** > **Scrape Website**.
  </Step>

  <Step title="Enter the start URL">
    In **Website URL**, enter the page to start from, for example `https://example.com`.
  </Step>

  <Step title="Set the depth">
    In **Recursion Depth**, enter a number from 1 to 5. Depth 1 adds the start page, the pages it links to, and the pages that those pages link to.
  </Step>

  <Step title="Set the page cap">
    In **Max Pages to Scrape**, enter the most URLs to add. The field cannot go above the pages left in your quota.
  </Step>

  <Step title="Limit the crawl (optional)">
    Fill in the filter fields you need. See [Crawl options](#crawl-options).
  </Step>

  <Step title="Start the crawl">
    Select **Add Links**. The message "Added website to the queue" appears.
  </Step>
</Steps>

## Import a sitemap

<Steps>
  <Step title="Open the form">
    In the chatbot sidebar, go to **Website Links** > **Add from Sitemap**.
  </Step>

  <Step title="Enter the sitemap URL">
    In **Sitemap URL**, enter the full address, for example `https://example.com/sitemap.xml`.
  </Step>

  <Step title="Set the page cap">
    In **Max Pages to Scrape**, enter the most URLs to add. The field cannot go above the pages left in your quota.
  </Step>

  <Step title="Set options (optional)">
    Fill in the filter fields you need. See [Crawl options](#crawl-options).
  </Step>

  <Step title="Start the import">
    Select **Add Links**. The message "Added sitemap to the queue" appears.
  </Step>
</Steps>

<Warning>
  SiteGPT only adds sitemap URLs that have the same hostname as the sitemap. For example, a sitemap at `https://example.com/sitemap.xml` that lists `https://www.example.com/...` pages adds no pages. Use the sitemap address with the same hostname as the page URLs.
</Warning>

A sitemap index that points to other sitemaps is supported. SiteGPT follows the nested sitemaps.

## Add a list of links

<Steps>
  <Step title="Open the form">
    In the chatbot sidebar, go to **Website Links** > **Add Multiple Links**.
  </Step>

  <Step title="Paste the URLs">
    In **Links**, enter one URL per line.
  </Step>

  <Step title="Set options (optional)">
    Fill in the scrape options you need. See [Crawl options](#crawl-options).
  </Step>

  <Step title="Add the links">
    Select **Add Links**. The message "Added links to the queue" appears.
  </Step>
</Steps>

You cannot add more URLs than the pages left in your quota. The form shows how many you can add.

Lists of links do not refresh on a schedule, even if you pick an **Auto Sync Frequency** in the form. To refresh them, [resync](/docs/guides/content/keep-content-up-to-date#resync-content-now) them.

## Crawl options

The forms share most options. Some options exist only on some forms. In the table, **All** means the Website, Sitemap, and Multiple Links forms.

| Option                                  | Forms            | What it does                                                                                                                                                         |
| --------------------------------------- | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **URL paths to include**                | Website, Sitemap | Adds only URLs that contain one of these texts, one per line. `/blog` matches any URL with `/blog` in its path. It is a plain text match, not a wildcard or pattern. |
| **URL paths to exclude**                | Website, Sitemap | Skips URLs that contain one of these texts. Exclude is checked before include.                                                                                       |
| **Allowed Domains**                     | Website          | Adds links to these other hostnames, one per line. SiteGPT adds those pages but does not follow links on them.                                                       |
| **Extract main content only**           | All              | On by default. Keeps the main content of each page and leaves out headers, footers, sidebars, and navigation.                                                        |
| **Skip pages that are already trained** | All              | Leaves pages that are already in the chatbot's content alone instead of adding them again.                                                                           |
| **HTML/CSS selectors to include**       | All              | Keeps only content inside matching elements, one selector per line. For example, `main` or `.content`.                                                               |
| **HTML/CSS selectors to exclude**       | All              | Removes matching elements before training, one selector per line. For example, `footer` or `.sidebar`.                                                               |
| **Custom Headers (Optional)**           | All              | Sends HTTP headers when SiteGPT fetches each page. Enter one `Name: Value` per line.                                                                                 |
| **Auto Sync Frequency**                 | All              | Crawls again on a schedule. It has no effect for a list of links. See [Keep content up to date](/docs/guides/content/keep-content-up-to-date).                            |
| **Auto Scan Frequency**                 | Sitemap          | Checks the sitemap on a schedule for added and removed URLs. See [Keep content up to date](/docs/guides/content/keep-content-up-to-date).                                 |

The auto-sync and auto-scan fields show only if your plan includes them.

### Hostnames must match exactly

A crawl follows links only on the exact hostname of the start URL. `www.example.com` and `example.com` are different hostnames. So are subdomains such as `docs.example.com`.

* Start the crawl on the hostname your pages use.
* To also add pages from another hostname, list it in **Allowed Domains**.

### Pages behind a login

SiteGPT sends your custom headers each time it fetches a page. You can use this to add pages that need a login, such as pages for signed-in customers:

1. Sign in to your website in your browser, and copy the session cookie or access token that your site uses.
2. Choose **Add Multiple Links** and paste the exact URLs of the pages you want.
3. In **Custom Headers (Optional)**, add the header, for example `Cookie: session=YOUR_SESSION_VALUE` or `Authorization: Bearer YOUR_TOKEN`.

Keep these points in mind:

* Add the pages as a list of links. Custom headers are not sent while SiteGPT looks for links to follow, so a crawl or a sitemap cannot find pages that are linked only behind a login.
* Session cookies and tokens expire. When they expire, a resync of these pages fails. Add a fresh value to update the pages.
* Everything you add can appear in answers to anyone who can reach the chatbot. Only add private pages to a chatbot whose visitors may see that content.
* Custom headers are available in the dashboard only. The API and CLI accept only the `User-Agent`, `Accept`, and `Accept-Language` headers.

You can also [upload the content as files](/docs/guides/content/upload-files) or [add a text snippet](/docs/guides/content/add-text-snippet).

### What a crawl does not do

* It does not read `robots.txt`, `nofollow`, or canonical tags.
* It does not add image or video files as pages. Image alt text in a page stays in the text.
* URLs with different query strings, such as `?utm_source=...`, count as different pages. Exclude them with **URL paths to exclude**.

If your site links to PDF files that the chatbot must learn from, [upload the PDFs](/docs/guides/content/upload-files).

PDF files that the crawl finds on your site are added too. SiteGPT reads their text, and each PDF counts toward your pages like any other document. Scanned PDFs that contain only images give no text. Tables and complex layouts in a PDF can lose their formatting. PDFs on other domains are added only if you list that domain in **Allowed Domains**. To leave PDFs out, add `.pdf` to **URL paths to exclude**.

## Check that it works

1. Go to **Website Links** > **Links List**.
2. Wait for the new links to change from **Backlog** or **Processing** to **Success**.
3. Ask the chatbot a question that one of the new pages answers.

## Fix failed links

A row with a **Failed** badge could not be added. Point to the **Failed** badge to see the reason. Common reasons:

<AccordionGroup>
  <Accordion title="The website blocked access to this page">
    The site blocks automated visitors, or the page needs a login. If you control a bot-protection service, ask your site admin to allow SiteGPT's requests. For pages behind a login, see [Pages behind a login](#pages-behind-a-login).
  </Accordion>

  <Accordion title="The page could not be found (404)">
    The URL is wrong or the page no longer exists. Check that the URL opens in a private browser window. Delete the link if the page is gone.
  </Accordion>

  <Accordion title="The page took too long to respond">
    Resync the link later.
  </Accordion>

  <Accordion title="Limit Reached">
    Your plan's page limit was reached. Delete content you do not need, or upgrade. See [Plans and limits](/docs/reference/plans-and-limits).
  </Accordion>

  <Accordion title="The crawl added pages I did not want">
    Delete the unwanted links. Then add the site again with **URL paths to exclude** or **URL paths to include**, and turn on **Skip pages that are already trained**.
  </Accordion>

  <Accordion title="Pages contain menus, cookie banners, or image data">
    Add the unwanted parts to **HTML/CSS selectors to exclude**, for example `nav`, `footer`, or `img`. Then resync the links.
  </Accordion>
</AccordionGroup>

To try failed links again, select them in **Links List** and select **Resync N Selected**. To try all failed links, use the resync button on the **Failed** status card.

## Change settings for links you already added

1. In **Links List**, select the links.
2. Select **Update Config**.
3. Change the scrape options.
4. Select **Update Configuration**. SiteGPT adds those URLs again with the new settings.