> ## Documentation Index
> Fetch the complete documentation index at: https://sitegpt.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Train on website links

> Crawl your website, import a sitemap, paste URLs, or add YouTube videos.

**Website Links** is where your chatbot learns from pages on the web. Open
your chatbot's sidebar and expand **Website Links** to see the four ways to
add links, and **Links List** to manage what is already there.

## Four ways to add links

<Tabs>
  <Tab title="Scrape Website">
    Give SiteGPT your site's homepage (or any starting URL) and it crawls the
    site by following links.

    1. Open **Website Links** → **Scrape Website**.
    2. Enter the starting URL.
    3. Set **Max Pages to Scrape**. The crawl stops at this many pages or at
       your remaining [pages quota](/docs/features/pages-quota), whichever is lower.
    4. Set **Recursion Depth**, which is how many clicks away from the
       starting page the crawler may go.
    5. Optionally restrict the crawl with **Allowed Domains**.
    6. Start the scrape. Progress appears in **Links List** as pages are
       discovered and trained.

    Best for: getting broad coverage of a site quickly.
  </Tab>

  <Tab title="Add from Sitemap">
    Point SiteGPT at your `sitemap.xml` and it imports the URLs listed there.

    1. Open **Website Links** → **Add from Sitemap**.
    2. Enter the **Sitemap URL**, for example `https://example.com/sitemap.xml`.
    3. Set **Max Pages to Scrape** if you want to cap the import.
    4. Optionally set an **Auto Scan Frequency**. SiteGPT re-reads the sitemap
       on that schedule and imports any new URLs it finds.
    5. Start the import.

    Best for: sites with a maintained sitemap, and for picking up newly
    published pages automatically.
  </Tab>

  <Tab title="Add Multiple Links">
    Paste an exact list of URLs, one per line, into the **Links** field.

    Best for: precise control over what the chatbot learns, or adding a
    handful of important pages from another site (for example, your status
    page or a partner's docs).
  </Tab>

  <Tab title="Add YouTube Videos">
    Paste YouTube URLs into the **YouTube URLs** field. SiteGPT trains on the
    video's captions.

    * Videos must be public or unlisted and must have captions.
    * The chatbot learns from the transcript, not the audio or visuals.

    Best for: product walkthroughs, webinars, and tutorial content.
  </Tab>
</Tabs>

## Scrape options

The website, sitemap, and multiple-links flows share these options:

| Option                            | What it does                                                                                                                                                                          |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Extract main content only**     | Keeps the main article content and drops navigation, sidebars, and footers. Leave this on unless important content lives in those areas.                                              |
| **HTML/CSS selectors to include** | Train only on matching elements, for example `.docs-content`.                                                                                                                         |
| **HTML/CSS selectors to exclude** | Skip matching elements, for example `header, .cookie-banner, #nav`.                                                                                                                   |
| **Custom Headers (Optional)**     | Sent with every request, one `Header-Name: value` per line. Use this for password-protected staging sites or bot-protection allowlists.                                               |
| **Auto Sync Frequency**           | Re-crawl on a schedule. Effective for website crawls and sitemap imports; for pasted links, set per-link schedules from **Links List** instead. See [Auto-sync](/docs/features/auto-sync). |

## Editing a link's settings

Open any link in **Links List** to change its scrape options or sync
frequency. Saving re-imports that URL with the new settings; the page count adjusts
to the content's current length.

To update the configuration of many links at once, select them in the list
and use **Update Config**.

## Fixing failed links

A **Failed** badge means the page could not be fetched or processed. Open the
row to see the error. Common causes:

<AccordionGroup>
  <Accordion title="The site blocked the crawler">
    Bot protection (Cloudflare and similar) can refuse automated requests. If
    you control the protection, allowlist SiteGPT's requests or use **Custom
    Headers** with a token your firewall accepts.
  </Accordion>

  <Accordion title="The page requires login">
    Pages behind a login cannot be crawled. Export the content and upload it
    as a [file](/docs/content/files), or paste it as a
    [text snippet](/docs/features/text-snippets).
  </Accordion>

  <Accordion title="The URL is wrong or redirects away">
    Check that the URL loads in a private browser window and update the link.
  </Accordion>

  <Accordion title="You ran out of pages quota">
    Imports stop when your plan's pages quota is used up. Free quota by
    deleting content you do not need, or upgrade. See
    [Pages quota](/docs/features/pages-quota).
  </Accordion>
</AccordionGroup>

After fixing the cause, select the failed links and use the **Resync** button (it shows the
selected count).