Skip to main content
Auto-sync jobs automatically update your chatbot’s knowledge base by periodically re-crawling your content sources.

How it works

When you add content from URLs, sitemaps, websites, or YouTube channels, you can enable automatic syncing to:
  • Detect new pages or videos
  • Update modified content
  • Remove deleted content
  • Keep your chatbot’s knowledge current

Sync frequencies

Choose how often to sync your content:

Daily sync

Updates content every 24 hours. Best for:
  • Frequently updated blogs
  • News sites
  • Documentation that changes often
  • Active YouTube channels

Weekly sync

Updates content every 7 days. Best for:
  • Marketing websites
  • Product pages
  • Knowledge bases with moderate updates
  • Educational content

Monthly sync

Updates content every 30 days. Best for:
  • Static websites
  • Infrequently updated content
  • Large content libraries
  • Cost-conscious implementations

Manual only

No automatic syncing. You control when to update content manually.

Setting up auto-sync

For new content

When adding content:
1

Add your content source

Go to Content and add URLs, sitemap, website, or YouTube channel.
2

Choose sync frequency

Select Daily, Weekly, Monthly, or Manual from the dropdown.
3

Complete setup

Finish adding your content. Auto-sync will begin based on your selected frequency.

For existing content

To enable auto-sync on existing content:
1

Go to auto-sync jobs

Navigate to Content > Auto-Sync Jobs in your dashboard.
2

Find your content

Locate the content source you want to auto-sync.
3

Update frequency

Click the sync frequency dropdown and select your preferred schedule.
4

Save changes

The new sync schedule takes effect immediately.

Managing sync jobs

View all sync jobs

The Auto-Sync Jobs page shows:
  • All content sources with auto-sync enabled
  • Current sync frequency
  • Last sync date and time
  • Next scheduled sync
  • Sync status (successful, failed, in progress)

Filter by source type

Filter jobs by content type:
  • URLs
  • Sitemap
  • Website
  • YouTube

Filter by frequency

View jobs by sync schedule:
  • Daily
  • Weekly
  • Monthly
  • Manual only

Sync statistics

Track sync performance:
  • Total jobs - Number of auto-sync jobs configured
  • Active jobs - Jobs with automatic syncing enabled
  • Successful - Jobs that completed successfully
  • Running - Jobs currently in progress

Sync job details

Click on any sync job to view:

Job information

  • Content source URL or identifier
  • Sync frequency
  • Date created
  • Last modified

Sync history

  • All previous sync runs
  • Start and end times
  • Status (completed, failed, in progress)
  • Pages/videos processed
  • Changes detected

Sync results

  • New content added
  • Updated content
  • Removed content
  • Errors encountered

Troubleshooting

Sync failures

If a sync job fails: Check source availability
Ensure the website or content source is accessible and hasn’t moved.
Review error messages
Click on the failed sync to see specific error details.
Verify permissions
Confirm you have access to the content source (e.g., YouTube channel is public).
Check rate limits
Some websites may block frequent crawling. Consider reducing sync frequency.

Missing content

If new content isn’t appearing: Wait for next sync
Content updates only during scheduled syncs or manual triggers.
Check sync status
Verify the last sync completed successfully.
Review content filters
Ensure your content isn’t being filtered out by URL patterns or exclusions.
Trigger manual sync
Force an immediate sync to test if content is detected.

Slow syncs

If syncs take too long: Reduce scope
Limit the number of pages or videos being synced.
Increase frequency
More frequent syncs process fewer changes each time.
Check content size
Large pages or videos take longer to process.
Review server response
Slow source websites delay sync completion.

Best practices

Choose appropriate frequencies

  • Daily - Only for content that changes daily
  • Weekly - Good balance for most use cases
  • Monthly - Sufficient for stable content
  • Manual - For complete control over updates

Monitor sync health

Check your Auto-Sync Jobs page regularly to:
  • Identify failed syncs
  • Review sync patterns
  • Optimize sync schedules
  • Remove unnecessary jobs

Optimize content sources

  • Use sitemaps instead of full website crawls when possible
  • Limit URL lists to essential pages
  • Exclude irrelevant sections of your website
  • Remove outdated content sources

Balance freshness and costs

More frequent syncing:
  • Keeps content more current
  • Uses more processing resources
  • May hit rate limits
  • Increases costs on higher-tier plans

Pricing

Auto-sync availability depends on your plan:
  • Daily sync - Available on Pro and Enterprise plans
  • Weekly sync - Available on all paid plans
  • Monthly sync - Available on all paid plans
  • Manual only - Available on all plans including Free
Check your plan details for specific limits on the number of sync jobs.

Next steps

I