Public URLs
Public URLs let you add web content to your chatbot's knowledge base by scraping website pages. This is powerful for keeping your AI updated with current information from your website, documentation, or help center.
Single URLs vs Sitemaps
You can add content in two ways:
Single URLs
Add individual web pages that contain important information. Perfect for:
- Product documentation
- FAQ pages
- Policy pages
- Important announcements
Sitemaps
Add an entire sitemap to scrape multiple pages at once. This is ideal for comprehensive websites, but requires careful planning.
Choosing Content Wisely
Not all web content is equally valuable for your AI. Some pages can actually hurt performance:
Good Content
- Static documentation pages
- Help center articles
- Product specifications that don't change frequently
- Company policies and procedures
Problematic Content
- Blog posts (often time-sensitive and not relevant to customer questions)
- Product pages with dynamic pricing/inventory (AI might give outdated information)
- News sections
- Event calendars
- User-generated content areas
The Double-Edged Sword
For very large websites (hundreds or thousands of pages), consider creating a secondary sitemap containing only the most important, stable content pages. This gives you precise control over what your AI learns.
Keeping Content Current
Your URLs are automatically rescraped every 2 weeks to keep information current. You can also manually trigger rescraping when needed.
Manual Rescraping
- Click the "Rescrape" button next to any URL
- Wait at least 5 minutes between manual rescrapes
- Useful when you've updated content and want immediate refresh
Automatic Updates
- URLs are rescraped every 2 weeks automatically
- Failed scrapes are marked as warnings and stop auto-rescraping
- Manually rescrape failed URLs to re-enable automatic updates