Why Isn’t My Chatbot Crawling All Pages from My Website?
If your chatbot isn’t pulling information from all the pages on your website, don’t worry — this is a common issue with a few possible explanations and easy solutions! 🛠️
Common Reasons the Chatbot Misses Pages
1️⃣ Crawler Page Limit
- On the free AI chatbot plan, the built-in crawler can access up to 50 pages.
- This limit doesn’t apply to pages scanned using the Website Search feature.
2️⃣ Pages Aren’t Linked from the Homepage
- The crawler discovers pages by starting at your homepage and following internal links.
- If certain pages aren’t linked from the homepage (or other linked pages), the crawler may never find them.
3️⃣ Content Loaded Dynamically with JavaScript
- Pages that use JavaScript frameworks (like React, Angular, Vue) may load content dynamically.
- Standard crawlers might not see this content if it’s not rendered on the initial HTML page.
4️⃣ The Website Is Protected by Cloudflare CAPTCHA
If your website uses Cloudflare or another service that presents visitors with a CAPTCHA or security challenge, automated crawlers — including chatbots — will be blocked.
How to fix it:
Temporarily disable CAPTCHA protection and recrawl the pages with your chatbot. Once the crawling process is complete, you can re-enable the protection. This allows the chatbot to index your website content without permanent security changes.
🛠️ Solutions: How to Get Your Chatbot to See All Website Pages
🗺️ Add a Sitemap (sitemap.xml)
- Generate and upload a sitemap (sitemap.xml) for your website.
- Add the link to your sitemap in the chatbot’s Data Sources settings.
- Sitemaps make it easy for the crawler to find and index all important pages, whether they're linked from the homepage or not.
📝 Add Important Pages Manually
- In the Data Sources settings, you can add specific URLs or entire folders for indexing.
🔍 Use the Website Search Feature
- The Website Search feature lets your chatbot access any page indexed by Google or Bing, even if it’s not discoverable via normal crawling.
- Each Website Search request uses 5 messages per request from your quota.
If you’ve already enabled this, your chatbot can answer using a wider range of website data.
⚠️ Important Details to Remember
- The 50-page crawler limit only impacts the built-in crawler, not Website Search.
- Pages requiring login or those heavily dependent on client-side JavaScript may not be crawlable.
- Sitemaps should be kept up to date when you add or remove website content.
- If your site blocks bots (via robots.txt or security plugins), the chatbot crawler may be unable to access your pages.
💬 Still Need Help?
If you have more questions about getting your chatbot to crawl your website, or need assistance setting up your sitemap or settings, contact our support team — we’re here for you! 😊