Why Isn’t My Chatbot Crawling All Pages from My Website?

If your chatbot isn’t pulling information from all the pages on your website, don’t worry — this is a common issue with a few possible explanations and easy solutions! 🛠️

Common Reasons the Chatbot Misses Pages

1️⃣ Crawler Page Limit

  • On the free AI chatbot plan, the built-in crawler can access up to 50 pages.
  • This limit doesn’t apply to pages scanned using the Website Search feature.

2️⃣ Pages Aren’t Linked from the Homepage

  • The crawler discovers pages by starting at your homepage and following internal links.
  • If certain pages aren’t linked from the homepage (or other linked pages), the crawler may never find them.

3️⃣ Content Loaded Dynamically with JavaScript

  • Pages that use JavaScript frameworks (like React, Angular, Vue) may load content dynamically.
  • Standard crawlers might not see this content if it’s not rendered on the initial HTML page.

4️⃣ The Website Is Protected by Cloudflare CAPTCHA

  • If your website uses Cloudflare or another service that presents visitors with a CAPTCHA or security challenge, automated crawlers — including chatbots — will be blocked.


    How to fix it:

    Temporarily disable CAPTCHA protection and recrawl the pages with your chatbot. Once the crawling process is complete, you can re-enable the protection. This allows the chatbot to index your website content without permanent security changes.

🛠️ Solutions: How to Get Your Chatbot to See All Website Pages

🗺️ Add a Sitemap (sitemap.xml)

  • Generate and upload a sitemap (sitemap.xml) for your website.
  • Add the link to your sitemap in the chatbot’s Data Sources settings.
  • Sitemaps make it easy for the crawler to find and index all important pages, whether they're linked from the homepage or not.

📝 Add Important Pages Manually

  • In the Data Sources settings, you can add specific URLs or entire folders for indexing.

🔍 Use the Website Search Feature

  • The Website Search feature lets your chatbot access any page indexed by Google or Bing, even if it’s not discoverable via normal crawling.
  • Each Website Search request uses 5 messages per request from your quota.

  • If you’ve already enabled this, your chatbot can answer using a wider range of website data.

⚠️ Important Details to Remember

  • The 50-page crawler limit only impacts the built-in crawler, not Website Search.
  • Pages requiring login or those heavily dependent on client-side JavaScript may not be crawlable.
  • Sitemaps should be kept up to date when you add or remove website content.
  • If your site blocks bots (via robots.txt or security plugins), the chatbot crawler may be unable to access your pages.

💬 Still Need Help?

If you have more questions about getting your chatbot to crawl your website, or need assistance setting up your sitemap or settings, contact our support team — we’re here for you! 😊

Did this answer your question? Thanks for the feedback There was a problem submitting your feedback. Please try again later.

Still need help? Contact Us Contact Us