How to Let AI Search Engines Find Your Website
TL;DR
AI tools like ChatGPT, Claude, Gemini, and Perplexity rely on websites to provide accurate information. If your site’s robots.txt file blocks them, your content won’t appear in AI search results. You can fix this easily by updating a few lines in your file—allowing AI tools to see your public pages while keeping private areas protected.
AI-powered search engines are changing how people discover information online.
But many websites still block AI crawlers without realizing it.
If your robots.txt file tells AI bots to stay out, your site might be invisible to the next generation of search tools. The fix takes only a few minutes, and it can help your content appear in AI-driven summaries and recommendations.
What a robots.txt File Does
Every website includes a small but powerful file called robots.txt. It tells search engines and crawlers which pages they can access and which to skip.
You can check yours by visiting: https://yourwebsite.com/robots.txt
Step 1: Check Your Current Settings
Open your robots.txt file and look for any lines that block bots, such as
User-agent: GPTBot
Disallow: /
This line tells ChatGPT’s crawler, GPTBot, that it can’t view your content.
If you see similar lines for other AI tools, they’re also blocked
Step 2: Update the File to Allow AI Crawlers
To let AI search engines view your public pages, remove those “Disallow” lines or update them.
A clean, safe setup for most websites looks like this:
User-agent: *
Disallow: /config
Disallow: /account/
Disallow: /api/
Allow: /
Sitemap: https://yourwebsite.com/sitemap.xml
This keeps private folders off-limits but lets AI tools and traditional search engines access your main pages. If your website is managed on a platform like Squarespace, Wix, or WordPress, you can usually find and edit your robots.txt file in the site settings or SEO tools section.
Note for Squarespace users: Squarespace automatically generates your robots.txt file. In version 7.1, you can manage crawler access under Settings → Website → Crawlers by unchecking the options for “Block Search Engine Crawlers” and “Block Known Artificial Intelligence Crawlers.” (In older 7.0 sites, you can also modify the file directly if Developer Mode is enabled.)
Step 3: Decide Who to Allow or Block
If you want more control, you can list specific AI crawlers individually.
# Allow ChatGPT
User-agent: GPTBot
Allow: /
# Allow Claude
User-agent: Claude-Web
Allow: /
# Block Perplexity
User-agent: PerplexityBot
Disallow: /
Official bot documentation:
Step 4: Save and Test Your Changes
After editing, save your file and visit:https://yourwebsite.com/robots.txt
Confirm your updates show correctly.
You can also test your setup using Google Search Console or similar crawl tools to ensure your file is readable and follows the correct format.
Step 5: Understand What Access Means
Allowing AI crawlers doesn’t give them control over your site; it simply lets them see public pages the same way Google or Bing does.
Not every crawler follows rules perfectly, but most major AI companies respect robots.txt.
You can always edit or remove permissions at any time if you change your policy.
Note: While major AI tools and search engines follow the instructions in your robots.txt file, compliance is not guaranteed. Some crawlers may ignore the file or access your content through alternate paths. Allowing AI access increases your chances of visibility, but it does not ensure full indexing or usage in AI-generated summaries. Regular content quality, site authority, and crawlability all still matter.
Example Setup
Here’s a simple, clear configuration that works for most small business or portfolio websites:
User-agent: *
Disallow: /admin/
Disallow: /login/
Disallow: /private/
Allow: /
Sitemap: https://yourwebsite.com/sitemap.xml
This protects sensitive directories while keeping your main content open to indexing and AI discovery.
Understanding the Example Setup
Not sure what each line means? Here’s a quick guide:
User-agent: tells which bots the rule applies to. Using an asterisk (
*) means “all crawlers.”Disallow: blocks bots from looking at specific folders or pages. For example,
/admin/keeps private dashboards hidden.Allow: lets crawlers view everything not listed under “Disallow.”
Sitemap: shows bots where to find your sitemap so they can navigate your site more efficiently.
Tip: The order matters. If a folder is both allowed and disallowed, the more specific rule usually wins.
Updating your robots.txt file is just one step toward improving online visibility. Combined with strong content, accessible design, and SEO best practices, it helps ensure AI and search engines can understand and surface your work.
The Bottom Line
Your robots.txt file shapes how both search engines and AI tools see your website.
If you want your content to appear in AI-generated results and summaries, make sure those crawlers are not blocked. Platforms like Squarespace 7.1 manage this file automatically, so you do not need to edit it manually. Confirm in your site’s crawler settings that reputable AI and search engines are allowed.
With a few small updates, you can keep your site discoverable in the era of AI-driven search while staying in control of what remains private. AI crawler names and access policies can change. Check your file every few months to stay current.