{ robots.txt Generator }

// generate robots.txt files visually, no guesswork

Generate a robots.txt file visually. Add multiple user-agents, allow and disallow paths, set crawl delay, link your sitemap, and download the ready-to-deploy file. Free, no sign-up.

// QUICK-START TEMPLATES
// GLOBAL SETTINGS
Used by some search engines (Yandex)
// USER-AGENT RULES
Quick add:
// robots.txt PREVIEW 

    


    
    

      

        // VALIDATION HINTS
      

      

    


    
    

      

        // SYNTAX REFERENCE
      

      

        
DirectiveMeaningExample

        
          
User-agentWhich bot the rules apply to. Use * for all bots.User-agent: *

          
DisallowURL path(s) the bot should not crawl. Blank = allow all.Disallow: /admin/

          
AllowOverride a Disallow for a more specific path.Allow: /admin/public/

          
Crawl-delaySeconds between bot requests (not honoured by Google).Crawl-delay: 10

          
SitemapFull URL to XML sitemap. Can appear multiple times.Sitemap: https://…/sitemap.xml

          
HostPreferred domain version (Yandex only).Host: example.com

        
      

    


  










  

    

      
HOW TO USE

      
    
        
  1. 01
    Start with a template
    Click a quick-start template to pre-fill common configurations. "SEO Safe" is a good starting point for most websites — it allows search engines while blocking admin and private paths.
    2. 
        
  2. 02
    Configure agent rules
    Add or modify user-agent blocks. Use the quick-add buttons to target specific bots. Add Allow and Disallow rules with the + buttons inside each agent card.
    3. 
        
  3. 03
    Download and deploy
    Click Download to save robots.txt and upload it to the root of your domain at https://yourdomain.com/robots.txt.
    4. 
      

    

    

      
FEATURES

      

        Visual builder
        Bot presets
        6 templates
        Allow / Disallow
        Crawl-delay
        Validation hints
      

      
USE CASES

      
    
        
  • 🤖 Block search engines from indexing staging sites
    • 
        
  • 🤖 Prevent crawling of admin and private paths
    • 
        
  • 🤖 Block AI training bots (GPTBot, CCBot)
    • 
        
  • 🤖 Allow only specific bots for bandwidth control
    • 
        
  • 🤖 Set crawl delay for high-traffic sites
    • 
      

    

    

      
WHAT IS THIS?

      
robots.txt Generator builds a valid robots.txt file using a visual form. Add user-agent blocks for specific bots or all crawlers, configure Allow and Disallow path rules, set a Sitemap URL, and download the complete file — ready to place in the root of your website. No syntax knowledge required.

      
RELATED TOOLS

      

        🏷 Meta Tag Generator
        ✔ HTML Validator
        ⚡ HTML Minifier
        🔗 URL Encoder
        { } JSON Formatter
      

    

  







  

    
FREQUENTLY ASKED QUESTIONS

    

      

        

          What is robots.txt?
          
robots.txt is a plain text file that lives at the root of a website (e.g. https://example.com/robots.txt) and tells compliant web crawlers which URLs they are allowed or not allowed to access. It follows the Robots Exclusion Standard and is the primary mechanism for communicating crawling preferences to search engines and other bots. Not all bots respect it — it is a directive, not an access control mechanism.

        

        

          Does robots.txt prevent Google from indexing a page?
          
No. robots.txt prevents crawlers from accessing a URL, but it does not prevent them from indexing it. If other pages link to a disallowed URL, Google may still index it based on those links, without crawling its content. To prevent indexing of a page, use the noindex meta robots tag or the X-Robots-Tag HTTP header in addition to (or instead of) robots.txt.

        

        

          What does User-agent: * mean?
          
The asterisk wildcard matches all crawlers that do not have a more specific user-agent block. It is the correct way to set default rules that apply to all bots. You can combine a wildcard block with specific bot blocks — the most specific matching block is the one that applies to a given bot. Order matters: place the wildcard block before or after specific blocks, but each block stands alone.

        

      

      

        

          How does Disallow work with paths?
          
Disallow: /admin/ blocks the /admin/ path and everything under it. Disallow: / blocks the entire site. Disallow: with an empty value allows everything (this is how search engines interpret a blank Disallow). Paths are prefix-matched: Disallow: /page would block both /page and /page-two. Google also supports limited wildcard patterns using * and $ within paths.

        

        

          Should I block GPTBot and other AI crawlers?
          
This is a personal or business decision. OpenAI's GPTBot, Anthropic's ClaudeBot, and Common Crawl's CCBot are used to train AI language models. If you do not want your content used for AI training, adding Disallow rules for these bots is the appropriate mechanism. Note that compliance is voluntary — legitimate AI companies like OpenAI and Anthropic have stated they respect robots.txt, but this cannot be enforced technically.

        

        

          Does Crawl-delay work with Googlebot?
          
No. Google does not support the Crawl-delay directive in robots.txt. To reduce Googlebot's crawl rate, use Google Search Console's crawl rate settings instead. Other crawlers — including Bingbot, DuckDuckBot, and many third-party crawlers — do honour Crawl-delay. The directive specifies the minimum number of seconds the bot should wait between consecutive requests to the server.

        

      

    

  







  

    
robots.txt Generator — Build and Download robots.txt Free

    
A correctly configured robots.txt file is one of the first things to set up on any website. It tells search engines which parts of the site to crawl and which to skip, helps manage crawl budget for large sites, and is the primary tool for blocking AI training bots from accessing your content. Getting the syntax wrong — an incorrect path, a missing blank line between blocks, or a misnamed directive — can silently cause crawling issues.


    
The robots.txt File Format

    
A robots.txt file consists of one or more blocks, each beginning with a User-agent directive. Each block applies to the specified bot and contains any number of Allow and Disallow directives. Blocks are separated by blank lines. Global directives like Sitemap and Host appear outside any user-agent block, typically at the end of the file. Lines beginning with # are comments and are ignored by parsers.


    
Crawl Budget and Large Sites

    
Search engines allocate a fixed crawl budget to each domain — a limit on how many pages they crawl and how frequently. For small sites this is rarely a concern, but for large e-commerce sites, news sites, or sites with extensive URL parameter variations, managing crawl budget becomes important. Disallowing low-value URLs — paginated filtered pages, internal search results, session-ID URLs, print versions — conserves crawl budget for content that should be indexed.


    
Blocking AI Training Bots

    
Since 2023, several AI companies have introduced user-agent strings for their training crawlers. OpenAI's GPTBot crawls content for ChatGPT and other OpenAI models. Anthropic's ClaudeBot crawls for Claude. Common Crawl's CCBot feeds many open-source models. To opt out, add a Disallow: / rule for each of these user-agents. This is the officially supported method for controlling AI training data collection.


    
Testing Your robots.txt

    
After deploying a robots.txt file, verify it with Google Search Console's robots.txt Tester, which shows you how Googlebot interprets your rules and lets you test specific URLs. Bing Webmaster Tools offers a similar feature for Bingbot. Always test after making changes, especially when using wildcard patterns, as incorrect patterns can accidentally block important content from being crawled.

  







☕