Robots.txt Generator

Search Engine Optimization

Robots.txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.



About Robots.txt Generator

Search Engines uses robots (User-Agents) to crawl web pages. The robots.txt file is a text file that defines which parts of a domain can be crawled by a robot. In addition, the robots.txt file can include a link to the XML-sitemap.

Whenever search engines crawl a website, the first thing it looks for is a robots.txt file in the domain root. If found, the search engine then reads the file’s list of directives to see which directories and files, and also check for the ones that are blocked from crawling. This file can be created with a robots.txt file generator

This Free Tool allows you to easily create robots.txt file for your site.

you can create and copy the text or create and save the file.

Enjoy!

 

Suggested read: Robots txt Generator And How to Use it for Website Indexing




Create a free website with GoDaddy!