Robots.txt Generator

Search Engine Optimization

Robots.txt Generator

Default - All Robots are:  
Sitemap: (leave blank if you don't have) 
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo MM
  Yahoo Blogs
  DMOZ Checker
  MSN PicSearch
Restricted Directories: The path is relative to root and must contain a trailing slash "/"

Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.

About Robots.txt Generator

Search Engines uses robots (User-Agents) to crawl web pages. The robots.txt file is a text file that defines which parts of a domain can be crawled by a robot. In addition, the robots.txt file can include a link to the XML-sitemap.

Whenever search engines crawl a website, the first thing it looks for is a robots.txt file in the domain root. If found, the search engine then reads the file’s list of directives to see which directories and files, and also check for the ones that are blocked from crawling. This file can be created with a robots.txt file generator

This Free Tool allows you to easily create robots.txt file for your site.

you can create and copy the text or create and save the file.



Suggested read: Robots txt Generator And How to Use it for Website Indexing

Create a free website with GoDaddy!