Robots.txt Generator

Default - All Robots are:

Crawl-Delay:

Sitemap: (leave blank if you don't have)

Search Robots:	Google
	Google Image
	Google Mobile
	MSN Search
	Yahoo
	Yahoo MM
	Yahoo Blogs
	Ask/Teoma
	GigaBlast
	DMOZ Checker
	Nutch
	Alexa/Wayback
	Baidu
	Naver
	MSN PicSearch

Restricted Directories:	The path is relative to root and must contain a trailing slash "/"

Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.

About Robots.txt Generator

Search Engines uses robots (User-Agents) to crawl web pages. The robots.txt file is a text file that defines which parts of a domain can be crawled by a robot. In addition, the robots.txt file can include a link to the XML-sitemap.

Whenever search engines crawl a website, the first thing it looks for is a robots.txt file in the domain root. If found, the search engine then reads the file’s list of directives to see which directories and files, and also check for the ones that are blocked from crawling. This file can be created with a robots.txt file generator.

This Free Tool allows you to easily create robots.txt file for your site.

you can create and copy the text or create and save the file.

Enjoy!