What is the Robot Text File?


Visitors: 374

The robot text file is used to disallow specific or all search engine spider’s access to folders or pages that you don't want indexed. Why would you want to do this?

You may have created a personnel page for company employees that you don't want listed. Some webmasters use it to exclude their guest book pages so to avoid people spamming. There are many different reasons to use the robots text file.

How do I use it?

You need to upload it to the root of your web site or it will not work - if you don't have access to the root then you will need to use a Meta tag to disallow access. You need to include both the user agent and a file or folder to disallow.

What does it look like?

It's really nothing more than a “Notepad" type . txt file named “robots. txt"

The basic syntax is

User-agent: spiders name here
Disallow:/ filename here

If you use
User-agent: *
The * acts as a wildcard and disallows all spiders. You may want to use this to stop search engines listing unfinished pages.

To disallow an entire directory use

To disallow an individual file use

You have to use a separate line for each disallow. You cannot for example use

Disallow:/file1.htm, file2.html

You should use


For a list of spider names visit http://www.robotstxt.org/wc/active/html/ Make sure you use the right syntax if you don't it will not work. You can check you syntax here http://www.searchengineworld.com/cgi-bin/robotcheck. cgi

For help on creating robot text files there is a program call robogen. There is a free version and an advanced version, which costs $12.99 http://www.rietta.com/robogen/

Alan Murray is a Certified Internet Webmaster Professional and Provides SEO Services and Website design.



Article Source:

Rate this Article: 
Forex Robot Forcast Software 3 Critical Factors to Chose the Best Forecast ..
Rated 4 / 5
based on 5 votes

Related Articles:

How To Check - Validate A Robots Text File

by: Mike Tekula (June 11, 2007) 
(Internet and Businesses Online)

Microsoft Great Plains Integration Manager - Working With Text File

by: Andrew Karasev (December 30, 2004) 
(Computers and Technology/Software)

Online Conversion Tool - Services To Convert From Text to PDF File Format

by: Nicola Deiana (July 02, 2007) 
(Computers and Technology)

The Robots Text File Or How To Get Your Site Properly Spidered, Crawled, ..

by: Andrei Smith (November 16, 2005) 
(Internet and Businesses Online/SEO)

Search Engine Tips & Tricks: Create a Robots Text File for Your Web Site

by: Sandra Waggett (September 22, 2005) 
(Internet and Businesses Online/SEO)

Forex Autocash Robot Review - Is Forex Auto Cash Robot a Scam?

by: William Barnes (September 30, 2008) 
(Finance/Currency Trading)

Importance of Keywords in Anchor Text or Title Text

by: Purva Mewar (June 19, 2005) 
(Internet and Businesses Online/SEO)

Text Loans- Acquire Finances By Sending A Text

by: Roselynn Jeff (October 25, 2011) 
(Finance/Personal Finance)

Cell Phone Reviews-To Text Or Not To Text

by: Wycliffe Williams (December 06, 2006) 

Forex Robot Forcast Software 3 Critical Factors to Chose the Best Forecast ..

by: Nigel Banks (June 19, 2008) 
(Finance/Currency Trading)