Robots.txt & How to Stop Search engines From Crawling Directories on your Server

How do I create robots.txt?

Very easy. Create a new document in notepad or your favorite plain text editor. Save as : robots.txt.

How do I prevent a search engine from crawling a directory on my FTP?

Use the following text within your robots.txt file to prevent all search engine bots from spidering of one or more directories on your FTP.

User-agent: *
Disallow: /cgi-bin/
Disallow: /secretfolder/

What is a User-agent?

User agent, with respect to robots.txt : is the name of the search engine robot. Actually theres a little more to it then this but for our purposes this will do. An asterix (*) means : All. Or we could get more specific like limiting a specific bot while allowing others. See the next question.

How do I allowing or disallow specific search bots (User-agent).

Below is an example of how you would disallow Googlebot while allowing any other search engines :

User-agent: Googlebot
Disallow: /cgi-bin/
Disallow: /secretfolder/

Next is an example of how you would allow Googlebot, but no other search engine bots :

User-agent: Googlebot
Disallow:

User-agent: *
Disallow: /cgi-bin/
Disallow: /secretfolder/

Where do I upload robots.txt?

Upload robots.txt to your public_html (wwwroot) or equivilant directory.