|
Robots.txt & How to Stop Search engines From Crawling Directories on your Server
How do I create robots.txt?
Very easy. Create a new document in notepad or your favorite plain text editor. Save as : robots.txt.
How do I prevent a search engine from crawling a directory on my FTP?
Use the following text within your robots.txt file to prevent all search engine bots from spidering of one or more directories on your FTP.
User-agent: *
Disallow: /cgi-bin/
Disallow: /secretfolder/
What is a User-agent?
User agent, with respect to robots.txt : is the name of the search engine robot. Actually theres a little more to it then this but for our purposes this will do. An asterix (*) means : All. Or we could get more specific
like limiting a specific bot while allowing others. See the next question.
How do I allowing or disallow specific search bots (User-agent).
Below is an example of how you would disallow Googlebot while allowing any other search engines :
User-agent: Googlebot
Disallow: /cgi-bin/
Disallow: /secretfolder/
Next is an example of how you would allow Googlebot, but no other search engine bots :
User-agent: Googlebot
Disallow:
User-agent: *
Disallow: /cgi-bin/
Disallow: /secretfolder/
Where do I upload robots.txt?
Upload robots.txt to your public_html (wwwroot) or equivilant directory.
|