Re: [Off Topic] robots.txt

[Joe Savelberg <joe@xxxxxxxxxxx>]

Dan,

All the information you need should be available in the official 
robots.txt FAQ at:

http://www.robotstxt.org

You can also see any robots.txt file of any site by typing 
http://www.domain.com/robots.txt -- that way you get some real-life 
examples.

There are also some ways to add Meta-Tags that prevent robots from 
crawling the site. Please do remember not to put any sensitive 
information (such as hidden directories) in your robots.txt as 
everybody will be able to see them.


Joe.

At 08:46 -0600 29/07/2003, Dan Tappin wrote:
> Can some one point me to a good robots.txt FAQ and / or post an example file?
>
> I am looking for a simple one to limit the robots to my home page 
> and try to prevent the crawling through my whole site.
>
> Thanks,
>
> Dan
>
> ---------
> Tenon Intersystems' iTools Mailing List
> To unsubscribe: send mail to itools-request@xxxxxxxxxxxxxxx
> with the body only containing: unsubscribe
> Find the searchable mailing list archives at:
> http://itools.blackpepper.co.nz/


---------
Tenon Intersystems' iTools Mailing List
To unsubscribe: send mail to itools-request@xxxxxxxxxxxxxxx
with the body only containing: unsubscribe
Find the searchable mailing list archives at:
http://itools.blackpepper.co.nz/