robots.txt - Forbidden Directories

The Laboratory for Dependable Distributed Systems at the RWTH Aachen has gathered some statistics about directories forbidden by robots.txt files. There are some interesting entries there, e.g. "private", "secure", ...

Many people don't realize that the robots.txt files contain nothing more than hints for spiders, they do not "protect" you in any way. On the contrary, you're pointing potential attackers right where you absolutely don't want to have them.

(via disLEXia 3000)

Syndicate content