All the search engines follow these robot directives, which, while not a perfect security solution, limit the entry points used by the bots and spiders that index the Web.

FORBES: Google Me Not