How do I prevent HotBot from indexing my site?

A robots.txt file removes pages from HotBot. You can place this file on your site to tell search robots which directories they should add to their databases and which they shouldn't index.

If you prefer that HotBot not index your site, ask your webmaster to create a robots.txt file for your site. HotBot's crawler will fetch and obey this command file. It will obey any entry with a user agent of "*" or containing the word "Slurp" (the name of HotBot's crawler).

You can find the robots.txt file standard here.

HotBot also honors the robots "noindex" meta tag, which keeps HTML files out of HotBot's database index. This can be added to the head section of an HTML document, as illustrated below. You can also remove the offending page from your Web server or restrict public access to your server (talk to your system administrator or Web site hoster). Your page listing will remain on HotBot until the next time our Web crawler visits your site, which can take up to 60 days.

For the security of people submitting pages to HotBot, we do not manually remove pages from our index. So, if you find information on a page that concerns you in some way (e.g., your personal information is displayed or you find instances of trademark infringement), write to the webmaster of that particular site.

For more information, see Webmonkey Tutorials.