To prevent unwanted content from the research indexes, webmasters can instruct spiders not to crawl specific files or directories with the normal robots.txt file in the root Listing of your domain. On top of that, a web site might be explicitly excluded from a internet search engine's database by using a meta tag unique to robots (typically ). Ever