Question 133. Is There Any Way To Prevent A Spider From Grabbing Urls That You Want To Keep Off Search Engines?
Absolutely - there are many ways, and you should use them all. For a quick overview, search for robots or spiders on Google, visit The Web Robots Page, or visit B.4 Notes on helping search engines index your website from the World Wide Web Consortium. The most fool-proof method to block spidders is to password protect any files that you don’t want indexed by the search engines. See Can a search engine index pages that are password protected.
