Spider

A Search Engine Spider is a computer software program used on the World Wide Web that searches the contents of the web. Spiders sort of take a picture of each webpage in the World Wide Web (by downloading the HTML code on each page). This information is then used by search engines to create indexes and to create Search Engine Results Pages (SERPs), those pages of search results you get when you type in a query into a search engine.

Spiders are known by many names, but they all do the same thing. Other names for search engine spiders include: Bot, Robot, Scutters, Crawler, Ants, Worms, Web Scutters, Automatic Indexers, Web spiders, and Web robots. You can also see Crawler and Bot for more information on this term.

Spiders are the empowering force behind search engines. Without Spiders, search engines like Google and Yahoo could not function as they do today. Instead of the Internet bringing you the world at your fingertips, a world without Spiders would mean we'd need to physically search through Internet "phone books" ourselves, category by category, and city by city.

Not all Spiders are good, though. Some Spiders (those not affiliated with legitimate search engines) crawl through email contacts, blogs, newsrooms, and forums for personal information. This information can then be used to deliver spam to email addresses, or disable websites or do other damage.

Some types of bad web spiders include Viruses, Worms, and Spam bots. You may have heard of CAPTCHA, which is a popular security feature that keeps most bad spiders from damaging websites or gathering information without permission (such as collecting email addresses and then spamming the emails).