Jenna Ryan, President
The Marketing Shop.com
November 2006

Search Engine Crawlers

A Search Engine Crawler is a program or algorythm sent out by search engines to the various websites over the internet. These robots, also called "spiders," go about the web, "crawling" or "spidering" webpages in search of textual content.

How Search Engines Gather Content

It is no longer necessary to "submit" your website to search engines in order to get listed. Today, web crawlers are everywhere searching for relevant web content to appease hungry searchers.

Search engines don't provide content to the searcher in real time. Your website must be "indexed" or "cached" in the search engine's database before anyone searches for your website. This is a process that takes time.

At any given time, there can be a search engine on your website, trying to index the content that's there. The search engine crawler cannot read everything that's on your website, and will skip unreadable content that's not text such as Flash, Images, Graphics, Javascript, Frames or Database Drive-Dynamic Content.

Website Content

The crawler accounts for all the content on the page and determines the relevancy for certain topics.

Google Search Engine Crawlers

Googlebot is Google's Search Engine Crawler.

Yahoo Crawlers

Yahoo Surp Crawler - SEO Chat Article

Web Crawlers - Wikopedia

Major Search Engines & Directories - SearchEngineWatch