Posted by admin in By Jenna Ryan – November 2006 – The Marketing Shop.com
A Search Engine Crawler is a program or algorythm sent out by search engines to the various websites over the internet. These robots, also called “spiders,” go about the web, “crawling” or “spidering” webpages in search of textual content.
How Search Engines Gather Content
It is no longer necessary to “submit” your website to search engines in order to get listed. Today, web crawlers are everywhere searching for relevant web content to appease hungry searchers.
Search engines don’t provide content to the searcher in real time. Your website must be “indexed” or “cached” in the search engine’s database before anyone searches for your website. This is a process that takes time.
At any given time, there can be a search engine on your website, trying to index the content that’s there. The search engine crawler cannot read everything that’s on your website, and will skip unreadable content that’s not text such as Flash, Images, Graphics, Javascript, Frames or Database Drive-Dynamic Content.
Website Content
The crawler accounts for all the content on the page and determines the relevancy for certain topics.
Google Search Engine Crawlers
Googlebot is Google’s Search Engine Crawler.
Yahoo Crawlers
Yahoo Surp Crawler – SEO Chat Article
Web Crawlers – Wikopedia
Major Search Engines & Directories – SearchEngineWatch
