What is crawler?

A web crawler or more commonly known as a Web Spider or a Web Bot . The crawler is an automated program that is designed to scan the World Wide Web. These crawlers are used by search engines like Google, Yahoo or MSN. It provided these engines with update knowledge of the sites that are on the web. For example, if a site has been crawled before and has received a index rank in the engine and three months later when the crawler comes back it sees something different that could affect the ranking.

The crawlers begin with a list of sites to scan, these are called seeds. When the crawler comes to the sites it first identifies the hyperlinks on the page and then puts them on another list of the Url's called the crawl frontier. There are a couple of factors that may affect the way a crawler crawls a site. If there is a large volume of information it may take the crawler a bit of time to send the downloads. So the crawler will have to see what information it wants to send down first.

Another thing a crawler will notice is if the site has undergone a change since the last time it crawled that site. There may have been pages added or removed during this time. Which is a good thing because a search engine will want to see a fresh site as long as when you remove pages, hopefully they won't be pages that the sites fond to be a key factor in giving the site its rank.

In the past year or so there has been an up flow of pages being created by server scripts . These are difficult for a crawler to identify . If it does recognize the language at all then you may only get a small return for original content . In turn the rank you had may fall. Crawlers may only choose to review sites that have HTML and stay away from the MIME types . The crawler can make a request to view only the HTML sites , the crawler can end up only scanning the sites that end in .html, .htm, .asp, .php and a slash.

Your goal is to have the crawler come to your site in order for you to get a good rank. By having something on your on your site that a crawler will not see kind of defeats the purpose of having a site. It can take a crawler weeks or a few months to change a rank but in that time there could be changes that have been made to the site. The crawler will already have taken this into consideration. You may have added or removed some content. Try to keep the site the same until your index changes. You can add pages because search engines love new content.

Some people may think that a crawler could not possible know if the content on a site is fresh but they are programmed with formulas that help them figure it out. They can determine the freshness and the age of the information. Some of the different types of crawlers are the RBSE , the Web Crawler, The World Wide Web Worm , the Google Crawler , Cobweb PolyBot , Mercator and the Fast Crawler. You can expect to get a visit from the crawler at least every three months. Some people believe that if you already have a good ranking that it won't change your rank but it can and has over the years. Nothing is guaranteed on the web.