How Quickly do Web Search Engines Get Updated?

When I put my first page on the web, I of course checked to see how quickly the various search engines incorporated it. And I checked ... and checked ... and am still checking!

I had this silly notion that the web crawlers used by the search engine were enormously efficient, and grabbed every page within a few days. Not so! Even after submitting my URL to 4 different search engines, only 2 of them have found my page 3months later!

The following table reports on how quickly some search engines have found that first page. I will collect data about how quickly they find this page, as well, since it might appear much more quickly after they have "found" my site.

Update on 10/13/96: Once again, I was wrong. Only excite has found the new page after a month and a half!

Delay (In Days) For A Web Page To Appear In Various Search Engines

Delay (days) for  
Search Engine My Home Page This Page Comments
Alta Vista ~36 >51
Excite ~48 ~15
Hotbot >90 >51
InfoSeek >90 >51 Infoseek twice has had my home page, but each time has quickly lost it!! It currently does not have it indexed.
Lycos ~29 >51
Magellan >90 >51
WebCrawler >48 >51
Yahoo >90 >51

For reference, the date that I first put my home page on the web was 7/13/96, and the date that I put this page on was 8/22/96. The way I conducted the search was to search for "Tom Chester" for my home page, and for "+delay +appear +updated +search +engines +Lycos" for this page.

Several years after I did this study, researchers Steve Lawrence and C. Lee Giles (Nature, July 8, 1999) reported that it took an average of 6 months for a page to be indexed by search engines.


e-mail: Tom Chester