Index Penetration & Search Engine Saturation

Spider Stats

How Often do Spiders Crawl?

Search engines try to index the web as a whole once or twice a month. Portions of the web they deem to be unimportant may be visited once every few months. Portions of the web they view to be important and/or frequently changing get indexed far faster. Frequently crawled pages must be considered important because search engines keep spending computational power to keep them fresh.


Full training module available to subscribers.

Already have a premium account? Please log in using the Log In link at the top of the left column.


by Aaron Wall

January 3, 2008