SEOmeter.com offers a convenient online tool that allows webmasters to monitor search engine's crawling activities on their website. With each website listed with us, our tool records in the database search engine's individual cache update events for the website, and display the history of cach updates (i.e., search engine's crawling history) in a user-friendly graph.

It is usually the case that a website "trusted" by search engines is crawled by search engine robots more frequently. This trust, in turn, is reflected in the website's search engine ranking. Therefore, by monitoring search engine's crawling activities, webmasters can gauge the search engine's trust in their website. SEO marketers can also utilize our service and quantify the benefit/effectiveness of their marketing campaign by checking the historical up/down trend of search engine crawling rate of their customer site. Finally, our historical chart is useful to detect and diagnose any site specific problem that interferes with search engine robot's crawling.

Currently we monitor Google search engine only (by using [cache:] search command). Among all major search engines in the Internet, Google is the only search engine that allows public access to the caching timestamp of cached URLs. With Yahoo, one needs to register their website in Site Explorer to be able to monitor the cache timestamp of their website's content. MSN and Ask.com only offer the cache date (but not timestamp) of individual webpages. We plan to incorporate more search engines into our monitoring system when other search engines start to make detailed cache timestamp publicly available.

The history graph visualizes how the crawl cycle (CC) of a given website changes over time. The CC is the time between two consecutive crawls (i.e., cache updates) done by search engine robots. Currently we display the last 3-month history of crawl cycles for a given website. The x-axis of each dot in the figure represents the timestamp of search engine's crawling event. The y-axis of the dot is a past one-month average of crawl cycle from that date. So for example, CC=2 on Dec. 20 means that between 11/20 and 12/20, search engine crawled the given website once every other day on average.

Please refer to our blog post that describes what determines search engine's crawl rate.