Are You Getting Crawled Deeply?

Is there a way to know if Google or other search engines have [tag]crawled[/tag] your site's pages deeply recently? Yes and the easiest way to find out is to conduct a search for a specific line of text that is unique to that page and be sure to enclose the speific line in "quotes" and nothing else. The quotes tell the search engine to search for exactly that set of words in that exact order. If that page shows up in each of the major search engines for that search you wiull know the page has been indexed.

Now a second important consideration is how recently the page has been [tag]indexed[/tag]. When a page has been recently crawled and indexed, that indicates the search engine places a higher priority on maintaining a recent copy of the page because the page is considered important. On the other hand, if the page hasn't been crawled in a month or more, that indicates the page isn't very important to the search engine due, in many cases, to a lack of incoming links.

To see how recently a search engine has indexed a page, click the little cached link that's displayed below the page's listing in the search results. This will retrieve the search engine's copy of the page that is stored on its server. You can also view your server logs or use your web analytics tools to see if and when any specific search engine last crawled your pages. This is the most efficient method when tracking a large number of pages.

Google and Microsoft both show the exact date they last indexed the page but Yahoo doesn't. That's why some webmasters include dynamic code that displays the current date somewhere on the page. That way, when Yahoo grabs a copy of the page, they also grab the date which can then be easily used to determine when Yahoo cached the page.