Be a Spider Magnet
Best practices for getting your Web site more fully indexed
By Michael Gorman
A large gap exists between the content available on the Internet and the information captured and accessible to users of popular search engines. Google, Yahoo! and MSN still are only capturing and indexing a small percentage—estimates are around 5 percent to 10 percent—of the Web pages available online.
For Fortune 2000 marketers, the gap between online content created versus content indexed is growing as Web sites become larger and more complex. Specifically, innovations in site design, which often improve the user experience for those who visit a site, also can make those visits less likely. Some innovations, such as session IDs, throw up barriers to search engines seeking to crawl and index a site's contents. Without search engine visibility, a site's pages appear less frequently in search results, leading to less traffic.
Check Your Vital Signs
To get a sense of how well search engines are indexing your site, visit a search engine's site. For example, Marketleap (a Digital Impact company) offers a Search Saturation Index among its free search assessment tools. Input your domain name into this reporting tool (leave out the "www.") to learn how many of your Web pages are indexed in the major search engines' directories right now.
If you find that the total number of pages indexed is substantially less than the number of pages in your Web site, something about the way your site is designed may be preventing search engines from fully exploring and cataloging your Web site.
Make it Easy for Search Engines to Index Your Site
The good news is that even if you haven't previously taken search engines into account as one of the highly valuable visitors to your Web site, there are some best practices you can follow to dramatically improve search engines' ability to easily find and index your site.