Search and Destroy – How to Reclaim Relevancy from Search Spiders and Data Mining Bots
Posted in Content Management,SEO/SEM,Web Analytics,Web Experience Management by Brian Bolton on September 20th, 2011
Identifying robots more effectively - even indirectly enlisting their help - will distill your results, and solidify your site's relevancy.
In a recent post, I spent some time exposing what we’ve since been calling the “bot quandary,” which, put simply, is the tendency for a sites’ analytics to become skewed – sometimes significantly – because indexing visits from bots and spiders are counted by search engines. In this post, I will spend some time on a few things that can be done to help reduce that skew in order to clarify your site’s actual relevancy.
Take aim and select your target, then execute
As I pointed out in my prior post, these spider visits affect site analytics, often significantly. In order to get a more accurate picture of your site’s true relevancy, it’s necessary to remove them from the count. But – and this is important – you don’t want to block the bots completely, or you’ll stop being indexed altogether. Of course, this can be a much worse fate than some fairly skewed numbers. (more…)






