Separating the bot from the customer will refine your site's analytics - sometimes more than you'd think.

Two words you hear constantly in content discussions are relevance and traffic – testament that these concepts, maybe more than any others, drive the life of your site. Usually, but not always, the two form a symbiotic relationship: relevance definitely drives traffic – in many different ways – and usually our focus is on how to harness that power for good.

For this post, though, I want to focus on another, less obvious – and much less intuitive – relationship between content and traffic: the concept behind the relevance of traffic.

It’s not something web professionals have examined readily – or very heavily – as search has grown, but the concept is gaining steam around analytics and SEO discussions lately.

Here’s a pine-nut-sized explanation of the problem:

• All websites – and especially niche sites – depend heavily on search to survive, and publishers spend countless dollars and hours pouring over analytics and affecting optimization to be sure their SERPs stay up near the top;
• These numbers are constantly changing as search engine spiders – or robots (“bots” also works) – crawl across the ‘net and index pages to add to search databases;
• Bot and spider visits are counted in the overall site SEO picture, which skews final numbers – sometimes significantly.

Obviously, it’s necessary to allow the bots access to your site (at least those that are actually indexing, as opposed to “scraping”) – kind of like it’s necessary to answer the census taker at your front door. The question now, then, is about the relevancy of those numbers. The fact is, actual, human visitors are the only real numbers you – or search engines – should really be concerned with.

At the end of the day, bots are never going to turn into conversions, no matter what sort of freebie you’re offering to entice them to register.

How did we end up here?

Sometimes, working at web speed, we tend to lose sight of some of the internet’s basic nature. This is one of those cases. The nature of the web has developed so quickly – and from so many different aspects – over the last few years that we now find ourselves in a sort of quandary. How’d we end up developing something that works, some would say, just too damned well? Put simply, we made spiders just a tiny bit too fast and smart for our own good (Hmmm – does that sound a little apocalyptic? Didn’t mean it to).

Until recently, bots weren’t smart or fast enough to execute JavaScript, which is one of the staple ingredients of most analytics packages. Since they only visited to index, but didn’t have the power to “pull the trigger” of the count process, they weren’t processed as a visit. Now, they’re not only much faster, many are intelligent enough to process a complete click stream.

So, your mission, should you decide to accept it, is to reduce the skew from the bot visits as much as possible – without losing the benefit of indexing. That done, you’ll begin to present your analytics whizzes, and particularly your customers, with a more accurate, bona fide picture of your site’s overall relevancy.

Let the experts handle it for you

There are a number of ways to complete the mission. You can adjust your analytics package to filter out IP addresses from which these bots are coming, or you can use an identifying string to see who they are and where they’re from before deciding to block them or let them onboard (more on these processes in another upcoming post – stay tuned).

iAPPS, with the launch of Analyzer in version 4.6 – the newest version of our WEM suite – is approaching the bot quandary in a really innovative way for its Software-as-a-Service (SaaS) customers. iAPPS developers are actually implementing rigorous methods to keep the wrong bots from being counted, behind the scenes, for all hosted customers. It’s a pretty gutsy move, especially considering that, so far, this whole “robot relevancy” idea isn’t very widely understood.

Case in point: once those visits stop being counted, customer traffic numbers are almost certainly going to decline – and might decline significantly. The move also means taking on a long-term level of support – in addition to the already comprehensive level the company offers its SaaS customers – with the need to constantly update options, IP addresses, etc.

“Not a problem,” says Anthony Wilson, Bridgeline’s Vice President of Delivery. “The quality and level of support we offer our SaaS customers is already top-notch, and we’re ready to take this on. After all, we’re the experts. This type of support is exactly what our customers deserve.”

Coming soon: Search and Destroy

Stay tuned for another post about the bot quandary, where I’ll discuss more options for reducing the skew in your analytics using Analyzer.