By Brian Harris
I recently ran a very successful A/B test scenario that I felt compelled to share with you. I call it successful because although the results were somewhat inconclusive, I found the experiment to be a stimulating exercise, and, as you read on, you’ll see how the unusual viral nature of social media skewed this A/B test.
[Related Article: 4 Reasons to Start A/B Testing Your Website Today!]
Background: How this all got started
It all started on August 14, in a discussion with Daniel Klotz of YDOP (really great folks, by the way!).
Why short URLs matter to me…
Let me be clear: as a marketer, not tagging and/or shortening URLs is a frightening prospect. In order to measure my audience (in this test, a mix of Twitter Followers and subject trawlers interested in my Tweet topics) engagement, I shorten the URLs I put in both my own and my clients’content, including social media posts. In doing so I can keep an eye on factors such as which subjects are yielding the most engagement.
For digital marketers, to stop measuring these clicks is to stop seeing their analytics. Only two of the clicks in the above graph were from my experiment, but I can see they were on the topics of steampunk goggles and renaissance hackers. Safe to say, as a results-driven marketer, I’ve since Tweeted again on Steampunk swag.
To stop shortening URLs is tantamount to losing this precious data; therefore the relevant conundrum is that (to most SEO marketing experts) engagement and conversion are easily as valuable as key performance indicators as this measurement data is to understanding performance. I won’t let go of my valuable analytics data unless I can prove more engagement.
Abstract: Shortened URL A/B test Approach
After my conversation with Daniel, I developed a more rigorous, regimented Tweeting pattern. I normally scan the day’s headlines for intriguing developments on a number of my favorite topics, and I also did a fair amount of more open-ended retweeting. If I post a link to a blog about a given topic (let’s say “robots”), I post another again shortly afterwards to a story of similar interest to the robot-loving community: one Tweet (A) containing a shortened URL, and the other (B) without.
I keep these paired Tweets about 10-20 minutes apart, putting them in the same relative time period to evenly match their audience as much as possible on a venue such as Twitter. I Tweet what I consider to be equally enticing headlines, and I use similar language patterns.I also alternate whether I post A or B first to keep the integrity of the results balanced.
An Anomaly? The Perez Hilton Effect -- Yes, THAT Perez Hilton
One anomalous event occurred in the middle of my testing. I retweeted a post from celebrity columnist, Perez Hilton. That post blew up virally: getting reverse Retweeted by Perez himself, as well as a few others, and incurring some unpleasant Tweets and Direct Messages from Twitter users who disliked the post. Perez’s mention also caused my number of followers to jump up by 10.
In terms of my study, it’s important to note that the retweet included a shortened URL (so technically, it was an ('A') Tweet). To try and balance it out, I also later retweeted a different celebrity column’s post, adding similar commentary and the same hashtag. For balance, I made sure the long-form of the link was Tweeted (making it a ('B') Tweet). As most Twitter users will attest, it’s hard to get lightning to strike the same place twice, and this was no different, my second post did not get any pickup.
Scoreboard: 'A' beat 'B', 4-0 (out of a possible 21-21)
By the end of my trial week, I had a trend: for better or worse, my short URL containing (A) Tweets got all of the engagement. Here is the data from Twitter analytics.
Any digital/social media marketing specialist will tell you that there are all sorts of issues with the way I conducted this experiment.
- I didn’t perform this experiment with 100% machine-like consistency. This has several facets
- Although the A/B pairs were on the same subject, each was to a different subtopic, so there were variations in headlines and subsequent attractiveness to each Tweet’s audience. While I tried to use matching tone, voice, tense, and perspective for each, subtle variations naturally occurred.
NOTE: As a rule, I won’t post duplicate content to any venue. So I would never recommend the practice of making identical A/BTweets with the only variant being the link type.
Tweets were only roughly 10-20 minutes apart. Some were further apart than that, and some were closer together.Technically speaking, unless they go out at the exact same time, their Twitter audience will be different, and even then it’s hard to say what random factors might still give one an advantage over another.
Twitter is an inhospitable content environment. A Tweet lasts 26 minutes on average. No two Tweets can be expected to have the same engagement, no matter how similar.
NOTE: Because my policy (and social media best practice) is to space Tweets out time wise to maximize an account’s reach, I would never recommend the practice of Tweeting two Tweets at the same time.
The Findings, In Depth:
During the test my Twitter account received:
42 A/B Tweets were posted during the experiment window
- 10 Mentions (this KPI rose during the experiment, in part because of the discussion with Daniel Klotz about this topic)
- 16 Follows
- 11 Unfollows
Tweet Engagement Tabulated:
- 4 ('A') Tweets received Engagement as measured by Twitter >Analytics (see table).
**All associated with shortened URLs (88% from Perez Hilton retweet)
- 12 Favorites**
- 11 Retweets**
- 2 Replies**
- 2 Clicks**
(NOTE: The only clicks I would be able to report on are from Short URLs; I am unaware of any way to measure clicks unless they were to my own content and in this experiment, all links pointed to external sites)
- If you’re interested in conducting a similar experiment, I recommend you do it using accounts with more of a following and in more than one venue. My conversation with Daniel was about Twitter, but this experiment broadened to span across Facebook, Twitter, Google+, and LinkedIn, as well as on a blog.
- A month (or two) of running the experiment would provide considerably more conclusive data. I plan to keep plugging away with my test to see if A and B balance out.
- I firmly believe Daniel’s experience was 100% valid, but in the week of my experiment I had the opposite results. For this reason, you’ll note that this blog post is full of links, and each and every one is shortened.
Let us know what you’ve experienced!
Have you done any A/B testing lately? What were the results? Anything else you would suggest to create a more balanced A/B Twitter test? Do you have any experience with improved engagement on unshortened links?
We’d love to hear from you! Gives us a shout out on Twitter or in the comments section.