A/B Testing Short URLs on Twitter, or The Time Perez Hilton Ate My A/B Test Homework

I recently ran a very successful A/B test scenario that I felt compelled to share with you. I call it successful because although the results were somewhat inconclusive, I found the experiment to be a stimulating exercise, and, as you read on, you’ll see how the unusual viral nature of social media skewed this A/B test.

[Related Article: 4 Reasons to Start A/B Testing Your Website Today!]  

Background: How this all got started

It all started on August 14, in a discussion with Daniel Klotz of YDOP (really great folks, by the way!).

Why short URLs matter to me…

Let me be clear: as a marketer, not tagging and/or shortening URLs is a frightening prospect. In order to measure my audience (in this test, a mix of Twitter Followers and subject trawlers interested in my Tweet topics) engagement, I shorten the URLs I put in both my own and my clients’content, including social media posts. In doing so I can keep an eye on factors such as which subjects are yielding the most engagement.

My bitly.com click statistic for the week of the A/B digital marketing experiment

For digital marketers, to stop measuring these clicks is to stop seeing their analytics. Only two of the clicks in the above graph were from my experiment, but I can see they were on the topics of steampunk goggles and renaissance hackers. Safe to say, as a results-driven marketer, I’ve since Tweeted again on Steampunk swag.

To stop shortening URLs is tantamount to losing this precious data; therefore the relevant conundrum is that (to most SEO marketing experts) engagement and conversion are easily as valuable as key performance indicators as this measurement data is to understanding performance. I won’t let go of my valuable analytics data unless I can prove more engagement.

Abstract: Shortened URL A/B test Approach

After my conversation with Daniel, I developed a more rigorous, regimented Tweeting pattern. I normally scan the day’s headlines for intriguing developments on a number of my favorite topics, and I also did a fair amount of more open-ended retweeting. If I post a link to a blog about a given topic (let’s say “robots”), I post another again shortly afterwards to a story of similar interest to the robot-loving community: one Tweet (A) containing a shortened URL, and the other (B) without. 

I keep these paired Tweets about 10-20 minutes apart, putting them in the same relative time period to evenly match their audience as much as possible on a venue such as Twitter. I Tweet what I consider to be equally enticing headlines, and I use similar language patterns.I also alternate whether I post A or B first to keep the integrity of the results balanced.

An Anomaly? The Perez Hilton Effect -- Yes, THAT Perez Hilton

One anomalous event occurred in the middle of my testing. I retweeted a post from celebrity columnist, Perez Hilton. That post blew up virally: getting reverse Retweeted by Perez himself, as well as a few others, and incurring some unpleasant Tweets and Direct Messages from Twitter users who disliked the post. Perez’s mention also caused my number of followers to jump up by 10. 

In terms of my study, it’s important to note that the retweet included a shortened URL (so technically, it was an ('A') Tweet). To try and balance it out, I also later retweeted a different celebrity column’s post, adding similar commentary and the same hashtag. For balance, I made sure the long-form of the link was Tweeted (making it a ('B') Tweet). As most Twitter users will attest, it’s hard to get lightning to strike the same place twice, and this was no different, my second post did not get any pickup.

Scoreboard: 'A' beat 'B', 4-0 (out of a possible 21-21)

By the end of my trial week, I had a trend: for better or worse, my short URL containing (A) Tweets got all of the engagement. Here is the data from Twitter analytics.

Tweet Data




A or B

4 things CIO's should ponder about #BigData - zd.net/152RshC via @ZDNet and @shaunconnolly (filling in for @andrewbrust)





Did I RT this yet? Just in case I didn't... “@PerezHilton: .@kanyewest Gets DENIED!!! perez.ly/16RDD1v pic.twitter.com/XxKcjD6s9w#BURN!!





Robot goes where police officers can’t - bit.ly/16xGU9O via @PostTrib #robots





RT @Boston_SciFi: Looks like #TomServo has his Sf39 ticket and will bring a friend, Richard Bartlett. fb.me/1ijwtvgSt #bostonscifi





Notable Disclaimers:

Any digital/social media marketing specialist will tell you that there are all sorts of issues with the way I conducted this experiment.

  1. I didn’t perform this experiment with 100% machine-like consistency. This has several facets
    • Although the A/B pairs were on the same subject, each was to a different subtopic, so there were variations in headlines and subsequent attractiveness to each Tweet’s audience. While I tried to use matching tone, voice, tense, and perspective for each, subtle variations naturally occurred.

    NOTE: As a rule, I won’t post duplicate content to any venue. So I would never recommend the practice of making identical A/BTweets with the only variant being the link type.

  2. Tweets were only roughly 10-20 minutes apart. Some were further apart than that, and some were closer together.Technically speaking, unless they go out at the exact same time, their Twitter audience will be different, and even then it’s hard to say what random factors might still give one an advantage over another.

  3. NOTE: Because my policy (and social media best practice) is to space Tweets out time wise to maximize an account’s reach, I would never recommend the practice of Tweeting two Tweets at the same time.

  4. Twitter is an inhospitable content environment. A Tweet lasts 26 minutes on average. No two Tweets can be expected to have the same engagement, no matter how similar.

The Findings, In Depth:

Twitter Account Engagement Analytics
During the test my Twitter account received:

  • 10 Mentions (this KPI rose during the experiment, in part because of the discussion with Daniel Klotz about this topic)
  • 16 Follows
  • 11 Unfollows
  • 42 A/B Tweets were posted during the experiment window
    • 4 ('A') Tweets received Engagement as measured by Twitter >Analytics (see table).

    Twitter Account Engagement Analytics Pie Chart
    Tweet Engagement Tabulated:
    • 12 Favorites**
    • 2 Replies**
    • 2 Clicks** 
    **All associated with shortened URLs (88% from Perez Hilton retweet)

    (NOTE: The only clicks I would be able to report on are from Short URLs; I am unaware of any way to measure clicks unless they were to my own content and in this experiment, all links pointed to external sites)

    • If you’re interested in conducting a similar experiment, I recommend you do it using accounts with more of a following and in more than one venue. My conversation with Daniel was about Twitter, but this experiment broadened to span across Facebook, Twitter, Google+, and LinkedIn, as well as on a blog.
    • A month (or two) of running the experiment would provide considerably more conclusive data. I plan to keep plugging away with my test to see if A and B balance out.
    • I firmly believe Daniel’s experience was 100% valid, but in the week of my experiment I had the opposite results. For this reason, you’ll note that this blog post is full of links, and each and every one is shortened.

    Let us know what you’ve experienced!

    Have you done any A/B testing lately? What were the results? Anything else you would suggest to create a more balanced A/B Twitter test? Do you have any experience with improved engagement on unshortened links? 

    We’d love to hear from you! Gives us a shout out on Twitter or in the comments section.

    2 Responses to "A/B Testing Short URLs on Twitter, or The Time Perez Hilton Ate My A/B Test Homework"

      • Brian Harris
      • September 28, 2013

      Daniel, agreed. I think in any venue, fact-based requires a greater sample base. I didn't exactly post 1,000 Tweets with and 1,000 without. To that end we may or may not be conducting this study on my team on a greater scale, across a broader set of venues with more trackable data. I can neither confirm or deny. ;-) I will say that personally, I only use short URLs in venues where I otherwise cabnot get data. Thank you however for the inspiration and the challenge!

      • Daniel Klotz
      • September 26, 2013

      Great post, Brian. I love the evidence-based approach you took to responding to my opinion-based tweet. At the very least, you've shown that this isn't an open-and-shut case and deserves further study. It's also cool that you got a RT from Perez Hilton! I'd be interested in a study where you have access to the sites' Analytics so that you can track and compare click-through rates on the links as another engagement metric.

    Add a Comment

    * Required

    Note: Your comment must be approved before it will be displayed on this page.

    Back to top

    Stay Connected

    • LinkedIn
    • RSS Feed

    Knowledge Center