• NZ 03-441-3223

Measuring Social Media Success: 3 NY Wine Stores and Social Mention

In the previous post I looked at three NY wine stores to show social media monitoring and measurement in practice. They were Chambers Street Wines, Frankly Wines, and New York Vintners all near Chambers Street in Manhattan.


In terms of social media I found them to be quiet, promising, and over-achieving respectively. I gave them a few suggestions based on what their local competitors were doing and their own strengths.

In this post I’d like to go into the tool I used, Social Mention, in a little more detail with the intention of seeing if it can robustly measure social media success.

Here’s a table I’ve compiled from my own general internet marketing knowledge combined with various social media experts. For more about this see A Wine Retailer’s Perspective on Social Media: why bother?.

Table: Social Media Objectives (click table to enlarge)
Social Media Objectives Table - a Wine Retailer's Perspective

Can SocialMention help?

Can it help measure the success of your blog, twitter, forum posts, facebook and external commenting strategies? Can it act as a robust measurement tool of social media strategy effectiveness?

Let’s go through the objectives I outline in the table.

Objective: Awareness

In other words, if no one knows you then they won’t buy from you.

Metric: Share of Voice

Share of Voice = Brand Mentions / Total Mentions of Brand and Competitors
Social Mention: Strength
– your search phrase mentions divided by a popular search phrase, I think ;).

Here’s how Social Mention explains it,

  • Strength is the likelihood that your brand is being discussed in social media. A very simple calculation is used: phrase mentions within the last 24 hours divided by total possible mentions.

Which is not the best answer.

However I’ve dug through the help forum and I believe what they do is look at the last 24 (or 48?) hours, find a high mentioned phrase and use that as an index denominator. So if Obama is the phrase that is mentioned a lot in the last 24 hours, then Strength is: your phrase mentions / Obama mentions.

For the purposes of my NY wine shop comparison I ended up doing three separate searches and compiling the results into a spreadsheet (deleting the poor video search results) and making my own Share of Voice report in the spreadsheet below (near bottom).

Which I’m happy with. But I’m not convinced that the Social Mention’s Strength is a terribly good way of measuring share of voice. Perhaps in the sense of cutting through general social media noise, but not versus the much more important direct competitors’ messages.

So, social mention can help with measuring Share of Voice but through compiling the results yourself not by using the Strength measure.

Constructive Feedback to Social Mention: please have an option where there is a number of search phrase fields: one for your brand, a few others for direct competitors. Then use the results of the competitors to create the Strength index. Also you really need boolean operators to make sure your users can cover all name variants.

Objective: Information Gathering

In other words, although people may know you they don’t consider you as a worthwhile option, and are not gathering information about you.

Metric 1: Active Advocates

Active Advocates = (# of Active Advocates in past 30 days) / Total Advocates
Social Mention: see comment below

Metric 2: Advocate Influence

Advocate Influence = Unique Advocate’s Influence / Total Advocate Influence
Social Mention: see comment below

Social Mention identifies Top Users. For example a Top User for Chambers Street Wines is Lyle Fass who has Chambers Street Wines listed in his Blogroll which appears in the sidebar of every one of Lyle’s posts. There are 79 other blogs listed including “The Onion” and “Modern Warfare – the Dating Game”. As a divorcee the latter is interesting but I, er, digress…

Now I’m not saying that he’s not an advocate for those blogs on his blogroll. Rather he shouldn’t be listed as an advocate based purely on a blogroll listing. In fact as I started to dig into his blog I saw that he was actually a keen blogger about Chambers, but those posts weren’t the ones listed on Social Mention.

So, sorry Social Mention, can’t use you for the Advocacy measurement.

Constructive Feedback to Social Mention: Hmmmn, how could you solve this one? I’d perhaps include only those links from the blog post content section not sidebars. I’d also weight various links, so a post would be worth more than a tweet from the brand.

Metric 3: Sentiment Ratio

Love...on the beach

Sentiment = Love

Sentiment Ratio = (Positive : Neutral : Negative Brand Mentions) / All Brand Mentions
Social Mention: Sentiment (Positive: Negative)
– on the face of it Social Mention does this well but as you start looking at examples you see it struggles like most other tools with this tricky area of semantic language analysis.

Social Mention says the following:

  • Sentiment is the ratio of mentions that are generally positive to those that are generally negative.
  • The sentiment algorithm’s success rate is somewhere between 60%-80%. That means for every 10 mentions it will only mark sentiment correctly for 8, and 2 will be inaccurate. This is within a generally acceptable margin or error.
  • Calculating sentiment accurately is an extremely difficult process and arguably achieving 100% accuracy is impossible.
  • Sentiment is based on a list of keywords and phrases.

Cast your eye over the mentions. It’s not very accurate huh? Probably best to ignore the Sentiment ratio and compile it yourself. If you do this regularly it will only be a 5 minute job.

There is a tool called ConsumerBase that I’m looking into that may have the best algorithm in the business. However they may be more for internet research than practical social monitoring for small retailers.

Constructive Feedback to Social Mention: Nice try but no cigar. Is there some other service you could use here that has better linguistic analysis?

Objective: Engagement

In other words do you interact with your audience? Are you forming direct connections in the same way you would, however fleetingly, with a customer in-store?

Metric 1: Audience Engagement (also Loyalty Objective)

Audience Engagement = (Comments + Shares + Trackbacks) / Total Views
Social Mention: Passion
– passion is calculated based on the number of authors who mention your search terms more then once

Here is what Social Mention says,

  • Passion is a measure of the likelihood that individuals talking about your brand will do so repeatedly. For example, if you have a small group of very passionate advocates who talk about your products or brand all the time you will have a higher Passion score.
  • Conversely if every mention is written by a different author you will have a lower score.

I’ve been critical about Top Users above. Mainly because it counts blogroll mentions. Here’s another rant, ahh, I mean example.

What most social media monitoring tools do is count each mention as equally worthwhile. Yet a customer service tweet from the store is worth nothing beside a post from an authority newspaper blog.

I saw this first when using another social media tool called SM2. I had been looking at the Alphabet Wine Co wine store in the East Village , owned by Keith Beavers. He was taking an interesting approach to wine stores and seemed very hip with social media, as well as being … very hip.

The problem was that when I looked at numbers alone, he outshone the other stores because he was a prolific tweeter. And when Eric Azimov, a wine writer, gave it a favorable review on the NY Times Diners Journal blog – a super influential authoritative source – the tweets drowned out this single blog post. Yet this blog mention should have been a tsunami in the social media sea compared to Keith’s tweet ripples.

So my point is using unweighted numbers are not an accurate way to track social media. These social monitoring tools need to weight each mention for relevant authority a bit like google does with its search algorithm.

And that can service as my constructive advice to Social Mention as well. 🙂

Metric 2: Conversation Reach

Liverpool Station Crowd Blur
Conversation Reach = Total People Participating / Total Audience Exposure
Social Mention: Reach
– the number of unique authors referencing your brand divided by the total number of mentions.

They also called it the range of influence. The problem is I don’t trust the Top Users as mentioned above.

Constructive Feedback to Social Mention: see my Advocate Influence suggestions above. There is a need to identify true advocates not just blogroll mentions. Perhaps with weighting each type of mention to give a weighted score.

Metric 3: Topic Trends

Topic Trends = # of Specific Topic Mentions / All Topic Mentions
Social Mention: Strength
– big tick here for Strength, that will work and show how popular you are versus others.

A couple more issues

Accurate Search. For my search terms Google video search was inaccurate. So ignore this part of the tool (I’ve reported the problem to Social Mention, and am waiting on a response).

Advanced Search. Boolean operators are needed.

I would prefer to be searching for something like “Chambers St Wines OR Chambers Street Wines OR Chamber St Wines OR Chamber Street Wines”. Or even better (Chambers OR Chamber) AND (St OR Street) AND (Wine OR Wines)”.

Instead I’m just searching for “Chambers Street Wines” and am probably missing out a few.

In Summary

Overall Social Mention does a great job for a small business, it allows you to track what is being said about your store and monitor what competitors are doing. All for free. But it does have some significant accuracy issues and doesn’t provide all the social media measurements I’m after.

So what’s your thoughts? Have I been too mean or is the criticism fair?

Photos courtesy of: Love…on the beach, Liverpool Street station crowd blur

Speak Your Mind