Sentiment is a complex beast, even for humans to decode. How many times are you never really sure if somebody is being positive or negative about something? How many times do you have to use non-verbal cues like their body language or facial expression? Cultural and linguistic factors play a huge role in our ability to understand what is meant. And this is why it is a difficult process to automate.
This is why sentiment reporting for social media discussions is problematic – giving a single ‘positive’ or ‘negative’ rating to a comment risks missing the real nuance. Even at a Tweet-level, assessing 140 characters as positive or negative can be wrong as often as it is right.
There is another way of looking at sentiment. Not to look at the comment, but to identify known elements within the sentence and look at how they are discussed. For example, let’s consider the following Tweet:
As much as I absolutely adore BBC Question Time, Steve Coogan is making me hate this week’s show so much. He’s making me really angry #bbcqt
As we read this, we know that it is neither positive or negative. It expresses both things depending on the object being discussed:
- ‘BBC Question Time’ is clearly being described positively (‘I absolutely adore [it]’)
- ‘Steve Coogan’ is clearly negative (‘making me really angry’)
- ‘This week’s show’ is also clearly negative (‘[I] hate [it]’)
So by breaking down the Tweet into elements like this we get a much more nuanced view of sentiment. And probably a much more useful one. If I am analysing what people say about an episode of BBC Question Time, for example, I might be more interested in comparing how people talk on Twitter about the issues raised, or the guests on the panel than I am generically about the tone of discussions during the show. Looking at sentiment at this object-level is more insightful and more actionable.
So for the episode first broadcast on 27 September 2012, we conducted an experiment to explore sentiment. Not of the show or general discussions but specifically to investigate people’s sentiment towards the five guest panellists.
What we analysed
- Using DataSift, we recorded all Tweets including the hashtag #bbcqt during the time the show was on air. This was a total of 21,651 tweets.
- A random sample of 20% of these was then taken, giving us a total sample of 4,266 tweets.
- This sample of Tweets was analysed using Semantria – this identifies the things (they call them ‘entities’) discussed in the Tweet and then gives a positive or negative score based on the context in which that entity is discussed.
- We isolated entities that were the five guests on the show – using all possible spellings of the following names:
- We then took a mean score for how positive or negative the context is in which each of these entities is discussed.
What we found
- The most discussed guest panellist this week was writer and comedian Steve Coogan who was explicitly mentioned in almost 7% of all Tweets about #bbcqt. But he was also discussed most negatively.
- The most positively discussed panellist was Labour MP Harriet Harman. She was also the only panellist who had a positive sentiment score overall.
- Liberal Democrat MP and Minister Danny Alexander was the second most negatively discussed panellist, with Jacob Rees-Mogg and then Kirstie Allsopp above him.
What we can learn about sentiment analysis
What can we learn from this? As with all research it is important to understand the biases of our sample – it could be that the audience who view the programme and discuss it on Twitter are more left-wing and so more sympathetic to Harman’s point of view. It may be that the discussions about Steve Coogan were coordinated by a small group of individuals who had an agenda against him and so biased his score down. And it may be that the relatively small instance of mentions of Kirstie Allsopp makes her score less reliable.
All of these are areas of potential bias that should be explored. But analysing sentiment at the object level like this gives us a much more nuanced understanding of how people were discussing BBC Question Time last night. And it allows us to have much more valuable discussions than just knowing that Tweets during the show were positive or negative.
Sentiment is a complex beast, as are the humans that are expressing it. To inform a real discussion and to have a real understanding of what may be happening in discussions online we need to stop thinking in terms of Tweets and posts and comments, and to start disaggregating the individual objects discussed and explore those instead.
The post A sentiment experiment: This week’s #BBCQT panellists on Twitter appeared first on FreshMinds.