A few months ago I happened to see a short social media insight report, written by a large, highly respected global research agency, for one of the world’s most iconic brands. It was very brief (5 slides in total) and formed part of a wider research report.
I was embarrassed for the vendor (not my own company, I hasten to add!). In those five slides were several claims so patently wrong that you wonder if anyone had their head screwed on when the report was written. They started by claiming that 99.9% of comments made on Facebook originated in the US – and that global mentions had a very heavy bias towards America as well.
They went on to paste in some automated sentiment charts which claimed that in some markets, social media reaction to the client’s highly entertaining, engaging promotional campaign was >97% neutral.
They also claimed that a sudden spike in online mentions of this major, engaging, global consumer campaign was due to coverage in a minor B2B magazine discussing a particular aspect of the production.
All of this – along with some other rather spurious claims – in five slides, lest we forget.
Let’s forget about the actual numbers for a minute. What concerns me is that the exec who wrote the report clearly never bothered to think about what the metrics meant – or to run a simple common sense test. Nor did the person who signed off the report. (It doesn’t reflect well on the client, either; did they not think to push back and ask what these numbers meant?) By all means report the numbers in good faith as provided by the tool you are using…but for goodness’ sake provide a footnote or caveat explaining the limitations. If reported “as fact”, anyone with an ounce of sense can rebut your findings.
Some basic understanding of how social media monitoring tools work can help explain those anomalies. These tools do their best with location detection – but it’s complex and far from easy to get right, and also platform specific. Facebook barely give away any metadata – so in most cases monitoring tools simply pick up the fact that Facebook.com is registered in the US and run with that. Similarly, automated sentiment tools tend to dump data in the “neutral” bucket if they aren’t sure – which depending on the dataset and language can often mean that almost everything is marked up as being neutral. As for the claim about the B2B magazine…I can’t explain that without seeing the raw data, but I’d imagine it’s due to duplicate mentions in the data.
I cite this specific example because I was frankly appalled at what a shoddy job this highly respected agency had done. But it’s representative of an endemic problem with poor-quality social media insights and monitoring – rubbish being peddled by technology suppliers and agencies is being met with client-side ignorance, resulting in an acceptance of poor findings…until somebody more senior does a review, realises the findings from social media are weak and/or unreliable, and blames the approach in general rather than specific failings. All this leads to a widespread mistrust in social media listening/insights. The damage doesn’t need to be done; it does need a little common sense, a willingness to go further than merely pasting charts directly from a tool without some sort of sense checking and interrogation of the data where appropriate, and some basic caveating and management of expectations. Most anomalies can be explained.
Social media research is a crowded space, and competes with many other emerging techniques for a share of limited client budgets. It is incumbent on all suppliers to push for better standards – as otherwise the mistrust can only grow and buyers will take their money elsewhere.
As we approach the final furlong of the race for the Scottish Independence referendum and rapidly approach another General Election, much excitable talk bubbles up once again about using social media as an election predictor; with the current fashion for presidential-style election debates, those are under the social media analysis spotlight too, with Twitter and other platforms providing a source of instant feedback and soundbites - cheaply or for free. Media organisations, research companies, political parties and casual observers alike all feast on instant statistics about who has "won". Needless to say, live debates provide a snapshot of how social media can give large-scale instant feedback - something which tickles the fancy of insight departments in companies and organisations the world over. Last night's EU debate on LBC between Nigel Farage and Nick Clegg was a good canvas to show how there are significant challenges to such an approach. To demonstrate why, I set up a quick search for the hashtags #NickvNigel and #LBCdebate, using social media monitoring tool Brandwatch. Incidentally, this isn't a tirade against such tools, which do exactly what they're supposed to. Instead, it's a call to arms: to make this data meaningful, we need to think very carefully about the context of such data, to clean it appropriately, and to treat is with extreme caution. If we take necessary steps, which may involve cutting out substantial proportions of the data, we may be able to get meaningful results.
The Blurrt "worm"
The LBC website has a "worm", courtesy of Blurrt. Sadly at time of writing the LBC website was creaking and the worm wasn't visible at all during the debate itself. All that was visible was the phrase "The requested URL /graphs/sentiment/ was not found on this server." The bolded word leaves me sad, but not as sad as the "how it works" page, which gives no information whatsoever on the methodology and a lot of explanation of some basic sampling theory - dressed up in such a way as to make it look intimidating to a non-technical audience whilst still explaining nothing useful. There is certainly a place for real-time analysis (although as Francesco D'Orazio points out succinctly, "If you can’t make decisions in real time there is no point in using real-time intelligence"); that real-time analysis must inevitably depend largely (or solely) on technology. As an advertisement for robust social media analysis, however, this is flawed, flawed, flawed.
There are several challenges which we need to consider.
1. Using hashtags as search terms
As this was a casual exercise, I opted for simplicity in my search term, opting initially for #NickvNigel (simply because this was the one appearing on my own Twitter feed) and later adding #LBCdebate, which I only spotted once it was mentioned by Nick Ferrari 10 minutes into the debate itself - a good thing I did, as #LBCdebate turned out to be the dominant hashtag:
This brings up one potential issue - retrospective data, which may not always be complete depending on how it's coming from Twitter. But there's a more fundamental problem. Almost by definition, the use of a hashtag implies prior knowledge of its existence, and generally also implies an affinity for the topic, and possibly good connections with others close to the topic. The casual LBC listener stumbling across the debate who chose to comment - very likely the unpartisan "floating voter" who we are so anxious to identify - will be unlikely to be found here. There are parallels in commercial social media research, too; do real people use hashtags like #danceponydance, or do they just talk about "the T-mobile ad"? (Hint: that's actually not a good example, as it's a rare occurrence of a campaign that has really taken off in social media. Much to my advertising research colleagues' frustration, not to mention that of my clients, the reality is that most campaigns barely get talked about at all.) Should we go with the easy option, or try to look at all tweets from the period referring to Clegg or Farage? Had I done the latter, the results might have been very different.
2. Coding: far from trivial
I dived in and manually coded 199 tweets. Simple, right? Not at all. There are myriad ways of doing this. This was a quick-and-dirty exercise on my part, but it's worth jotting down some of my assumptions, because even a quick-and-dirty bit of coding can rapidly prove a head-scratcher. I'm not claiming this is the "right" way to go about things! On the contrary, there are probably approaches which are far better, and some of my assumptions are probably way off the mark. For example, I could have focussed purely on tweets which made reference to the debate performance itself ("Farage is winning", "Clegg sounds nervous", etc).
I started by taking a sample of tweets using either hashtag, between 1900 (the start of the debate) and 2100 (an hour after the finish). The time period is arbitrary. My code frame was very simple: "Clegg", "Farage" or "neither". Broadly speaking, I defined "Clegg" as any tweet saying either something good about Clegg or something bad about Farage, and "Farage" vice versa; "neither" was any comment which gave nothing away. Any retweet of an official party account I automatically set to being "for" that party (mercifully both Labour and Tory HQ seemed to be very quiet); retweets of mainstream news accounts, without added comment, I set to "neither" unless the tweet reported something obviously critical. This approach was pretty self-explanatory to begin with, but there were snags aplenty.
This tweet is clearly making a political point, but for which side?
How many of the 2 1/2 million brits abroad work for peanuts #LBCdebate
— nick chapman (@nickchapman1963) March 26, 2014
How about this?
Funny enough Nigel, we're Scottish and we believe the best people to govern Scotland are Scottish. #indyref#LBCdebate
— Keith R (@TheHornyHaggis) March 26, 2014
3. Are opinions representative of Twitter? Of the wider population? Even of the tweeters talking about the issue?
Coding social media verbatim is tricky at the best of times and whether a manual, automated or machine-learning approach is taken, clearly needs a lot of thought. However, even if we assume an optimal coding strategy, there's a deeper-seated problem, and this comes back to the question which old-school market researchers always ask about social media data: But is it representative?
When asked that question, I generally fall back on a standard response: "Probably not...but does it matter?" There are so many unknowns, but survey respondents aren't exactly representative either ("yes, of course I'll spend 45 minutes for little or no reward answering questions about my mortgage provider")
The problem is not a question of demographic representivity, but more "to what extent do the views expressed on tweets represent the views on Twitter?" The first and most obvious point is that people only tweet about stuff they care about. Hence we'll have to stick with surveys for our mortgage provider research. Do the tweets represent the underlying opinions? Probably not - it's only the things that delight/outrage people the most that actually get posted. People don't necessarily offer up unprompted opinions unless they feel the need to broadcast them.
But studying political tweets is even more problematic.
4. Activists dominate proceedings
Of the 198 tweets I analysed, 153 gave some sort of opinion one way or another. I looked at the profiles of these 153 tweeters to see if I could find anything out about them. A Twitter profile gives you 160 characters to define yourself. After going through a few, it seemed to me that they could be divided into four categories:
Activist
Politician
Journalist
Other
I decided to code anyone as an "activist" whose profile showed an obvious leaning towards a particular political party or ideology. My reasoning was that anyone who uses up some or all of their 160 character bio to state their political leanings would be likely to be pretty dyed-in-the-wool. Some were a grey area: there were plenty who were self-described as "interested in politics" who I coded as "other", while anyone who said things like "socially liberal" or "Europhile" I placed in the "activist" bucket. "Politician" means anyone whose bio states that they are an MP, MEP, Councillor and so on; prospective candidates were problematic, although anyone who was borderline would end up in the "activist" category anyhow. "Journalists" were mostly self explanatory.
The breakdown of "opinionated" tweeters is as follows:
No less than 36% of the tweets were written (or retweeted) by tweeters were self-described as being politically polarised*, with another 3% being journalists.
Does that skew our sample? Of course it does - massively. There is a substantial minority of politically savvy, active cyberwarriors sticking up for their man. It's true of the #IndyRef debate as well. Never mind the demographic breakdown of Twitter - it's the propensity of people to tweet about what matters to them that is more important. The sample is biased away from casual listeners and floating voters, and towards a polarised, politically charged audience. Shortly before the debate began, Lib Dem Digital Communications lead Bess Mayhew sent out an email to supporters which said "LBC are running a “Twitter worm” which tracks who is winning the twitter battle. Nick needs your help to come out on top, so lets get tweeting!" In a world increasingly judged in this way, groups will always look for ways to game the system.
There's one further consideration to take into account which I've also not dealt with here - multiple tweets by the same person. As an example, Peter Chalinar (@TaleahPrince) tweeted nearly 200 times yesterday about the debate (mostly retweets of others) - mostly strongly in favour of Farage, whilst Lib Dem MEP Rebecca Taylor notched up nearly 150 tweets. While neither of them turned up in my sample of 198, there were several people whose tweets appeared twice. De-duplicating authors is another step in social media analysis which might want to be taken, depending on the objectives.
* Of course it could be argued that anyone tuning into an hour-long programme on a political issue that isn't even considered to be in the top 10 issues facing Britain today according to the Ipsos MORI issues index would be likely to be a bit of a politics nut anyhow.
So what about the results?
What about them? Hopefully I've demonstrated that without some careful methodological thought, the results are pretty meaningless, and my own system was not thought through in detail - I simply wanted to point out some issues. For the record, the Blurrt worm seems to have done reasonably well at picking up sentiment expressed towards particular issues as the debate went on, and called it overall in favour of Farage, mirroring the snap Yougov poll taken immediately after the debate. My own results were rather different:
Topline figures
Clegg 44%
Farage 33%
Neither 23%
Ignoring the "neithers", this boils down to
Clegg 58%
Farage 42%
What about if we exclude politicians and activists from our sample? This reduces the sample of opinionated views from unpolarised people down to a rather meagre 94 (less than half of our original sample size)
As it turns out, and somewhat to my surprise, there was actually very little effect, with the results now amended to
Clegg 55%
Farage 42%
Perhaps implying that the cyberwhipping on both sides was equally effective.
How do I explain the discrepancy between my own results and the worm (and indeed the poll)? It's hard to say. There were a few hashtag "hijacks" - people talking about issues which came up in the debate which were not directly related to the EU; notable examples included Scottish independence and gay marriage, where there were several tweets critical of Farage - by my own rules I coded these as "wins" for Clegg but perhaps these could have been excluded from the sample or coded as "neither". There were several tweets reporting the Yougov poll result which I categorised as neutral as they were merely reporting the mainstream media outlet - I could have coded these as being for Farage, which would have boosted his score a few points. Other than that, there are so many variables that I find it difficult to pinpoint.
Perhaps Sky's primitive method was best?
Sky News opted for a simple approach - they posted a couple of tweets, one in favour of Farage, one for Clegg, and asked for retweets to endorse. This direct approach - closer to a traditional market research technique - might work better in such circumstances, and indeed this was in line with the poll (and the worm):
Where does this leave political social media analysis?
Overall, then, I believe there are multiple issues with political social media samples, although with appropriately thoughtful handling I do think these issues can be overcome. There is certainly a place for fast-turnaround or real-time analysis which presents significant challenges, although once again these are not insurmountable. Watch out for the next debate on the BBC, for which no doubt there will be more furious analysis and debate.
I've written an article for Brand Republic about some of the recent work that we've done at Ipsos MORI alongside Brandwatch. It focusses on the importance of setting norms and benchmarks when working with quantitative social listening research data.
*** Update: I have also written a piece on similar themes for Research magazine. A shortened version is in the print magazine, or you can view the complete article here. ***
Interesting reading and plenty more food for thought. My thought palate is salivating. Have they got it right? My printout is covered in pink highlighter markings; will post some reactions when they come together in my head a bit more coherently. There is no simple quick-fix answer here.
Debate on the ethics of social media research has flared up in recent months with some eminent names taking diametrically opposed points of view.
A good starting point is the lively debate surrounding Brian Tarran's excellent post on Research Live. There have also been a couple of good posts on the Digital MR blog recently which address the pertinent
issues head on. They are clearly worried that new guidelines will restrict their ability to do their job effectively, and leave them vulnerable to providers from non-traditional research backgrounds who may not be subject to the straitjacket of a code of conduct, and therefore be able to provide research solutions quicker and more cheaply, which is definitely the trend. Their worries are certainly valid.
My own take on it is this. The principle of informed consent should still be the starting point. There are a lot of people making loud noises about social media research being "different" from traditional market research. This is true...up to an extent. But my worry is that the motivations for wanting to water down the restrictions on data usage are business ones rather than ethical ones. "If we restrict ourselves then there are non-MR companies out there who will move into our space" simply does not wash as an excuse for lowering standards.
Ray Poynter has made a series of thoughtful posts on the issue and neatly breaks down the issues. In August he wrote:
"The benefits of traditional market research ethics were that they allowed some exemptions to laws (e.g. data protections laws, laws about multiple contacts, laws about phoning people who were on ‘no call’ lists), increased public trust, and allowed market research to get close to a scientific model – for example to use concepts such as random probability sampling and statistical significance. Complying with codes of ethics incurred extra costs, but they also brought commercial benefits. The ‘proper’ market research companies could do things the non-research companies could not - so there was a commercial argument in favour of self-regulation, codes of conduct, and professional conduct bodies."
Why can't this continue? Annie Pettit reported that Jillian Williams from the Highways Agency, said that anonymity is important to clients as they will take the flak rather than the research industry. Ray then appears to contradict himself slightly by saying "If market research companies abide by the old ethics, in particular anonymity and informed consent, they will not be able to compete for business in most areas where market research is growing. This is because there will be no commercial benefits that will accrue to sticking to rules and ideas that nobody else does." Surely the majority of clients, if they are looking for a genuine market research study, will want to stay firmly within the "rules" whatever they might be. There was an almighty stink when Nielsen Buzzmetrics were found to have scraped a healthcare forum that was ostensibly private. I actually had some sympathy for them - they were exploring new ways of collecting data, which in itself is quite legitimate - they'd just made a mistake in the execution and hadn't thought hard enough about the wider implications. They took the rap rather than the end client that time, but no client wants to be caught up in a grubby web scraping scandal.
Anonymity is a sociological issue that's very a la mode - there's an interesting post on the ever-excellent Face blog about current trends for real names versus pseudonyms; meanwhile debate rages over Google+'s insistence on real names. What about agencies using monitoring services such as Sysomos or Radian6 or in-house tools? These generally provide the capability to drill down to individual posts, tweets and so on, which can be sent directly to the end client. Perhaps some sort of deals could be set up with the dashboard providers whereby data is automatically anonymised in certain situations. And what about client-side monitoring, which may be informal reputation management/PR or a more in-depth research project. We must be careful not to set guidelines that are restrictive merely because the technology is so good. The principles should apply no matter what fancy new algorithms (buzzword...ugh) are created.
There is also a difference between qualitative and quantitative data. There is an enormous gap between a qualitative study which drills down to individual tweets, forum posts or Facebook status updates and sends them - warts, personal details and all - to the end client, and a large-scale overview of aggregated sentiment-analysed anonymised data which may say nothing more than "there has been a 17% uplift in sentiment from Yorkshire women on Twitter towards the value for money of Fabreze in the last 6 months" or whatever. (What is Fabreze, by the way? It's something which I know my girlfriend spends money on and is almost certainly totally unneccessary - beyond that I haven't got a clue).
The next question over anonymity surrounds platforms. Bloggers, for example, are posting opinions which they want to be heard; furthermore, bloggers generally have an easy choice whether to remain anonymous or not. Many do, others are quite happy to be identifiable. In my book they're about as close as you can get to "fair game". Forums are somewhat similar. At the other end of the scale, you have Facebook; I would hazard a guess that many people whose profiles are set to public are actually unaware of the fact, and have simply been confused by Facebook's ever-changing T&Cs, not to mention their tendency to play fast and loose with privacy. Add the fact that Facebook profiles are usually in real names - and easily identifiable with photos and so on - and this adds up to an ugly mixture of possibly unwanted intrusion combined with ignorance of the fact. A far cry from the "informed consent" principle if researchers start harvesting their data for business purposes.
Then there are idiosyncracies of the social networks. Should there be a difference between the attitude to privacy of someone saying "I wish Nature valley cereal bars were sweeter" and "I wish @NatureValleyUK cereal bars were sweeter"? Is the second option crying out for attention - by researchers?
"Finally a specific minor detail which is most important from a DigitalMR perspective is this: when using quotes in MR reports, we (MR agencies) should not be asked to mask the handle/meta data of a person who posted a comment on a public website – if that website states that posted comments can be viewed by anyone."
I think this depends on what is being done with the data. If the data is quantitative then I believe it should be anonymised - at least before it reaches the end client who needs to make the business decisions that follow the research. For qualitative data perhaps another set of rules should apply;
Ultimately I suppose the question needs to be asked "what are the purposes of these ethical codes anyhow?" I've even heard people criticising the Data Protection Act itself - this smacks of tobacco companies criticising smoking regulations. The Data Protection Act was drafted to bring UK law into line with EU privacy directives and the European Convention on Human Rights. These are fundamental directives; they are universal. They provide for people to be able to live their day-to-day lives in a normal way. They enshrine into statute principles of common decency which are inherently part of human nature. Thanks to UK implementation such as the Data Protection Act and Human Rights Act, we are able to do this. The Code of Conduct must use these principles of common decency as its starting point, and leave "but other people are doing it" wheedles to the minor details. The ever-excellent Annie Pettit speculated the other day that a lack of grounding in the "old" ethical MR principles has led to a slackening of attitudes towards privacy. This sounds very plausible, but a lot of it seems just to be a frustration with, or fear of, not being able to work efficiently, particularly if there is "competition" out there coming from a different background who will cheerfully sweep up the work without having to worry about pesky obstacles like common decency.
All this still doesn't quite square with the fact that this social media data is publicly available, sitting there for the world to see, and common sense would seem to dictate that it would be daft to deliberately close our ears to mountains of conversations that are taking place in the public domain. It is undeniable that it is impractical to contact thousands of people individually and ask them whether the sentiment expressed in their Facebook status yesterday may be used for market research purposes. It is also unlikely that many people will feel there's much of an intrusion of privacy from Jack Daniel's picking up on the fact that someone has publicly moaned about it being too expensive, and using that to influence their pricing stategy. But it must be done in such a way as to minimise disruption to people's lives and not fuel speculation that businesses are running slipshod over personal data. Is there a difference between "private" and "personal"? I think so, and perhaps it's a definition that needs to be made explicitly. In general we may need to re-think the "informed" concept and define in what situations "informed" means "explicitly told personally".
I think there are direct parallels between the issues faced by social media researchers, and the police and the Regulation of Investigatory Powers Act (RIPA): for intrusive "directed surveillance" authority from RIPA is required - because that involves targeted "stalking" if you like, of a particular person. You also need RIPA authority for similar work online. But there's no requirement for a RIPA for simple day-to-day casual monitoring. If an officer in plain clothes spots someone doing something he regards as suspicious, there's no need for a court authorisation to discreetly follow that person down the road to find out what he's up to.
As Steve Cooke of Digital MR points out, it is true that social media listening is different to other forms of social media research such as communities. But offline ethnography is subject to pretty strict controls and to informed consent principles. Social media conversations - even "person to person" conversations such as @messaging on Twitter - may be in the public domain, but any offline conversation in public is monitorable if you have a big enough pair of ears. Social media listeners must be careful that the sensitivity of their "ears" doesn't mean they abuse their power. Perhaps there is a case for abandoning long-standing principles - but it shouldn't be merely for convenience purposes.
I conducted an informal five-minute focus group on Friday afternoon with colleagues. The topic of discussion was: imagine you go to sit upstairs on the bus. Every double seat has one person sitting in it, so you have the choice of the whole top deck, but you'll have to sit next to someone. Who do you sit next to? Instant reactions included "completely random", "somewhere near the front", "over the back wheel", "the hottest girl" and "not next to anybody fat or smelly". All fairly predictable stuff.
When pushed a little further, people started to realise the subtleties of the decision. Did they sit on the left or on the right? Next to men or women? What if there were several places that all looked as good as each other?
A couple of people stubbornly refused to believe their choice was random at first, but had to admit that they had to make a conscious choice to actually do the action of sitting down. The discussion was fascinating, with several key areas coming into play.
The most cited motivations for choice were "someone ordinary", alongside "convenience". Ordinary meant not fat or smelly, not taking up the whole seat with bags, and no loud music. Even the blokes who said "I go straight for the most attractive girl" managed to elaborate: when I asked "wouldn't it look a bit obvious going straight for the hottest girl when you've got the whole bus to choose from?" everyone agreed, amending their choice to "across the aisle", "the most obtainable girl" (!) or "in which case I'll go for the second most attractive". Feel privileged, ladies.
Then a thought occurred to me. With as deadpan a tone as I could muster, I asked if they tended to sit next to white or black people. Everyone initially insisted this didn't cross their minds, and I was careful not to push anyone to say anything indiscreet...but then one of the (white) girls admitted she probably sat next to white people more of the time, which led to one or two other people mumbling something similar.
When I asked why this was, she came up with a fascinating piece of insight: "I think I try to sit next to people who are similar to me." I was delighted and leapt on this; it tallied with the fact that she had already said she tended to sit next to women (and perhaps gave a little more insight into what people meant by ordinary). Another girl separately said that she would sit next to people "about my age or a bit older." This all tied in neatly with Thomas Schelling's theories about racial segregation: a very slight preference to be amongst people like ourselves can result in near-complete racial segregation which can sunder a whole city. I wonder what Rosa Parks would make of a theory that perhaps segregation on buses could be more naturally occurring than one might think?
According to Robert Cialdini, we subconsciously lean towards choices that remind us of ourselves (although I reserve the right to remain sceptical about nominative determinism, as explained by Wired this month). If this is true, what questions does this answer for people wanting to affect decisions? Is this why Dove's campaign for real beauty struck a chord because people saw themselves in the ads? Or is that going too far?
So there seemed to be rational motivations (I could suggest loads of others - from window/aisle to proximity to the emergency exit to wanting to pretend-drive the bus from the front seat) and less rational ones. But what else could be affecting our decisions? For example, let's amend the parameters slightly - this time a couple of people get on in front of you. Perhaps their choices affect yours (the Herd effect). If you are with friends, how would that affect your decision (and how would your presence affect theirs?) What about if you were on the phone and therefore slightly distracted? How does experience affect your decision - would someone who takes the bus every day make a different choice to someone who has never taken a bus in their life?
But here's a tester: would people choose the same seat again, given the same initial conditions? Asking people they thought "no" but then these were the same people who thought their choice was random in the first place. But do they have a point? Stochastic choice models would suggest that yes, there will be a "random" element involved to a certain extent. Thinking about this, my gut reaction was to think "well of course that makes sense, with 20 seats to choose from it's hardly likely that you'd choose the same one each time" - but surely the fact that the probability of choosing the same seat ten times running is a function of the number of possible outcomes suggests that there is, in fact, a chance that with only two choices you wouldn't go for the same one each time. In general, though, it seems that stochastic models for decision making are generally preferred among academics. Comparable to quantum mechanics, they imply that any input-output model for a decision can only give a probability that a certain decision will be taken, given a certain set of initial conditions.
You could extend this psychology of seats on buses. On an emptying bus, at what point does it become appropriate to move away from the person next to you into an empty seat? And at what point do you become irritated if your new-found companion insists on staying put, rather than moving into a free double seat? In addition, if you're sitting on your own, presumably you breathe a sigh of relief when people decide to choose someone else (according to what you've learned today, you need to look as little like that person as possible!) but do you get paranoid if you are the last person to be chosen? I certainly do! All this is very similar, of course, to the etiquette of choosing a urinal - every self-respecting bloke should know this, but if not, then have a go at this game...
We make odd choices and have odd motivations depending on our circumstances. For some reason I'm reminded of a time, years and years ago, when I was in the local organic/health food shop with my dad - you know, the sort of place with business cards advertising reiki and aura therapy. In the vegetable section there were two boxes next to each other: Carrots (Dirty) and Carrots (Washed). The Carrots (Dirty), which were covered in soil, were more expensive than Carrots (Washed)! Another example of creative pricing!
The bus discussion evolved somewhat with my girlfriend in the pub (the Black Lion on Kilburn High Road, which is a cracking place). A couple of days previously Rachel had picked up some beers in the supermarket. We did our best to deconstruct the process.
She started out by looking to see if they had any Peroni, because she knew I like Peroni (bless her). They didn't have any in multipacks, so she looked at what was on special offer. There were a few options. She only looked at bottles - not cans. Why? Not sure, she was in a rush, and tired. How did she choose the crate of bottles, then? Some were 6 for £5, others were 8 for £6 which she thought was a better bargain. Did she look at the volume of the bottles? No. Why not? She was in a rush, and tired [she was becoming increasingly irritable by this point in the discussion!] Did she consider standard "session" lagers (Carlsberg, Carling, Fosters), or just premium lagers (Heineken, Stella, San Miguel)? Just premium lagers. Even though her primary motivation appeared to be cost? No, just premium lagers. Would she still agree that price was her primary motivation? Rachel glanced at me, then meaningfully at her empty glass, then at the bar, and then at me again. I took the hint.
------------
I've written much briefer post on decision making here and there's another half-written one in the pipeline - watch this space.
It was a tweet from Rags Srinivasan, via Leigh Caldwell, that got me wondering. "If a designer shoe goes up from $800 to $860, who notices?"
Certainly not me; I operate at the grubbier end of the shoe market. Caldwell, however, commented, "When behavioural economics can answer this, we've won." Caldwell's blog Knowing and Making and Twitter feed are both essential reading; they're the sort of effort that make one wish they had studied economics at university (or even at school). I know next to nothing about either pricing or behavioural economics (Caldwell's specialities) but I have read snippets by Ward Edwards and others on the marginal utility of money and perceived price, and it set me on a daydream.
The starting point is that the actual price of a product, and the price that the consumer thinks it costs are different entities. The relationship between actual price of a good, and its perceived price by a consumer is not necessarily linear. It might also depend on the product, sector, economic circumstances or, of course, the individual consumer and his whims. If you ask people "which do you think is more expensive, good A or good B?" for a range of products, then you can roughly calibrate a "perceived price" scale - which, after all, is going to drive the purchase decision far more than the actual price of the product.
This got me wondering: if the perceived price depends on the individual consumer, for a given product, under given test conditions (economic circumstances, etc), if you were to plot a histogram of those perceived prices, what would the dispersion of the curve be? And how does perceived price change depending on the consumer's income, and indeed over time?
And how would those dispersion curves be affected from product to product? This would be particularly interesting when comparing two products in the same sector. The acid-test question would be, if two competing products happened to be the same price, which would be perceived as more expensive?
These aren't just "nice-to-know" theoretical questions. If your company's product is the same price as a competitor, but people think it's more expensive, then this will very likely have an effect on the purchase decision, all other factors being equal. Although whether it's a good thing to be perceived as more expensive is not necessarily clear cut, either; it might be offputting (consumers might opt for the commodity they think is cheaper) but on the other hand if "more expensive" equates to "more desirable" then it may be an advantage ( - although then, presumably, that brand's prices are set too low.
It is logical that the relationship between actual and perceived price would vary across product categories, frequency with which they are bought, price as a percentage of disposable income, and whether the product is an "essential" commodity or a luxury.
So perceived prices - and indeed perceptions in the rate of change of price - will surely be different between bus fares, beer, washing powder, a packet of crisps - or designer shoes. Perceived value for money may depend on external factors as well. The mix of online and offline word of mouth, advertising, and economic trends could all contribute.
It is one thing to ask "who notices?" but the other question, of course, is "at what point does it become a problem?" At what point does a consumer go elsewhere to a competitor, or change their habits? What are their motivations for buying the product in the first place, and where does value for money (perceived, of course) fit into the decision-making process?
This leads on to a further question: how often do people compare changing prices across categories? Aside from the odd nostalgic "I remember when a pint cost less than a loaf of bread", I suspect it's not very often - after all do we really compare the utility and value for money of a litre of petrol compared to a litre of Coke? How often do we consider the price of a month's travelcard, and evaluate it versus a month's electricity? Consciously, I'd argue, rarely - but subconsciously? And even if we do, how often do we make conscious decisions to spend money on one luxury over another, or to sacrifice a luxury for a commodity, or even vice versa? Surveys ask blandly "Do you think you have cut back your spending in the last 3 months?" but without probing the thought processes that go into belt-tightening or splurging, those sorts of questions strike me as next to meaningless. Knowing that people have less disposable income is one thing, working out behavioural patterns and irrational decisions, and how to make sure that as the manager of a gym your customers sacrifice a meal in a restaurant each month, or even shiver in darkness, rather than give up their membership - that's the sort of question we need to be asking. Are those purchasing decisions rational in a recession, and how can we find them out?
Putting people on the spot in surveys, asking "is product X good value for money?" always seems contrived, and I have instinctive doubts about the validity of answers when compared to an unprompted population. Worse still are questions along the lines of "how much would you be willing to pay for product X?" I have filled in surveys of this type before, and consistently give a figure lower than what I'd "really" be willing to pay, in a subtle effort to drive down the price of my favourite products, and I suspect this will be true of many people. I certainly can't imagine anyone saying "I think this product should cost more than it already does" - although perhaps someone can enlighten me to the contrary!
A lot of questions, then. One final one - where can I read more about the psychology of pricing, and hopefully find the answers to some of them?
The power of premium brands, eh. The other day, my girlfriend (who drives to work) and I were walking along the pavement when she grabbed my arm, turned towards a thirty-something power-dressing woman clutching a coffee, and muttered to me wistfully, "I wish I got the train to work in the morning, so I could clutch a Starbucks cappuccino on my daily commute."
Then, after a moment, a nervous laugh: "I got so carried away, I forgot I don't even like coffee!"
Transferring a successful restaurant brand into a supermarket staple, without losing brand values, is a tricky balancing act. Pizza Express have negotiated the tightrope well without losing their vision, even branching our into sundries such as dressing.
Rather surprisingly, Pizza Express feature in the "top ten most working class brands" as reported in a recent study by research/strategy agency Britainthinks which looked into the differences between a self-defined middle class and working class.
Interestingly, according to Britainthinks, 71% of Britons consider themselves middle class, although according to the National Readership Survey, 55% of the population would be defined as "middle class" according to the well established standard NRS social grading system.
All the usual C2DE suspects are there - KFC, Iceland, The Sun - but celebrants of rocket and parma ham, Pizza Express sneak into the top ten. Deborah Mattinson of Britainthinks wondered if the launch of supermarket products might have had something with Pizza Express's new-found fame as a working class icon.
Marketing textbooks are littered with examples of brands launching in new markets, or launching new product lines, diluting their brand values, and losing brand equity as a result. Pizza Express took the gamble of launching into a crowded market with their supermarket pizzas and are seemingly as strong as ever; an even tougher challenge is faced by Starbucks, who launched their VIA instant coffee brand in the UK last year.
Where Starbucks lead in the social media space, others follow - their MyStarbucksIdea co-creation concept spawning hundreds of case studies across Slideshare - but, according to Starbucks head honcho Howard Schultz, instant coffee has been in the pipeline for twenty years (although I note that a caramel flavour has been introduced partly following a suggestion via the community).
But how exactly do you launch a product like this into such a crowded space? The Internet Advertising Bureau have published a little case study video of an ad tracking study undertaken by GfK.
A promotional piece by the IAB it may be, but the research clearly shows an augmentation in reach with online advertising, and demonstrating the success that digital ads have in improving both product awareness (up 19% compared to the control group), and also brand favourability and purchase intent. Product awareness was already quite high amongst the target female audience, and with this group it was purchase intent which was boosted most. The video touched on the differences between portals (for high reach), lifestyle sites (where consumers are really engaged with the site, for a longer period of time) and social media; it was inferred that social media advertising gave the best value for money in terms of driving brand favourability and purchase intent for a low cost. Food for thought.
In the last few days an American study for Google has shown that search, rather than social media, is the biggest driver of word of mouth.
The timing of this announcement was interesting, as it coincided with the launch of the Google+ social network. Forget all the excitable "Is it a Facebook killer" chatter; it is a logical progression for a search engine to move towards more human-generated content - based on both your own preferences and those of your contacts.
I haven't played with Google+ yet (OK, OK, what I mean is I haven't had an invite...DAYS behind the times, darling) but my hope would be that rather than merely offering an alternative to established social networks, that it would integrate heavily with other Google products, and I imagine that Google themselves are thinking the same way. Much is being made of the "circles" concept, but I would think that there is more at stake with heavy integration with YouTube, Blogger and, yes, search. The "+1" concept, alongside the recent trend for including acquaintances' tweeted link in search results already means that Google are actively "socialising" their search offering. In an excellent blog post, Simon Mainwaring says that "search...is becoming increasingly inward facing, with the individual as filter."
As a colleague of mine pointed out, this may limit our own personal web somewhat, if we are restricting ourselves to search results based on the preferences of others. He has a point; the fact that we are naturally social creatures, I hope, won't be an excuse to diminish the breadth of our web universe. However, this provides an immediate reference point - if time is short, then a link that your friend has recommended is likely to be your first port of call over anything else. I hope this can be extended into Google+ to make it a content-driven social network; Facebook has no handy way of storing your favourite content in one place ("Likes" are a mess), while social bookmarking sites are sprawling and geek-heavy. If Google get this right, Google+ could be a Digg killer.
But it seems that search itself is a social phenomenon, which leads us back to the Keller Fay study. They don't go into methodological details but claim that "conversations referencing search are thought by consumers to be more credible and more likely to purchase, compared to those that reference social media." They emphasise that offline word of mouth dominates online; but that both TV and internet content drive those conversations (online and offline). And of the internet content that drives conversations, it's the stuff found in search engines that is the most influential.
This goes slightly against the fashionable line of thinking that all conversations just happen organically, that we're purely influenced by those around us, and that there are no tangible drivers; I think it also makes sense. We ARE all capable of thinking for ourselves, of looking for things we like, then finding them (and I'd say this study vindicates Google's search algorithm somewhat if we're liking the things we find via search!) and sharing our favourite content. After all, isn't it natural to TRY and influence people?
This research is a clear warning that "buzz" agencies need to do more than just creating some snappy content, shoving it on Twitter and Facebook and waiting for the rest to happen organically. Ultimately, if brands are measuring their word-of-mouth success purely by numbers of retweets and Facebook Likes then they will fail; the interactions between our online conversations, offline conversations and, critically, the actions we take as a result, that are most important. It transpires that the internet and TV have equal importance in influencing consumer conversations, but that the internet is used as an information checking tool (as one might expect) and that search, in particular, has more bite: "conversations referencing search are thought by consumers to be more credible (+25%) and more likely to lead to purchase (+ 17%), compared to those that reference social media."
By subtly integrating their various products yet keeping them disparate, boosting their core search product, and offering a simplified interface to social networking, Google are taking the fight back to Facebook - and it seems they have a new lease of life.
Update: this article in the International Business Times is worth a read.
The public and political reaction to the News of the World phone hacking has been unprecedented. It's rare that the response to a political or media issue is so unanimous and hostile. While there is lots of hyperbole and hand-wringing, I feel this is one of those times where it is entirely justified. Vince Cable must be feeling very smug at the moment. But brands who advertise in the paper will be nervously judging the mood and trying to work out what urgent changes they need to make to their marketing tactics.
Lot of research will be in progress at the moment to determine just how toxic an association with the NOTW actually is. In monetary terms, how does the loss of cash (from buying the ad space) and value generated by the advertising, weigh up against the loss of brand equity from the downturn in corporate reputation? As Keith Trivitt points out, brand reputations can take years to build but can be tossed away in a matter of days.
This research might be asking how seriously the whole episode is perceived; how the NOTW reputation has suffered; how consumers would be disposed towards brands who advertise in the NOTW; and, perhaps most importantly, to get a sense of how long this whole shitstorm will last for. My guess is that brands would be best advised to pull their ads from the NOTW with immediate effect, sit tight and monitor the situation in the coming days, and quietly carry on as normal after everything's blown through. The losses could be measured in the tens of thousands in most cases - chickenfeed to many brands. This can be offset by the uplift in brand equity as the result of a "good" (in the eyes of the Twittersphere) response.
Ford is a case in point. A solid, firm response was met with a positive reaction online, notwithstanding the point that their media buying agency, Mindshare, are simply putting more ads in the Screws' sister paper, The Sun. Other brands can minimise negative sentiment with decisive reactions. Yes, there is an argument against doing anything rash. But this is a world where "rash" and "social media" are bosom buddies.
The Co-Operative would do well to learn that. Contacted early for a reaction, a spokesman gave the rather blunt reply, "These are allegations. We have no plans to withdraw our advertising."
This was badly misjudged for several reasons. Firstly, they badly judged the prevailing wind. I have never seen a social media backlash as savage and prolonged as this one (most flare up and die down in a few hours). Next, the Co-Op's brand is built on a central platform of an ethical stance. If you shout loudly about "taking ethics to the next level" then the last thing you want to be doing is letting people actively associate you with such a putrid affair. To repeat the point: brand reputations are built carefully over a period of years...why throw it away? Thirdly, the fact that at the time they were only "allegations" is neither here nor there as far as the brand is concerned. A brand is simply "a collection of perceptions in the mind of the consumer" according to Nigel Hollis. That's all; just a set of whims, visions, discernments, not rational but only in our minds. The Co-Op does indeed have a strong brand identity, but given that that identity itself is irrational, why justify the marketing tactics with such a wooden, rational response? It sounded as if the Co-Operative were trying almost to stick up for the Screws - I can think of no logical reason why. It is instructive to note that apparently more tweets were sent to the Co-Op on Tuesday than to any other NOTW advertiser.
As for the loss in revenue, there is likely to be a short term downturn in readership. On the other hand, I wouldn't be surprised if there were a lot of people who went to the NOTW site to take a sneaky look (I must confess to this) only to be knocked back by the paywall. So online traffic is unlikely to pick up in the short term. Inevitably, it leaked out that Ford's media agency, Mindshare, had just transferred their ads into the Sun; brands which take that risk (or indeed any Murdoch title) should be poised to drop that hot potato at short notice depending on what unveils in the next few days. The situation continues to develop at volcanic pace.
One interesting side note from a social media perspective is the lack of propagation of opinion from Twitter through to Facebook. I follow a variety of people on Twitter across marketing, research, social media, political, musical and theatrical circles, as well as a handful of local people, and at times nearly half of my Twitter feed has been part of the #notw feed. On Facebook, however, where I'm friends with 400 people in a balanced cross-section of society (OK, I know, I would say that) there was almost no sign.
Most importantly of all, has this rage extended offline? As I type, a feature on Today asks residents of a London estate, who buy the NOTW, their opinions. Based on a convenience sample of course, and unscientific in every way, but all the people spoken to thought the behaviour was "disgusting" or similar, while several said they wouldn't buy the paper at the weekend (although one said she still would). They weren't asked about the brands, so advertisers will have to await the results of their questionnaires to determine the likely loss in brand equity.
Finally, a few articles to read: some fantastically savage vitriol and anti-Murdoch hostility from Peter Oborne, Matthew Norman and Damien Thompson. The coverage in Marketing Week, spearheaded by Lara O'Reilly,has also been excellent.
For several weeks I've been making painfully slow progress through Francesco Nicosia's classic work from the 60s, Consumer Decision Processes. It's an academic work in a field where I have no formal training, so I'll forgive myself the slowness, although to be fair my eyes occasionally glaze over when reading it on the Overground after a long day at work.
I've still got a long way to go on it (estimated finish time: 2013) but although it's not always edge of the seat stuff (by page 80 the reader has just been informed that a decision starts with a goal and ends with an act) it is a carefully constructed breakdown of how decisions might work. Nicosia reviews the existing literature but it was the ideas of Paul Lazarsfeld that grabbed me.
Lazarsfeld's scheme, from 1935, basically postulates that at time T you have an individual with feelings or situations I(T) in his environment E(T). The environment acts on those feelings and situations, and helps shape them in turn. Of course, only relevant environmental concerns will have any effect on the individual.
For example, E could be anything from some word of mouth, to a change in personal circumstances, to an ad, to a change in product availability, to an event, to a product attribute. So E feeds into I, which in turn is changed, so another "set" of environmental variables will come into play, and so on in a constant iterative process, until the individual preferences reach a critical point leading to some tangible action, and the decision is made. It sounds lovely and simple, but the key point is that all these variables play off one another; so one particular variable will only have an effect on, or be relevant to, another variable depending on what stage of the process you are at.
I feel the marketing implications of this scheme are clear. Imagine first the universe of individual variables (circumstances, opinions and so forth), alongside the universe of environmental variables. The trick is to draw up some sort of infinite Venn diagram, and work out which variables interact with which other variables, and under what circumstances. The marketeer can then consider which of those variables he has control of, and apply them at the appropriate point in the decision process. But the hard bit is realising that each individual circumstance will be different; so are there patterns, or general rules, that can be drawn - and indeed are the decisions that are being taken to purchase the product in question the same or different?
Of course the environmental variables such as product attributes themselves are not constants; because it's the subjective opinion on product benefits that matters. Which neatly ties in with what Ward Edwards and others were looking at in the 50s (my progress through Nicosia is supersonic compared to the rate I'm reading Amos Tversky-edited Decision Making!) I won't pretend to know anything at all about microeconomic theory, but the key to "utility curves", marginal utility and a value-to-cost ratio is that the value or utility of a product is subjective. Even something like price is subjective; the perceived cost is more important than the actual cost when it comes to decision making.
I did actually set my alarm for 4am with the intention of catching some of the Aussie/Far East sessions of the NewMR festival, however willpower (or lack thereof) won the day and I could only haul myself out of bed in time for the 9am GMT sessions.
Brainjuicer's John Kearon kicked things off with a presentation on his "research robots" or DigiViduals. I'd already seen a presentation online on the same subject, but preferred this new one. Basically the concept is that you create a virtual persona, consisting of any attributes you like: behavioural traits, attitudes, tone of language, personality types, lifestyle choices...with or without more traditional sampling attributes like demographics. With your subject created, you go and scan various forms of social media (Kearon always starts with Twitter, but it can go to shopping sites, forums, YouTube...) looking for REAL people whose personal characteristics, as evidenced by the content they have created, "match" our virtual person. Then, you can simply lift content from those people and analyse the relevant parts to your study.
This is a brilliant concept. Sampling can move away from the old "middle class mothers who read magazines" to groups who share characteristics that are much more tactile. I wondered about the volume of data that you'd have to go to before you'd get people who form a "close enough match". Kearon pointed out that while much of the work is done automatically by the research robots, there's still a lot of manual data cleaning to be done.
I imagine in practice there is some sort of "threshold" that people have to meet to be counted in. For example, perhaps they meet at least 60% of the attributes, or else they are 40% more like our digividual than the "average" person.
The real beauty in this is that your digividual can be a totally artificial construct, not based on any real people at all; in fact, it could be an experiment to find a type of person who you don't know exists. The potential for discovering new or niche markets is endless.
Talking of artificial situations, this led nicely into Tom Ewing's presentation comparing research methods with gaming. In the last year or two, more and more commentators have predicted that online gaming will really take off to new levels in the next few years thanks to the social side. All sorts of games - whether web-based, console-based or whatever - have had a new lease of life thanks to the social aspect. Ewing mentioned FourSquare as the ultimate example (I'm only just about to get my first posh phone so I barely know anything about it!); I was surprised he didn't mention Second Life (does anyone actually play that any more? You heardly hear about it these days).
Ewing rattled through a series of nice analogies - but there was a linear theme about showing how research can learn from the best games which keep their players entertained and engaged. He pointed out that a game like chess, whose mechanics are simple and dull, has millions of possible game scenarios, which quickly become complex and involving, requiring a lot of thought and effort on the part of the player (or respondent!). He also pointed out that different people have different motives for playing games, and that good game designers can take this into account; similarly, research respondents have different reasons for giving up their time, and the canny research designer will bear this in mind and try to take advantage.
He makes the point that Sonic the Hedgehog would be a dull game if there was a constant progress bar! However, the concept of levels in games means different things to different people and a sense of achievement (and therefore the effort that goes in to fulfil that achievement) varies from person to person. Monopoly playing styles also vary - people's approach to risk results in very different ways of taking the game on.
Ewing also showed he similarities between gaming and research like simple mobile tasks/apps and also community building. While the analogies came thick and fast, the presentation was full of real-world suggestions for ways that researchers could actually go away and make their projects more interesting for respondents tomorrow.
"Gamey" was how the next presenter, Jon Puleston, described some projective techniques and again this presentation was full of practical ideas of how to improve data quality. He recently undertook a study showing increases in respondent productivity as a result of changes made to online survey designs. Imagery and snappier introductions both made a significant difference, but most interesting were the increase in data quantity/quality from using more interactive, projective techniques. One in particular (where researcher and respondent trade ideas one-for-one) was shown to be particularly effective, as was the game of "put yourself in someone else's [the client's?] shoes..." A very nice presentation.
Completing the first mini-session was Graeme Lawrence of Virtual Surveys. His presentation seemed to have less of a structured narrative, but was no less interesting for that. It was all about "not just listening"; the point that successful "NewMR" needs to be a mixture of large-scale, passive listening/monitoring ("why ask some when you can listen to all?") - but also more proactive asking of questions. I suppose this must vary depending on the subject - there are some areas where there are vast volumes of data already out there, but others where respondents need to be prompted and pushed. I suppose there's less noise to eliminate once people have more of a focus - at the expense of things being a bit less natural (looking forward to Mark Earls's keynote later - my rather verbose review of his book here). One example he gave showed some data on "where else" on Facebook fans of a particular page go - does anyone know what tool was used to get that insight? He gave examples of Facebook fans of Next and H&M providing opinions and insight - it occurred to me that here you are restricting yourself to brand loyalists. It doesn't necessarily work for all brands, either: people may be shy to become Facebook fans of a feminine hygiene product or political party, for example.
After a short coffee break, Annelies Verhaeghe gave a terrific talk on research using social media. I loved her initial analogy of a house of cards - companies are throwing themselves into social media without having a clue about best practice, then getting surprised when things go catasrophically wrong. My current line of work is closer to PR than MR but the facepalm horror stories come thick and fast. She quickly moved on to the issue of representativeness of online and NewMR techniques - a subject dealt with at some length by Ray Poynter in his excellent Handbook of online and social media research. Her main point was that we don't know who is talking. Real people become personae, defined by their content and personalities rather than their demographics. But haven't we heard something like that before? It's all about John Kearon's digividuals again. The sampling goalposts haven't been taken away, just moved along. She also talked about the fact that most sampling online is convenience sampling, and touched on issues of data quality (for example content that is "quoted" or duplicated). There was also a very nice graphical representation of data volume vs sentiment for a particular topic.
Rijn Vogelaar followed with his take on opinion leaders or "Superpromoters" as he calls them. He divided thoughts up into conscious, subconscious, and brand opinions. Personally I found myself a little skeptical - for a start I'm not convinced that there are an elite few brand evangelists who shape community opinions, but also because I'm not convinced that the opinions of the blind optimists, the hardcore fans, are necessarily the most important: aren't the drifters, the disloyal, and the indifferent, of more interest?
Rich Shaw finished up the second mini-session with a presentation about the "hacker ethic". I must admit I missed most of this - initially distracted by "NewMR chatup lines" on Twitter, and then by the gas man knocking on the door. I'll come back to it.
Academic researcher Dr Agnes Nairn gave a great overview of the ethical issues surrounding new research techniques in a talked entitled Oi, you took that without asking! Her own work is concerned with children, and she brought up practical concerns about getting the appropriate level of consent from both the child and their parents (by phone: consider mebeingmymum@gmail.com!!!) There is also an issue of data protection: I was pleasantly surprised at the level of confidence in the police dealing with personal data, but market researchers were at the bottom of the trust pile - way behind bankers.
She moved on to the central issue of informed consent. The old rules have been thrown out of the window where social media monitoring is concerned. It is difficult to inform people for whom you have no point of contact (for Facebook, forums etc) or details (Twitter), particularly if you are collecting data on a very large scale. The level of intrusion also varies on a sliding scale: there is a world of difference between taking one person's personal essay, quoting it in client meetings and using it to influence decisions on he one hand, and merely using a sentiment analysis tool to add an opinion to a set of positive/negative sentiment aggregate data at the other. I also have some sympathy with the view of Mark Zuckerberg who caused a storm when he said that people in the Facebook generation are less bothered about privacy and more inclined to open up their lives in public online; yes, of course he has an ulterior motive, but I get the feeling that he's mostly right despite the noisy protests of various pressure groups.
Henrik Hall's chat with Ray Poynter wasn't really relevant to me, but Bernie Malinoff's presentation on the pitfalls and differences between different approaches to online surveys was interesting. Incredible that two similar methodologies, with only some small tactical differences, can give completely different results. It's the sort of thing the research industry needs to tackle quicksticks to avoid being seen as a waste of time and money by clients. Ian Ralph's practical talk on smartphone research was also interesting, although as a non-practitioner I find these highly tactical discussions a struggle to keep up with.
Betty Adamou finished with a brilliantly rousing call to arms for Facebook research. She made some bold claims about young people - email is as dead as the CD, for example - and pointed out that researchers must make the effort to reach out to respondents, not the other way round. She made some great points about the sorts of times and places respondents might want to take on a piece of research: at a bus stop, for example, or waiting for a late-running boyfriend. I'd love to see some "situation-based" research. She also said that researchers need to be more flexible about adapting to the way young people behave, especially online - by embracing things like txt spk and smilies.
The evening session features one of my recent heroes, Mark Earls, and lots more goodies: I can't wait. If it's half as good as today's session then it'll be a very enjoyable few hours.
I have cross-posted this on the NewMR site as a blog post.
Richard Feynman is the scientists' scientist. He was one of the greatest physicists of all time (seventh in Physics World's 1999 poll, behind Einstein, Newton, Maxwell, Bohr, Heisenberg and Galileo) - brilliantly explaining the relationship between photons and electrons in a fundamental area of particle physics called quantum electrodynamics which he basically invented. His squiggly pictures, now known as Feynman diagrams, can help explain complicated particle interactions in a non-mathematical way to the extent that someone with only a basic knowledge of physics could understand them.
The Feynman diagrams were a classic example of what made him not just a boffin, but a great scientist and educator. Feynman's approaches and attitudes were second to none. One of the best £90s I ever spent was, as a student, when I splashed out for the Feynman Lectures on Physics. If any physics undergraduate (or even A-level student) happens to read this, do yourself a favour: the three volumes are utterly inspiring, although beware - his methods are frequently unorthodox and can sometimes become ferociously difficult. I had a love-hate relationship with physics, but keep coming back to the Lectures time and time again.
Feynman has always been a bit of a hero of mine, and so when the latest batch of books I ordered online plopped on the mat, rather than consigning The meaning of it all to the end of the rapidly expanding backlog, I gobbled it up over a couple of days on my commutes. It is a collection of three lectures he gave in 1963 on the relationship between science, religion and uncertainty. It's a slim volume, but essential reading whether you've made up your mind on metaphysical and ethical questions, or whether you have an open mind. In an era dominated by shouty shock-jocks like Richard Dawkins, Feynman's quiet reflections really do stand out. Furthermore, I'm convinced that his opinions will be relevant to commercial research of all sorts as well as pure scientific work.
The general thrust of Feynman's argument is a celebration of uncertainty. The claim that ignorance and uncertainty are not something to be ashamed of, but quite the opposite - an exciting problem fresh to be solved - is quite refreshing. He comes across as a pure scientist: tackling problems for the problems' sake, yet appreciative of the practical implications of scientific research. One aspect that it's enlightening to hear from the mouth of the great man is just how "unscientific" the scientific process can be: rather than "Eureka" moments, scientific research is an iterative process, slowly proving the old rules wrong; Sherlock Holmes knew what he was talking about when he said "Eliminate all other factors, and the one which remains must be the truth." Slowly building on established knowledge, confirming and disproving different strands by experiment (trial and error, if you like), is the way knowledge is furthered.
While Feynman, of course, is concerned primarily with things that can be measured and evaluated quantitatively (or even qualitatively), he makes the point that it's not only measurable aspects which are important:
But if a thing is not scientific, if it cannot be subjected to the test of observation, this does not mean that it is dead, or wrong, or stupid. We are not trying to argue that science is somehow good and that other things are somehow not good. Scientists take all those things that can be analysed by observation, and thus the things called science are found out. But there are some things left out, for which the method does not work. This does not mean that those things are unimportant. They are, in fact, in many ways the most important. In any decision for action, when you have to make up your mind what to do, there is always a "should" involved, and this cannot be worked out from "if I do this, what will happen?" alone. You say "Sure, you see what will happen, and then you decide whether you want it to happen or not." But that is the step the scientist cannot take. You can figure out what is going to happen, but then you have to decide whether you like it that way or not.
What of consumer research, then? To me, that implies that the business decisions which are made as the result of research still need to be bold ones, and that no matter how robust the research methodology itself may be, if the research brief is lousy or pointless then the research is wasted. Nothing new there then. But also that the research in itself might not be enough to show clearly whether the benefits of international expansion, or introducing a new product line, or scrapping dress-down Fridays, outweigh the side-effects. The research might demonstrate what the benefits and side-effects are, and even quantify them. But as for making that instinctive judgement? Market research's usefulness might be limited there. What Feynman is saying is that even from a scientist's point of view, that instinctive judgement isn't something to sniff at.
Having recently read Ben Goldacre's Bad Science, it occurs to me that there are a lot of parallels between the two - and can be summed up as "common sense". Objectivity. Looking for ways to disprove your work rather than proving it. These are basic groundrules.
More on uncertainty:
...doubt and uncertainty [are] important. I believe that [they] are of very great value, and extend beyond the sciences. I believe that to solve any problem that has never been solved before, you have to leave the door to the unknown ajar. You have to permit the possibility that you do not have it exactly right. Otherwise, if you have made up your mind already, you might not solve it.
Feynman is a pure scientist, through and through. He asserts that
work is not done for the sake of an application
which I agree with 100%. But, since I'm corrupting this post horribly with comparisons to commercial/consumer research which would probably horrify the great man, I'm not sure that I agree that this statement is true for market research. Sure, if MR and consumer research help boost the pool of knowledge surrounding consumer behaviour, then great. But while the foibles and complexities of consumer behaviour are undoubtedly fascinating (whether the research is conducted by market researchers or academic psychologists), the implications of its results are more important. Market research is definitely an applied science. A MR project is generally undertaken to fulfil a purpose - is that something to be ashamed of? It is a tool that provides insight into consumer behaviour and attitudes so that decisions can be made based on the findings. To paraphrase Feynman, the scientist (or market researcher) can provide the answers to questions like "what will happen if I do X?" or "what should I do in order to achieve Y?" Of course, whether Y is a desirable outcome or not, is outside the researcher's remit, and only the decision-maker can tell. But it's the transition from pure research to marketing applications that is where research is so exciting, for me.
To every man is given the key to the gates of heaven. The same key opens the gates of hell.
That sounds like the sort of thing one of those "inspirational" business blogger "gurus" like Seth Godin might say. It wasn't. It was something a Buddhist monk once told Feynman, who, rather than making a sweeping statement based on it, interprets it in the context of ethics: top-level scientists have the capability to do something really dangerous with their expertise. Feynman should know: he worked on the Manhattan Project at Los Alamos.
He says that whilst making vague claims is next to useless, making bold assertions, which can then be proved either wrong, or correct until proven wrong (as with Newton's laws) is the way to go about scientific research:
The more specific a rule is, the more powerful it is, the more liable it is to exceptions, and the more interesting and valuable it is to check...doubt is not a fearful thing, but a thing of very great value.
More generally, he lauds risktaking in research, saying that with such risks, reputations may be put on the line, but the greatest discoveries are made.
...you must be willing to stick your neck out...make a specific rule and see if it falls through the sieve.
The risk, of course, is that you are barking up the wrong tree completely. The parallels with the commercial world are obvious. Those researchers who try an outlandish methodology, or make a bold hypothesis, might end up getting nowhere. All kinds of agencies are experimenting with neuroscience, text analysis, eye tracking and all sorts of techniques with varying levels of sophistication. They sound sexy and therefore generate coverage (and thus sales), although there are a lot of sceptics who doubt the value of such techniques, particularly as, given their youth, by the sounds of things many of them are doing little more than floundering around in the dark.
I would argue, however, that is is the duty of big brands to invest in risky research - to invest in a hypothesis or methodology that could be badly flawed and provide zero ROI. But the likes of Unilever, Coca-Cola and Nike can afford to take such risks once in a while, and if the gamble pays off, that new methodology might give them deep consumer insight which rapidly gives them ROI and an edge over the competition. Perhaps more importantly, they should feel compelled to make these contributions to marketing, because brands as a whole may be able to benefit from the latest techniques in neuroscience, say, that would be too expensive for smaller companies to invest speculatively. Of course, this would rely on providers not flogging dud research.
Yet to invoke the spirit of Feynman requires that little spark, that little wow factor. There are plenty of case studies out there at the moment which just make you sit up and take notice. One example is this from Brainjuicer's John Kearon:
Crumbs. I don't know where to start with that one. Maybe it deserves another piece in itself.
The other piece of research, and this is something that REALLY made me sit up and take notice, was an academic study by researchers from Indiana University, entitled Twitter mood predicts the stock market. I've read the paper in full and will write a piece shortly - until then, if your imagination is running riot and you can't bear to wait, have a look at Wired who have covered it.
These analogies are horrible, I know, but I'm reading a lot about MR developments at the moment so forgive me for making comparisons.
And so to religion. Dawkins and his crew have made a lot of noise in the last few years with dismissive sweeping statements (the same, of course, can be said about the religious right). Feynman (who, let me reiterate, was one of the greatest scientists of all time) doesn't mince his words.
I agree that science cannot disprove the existence of God. I absolutely agree...belief in God and action in science [are] consistent.
For someone so mild, cannot is a strong word. It's a bold claim, an absolute claim. (It's also something I happen to agree with, but who am I to comment on the great man's veracity). He describes the wonder of seeing the mysteries of the universe come alive as
an experience which is very rare, and exciting
A young scientist, Feynman continues, may find
the religion of his church" inadequate to describe that kind of experience. The God of his church isn't big enough. Perhaps.
And the walls of belief might start to crumble. But at the same time, he argues that people who are not religious, by and large, share the same values as religious people:
It seems to me that there is a kind of independence between the ethical and moral views, and the theory of the universe.
Feynman splits religious thought into three: metaphysics, ethics, and inspiration. On the metaphysical aspect, he argues that scientific beliefs are perhaps less strong than religious ones (the principles of dogma and infallibility spring to mind here). Meanwhile
the uncertainity that is necessary in order to appreciate nature is not easily correlated with the feeling of certainty in faith, which is usually associated with deep religious belief.
This goes contrary to conventional wisdom, which says that it is science which provides absolute answers, and religion which leaves questions unanswered.
Feynman died in 1988 but the most obvious as yet unanswered question which still lingers regards the origin of the universe. As science and technology become ever more sophisticated, we know what happened closer and closer to the Big Bang. Minutes, seconds, tenths of seconds, millionths of seconds...we're not that far away from having a pretty good idea of what the universe looked like at one Planck time after the Big Bang (which I don't have time to explain...ho ho). But as for the Big Bang itself? All analyses seem lead to a singularity (infinity or impossibility if you will) - something which we cannot measure, cannot decipher, and it seems as likely as ever that it will be something that we can never understand. And what is so wrong with that?
