AEO Marketing Forums – Measuring Customer Satisfaction

It was great to catch up with some partners and clients at last weeks’ AEO Forums, it’s always a good focus for the start of another year.

With a loose customer experience theme running through the day, the Marketing Forum was not short of statistical references, after all, no-one knows how well they’re delivering customer experience unless they measure it.  In fact, all but one of the audience were using NPS or The Net Promoter Score® to give it its full moniker.

Of course, NPS is a very simplistic measure of customer loyalty and simple is good, right?  Well yes and no (get comfortable, this isn’t a quick explanation!).

You see the problem I found while listening to various sessions was how many times there were mentions of because of X and Y “our NPS increased by….”.  Unfortunately because NPS is a simplistic measure it doesn’t register your specific changes, it simply tells you whether the audience who respond to your survey criticise, remain impassive or promote their experience, it can (if the right follow-up questions are asked) highlight improvement opportunities, it can tell you what benefits you should be communicating to prospects and it can provide invaluable marketing guidance, what it can’t do is to tell you what aspects of delivery are responsible for an upturn or improvement in your score, you can hypothesise but ultimately it’s a guess.

NPS as an overall loyalty measure, can be heavily influenced by sample and since the majority of event NPS measures come from self-selected, online surveys these days, year-on-year score tracking is often subject to sample bias which inherently impacts the NPS.

Essentially you can’t trust the score unless you check, balance and validate the sample – unfortunately few do.  ‘So what’ you say, so long as our NPS is improving.  Well what if you’re missing a critical segment of your audience?  What if you’re only hearing from the people who can be bothered i.e. those with polarised experiences – detractors and promoters, how useful is your customer experience metric now?

More concerning and something we’ve heard a lot recently, is when NPS is aligned to appraisal objectives and bonuses.   You see, the problem is NPS as a simplistic measure of loyalty, really isn’t that simple.

Because two percentages are involved in the NPS calculation (detractors from promoters), both percentages (even in a validated sample) are subject to margins of error, meaning the overall range of potential scores in any given sample can be wide.

As a rough guide, even if you have a thousand responses (which many of you won’t have, especially for exhibitor NPS), you only really know your scores within 10 points (+/-5 each way).

I’ve illustrated an example of this below:

(Figure 1)

Show X 2015 2016
Survey sample 110 115
Audience size (visitors/exhibitors) 500 510
Response rate % 22% 23%
Confidence interval (unvalidated sample) at the 95% level +/-9.3% +/-9.1%

 

(Figure 2)

2015 2016
Promoters 36% 39%
Passives 9% 26%
Detractors 55% 35%
NPS -19 +4

 

So looking at figure 2, on the face of it, you have a +23 improvement in NPS from 2015 to 2016 (-19 to +4 scores) congratulations right?!  Well yes but not as you might imagine.  With samples of this size the scores are within their margin of error meaning the NPS upturn is not significant.    What is significant is the reduction in detractor’s year-on-year (down 36% or 20 percentage points year-on-year).

While it’s still positive in this instance, reverse it and how do you feel?  Your score looks like it’s gone down significantly or it’s just remained static – but has it really?  You’ve worked your backside off on this event, you listened to audience feedback, you’re confident you understand their needs and you’ve delivered what you truly believe is a fantastic event .  There doesn’t seem to be any rhyme or reason to the difference in scores so how do you justify it?

That’s why you need to understand more than the very ‘simple’ NPS calculation, you need a robust and reliable sample, you need to minimise sample bias wherever possible and you need to set targets/objectives that are cognisant of the error margins.

And ultimately, while NPS is a very good ‘simple’ measure of customer loyalty, it’s simply not enough on its own to provide reliable understanding of the customer experience, what works and what doesn’t.

If you want to know how best to do that, get in touch, we’d love to share our experience.

Posted by Lisa

ZING INSIGHTS LOGO BLACK 300x300

World Cup 2018 and using data in sport

With the World Cup only a matter of weeks ago, England fans all over the country are slowly returning to normality, while the players themselves will only have only had… Read more »

ZING INSIGHTS LOGO BLACK 300x300

Starting at Zing

I joined the team just over two months ago now and how time flies… It was nothing more than perfect timing that the position here at Zing appeared, just as… Read more »

LBS for blog

The importance of event evolution

With a 62-year history, you might have been surprised to hear the recent British Marine announcement that the London Boat Show 2019 will not be going ahead due to ‘insufficient… Read more »

GDPR

GDPR Guidance for Clients

GDPR – Guidance on using personal data for research purposes Under the new GDPR regulations coming into force on 25 May 2018, we, as data processors, will need assurances from… Read more »

GGH 300x300

Sponsoring GoGoHares 2018 #GGH18

Summer 2018 brings a city trail of 50 hares to celebrate Break’s 50th Birthday. It’s a New Year, we have a new look and we’re pleased to be supporting Break… Read more »

ZING INSIGHTS LOGO BLACK 300x300

New Look for a New Year

It’s hard to believe that it’s almost 7 years since we launched Zing Insights as an independent market research agency and we’ve experienced so much change in that time.   But… Read more »

Our Clients

We have extensive experience working across a wide range of industry sectors for many leading brands.

Contact Details

Victory House
Chequers Road
Tharston
Norwich
NR15 2YA

t / +44 (0)844 800 9588

e / hello@zinginsights.com