Common errors in customer surveys

What’s the problem?

I see the same mistakes – as well as some new ones – being made over and again in surveys and questionnaires. Most businesses and other organisations are dependent on surveys to a greater or lesser extent. They use them to find out what customers think of them, or what products they should be developing, or what issues need fixing… etc etc. Yet often those organisations are not getting accurate information. If survey questions are confusing or ambiguous, or constrain answer choices, they will be getting a skewed view of responses. It’s like a political poll asking, for example, ‘which candidate do you like’, rather than ‘who do you intend to vote for’. What do you actually want to know?

My advice is that if you are responsible for a survey of any sort, spend some time getting yourself up to speed with what makes for a good survey, and what some of the pitfalls are. It’s easy when you’re familiar with a topic to ask questions that respondents won’t necessarily understand, and it takes some self-discipline and customer knowledge to avoid the problem.

You can’t always entirely trust the ‘experts’ either. I’ve had many a debate with professional purveyors of surveys about their proposed wording for questions, as I’ve felt that they were

  • ambiguous
  • unclear to my particular customers
  • pointless
  • too similar to other questions
  • not offering adequate response choices
  • etc

You need at least to be able to judge whether the professional advising you really knows their stuff.

It pays to test a survey on a small sample before general release – and that means talking to people, and understanding how they interpret the questions, and whether it’s the interpretation that you intended.

There are many books on how to write questionnaires and surveys. One that I’ve read and can recommend is

The Complete Guide to Writing Questionnaires: How to Get Better Information for Better Decisions

Don’t expect a riveting read, it’s a textbook. But it does cover the ground with good examples.

Now I’ll describe how I decided to approach a survey question at British Airways that was more complicated than it seemed, and then I’ll give some examples from recent surveys that I’ve filled in.

The British Airways question

I was working on the wording for the ba.com site survey, and ended up with some convoluted logic. It wasn’t convoluted to the people filling it in (hopefully), as the sequence would make sense to them. It didn’t make a lot of sense though to colleagues and others who reviewed the questions, and I had to defend the structure many times.

When customers filled in the feedback survey on ba.com, we wanted to know if they were

  • a member of the Executive Club (the frequent flyer scheme, abbreviated to EC)
  • if so, which tier they were (Blue, Bronze, Silver, Gold)
  • if they weren’t, were they registered with a site login
  • if they weren’t registered at all

We could have gone with this:-

<Are you

  1. An Executive Club (EC) Member
  2. Registered on ba.com
  3. Neither
  4. Don’t know>

The problem with this is thatĀ some people don’t know if they are EC members. Generally, those who are know they are, as they’ve gone through the process of joining, but otherwise people could say, “How do I tell if I am?”. They might think that just registering on the site, or buying a plane ticket would give them membership. Internally within BA it came as a surprise to some that there could be this confusion.

It would be a little better to have

<Are you

  1. An Executive Club Member
  2. Registered on ba.com (but not an Executive Club member)
  3. Neither
  4. Don’t know>

The problem would still remain though that if you are either an Executive Club member, or registered, but weren’t sure which, you would answer ‘don’t know’, and then we wouldn’t have known whether you were registered at all.

It would also potentially confuse some Executive Club members who would think that they are both a member, and also registered.

What we went with was this.

<Do you have a login for ba.com?>

If the customer said no, skip to next question, if they said yes, then we asked

<Are you an Executive Club Member?>

If they said no, then they were registered, but not EC. If they said yes, we asked what Tier they were.

It’s still not perfect, but it does at least mean that we got better quality results on whether people were registered or not (without having to interpret what ‘registered’ means).

Other surveys

Limiting responses

One of the most frustrating things that I see in a survey is when none of the answers apply. I’ve lost track of the number of times I’ve told people that they need an ‘other’ option. Sometimes there is a closed set of potential answers – either you bought something on this visit or you didn’t.

I filled in a survey after having attended a Rock weekend at Butlins (it was great). I answered a question saying that the experience could have been better, and then I was presented with this question asking what could be improved.

Butlins survey

The problem is that my reason wasn’t any of these things, yet the only way to progress is to pick one. So I did. If they had an ‘other’ option which allowed me to enter text, I would have let them know that some of the behaviour from other guests who had had too much to drink had been annoying. But they’ll never know.

Butlins also asked this –

Butlins second survey question

The drop-downs were the same for each, and showed this-

Butlins options for what makes a good break

I would expect Butlins to have a good handle on what matters to their customers in general, but for me, Internet access can be a deal breaker for a holiday, and it’s not in the list. My wife always wants to know if there is a hair dryer in the room. These may not be our number one issue, but if you’re going as low as number five, then you risk missing out.

First Great Western (FGW) ask about reason for travel.

FGW reason for travel

I think it’s reasonable to assume that business, commuting and leisure account for the majority of train journeys. But what if you’re travelling to a funeral, or other reasons? It may be a small enough proportion that FGW think it’s not worth making the survey more complex by having an ‘other’, which they are entitled to do. But each time a respondent has to think harder about a question, it’s an additional point at which they are likely to drop out.

Task accomplishment

It’s fairly common on a site survey to ask what the purpose of visiting was (again, it can be problematic to assume you know all the answers), and then to ask whether you were successful.

Maplin and First Great Western both use Foresee to serve their surveys, and they take a different approach to each other.

Maplin success question
First Great Western success question

You can see that Maplin offer a ‘partial success’ option, which FGW don’t. It’s likely that for many sites a significant proportion of customers will be partially successful. With FGW, I might have come to buy a train ticket, and did so, but it wasn’t at the price or the time that I wanted. I count that as a partial success. By only offering the binary choice customers are forced to make a qualitative judgement which way to vote. When that happens, the survey owner loses useful information. That’s especially so if the customer votes for ‘success’, as then you don’t know there’s something that was an issue. You can still ask ‘was there anything else that would have improved your experience today’, but then you have a pile of verbatims, and the issues are lost from the headline reporting of the success question.

Complex questions

I filled in a Which? survey about my car. This was one of the questions, asking how I financed the purchase.

Which? car financing options

As with any web text, survey respondents don’t necessarily read the detail of each question. They will scan, and stop at the first answer they think applies to them. In this list, the choices for ‘Personal Contract Hire’ (the first option) and ‘Personal Contract Purchase’ (further down) are quite similar, and unless you are a wizard on car finance you have to read the detail to understand the difference. I suspect that Which? are going to get more responses to the first option than actually applies. You’ll get people like myself who have ‘Personal Contract Purchase’, who read just enough to decide that the first (and wrong) choice, applies to them.

In such cases, the two options should be next to each other. It doesn’t entirely solve the problem (it would further help if the order was randomised), but there’s more chance that people will spot the alternative, rather than just going with something that looks close enough.

Confusing questions

Time and again I’m filling in a survey and think ‘what do they mean by that?’. Often, these will be technical questions, or ones requiring a subjective judgement but no guidance is offered.

On Google maps I often answer questions about places I’ve visited. It seems that Google has a standard set of questions, some of which do puzzle me.

Google 1

What you and I consider to be ‘trendy’ can vary. Google may be ok with this, but I usually just go for ‘not sure’.

Google 2

I always struggle with this one. Is a more expensive pub ‘upscale’? There are probably venues that clearly are, like the Ritz, and those that clearly aren’t, like McDonalds, but where’s the line?

Google 3

Shouldn’t the question be ‘Is this place popularĀ with travelers?’. Each time I see this I have to stop and think about what it means. Anyway, how can I tell who is a traveler? Does it mean tourists? People just in transit?

Google seem to be experimenting with images as well. Here’s one question I was asked.

Google image choice

I can well imagine that Google could be experimenting with the automation of image choices. Nevertheless, whilst asking me which image is more ‘helpful’ (what does that mean? Should it be ‘representative’?), the picture on the left is of Windsor, rather than Slough. The picture on the right is of some offices just outside Slough. I don’t think either are ‘helpful’, although the one on the right is at least of Slough.

There could be some rhyme and reason to all this. All I’m doing here is pointing out some of the confusion these things cause to me, and readers can decide for themselves whether it’s useful or relevant.

Other issues

I filled in a Which? survey about pet insurance. We have a cat. The survey asks what type of cat it is.

Which? cat

As you can see, the response is selected with a check-box, but unlike radio buttons, check boxes are not mutually exclusive. This doesn’t make sense, as the cat can only be of one type. If you select more than one type, you get an error message.

Which? error

This could easily be avoided by using radio buttons. Whilst most people aren’t going to pick multiples, if you have a cross-breed you might pick two boxes, or you might pick one, and then spot a more accurate description, and go for that. The initial question also does not specify that only one choice can be made.

Viking sell office supplies. They also use Foresee to serve their survey. At the end of the survey this is what you see.

Viking survey completion

It’s good that it says thanks, but where do you think ‘Contact Us’ links to? I’d assumed it would allow me to contact Viking, as I’m answering their survey, but it actually links you to the Foresee site.

Many years ago we discovered that some of our customers were contacting our survey supplier in the mistaken impression they were contacting us. Worse, the supplier was responding directly, rather than passing the messages back. It needs to be clear who the contact is with. Own your own survey.

A couple of positives

I’ll finish off by pointing out a couple of positives things I’ve seen.

Butlins completion

This is a good(ish) sign-off from Butlins, thanking the customer for taking the time. It’s a shame that the message about entry into a prize draw is so small and barely readable. More could be made of it – and a happy picture would add to the experience.

FGW contact question

Customers who respond to a survey may be inclined to help out with further research. FGW ask if customers are willing to do so, and it’s possible to build up quite a database of willing customers that can be segmented by the responses to the survey. The wording could be tightened up and made a bit more visually appealing though.

Finally, from the Which? survey on pet insurance, there’s a question about the age of the cat. It’s good that there is encouragement to answer approximately if you’re not sure. It gives that bit of permission not to sit and agonise about being precise.

Which? cat age

This question reminds me of applying for car insurance years ago. Many insurers asked for the date when your licence was issued. In fact, all they were interested in was whether it was issued more than a certain number of years ago. It would have made my life easier if they had just asked that.

When do you stop referencing old research?

The issue

When I started out in UX there were some designs that I was convinced would work, some I wasn’t sure about, and some that I was convinced wouldn’t work. Simplistically, I think of it like this.

Old range of certainty

Then, as I learned more, and in particular did more research, I found that I wasn’t always right. Odd that. This had the effect of increasing my range of ‘might work’, as there are so often surprises. So the model changed to this.

New range of uncertainty

There were fewer designs that I thought definitively would or wouldn’t work (or would be better than an alternative), but I was more confident of those judgements.

As time went by (maybe there’s a song there…), people started to say things like ‘look, just because you did some research five years ago, doesn’t mean this design will fail now’. I don’t think it was my communication style that earned this response (but reference Discussing Design). So I tried to explain that there were some things that you discover through research that don’t change, or don’t change much, and that there were some things that were a result of context and time, and would be more likely to change over time. I tried to explain that if I was citing older research, it was only to illustrate a principle. Sometimes if I thought a design wouldn’t work it was on the basis that it broke a principle, and people assumed it was because of some old research.

I’ll give you some examples.

What gets old

Some years ago when we did usability testing on ba.com, it was common to find people who were surprised that you could book a hotel on the site. These days, it tends to be the other way round. People are used to the cross-sell. The fact of being able or not able to book a hotel on an airline website isn’t related to human psychology. It’s cultural and can be learned, so you can’t rely on old research.

Another more current and general example is the hamburger menu. It’s commonly used in mobile designs, although any research that I’ve seen says it’s most effective when used in conjunction with the word ‘menu’. Some designers maintain the style for desktop users. This makes no sense for a number of reasons. One is that since you have more space, you should expose the main navigation elements, rather than make users search, but also because you get many older users of desktop sites who have not learned the hamburger style. This is something that is likely to change over generations.

Another example that I found interesting was when we were doing some research in the US. In the checkout process the customer was offered the opportunity to get a credit card on which they would earn miles, and they could use it to pay for this flight. In our mockups we used the words ‘instant credit’. This was just after the sub-prime crash around 2008. Users reacted strongly against this phrase. When we changed the words to something more like, ‘get a card, earn miles, pay now’, the reaction was quite different. It was the same offer, just different positioning influenced by what was going on in the world at the time. It’s quite possible that at another time ‘instant credit’ would be more appealing.

What doesn’t get old

One example of what’s not going to change is the impact of grouping and separation, a fundamental principle of human cognition.

Here’s a snip from a search for ‘wheels’ on Cycle Surgery

Search for ‘Wheels’ on Cycle Surgery

Depending on screen size and resolution, a user could quite possibly scroll to a position like this. At a glance (which is all it should take), it looks like the upper price relates to the wheel below. They are spatially closer, and there is a lot of white space above the price. There are faint separating lines, which a user may or may not notice depending on the quality of their screen and eyesight. It’s confusing.

By contrast, here’s the same search result on Chain Reaction Cycles

Search for ‘Wheels’ on Chain Reaction Cycles

There is much clearer grouping of associated information, and separation from others.

Even better is Argos, using a card style.

Argos search result

So, to bring it back to old research. If I were to look at the page of results from Cycle Surgery, I’d say that at the very least it could be improved, not because of any research, but because it breaks the rules of how people look at things. I don’t need to do any research to figure that out. Items close together are related (we assume). Items with a strong separator are not related. Sometimes a poor design can cause conflict between these principles. Argos overcomes these issues.

To get more on those fundamental principles, have a look at some of the books I recommend on design, and behavioural economics.

Some things are in between

When people look at a list of flight choices, the vast majority start by trading off price and time. The cheap flight looks good, but you don’t want to get up a 4 am to catch it. 11 am is really convenient, but too expensive, so the 9 am is a reasonable compromise.

That’s the first calculation. After that, people will look at seat availability (can we all sit together), or is it the aircraft type that I want to fly on. Sometimes there’s a choice of airports, and so on. These things aren’t to do with the basics of human psychology, but they are deeply rooted, and are unlikely to change anytime soon, unless other factors become a lot more important for reasons I can’t currently think of. But it could happen.

I lost track of the number of times we had to reinvent the wheel to discover the same things because someone new was working on the design. From the point of view of the business, it’s not the most efficient use of corporate memory, but from the point of view of the designer, it does mean that they get validation from users rather than some bloke in the office who says he’s seen it all before. There’s a balance to be struck there.

In summary…

When you do research, some findings will be dependent on context, and can change over time, or with different personas. Others are constant, based on how our brains work. The third category is in between, where the design pattern might change in the future, but it would take something significant for that to happen.

When you review old research, ask yourself which category those findings fall into. Take it back to principles of psychology and design. The answer doesn’t dictate that you follow exactly the same design as was tested previously, but the fundamental approach will be directed by it.