When do you stop referencing old research?

The issue

When I started out in UX there were some designs that I was convinced would work, some I wasn’t sure about, and some that I was convinced wouldn’t work. Simplistically, I think of it like this.

Old range of certainty

Then, as I learned more, and in particular did more research, I found that I wasn’t always right. Odd that. This had the effect of increasing my range of ‘might work’, as there are so often surprises. So the model changed to this.

New range of uncertainty

There were fewer designs that I thought definitively would or wouldn’t work (or would be better than an alternative), but I was more confident of those judgements.

As time went by (maybe there’s a song there…), people started to say things like ‘look, just because you did some research five years ago, doesn’t mean this design will fail now’. I don’t think it was my communication style that earned this response (but reference Discussing Design). So I tried to explain that there were some things that you discover through research that don’t change, or don’t change much, and that there were some things that were a result of context and time, and would be more likely to change over time. I tried to explain that if I was citing older research, it was only to illustrate a principle. Sometimes if I thought a design wouldn’t work it was on the basis that it broke a principle, and people assumed it was because of some old research.

I’ll give you some examples.

What gets old

Some years ago when we did usability testing on ba.com, it was common to find people who were surprised that you could book a hotel on the site. These days, it tends to be the other way round. People are used to the cross-sell. The fact of being able or not able to book a hotel on an airline website isn’t related to human psychology. It’s cultural and can be learned, so you can’t rely on old research.

Another more current and general example is the hamburger menu. It’s commonly used in mobile designs, although any research that I’ve seen says it’s most effective when used in conjunction with the word ‘menu’. Some designers maintain the style for desktop users. This makes no sense for a number of reasons. One is that since you have more space, you should expose the main navigation elements, rather than make users search, but also because you get many older users of desktop sites who have not learned the hamburger style. This is something that is likely to change over generations.

Another example that I found interesting was when we were doing some research in the US. In the checkout process the customer was offered the opportunity to get a credit card on which they would earn miles, and they could use it to pay for this flight. In our mockups we used the words ‘instant credit’. This was just after the sub-prime crash around 2008. Users reacted strongly against this phrase. When we changed the words to something more like, ‘get a card, earn miles, pay now’, the reaction was quite different. It was the same offer, just different positioning influenced by what was going on in the world at the time. It’s quite possible that at another time ‘instant credit’ would be more appealing.

What doesn’t get old

One example of what’s not going to change is the impact of grouping and separation, a fundamental principle of human cognition.

Here’s a snip from a search for ‘wheels’ on Cycle Surgery

Search for ‘Wheels’ on Cycle Surgery

Depending on screen size and resolution, a user could quite possibly scroll to a position like this. At a glance (which is all it should take), it looks like the upper price relates to the wheel below. They are spatially closer, and there is a lot of white space above the price. There are faint separating lines, which a user may or may not notice depending on the quality of their screen and eyesight. It’s confusing.

By contrast, here’s the same search result on Chain Reaction Cycles

Search for ‘Wheels’ on Chain Reaction Cycles

There is much clearer grouping of associated information, and separation from others.

Even better is Argos, using a card style.

Argos search result

So, to bring it back to old research. If I were to look at the page of results from Cycle Surgery, I’d say that at the very least it could be improved, not because of any research, but because it breaks the rules of how people look at things. I don’t need to do any research to figure that out. Items close together are related (we assume). Items with a strong separator are not related. Sometimes a poor design can cause conflict between these principles. Argos overcomes these issues.

To get more on those fundamental principles, have a look at some of the books I recommend on design, and behavioural economics.

Some things are in between

When people look at a list of flight choices, the vast majority start by trading off price and time. The cheap flight looks good, but you don’t want to get up a 4 am to catch it. 11 am is really convenient, but too expensive, so the 9 am is a reasonable compromise.

That’s the first calculation. After that, people will look at seat availability (can we all sit together), or is it the aircraft type that I want to fly on. Sometimes there’s a choice of airports, and so on. These things aren’t to do with the basics of human psychology, but they are deeply rooted, and are unlikely to change anytime soon, unless other factors become a lot more important for reasons I can’t currently think of. But it could happen.

I lost track of the number of times we had to reinvent the wheel to discover the same things because someone new was working on the design. From the point of view of the business, it’s not the most efficient use of corporate memory, but from the point of view of the designer, it does mean that they get validation from users rather than some bloke in the office who says he’s seen it all before. There’s a balance to be struck there.

In summary…

When you do research, some findings will be dependent on context, and can change over time, or with different personas. Others are constant, based on how our brains work. The third category is in between, where the design pattern might change in the future, but it would take something significant for that to happen.

When you review old research, ask yourself which category those findings fall into. Take it back to principles of psychology and design. The answer doesn’t dictate that you follow exactly the same design as was tested previously, but the fundamental approach will be directed by it.

Sometimes, you need to design something really odd

I’m usually a little cautious when critiquing sites where I’m not familiar with the design rationale. I’ve often read a critique of a site that I’ve worked on, where the issues raised make sense from the point of view of that individual, but wouldn’t result in us changing anything as there was actually a good reason for the design they didn’t like. Usually, it related to the needs or behaviours of other personas that they hadn’t thought about, or an overriding commercial imperative.

Here’s an example.

In the checkout process on British Airways’ ba.com, the customer is asked for a phone number. Here’s the relevant bit.

Provide phone number on ba.com

It was often pointed out to us that we should have put the radio buttons above the number input field. An alternative would have been to allow the customer to put in a number, and then let them tell us whether it was a landline or mobile number.

In this case, if you hit ‘No’, then you see this.

Result of saying ‘I don’t have a mobile’

The reason the fields are in this order is that it was quite likely that no use at all would be made of a landline number. In a real emergency, or if you were high value, then maybe.

Even if we’d suggested strongly that it was in the customer’s own best interest to provideĀ  mobile number, some people still wouldn’t, for their own reasons. By doing it this way, people would see the request for the mobile number, see it was mandatory (so far), and maybe be a little pissed off, but would give a number. Many wouldn’t notice the ‘No’ option. Even if they saw the ‘No’ after typing in a mobile number, people being lazy, they’d just leave it.

If there were customers who really didn’t have a mobile, they had put so much effort into getting to this point that they would hunt around for a way round this problem, and spot the next bit.

Whilst we never A/B tested this one, I’m sure that the end result was that we collected many more mobile numbers than we would otherwise have done, and that we didn’t lose significant revenue.

Don’t default to ‘please select’ in drop-downs. Only use defaults when applicable to most users.

Here’s the name field entry from British Airways’ Executive Club registration form.

Name fields for BA’s Executive Club form

Here’s the equivalent for Virgin Atlantic

Name fields for Virgin Atlantic registration

Note that the drop-down for ‘title’ is blank for BA, but has ‘please select’ for VA. Users don’t need to be told to ‘please select’ if there’s a field that’s empty. There may have been a day when people were not used to drop-downs, and didn’t know what to do with them, but that’s not the case any more, for the vast majority of users.

Even more importantly though, users scan forms looking for empty fields. As people, we don’t methodically work through all the field names, picking them out from all the other clutter on the page. We take the easy lazy route. Which fields are blank? We don’t do this consciously, but we’re surprised when we hit the ‘enter’ button, and get an error. I have observed this often enough in usability testing. Even worse on the VA form, I have no idea why they would duplicate the field names in the input fields.

Further down on the BA page, we find a couple of fields that have been defaulted. The ‘country of residence’ and ‘language’ fields.

Defaulted drop-downs

In this case, it’s reasonable to default, as on entering the site the user has had to accept or change their country and language settings. The vast majority of users won’t change these fields, and so it would just be annoying for them to have to apply a setting that they’ve already done. Moreover, anyone who happens to want to change these entries, is likely to notice the disconnect with what they want. Again, it can be an unconscious thing.

Just by way of another example, here are some fields from the UK organ donor site. Right in the middle of a lengthy form, we find ‘please select’ again. The user just wants to get the thing done, and this sort of thing in the middle of a form increases the risk of errors.

Organ donor form

Check your forms. Are you leaving fields blank when they should be, and thinking intelligently about what to default? You should also be looking at your form analytics, and seeing which fields cause the most errors, so you know what to fix.

Big ‘look at me’ messages don’t work if they are not under the relevant heading

I’ve been looking at the job sites of various companies. I searched for ‘UX’ on the Royal Mail site, and got the following result.

Royal Mail job search results

I looked at the list of results, and, as has happened on other sites, found myself wondering what these results had to do with ‘UX’. I then noticed the big message above the results, saying there was nothing to match my search criteria. I’ve seen this in user testing.

The user is focused on their task. My task is to look at search results, not to look at headings or other stuff on a page. If you really want to stand the best chance of getting someone to notice content that is not on their main path, use red.

Whether it’s a list of results for flights, shoes, mp3 players, or dead Roman emperors, the natural tendency is, not surprisingly, to look at the results. Use red, or position the content beneath the results heading if you want people to notice other content.