When do you stop referencing old research?

The issue

When I started out in UX there were some designs that I was convinced would work, some I wasn’t sure about, and some that I was convinced wouldn’t work. Simplistically, I think of it like this.

Old range of certainty

Then, as I learned more, and in particular did more research, I found that I wasn’t always right. Odd that. This had the effect of increasing my range of ‘might work’, as there are so often surprises. So the model changed to this.

New range of uncertainty

There were fewer designs that I thought definitively would or wouldn’t work (or would be better than an alternative), but I was more confident of those judgements.

As time went by (maybe there’s a song there…), people started to say things like ‘look, just because you did some research five years ago, doesn’t mean this design will fail now’. I don’t think it was my communication style that earned this response (but reference Discussing Design). So I tried to explain that there were some things that you discover through research that don’t change, or don’t change much, and that there were some things that were a result of context and time, and would be more likely to change over time. I tried to explain that if I was citing older research, it was only to illustrate a principle. Sometimes if I thought a design wouldn’t work it was on the basis that it broke a principle, and people assumed it was because of some old research.

I’ll give you some examples.

What gets old

Some years ago when we did usability testing on ba.com, it was common to find people who were surprised that you could book a hotel on the site. These days, it tends to be the other way round. People are used to the cross-sell. The fact of being able or not able to book a hotel on an airline website isn’t related to human psychology. It’s cultural and can be learned, so you can’t rely on old research.

Another more current and general example is the hamburger menu. It’s commonly used in mobile designs, although any research that I’ve seen says it’s most effective when used in conjunction with the word ‘menu’. Some designers maintain the style for desktop users. This makes no sense for a number of reasons. One is that since you have more space, you should expose the main navigation elements, rather than make users search, but also because you get many older users of desktop sites who have not learned the hamburger style. This is something that is likely to change over generations.

Another example that I found interesting was when we were doing some research in the US. In the checkout process the customer was offered the opportunity to get a credit card on which they would earn miles, and they could use it to pay for this flight. In our mockups we used the words ‘instant credit’. This was just after the sub-prime crash around 2008. Users reacted strongly against this phrase. When we changed the words to something more like, ‘get a card, earn miles, pay now’, the reaction was quite different. It was the same offer, just different positioning influenced by what was going on in the world at the time. It’s quite possible that at another time ‘instant credit’ would be more appealing.

What doesn’t get old

One example of what’s not going to change is the impact of grouping and separation, a fundamental principle of human cognition.

Here’s a snip from a search for ‘wheels’ on Cycle Surgery

Search for ‘Wheels’ on Cycle Surgery

Depending on screen size and resolution, a user could quite possibly scroll to a position like this. At a glance (which is all it should take), it looks like the upper price relates to the wheel below. They are spatially closer, and there is a lot of white space above the price. There are faint separating lines, which a user may or may not notice depending on the quality of their screen and eyesight. It’s confusing.

By contrast, here’s the same search result on Chain Reaction Cycles

Search for ‘Wheels’ on Chain Reaction Cycles

There is much clearer grouping of associated information, and separation from others.

Even better is Argos, using a card style.

Argos search result

So, to bring it back to old research. If I were to look at the page of results from Cycle Surgery, I’d say that at the very least it could be improved, not because of any research, but because it breaks the rules of how people look at things. I don’t need to do any research to figure that out. Items close together are related (we assume). Items with a strong separator are not related. Sometimes a poor design can cause conflict between these principles. Argos overcomes these issues.

To get more on those fundamental principles, have a look at some of the books I recommend on design, and behavioural economics.

Some things are in between

When people look at a list of flight choices, the vast majority start by trading off price and time. The cheap flight looks good, but you don’t want to get up a 4 am to catch it. 11 am is really convenient, but too expensive, so the 9 am is a reasonable compromise.

That’s the first calculation. After that, people will look at seat availability (can we all sit together), or is it the aircraft type that I want to fly on. Sometimes there’s a choice of airports, and so on. These things aren’t to do with the basics of human psychology, but they are deeply rooted, and are unlikely to change anytime soon, unless other factors become a lot more important for reasons I can’t currently think of. But it could happen.

I lost track of the number of times we had to reinvent the wheel to discover the same things because someone new was working on the design. From the point of view of the business, it’s not the most efficient use of corporate memory, but from the point of view of the designer, it does mean that they get validation from users rather than some bloke in the office who says he’s seen it all before. There’s a balance to be struck there.

In summary…

When you do research, some findings will be dependent on context, and can change over time, or with different personas. Others are constant, based on how our brains work. The third category is in between, where the design pattern might change in the future, but it would take something significant for that to happen.

When you review old research, ask yourself which category those findings fall into. Take it back to principles of psychology and design. The answer doesn’t dictate that you follow exactly the same design as was tested previously, but the fundamental approach will be directed by it.