It’s not the analysis, it’s asking the right question that counts.

There are many people who can ask questions and get answers by analysing a set of numbers or by interviewing the users of a product. There are fewer people who know what the right questions are that will deliver a deeper level of actionable insight.

Identify the right questions

I recently read a book called ‘Everybody lies’ by Seth Stephens-Davidowitz. Most of it is a fascinating insight into what lessons can be learned from a really intelligent analysis of Google searches. The contention is that whilst people modify their answers in interviews and on questionnaires to put themselves in a good light they tell the truth in their Google searches. Here’s a quote:-

Early in the primaries, Nate Silver famously claimed that there was virtually no chance that Trump would win. As the primaries progressed and it became increasingly clear that Trump had widespread support, Silver decided to look at the data to see if he could understand what was going on. How could Trump possibly be doing so well? Silver noticed that the areas where Trump performed best made for an odd map. Trump performed well in parts of the Northeast and industrial Midwest, as well as the South. He performed notably worse out West. Silver looked for variables to try to explain this map. Was it unemployment? Was it religion? Was it gun ownership? Was it rates of immigration? Was it opposition to Obama? Silver found that the single factor that best correlated with Donald Trump’s support in the Republican primaries was that measure I had discovered four years earlier. Areas that supported Trump in the largest numbers were those that made the most Google searches for “nigger.”

If someone is racist, likes extreme porn, wants to find out how to make a bomb etc they often won’t be keen to immediately disclose these things to the first researcher who asks them – but they will be honest in their Google search and that dataset can be mined. By looking at searches on ‘where to vote’ or ‘how to vote’ a more accurate prediction was made of voter turnout in specific geographies. In areas in the US where abortion has become harder to access there is a spike in searches for how to do your own abortion.

Here’s a Guardian article on the book

One thing that’s clear from the book is that the insights didn’t just leap out from the massive dataset. Asking the right questions was critical.

In business I’ve met many excellent competent data analysts who can churn out impressive charts at a great rate. Many of those charts have been completely useless in business terms. If you find someone who understands the business well enough to be able to churn out meaningful in-depth insights then you need to keep them.

Whilst A/B tests are often valuable they are by definition binary. Does the blue or the green button work best? What if a red one would be better? Multivariate testing lets you try multiple variables at the same time but then you need more time and volume to reach statistical significance. So knowing enough to ask the right question in the first place based on previous knowledge and experience can make a difference.

One of the difficulties I find with A/B tests is that everything except the variables being tested tend to be averaged in the analysis. Maybe the test shows that the blue button works best, but hidden in the data is the fact that blue works best with frequent users and green works best with infrequent users. I recall a test done many years ago by a bank who found (once they looked) that the effective colour was different in the morning from the evening. Again, these things won’t just jump out from the data – someone has to think to ask the question.

This doesn’t just apply to data. I’ve written a previous blog post on common errors in survey design. I recall one time as well when we were interviewing customers in Germany on the design of the flight selling system on ba.com. One interviewee was indicating that he thought that design A was better for a particular page than design B. My colleagues seemed to take this at face value but I wasn’t convinced – there was something that wasn’t right. In my view it was obvious that B was better and I thought that we just weren’t asking the right questions. At the end of the interview I went in to follow up. Did he prefer A or B? A was the answer. Which was better? A was the answer. Which one should we implement? A was the answer. Which one was easier to use. B! It turned out that B was easier to use but he liked the look of A. So the action resulting that we nearly didn’t get was to maintain the usability of B and combine it with the visual appeal of A. It may seem obvious in retrospect but it was a good lesson in being clear about the difference between someone ‘liking’ a design (which really doesn’t mean much), or preferring the colours, or finding one easier to use etc. What question is it that you actually want an answer to?

There is currently a debate raging on whether screen time – and how much of it – has adverse effects on children. More data is being brought to bear over the many opinions that are freely available, yet there is still no consensus. The Oxford Internet Institute has recently released a study that found that screen time had little impact on ‘teen well-being’. And the World Health Organisation came out with a report that children under two should have no sedentary screen time at all. As The Verge points out though …the guidelines are less about the risks of screen time itself, and more about the advantages of spending time doing pretty much anything else.

In a recent edition of the BBCs Tech Tent podcast the most intelligent comment I heard from a guest on the show in relation to this issue is ‘I’m not sure we’ve asked the right questions yet.’

Minimise errors in a selectable list by placing similar choices together

When paying by card for an online purchase some sites recognise the type of card from the number. Others ask you what type of card you are using. Here’s a typical dropdown menu asking the customer to specify the type of card.

Dropdown list of credit/debit cards

There’s no apparent order to the list of items – it seems that someone doesn’t really think it matters. However, for those who have a Mastercard credit card there are two issues.

Firstly, since users typically have the attention span of a gnat some people are likely to just register the word ‘Mastercard’ on the first line and select it, resulting in an error on submission.

Secondly, whilst debit cards have the word ‘debit’, credit cards do not. This potentially leads to some ambiguity as to whether ‘Mastercard’ on its own refers to the credit card. It’s left to the customer to make that assumption. Note also that for the debit card it’s spelled ‘Mastercard’ and for the alternative it’s MasterCard. That’s just sloppy and will again cause some people to wonder if it’s significant.

In such a list the debit and credit version should be consecutive e.g.

  • Mastercard debit
  • Mastercard credit
  • Visa debit
  • Visa credit
  • Visa electron
  • Solo
  • etc….

Doing it this way makes it quite clear what each card is and also maximises the chance that the customer will see that there is more than one option for some card types.

This principle applies to any list that users need to choose from and is relevant to surveys. I once filled in a survey about how I had paid for my car. One of the early options was a complex description of a contract which looked like the right choice. I nearly selected it and many would have, but I scanned through the rest of the list and found another very similar description further down that was actually the correct one. Had the two options been consecutive in the list there would be more chance that users would spot that the first likely option they come across is not necessarily the right one. It would also have helped if the two options had been worded to highlight the difference.

The bottom line is that when presenting users with choices in a list

  • make the choices explicit
  • word each option to highlight differences
  • put options that could be confused with each other close together

The pros and cons of structured job interviews and competency-based questions

The interview

About ‘competency-based’ and ‘structure’

I’ve recruited a lot of people – mostly but not exclusively UX/UI designers and researchers. Over time, unsurprisingly, I’ve evolved my approach and whilst I didn’t start with them did adopt competency-based questions as the mainstay of my interviews.

Let’s make sure we’re all on the same page – when I talk about ‘competency-based’ I mean questions along the lines of ‘Can you think of a time when…’, and you ask when the candidate dealt with an uncooperative colleague, or did their best work etc.

The rationale behind such questions is that evidence of past behaviour is the best predictor of future behaviour, and that asking people about what they have actually done and getting them to be specific about it is better than asking what someone hypothetically might do in a given situation. It’s a good rationale.

Since I’ve been job hunting and have now been on the receiving end of such questions I’ve gained a more rounded view of the pros and cons of this approach. What it boils down to is this – interviewers can be more focused on following interview protocol than getting the best information from the candidate. It’s also tied in with having a structured interview. HR advisers quite rightly point out that if you want to be able to properly compare candidates then you need to be consistent in your approach and the questions you ask. It’s a scientific experiment.

When I first recruited people I had a chat and decided on not very clear criteria whether I thought they’d be any good. I actually got good feedback from my boss on the quality of the people that I took on so it wasn’t a disaster – and I’m still in touch with most of those people.

When I was subsequently presented with a structured discussion guide where I had to score the candidate on each section I was sceptical at first. However I did very quickly find it to be useful, especially where I was interviewing a number of people. As an aside I’ll point out that I would always interview with a colleague. This co-interviewer would usually be one of my team – it gave them the experience and development which they enjoyed and helped to validate my own impressions of the candidate. Having done the scoring on each candidate I was surprised how much it helped to clarify our thoughts.

Evolution

The thing is it’s not possible to control for all the variables of an interview in a way that may be possible in a scientific experiment. People react differently to questions and to context. I started off asking all the questions in the guide as they were written, and in that order. I quickly realised that caused some problems. The questions that were confusing or irrelevant could be re-written or dropped, which was done. But sometimes in answering a question a candidate would end up covering some or all of a later question without knowing it, so when it came to that subsequent question it wouldn’t make much sense just to ask it straight. So we might say something like ‘Apart from the thing you just told us about, what else did you do about x?’

The other thing was that people get nervous in interviews. When they do they get tunnel vision and their thinking closes down. We did what we could to try to make them feel comfortable but that only works so far. So sometimes we’d ask ‘Tell us about a time when…’ and the candidate would answer a completely different question, or rabbit on for ages getting mired in detail. The strict version of the protocol says let them talk, say thanks and move on to the next question. That’s what I did at first but very quickly realised that it wasn’t helping me or the candidate.

I started to give some nudges and sometimes just stopped the candidate outright and tried to re-focus them. I was interested in an answer to the question, not how well they coped with an artificial interview. I would prompt the candidate to ‘tell me a bit more about that’, or ‘that’s not quite what I’m getting at, is there a different example you can think of’.

On the receiving end

It’s been quite instructive being on the receiving end of this process. There’s a risk that the application process (see my post on that Internal and recruitment applications need as much UX as ecommerce) and interview style become major filters in their own right, rather than the candidate’s ability to do the job.

I’ve found some of the competency-based questions quite hard. To suddenly come up with an example of experience from a long work history that meets my (and the interviewer’s) understanding of what is actually being asked has been challenging at times. On one occasion I couldn’t really understand what the difference was between the questions I was being asked, and there was no guidance. So it’s not surprising that I waffled and the interviewers didn’t really get to hear what I was capable of.

I have also been asked to ‘Think of a time when…’, when the straight answer is ‘I can’t, because it never happened’. What I do is try to be honest about it but think of an analagous situation that might still score me a point or two. When I was interviewing people and that happened I would then fall back on ‘Ok, I understand that’s a situation you’ve never been in, so what do you think you would do if it occurred?’ At least then I find out if the candidate understood the context and appropriate actions rather than just giving him or her a low score and moving on.

Sometimes I’ll think of the ‘right’ answer to a question just as I walk out of the interview room. You could argue that the ability to think on your feet is an essential attribute of the job and it would be a fair point, but the context and nature of what you’re being asked about is different.

Introverts are typically reflective. If you’re having a meeting at work you’ll get the best input from the introverts if you let them know what you want from them in advance. Otherwise they’ll let you know after (or, often, not) that they’ve thought of something they should have said in the meeting. Interviews are no different. The format of ‘give me an answer now to an important question’ discriminates against introverts. I’m an introvert.

Top tips for interviewers

So here’s where I’ve got to in my thinking around job interviews.

  • Firstly and most important, as an interviewer never lose sight of why you’re there. You are trying to find the best candidate for the job, not the person who is best at interviews.
  • Use an interview guide as a guide not a script. The topics covered are what you need to find out about, and your job is to ask questions, prompt and direct the candidate so that you do find out. Don’t just stand back while someone digs their own grave.
  • Consider letting the candidate know in advance what the questions are – or at least the subject areas that are going to be covered.
  • Overall be flexible, subtle and nuanced in your questions. Try to understand the person in front of you rather than blindly following process.

Eight fundamental principles to underpin UX/UI design

The eight principles

When critiquing or explaining a design it can be useful to reference principles – they provide a North Star that you can fall back on when short of ideas and give a level of consistency to your work. They are not generally a tool for innovation and you can still produce an unusable design when trying to follow them, although it’s less likely if you do follow principles. A colleague who once asked (quite reasonably) for our design principles to be made clear thought that they could provide an algorithm to generate a design or be definitive about whether a given design was the ‘right’ one. It doesn’t work like that.

These are based on some work we did at British Airways with a big shout out to my colleague Lee Head who for many years was the lead (indeed the only) UI designer working on ba.com. Kudos. Lee was instrumental in pulling it all together.

The eight principles are

  • Design for customers
  • Design with knowledge
  • Focus on the task
  • Make it clear and simple
  • Make use of conventions
  • Be consistent not prescriptive
  • Avoid errors
  • Use design dimensions

I’m not claiming these are the only principles that you’ll need or that they can’t be improved but they are a useful starting point.

Design for users

For services to be successful they must be designed for the user and their goals. Take satisfaction and emotion into account.

This should be an obvious starting point. Personas can be a useful way of describing the target audience – and there maybe multiple personas. What do your users know or not know about your product? What devices are they using? What are their physical capabilities? How can you personalise the experience and generate positive emotions whilst avoiding negative ones?

Collate all the information you have on users and decide which are the relevant (and prioritised) dimensions. Then let than influence design.

Design with knowledge

Gather information that could improve design solutions. The more
knowledge we have the more likely we are to find a better design.

Start with the objective of the design. It’s remarkable how often you can ask a designer ‘what is this page for?’ and get a blank look. What’s the overall objective of the process, of the page, of the component?

Knowledge of the user and the objective should then lead to content. What content is needed to satisfy the objective?

‘Data driven design’ is an oft-used phrase, and rightly so. Collate existing metrics and research on user behaviour which shed light on what users want and how they want it. These can feed into success metrics. The knowledge doesn’t just have to be from your own organisation – competitor reviews play in here also.

Test early and often to ensure that you know you are doing the right thing, and keep measuring once live.

Focus on the task

Don’t make things hard for users. Break down tasks into manageable chunks and simplify the design.

Have a clear idea of what’s important. Secondary and edge cases may generate a need for additional content and a modified design but shouldn’t distract from the main (user) goal.

User choices and how to progress should be clear and standout, and the consequences of actions should be predictable. Too often I’ve sat frowning at a screen trying to figure out what I’m supposed to do next or how to make a selection. Real-world buttons stand out physically and they should stand out from a design. Flat design is a usability disaster.

On many sites you’ll see paragraphs of text followed by ‘Click here for more information’. Users scan links so they should be stand-alone and ‘click here’ just forces them to read more than is necessary. It can also be problematic for accessibility. Instead use ‘More information about product x’. Don’t embed the links in the middle of text or if you do repeat them away from the text so they stand out.

Only ask for information when it’s needed and don’t ask for it if it’s not going to be used. Provide information only where it’s needed and useful – and in the eyeline, not off to one side. Provide negative as well as positive information. Users need to know what they can’t do just as much as what they can do.

Maintain a ‘scent’ so that users know that they are progressing in the right direction.

I’ve found that for sequential processes such as a checkout, just having the word ‘continue’ on the continue button works better than e.g. ‘go to payment’ or ‘review basket’. It’s simpler, requires no interpretation and is quite sufficient. I would only differently label the final button e.g. ‘Pay now’. This principle may need to be modified if there are multiple possible ‘continue’ paths.

Make it clear and simple

Nothing destroys a considered design approach more than poor layout. A cluttered design is not only hard to use, it reduces satisfaction. We should strive for a clear and simple layout that makes the experience easier and more delightful.

Every element on a screen takes attention away from other things and so has to justify its own existence. Clarity of communication has to be a prime objective. So often you’ll see a useless introductory line or paragraph on a product page along the lines of ‘On this page you’ll find the best products of this type anywhere in the world and our goal is to serve you. See below for what we have to offer’. No one reads that. It’s clutter. Users go straight to the useful content.

The purpose and content of a page should be clear at a glance. Maintain an appropriate heading hierarchy, and don’t use heading styles purely to style the text, as that messes up accessibility.

When you include keywords in links ensure that that keyword is immediately visible when the user follows it so they know they’ve arrived at the right place.

Reduce visual noise – don’t have a variety of fonts, colours, shapes, carousels etc for the sake of it. For gods sake don’t centre long paragraphs of text.

Make use of conventions

Using design conventions can be a useful method of simplifying a design. Use them unless there is a very good reason not to.

You might think that a quirky navigation and unusual interactions reflect your personality or brand – but it would turn users off. The likes of Google, Microsoft, Facebook and Apple can get away with introducing new standard elements because of their reach. Most brands don’t have that leverage. Users just want to get their stuff done. They aren’t there to admire your design.

There’s a reason for radio buttons, checkboxes and dropdowns all existing. Understand what they are good for and what they aren’t good for. Use them appropriately. You don’t need new methods. Apply this practice to all design elements.

Innovate only if you can prove that users get it and it adds value.

Be consistent, not prescriptive

Develop a pattern/component library and use it consistently. Vary it only if the context requires it.

This needs collaboration between design and engineering. A proliferation of styles either visually or technically hurts the business and the user experience. Have a process and a decision maker responsible for allowing (or not) exceptions to the standard design.

Somewhat ironically, unless a design variation is large, designers will usually spot specific variations more than users, but nevertheless those variations can cause confusion and lead to a poorer impression of your brand.

Consistency applies to terminology as much as visual design.

Avoid errors

Good design minimises the risk of user error. When it can’t be prevented, ensure customers understand what to do to achieve success.

It drives me crazy when I use a form or a process with obscure instructions, instructions in the wrong place, instructions that aren’t needed, poor labelling, unintuitive order of fields and a whole litany of other issues. Then you finally press the button, wait for a long time, and then get an error message that could have been checked inline.

If you do present an error message say (as far as you can)

  • what has gone wrong
  • why it’s gone wrong
  • what the user can do about it

Use design dimensions

There are slightly varying lists and terminology, but much common agreement on the dimensions of design. Using them intelligently will greatly enhance the user experience.

  • Contrast/similarity
  • Positive/negative space
  • Line
  • Shape
  • Form
  • Colour (hue)
  • Colour value (tone)
  • Direction
  • Size
  • Texture
  • Grouping/separation (space)
  • Motion

Take grouping and separation as an example – many times on ecommerce sites I’ve been confused by the prices in a grid layout being closer to another item than the item it refers to.

You can use colour as a guide to product type e.g. electronics are all blue, clothing are all white – so long as it’s not a critical cue that would be missed by those of us who are colour blind.

I very much like Google’s Material Design as it does so much that’s right. The ‘card’ design does grouping and separation well and motion is used to help with usability.

Finally

It’s quite possible that someone else would come up with a different set of principles with different headings or structure – but I’m sure there would be considerable overlap.

The list above isn’t the final answer – it’s just intended to be useful.

The tyranny of design. Why you need to focus on objective outcomes in interface design.

In-flight entertainment – a story

Some years ago at British Airways there was a project to design an interface for a new in-flight entertainment system. My then-colleague and friend Mike Lock was project managing. Mike is the real godfather of digital usability at British Airways. Whilst I set up and developed the UX, Design and Research team, I did so off the back of what Mike had already done to sow the seeds of awareness and need. He says I’m too modest, but I call it as I see it.

British Airways’ in-flight entertainment screen

For the design of the IFE interface a small number of companies had been asked to present a concept. At the end of the presentations it was clear to Mike that only company X had grasped the issues around information architecture and navigation, although the visual design was a bit off. He was shocked to hear when going round the table that no-one else favoured company X. All the other (more senior) stakeholders in the room went for a more on-brand visual design. Mike felt overwhelmed – like all the big guns were pointed at him and he only had a cardboard shield to deflect the blows.

‘But’, said Mike, ‘the one you all like does look nice, but no-one will be able to use it. It’s not a design where the usability issues can be fixed – it needs throwing away and starting again.’ Sceptical faces were all he saw, but he kept going. ‘Although the design from company X isn’t quite on brand it is usable, and we can fix the design elements relatively easily.’ The meeting broke up with the stakeholders thinking that Mike didn’t get it, and there was no way they were going with company X. But Mike still wasn’t giving up.

That meeting was on a Friday and the group was going to reconvene on Monday to decide what to do. Mike was desperate to find a way to influence the decision towards the one he knew to be correct. Over the weekend he took the images from the presentations and turned them into clickable prototypes using Powerpoint. It was all that was available in those days. He then videod his mum ‘using’ the two interfaces. This was in the days of camcorders that recorded on tape. She couldn’t use the pretty design but got on pretty well with the from company X.

On Monday Mike played the video to the group. Ultimately he won the argument, company X got the contract and re-worked the design, and it was implemented on many aircraft. It left a bad taste in Mike’s mouth though. He had to work too hard to prove a simple point and taken a load of senior shit for it.

The lessons

Something similar to that scenario has played out again and again over time. There are lessons to be learned.

Firstly you can’t assume that your stakeholders get what it takes for an interface to be usable. It’s one of those contradictions in life – we’ll all swear at an interface that frustrates us, but (some of us) would still build an interface for our own companies that incorporate the same frustrations. It’s human nature – we’re often not aware of the causes of our emotions, and most people don’t analyse exactly what it is that they don’t like about a website or an app. They just ‘know’ that it doesn’t work for them.

Secondly, if you have business people who are (relatively) sane and rational it should be possible to influence their perception of effective design. They do actually want it to work. There are different ways of doing this, and sometimes it depends on the person as to what the best way is. Some people like to review a spreadsheet of analytics following a multivariate test, but usually the best way of snagging a stakeholder is for them to see a real customer being unable to use an interface that that stakeholder thought was ok. It hits at an emotional level that has impact. Get them to watch live research in person – but if you can’t, then show the video.  If you can, involve them in the setup of the research so they can’t quibble with the methodology afterwards.

Thirdly, agencies vary in their expertise. Some are better at UI, some excel in IA, some at ecommerce. It’s critical when engaging an agency to make sure they have the expertise to do what the client wants, and to be clear about what success looks like. I wrote another post on why agency/client engagements often don’t work.

The landscape today

Designing for mobile forces the designer to ruthlessly prioritise content and produce a compact design. Users focus more because there’s less to look at, and identify more issues with confusing and irrelevant copy which they would just ignore on a desktop screen. More people are using phones more of the time, but desktop isn’t dead yet. I don’t know if it’s a reaction to compact phone design, but desktop design seems to have gone the other way.

A short while ago I was talking to a senior business manager about her company’s desktop site. It had been designed by an agency and she had complained that there was too much scrolling. The agency had ‘explained’ that it was ‘modern design’. The business manager was right, there was too much scrolling.

It seems to be the vogue to have enormous images, lots of white space, and huge font sizes. If you have an ‘artistic’ site or a particular brand image all of this might be appropriate. However, for most ecommerce or informational sites it isn’t appropriate. Customers want to get in, do their stuff, and get out. They want the experience to be a perfect combination of ease, pleasure, succinctness, entertainment, effectiveness etc. And yes, that does include a site that’s pleasant to look at. But if they have to repeatedly scroll just to find out what’s on offer, or to find the information relevant to them, then it’s not achieving their goals and it’s not helping the company to win their business. There are some sites where I’ll go to read an article and I almost feel like I’ve been punched in the face by the huge font that’s difficult and unpleasant to read.

My perception is that many, if not most agency sites are culprits of implementing a triumph of design over communication. If they built their clients’ sites like they built their own then their clients would go out of business.

The bottom line

I’ve had excruciating debates with got-religion UI designers who can’t bear to see the excellence of their design debased and compromised in pursuit of mere money. They don’t disagree that the design impacts usability – they just think making the design right is more important. I’ve told them I’m not prepared to explain to the CEO that we chose to make lower profits so that we could adhere to the designer’s idea of a nice-looking interface.

I want to be clear that I’m not at all putting down UI. It’s absolutely essential. It’s just not the reason why we do all this work.

The reason for the existence of UX, UI, research, interaction design, information architecture etc etc is to be effective in the mission of the organisation paying for the work to be done, which is usually to make money and/or to communicate. We need to focus on the goals and objectives for the interface, where ‘on brand’ is a primary goal and ‘nice looking’ is a secondary goal.

It all needs to come together. Figuring out what works needs to be based on research, on facts – if anyone can stomach facts in a post-fact world.