The pros and cons of structured job interviews and competency-based questions

The interview

About ‘competency-based’ and ‘structure’

I’ve recruited a lot of people – mostly but not exclusively UX/UI designers and researchers. Over time, unsurprisingly, I’ve evolved my approach and whilst I didn’t start with them, did adopt competency-based questions as the mainstay of my interviews.

Let’s make sure we’re all on the same page – when I talk about ‘competency-based’ I mean questions along the lines of ‘Can you think of a time when…’, and then you ask for a time when the candidate dealt with an uncooperative colleague, or did their best work etc.

The rationale behind such questions is that evidence of past behaviour is the best predictor of future behaviour, and that asking people about what they have actually done and getting them to be specific about it is better than asking what someone hypothetically might do in a given situation. It’s a good rationale.

Since I’ve been job hunting and have now been on the receiving end of such questions I’ve gained a more rounded view of the pros and cons of this approach. What it boils down to is this – interviewers can be more focused on following interview protocol than getting the best information from the candidate. It’s also tied in with having a structured interview. HR advisers quite rightly point out that if you want to be able to properly compare candidates then you need to be consistent in your approach and the questions you ask. It’s a scientific experiment.

When I first recruited people I had a chat and decided on not very clear criteria whether I thought they’d be any good. I actually got good feedback from my boss on the quality of the people that I took on so it wasn’t a disaster – and I’m still in touch with most of those people.

When I was subsequently presented with a structured discussion guide, where I had to score the candidate on each section, I was sceptical at first. However, I did very quickly find it to be useful, especially where I was interviewing a number of people. As an aside I’ll point out that I would always interview with a colleague. This co-interviewer would usually be one of my team – it gave them the experience and development which they enjoyed, and helped to validate my own impressions of the candidate. Having done the scoring on each candidate I was surprised how much it helped to clarify our thoughts.

Evolution

The thing is, it’s not possible to control for all the variables of an interview in a way that may be possible in a scientific experiment. People react differently to questions and to context. I started off asking all the questions in the guide as they were written, and in that order. I quickly realised that caused some problems. The questions that were confusing or irrelevant could be re-written or dropped, which was done. But sometimes in answering a question a candidate would end up covering some or all of a later question without knowing it, so when it came to that subsequent question it wouldn’t make much sense just to ask it straight. So we might say something like ‘Apart from the thing you just told us about, what else did you do about x?’

The other thing was that people get nervous in interviews. When they do they get tunnel vision and their thinking closes down. We did what we could to try to make them feel comfortable but that only works so far. So sometimes we’d ask ‘Tell us about a time when…’ and the candidate would answer a completely different question, or rabbit on for ages getting mired in detail. The strict version of the protocol says let them talk, say thanks, and move on to the next question. That’s what I did at first, but very quickly realised that it wasn’t helping me or the candidate.

I started to give some nudges, and sometimes just stopped the candidate outright and tried to re-focus them. I was interested in an answer to the question, not how well they coped with an artificial interview. I would prompt the candidate to ‘tell me a bit more about that’, or ‘that’s not quite what I’m getting at, is there a different example you can think of’.

On the receiving end

It’s been quite instructive being on the receiving end of this process. There’s a risk that the application process (see my post on that Internal and recruitment applications need as much UX as ecommerce) and interview style become major filters in their own right, rather than the candidate’s ability to do the job.

I’ve found some of the competency-based questions quite hard. To suddenly come up with an example of experience from a long work history that meets my understanding of what is actually being asked has been challenging at times. On one occasion I couldn’t really understand what the difference was between the questions I was being asked, and there was no guidance. So it’s not surprising that I waffled, and the interviewers didn’t really get to hear what I was capable of.

I have also been asked to ‘Think of a time when…’, when the straight answer is ‘I can’t, because it never happened’. What I do is try to be honest about it but think of an analagous situation that might still score me a point or two. When I was interviewing people and that happened I would then fall back on ‘Ok, I understand that’s a situation you’ve never been in, so what do you think you would do if it occurred?’ At least then I find out if the candidate understood the context and appropriate actions rather than just giving him or her a low score and moving on.

Sometimes I’ll think of the ‘right’ answer to a question just as I walk out of the interview room. You could argue that the ability to think on your feet is an essential attribute of the job, and it would be a fair point, but the context and nature of what you’re being asked about is different.

Introverts are typically reflective. If you’re having a meeting at work you’ll get the best input from the introverts if you let them know what you want from them in advance. Otherwise they’ll let you know after (or, often, not) that they’ve thought of something they should have said in the meeting. Interviews are no different. The format of ‘give me an answer now to an important question’ discriminates against introverts. I’m an introvert.

Top tips for interviewers

So here’s where I’ve got to in my thinking around job interviews.

  • Firstly and most important, as an interviewer never lose sight of why you’re there. You are trying to find the best candidate for the job, not the person who is best at interviews.
  • Use an interview guide as a guide, not a script. The topics covered are what you need to find out about, and your job is to ask questions, prompt and direct the candidate so that you do find out. Don’t just stand back while someone digs their own grave.
  • Consider letting the candidate know in advance what the questions are – or at least the subject areas that are going to be covered.
  • Overall be flexible, subtle and nuanced in your questions. Try to understand the person in front of you rather than blindly following process.

Lloyds needs to link its databases to avoid a bad customer experience

I’ve just had one of those experiences that sadly fails to surprise. I have a Lloyds Bank credit card, and in attempting to do an online transaction with it, was presented with the Click Safe authentication screen. This told me that I would be sent an SMS message with a code that I should enter on the next screen.

Lloyds credit card

The problem was that the code was going to be sent to my old mobile number. Several months ago when I changed my mobile number I had logged in to my card account and updated it. Maybe it had regressed? I’ve checked but my online account was up to date. Clearly Lloyds have separate databases holding customer information, and they don’t always talk to each other. It’s the sort of incredibly annoying thing that still happens.

My penalty for being in this situation was that I had to phone the euphemistically named Customer Services, never something I’ve looked forward to with Lloyds.

I had to speak to three people, which included a gap in between the second and third where I was told I would have to wait at least ten minutes, and I could phone back another time. I don’t think there’s ever been a time when I’ve called when they haven’t been ‘extremely busy’.

Of course, I had to explain the issue three times. All of the staff tried to be helpful and polite. The last chap had to take me through some laborious authentication questions, and then told me that he’d updated the system to authorise the transaction I was trying to make, and he’d updated my phone number. I should wait five minutes, then try the transaction again.

I waited more than five minutes, and tried again.

Transaction error message

I was told that the authorisation had been declined by the bank.

I waited, and tried again. Same result.

I phoned again, and although this time I check to use a number specifically for credit cards, I still had to be put through to ‘credit cards’ after explaining the problem.

The chap on the phone then explained to me that the problem was with the expiry date of the card. I guess that the error message couldn’t have told me that, as it would help someone who was trying to make a fraudulent transaction. I was using a password protection tool to auto-fill my card details, and although the information in the database is correct, it was appearing incorrectly on the payment form, and I hadn’t noticed. I still don’t know why that happens. I tried again with the correct information, and this time it worked.

There are a couple of take-outs.

Firstly, there’s no excuse for the lack of a joined-up view of the customer by Lloyds. When I updated my account online with my new phone number, it should automatically feed into whatever Click Safe uses.

Then there’s the inevitable transferring of calls, and the need to explain the problem multiple times. There has to be a better way.

It’s not all down to the business though. I did submit incorrect information when I tried again. I didn’t check, and if anyone had asked me, I would have insisted that I had entered the correct information. I know from experience on both sides of the fence that people will often swear blind that they did or didn’t do something online, only for the system logs to prove differently. People can look very sheepish when presented with the evidence, but that’s how people are. It’s not acting in bad faith. We’re fallible creatures who don’t always understand our own motives, and who don’t notice what actions we take. As far as possible, it’s the job of UX (and CX offline) to mitigate the impacts.

If you ask for ideas, make sure you feed back

Have you ever taken part in an idea-generation session, and heard nothing more about it? I would guess that applies to many of you. It’s a good way of turning people off, and generating a ‘what’s the point?’ attitude.

Generating ideas

There are great ideas lurking in many peoples’ heads that just need an audience. Some people are good at talking – but that doesn’t necessarily mean they have the best ideas. If you have an idea session, the extroverts will be clambering over each other to get their thoughts aired, whilst the introverts will be sitting at the back having a good think. It’s generally a good idea to let people know in advance if you want ideas.

There are a few situations that I think of in relation to idea generation. One is within the context of a project. Maybe you’re working on a design for a new interface, and want some critique of what you’ve done so far. An approach here is to stick up the designs on the walls, and leave pens and sticky notes around so that people can leave their thoughts. If you also post up a change log with revised designs, then those contributors can see what, if anything, became of their ideas. It provides them with reinforcement to want to contribute some more. It’s a good idea as well to list those ideas that won’t be implemented. It helps people to know that although their idea won’t be adopted, that it has nevertheless been thought about. It also helps them to understand more about the context of use, so that their subsequent suggestions can be more relevant.

Another idea-gathering context occurs when an organisation has time out, such as an away day or business unit forum. Maybe there are some business presentations, and then, to get more ‘engagement’, everyone is invited to come up with good ideas either singly or in groups, and then to present these back. There might be some voting for the best ideas, or management will gather them up for consideration.

I can’t say that I’ve seen a lot of value from these efforts. It does get people talking and thinking about things which is good, but generally the follow up has been lacking. Companies don’t typically have a lot of spare resource, in time or money or people, and so if new ideas are presented, then something else has to stop in order for them to be implemented. Some companies may be better than others at this, but it takes effort to review, compare, persuade, change course. And so the initial enthusiasm gets lost in the daily grind. For it to stand a better chance, one or more individuals should have responsibility for getting the ideas properly assessed – and it should be in their objectives. Those individuals could be at different levels in the organisation – they don’t have to be senior, and it can be a good development opportunity. If they are junior though, they should have a senior sponsor who takes the role seriously.

This lack of feedback and visible impact causes demotivation and cynicism over time. When I started at British Airways as a young developer, I enthusiastically joined in a Process Improvement Plan workshop. I remember well one of the old hands shrugging his shoulders, saying he’d seen it before, nothing ever happened, but at least he’d join in and help. I thought it was a poor attitude. Years later I found myself in the role of the old hand offering similar views, the result of having been through the mill many times.

Yet positive things can happen. When I was UX Manager at BA, a colleague and I went to the contact centres in the UK and the USA to get feedback. We wanted to know what the agents thought of ba.com. We wanted to know what we could improve so that customers wouldn’t be phoning them up with complaints or problems that the agents didn’t even know existed. The contact centre management set us up with different groups, and we sat round a table to talk about what the issues were. We had assumed that there would be some common themes, but were surprised at how diverse the problems were – different by geography and also the role of the agent. The one thing that was common across the groups was the sense of cynicism, that they’d all been through this process before, and nothing ever happened.

My colleague and I documented all of the issues – not just some – on a spreadsheet. We took it back to base, and sat down with the rest of the management team, and agreed who would be responsible for fixing which issues. We also agreed that for various reasons, some were unlikely to be fixed, and we recorded it all in the spreadsheet. As we made progress with fixes, we periodically shared the spreadsheet, warts and all, with the contact centre staff. The response was incredible. People were so happy that we’d actually really listened to them, that we were actually doing something about it, and that we were letting them know about it. Even the old hands got enthusiastic.

This isn’t rocket science.

  • If you want ideas, let people know in advance
  • Provide a forum or method that lets everyone have their say, not just the loudmouths
  • Ensure someone has responsibility for gathering and feeding back
  • Immediately feed back on what the ideas were (and make sure you don’t place your own interpretation on someone else’s idea)
  • Feed back on progress to appropriate timescales
  • If you’re not going to do anything with the ideas, don’t ask in the first place

IOT requires a broader skill set than traditional product management

Whilst user interfaces have by definition been based on hardware, the increasing ubiquity of the Internet of Things (IOT) will likely challenge product managers to develop a broader range of skills. Once, you had a computer, built, let’s say, by IBM. There needed to be an operating system for it, and a young Microsoft came along with DOS, which stood for Disk Operating System. It was before Windows. One company made the hardware, another made the software, and if something didn’t work, there was always a chance that they could say ‘it’s not our fault, it’s the other guys’.

IOT typically encompasses a combined hardware and software offering from one company. Even if parts of the whole are contracted out for the manufacturing, there’s going to be a lead marketing company which takes responsibility for all of it. And the product manager needs to know enough to be able to manage production, software development, UX, and to work with colleagues in marketing. All of that, as well as understanding the market, competitors, customer needs, measurement and feedback processes etc. If hardware and software are developed in-house, being able to manage those processes can sometimes be as challenging as outsourcing – sometimes more so, if, say, the internal IT department has its own budget and autonomy. It’s always critical when engaging with a supplier, internal or external, to agree the terms of engagement up front. Have arguments, have disagreements, but have them before you start, so that once you’re on the path, you have a contract or terms of reference that everyone has signed off on. Everyone has to be aligned to commercial and customer needs.

I have a dog.

My dog

The dog is one of the family, and we want to keep her safe. We have bought her more than one satellite tracker that attaches to her harness. If we lose her, there’s a SIM card in the tracker that connects to an app, and we can see where she is in relation to us, on a map. It’s a great idea.

To make it work, a lot of different elements have to come together. Someone has to have the idea. It can’t just be a good idea, it has to be a commercially viable idea. So there has to be an understanding of the potential market, barriers to entry from competitors (if it’s new), cost and timescales of production, distribution, quality, and product features. That’s the hardware. There are similar consideration for the software, but it’s a different skill set to understand the UX of the app from the features of the hardware, or the technical infrastructure or data sources that are needed. The original tracker we bought wasn’t natively waterproof, but the company had brought out a case for it. Many dogs love to go swimming, as ours does, and they are going to be caught in the rain, so a waterproof tracker has to be high on the list of requirements, but adds to the expense. There’s an equation to be worked out.

Ultimately, the hardware was good, from our point of view. The software was a different story though. It seems that many companies outsource software development not only to India, but also China, as was the case here. The default location for the dog was always in China, which completely threw me when I loaded it up. The whole interface was un-intuitive. If you’re in a panic because you’ve lost your dog, your vision and thinking will shut down, and you have to have something that is utterly simple to use.

The point is that, as ever, the end-to-end experience has to work for the customer. I messaged the company repeatedly about the problems with the software, including the bugs. They didn’t help much. I don’t think they had the skills, or maybe the resources. I wonder if they even owned the IP on the software, or were committed to using a developer thousands of miles away with whom they had poor communication. That’s speculation of course, but something a product manager should be thinking of. Eventually the company stopped supporting the device, leaving me with some useless and expensive hardware, and a useless annual subscription for a SIM card.

Marketing and sales need to understand the benefits to the customer, as is the case with any product. When I realised I had to replace the tracker, I researched the market. One of my requirements was that the tracker should roam across mobile networks. There’s no point having an O2 SIM if there’s only a Vodafone signal. It should work if there’s a signal from any network. If there’s no signal, then you’re out of luck, but at least maximise the chances. This requirement means that either you provide your won SIM, or the built-in one has to be able to roam. One product I looked at comes from Europe. I messaged the very friendly people there, but was either unable to clearly communicate what the question was that I was asking (will it roam across UK networks), or they didn’t know. So I bought a different, more expensive one.

Some ideas will originate with the hardware, such as, say, a thermostat or speakers, and someone will come up with an idea that will link them to the Internet, make them controllable remotely, integrate with other functions, which requires a software interface (that doesn’t have to be an app, it could be voice, for example, or both). Equally you could start with a fitness app, and have an idea for a hardware tracker. In each case, the manager has to extend their skill set beyond their traditional competence to ensure that end-to-end experience.

IOT will also integrate with the latest tech buzzwords – robotics, AI, big data, AR, VR, and alternative interfaces, such as gestures and, particularly, voice. Interface designers will need to be clear about the underlying principles of good usability that apply across different implementations, and then understand how to apply those principles for each case. The product manager must be able to assess and trust that the designer has done their job.

New technologies demand new skills, and the IOT is no different in that respect. It does though need  product managers/owners and their bosses to recognise what that range of skills is.

Why client+agency engagements are so often failures, but it’s a secret

Whilst I’ve never worked agency-side, I have a perspective from client-side on why client+creative agency engagements are so often not as productive as they should be. I’ve worked with many creative agencies – as well as other suppliers, and there are some common themes that I’ve identified. There are issues on both sides.

In this post, I’m generalising. I’m sure there are some great supplier managers, and some great agencies, that really make things work. I have seen moments of sanity impinge on this process, but not as often as one would expect.

Senior managers drive it

Taking on an agency is usually driven by senior management. That’s not a surprise in itself. They are the ones carrying the can for effective results, and they have the budgets to pay for the work. The problem is that they often don’t involve the workers, don’t always understand the actual problem they are trying to solve, don’t know how to set the right goals… erm, etc.

Whilst the worker bees are, to mix metaphors, beavering away, they find that all of a sudden  there’s this agency that’s been taken on to do something or other, and the worker bees may or may not be expected to engage in the exercise. It’s not always clear.

Often, whatever the deliverable is, it’s the worker bees who are expected to manage, operate, and update it once the engagement is over. But there’s been little engagement or input from the client bees, so they feel no emotional involvement with the solution, and may even think that it’s the wrong answer.

This isn’t just an issue that applies to agency engagement – too often senior management will think that they know enough to drive a solution without talking to the people on the ground.

At the end of the engagement, it’s clearly not in the interests of the commissioning manager to say ‘Well that didn’t work’. If they do, then they typically have to blame the agency. Unless you have someone who is remarkably non-defensive and open to learning, they aren’t going to admit that the terms of engagement and processes weren’t right from the start. The result is that the same mistakes are made next time.

Takeouts:

  • Owning managers should consult their own staff to shape an engagement
  • Client staff should be involved throughout the engagement at all levels

Agencies load up the client team

Particularly with larger agencies, there are going to be a cast of players whose roles are never really quite clear, but who charge high fees for each hour worked. There’s a relationship manager, a relationship director, a project manager, a creative lead, an interaction designer, etc etc, and when there’s a meeting some turn up, and some don’t, and the client doesn’t get a lot of say in it. I have serious trouble believing that all these roles are either needed, and need to be engaged at the level they are. It’s certainly not Lean, and it costs a lot of money.

Takeouts:

  • Agencies need to be transparent about roles
  • Agencies need to clearly justify roles and billable hours
  • Everyone client side should understand who’s in the meeting, and why, not just the owning manager

Objectives aren’t clear, and most aren’t measured

I think it would be fair to pay an agency on results (or at least, have an element of it). It’s likely some do that. But it’s hard if the client organisation isn’t clear itself about hard business objectives that need to be met, and the role the agency can play. Obviously there’s a difficulty for an agency if the client refuses to implement their recommendations, which is why the mixed compensation model may be best. It’ll depend on the brief.

Some time ago I was strongly dissatisfied with what a marketing agency were doing for us. I talked to one of our internal marketeers (who agreed with me), and who arranged a meeting at the agency offices for the entire team working on our account – about 12 people I think. We had a discussion. At one point I asked how they knew if what they were doing for us helped us meet commercial goals. There was a slight pause of astonishment that such a question should be asked, and then it was carefully explained to me that the people in the room were all experts in their roles, and there was a large amount of cumulative experience brought to bear on the work. That was it. No reference at all to measurement, testing, metrics, anything. And it seems that both sides were ok with that.

A particular issue that I’ve come across is that I’ve often seen agencies obsessing over details of visual design, whilst completely missing the point that the interaction design was poor. Without a clear focus on commercial goals, and the right balance of skills on the project, there’s a risk of ending up with a visually delightful, commercially useless solution.

As a business, you need to be able to define quality standards for the work that an agency does, be able to police those standards, and have a process that kicks in if they are not met. Take note, anyone who plans IT outsourcing.

Takeouts:

  • The client needs to be clear about the business goals for the engagement
  • Business goals should be written into the contract and impact remuneration
  • How to measure the business goals may need to be planned, and measure put in place, they may not already exist
  • Everyone on the project (client and agency) should know what the goals are

Working processes are unclear

So often that it hurts, a contract between client and agency would be agreed, and then the discussion would start on how it was going to be achieved. Typically, agencies wanted a brief, and then wanted to go away for a time, and come back with the answer, and be paid. What we wanted was to collaborate. Have some of their people on site, work over there sometimes, have regular catch-ups to see how it was going. Entirely different assumptions were made about how it was going to be done, and neither the commissioning manager nor the contract team had any idea that this was an issue for the contract.

Takeouts:

  • Agree what the workflow will be between client and agency, and include it in the contract or as an appendix
  • Involve the people who will be doing the work in shaping the agreement

Agencies don’t think they need help from the client

This one frustrates me hugely. Almost without fail, the agency pitch in response to a client brief would mostly include platitudes about the business, their customers and processes that were already well known. That’s ok to the extent that it shows that the agency understand something, but it doesn’t help when it’s presented as the big reveal. Then, much of the rest of the pitch is stuff that’s already been thought of but simply won’t work for very good reasons, or is actually a bad idea. After that, there may be a nugget or two to latch on to, which the agency is told to go away and develop. It’s a massive waste of everyone’s time.

Many times I and colleagues have spoken to agencies as they worked on a presentation for the big bosses, offering our help. Talk to us, we said. Share what you’re thinking, and we can help you to take out the road bumps. We’ll tell you if we spot issues with your proposals, if there are political issues you may need to be aware of etc. We made it clear, truthfully, that we weren’t going to steal their thunder, and also that they didn’t have to take out advice. If I had been agency-side I’d have jumped at the offer. Sadly, I can’t think of a single agency that did, with predictable results.

Takeouts:

  • Clients should offer expert help in shaping agency proposals
  • Agencies should ask for it
  • Work collaboratively, leveraging the expertise of each organisation