It’s not the analysis, it’s asking the right question that counts.

There are many people who can ask questions and get answers by analysing a set of numbers or by interviewing the users of a product. There are fewer people who know what the right questions are that will deliver a deeper level of actionable insight.

Identify the right questions

I recently read a book called ‘Everybody lies’ by Seth Stephens-Davidowitz. Most of it is a fascinating insight into what lessons can be learned from a really intelligent analysis of Google searches. The contention is that whilst people modify their answers in interviews and on questionnaires to put themselves in a good light they tell the truth in their Google searches. Here’s a quote:-

Early in the primaries, Nate Silver famously claimed that there was virtually no chance that Trump would win. As the primaries progressed and it became increasingly clear that Trump had widespread support, Silver decided to look at the data to see if he could understand what was going on. How could Trump possibly be doing so well? Silver noticed that the areas where Trump performed best made for an odd map. Trump performed well in parts of the Northeast and industrial Midwest, as well as the South. He performed notably worse out West. Silver looked for variables to try to explain this map. Was it unemployment? Was it religion? Was it gun ownership? Was it rates of immigration? Was it opposition to Obama? Silver found that the single factor that best correlated with Donald Trump’s support in the Republican primaries was that measure I had discovered four years earlier. Areas that supported Trump in the largest numbers were those that made the most Google searches for “nigger.”

If someone is racist, likes extreme porn, wants to find out how to make a bomb etc they often won’t be keen to immediately disclose these things to the first researcher who asks them – but they will be honest in their Google search and that dataset can be mined. By looking at searches on ‘where to vote’ or ‘how to vote’ a more accurate prediction was made of voter turnout in specific geographies. In areas in the US where abortion has become harder to access there is a spike in searches for how to do your own abortion.

Here’s a Guardian article on the book

One thing that’s clear from the book is that the insights didn’t just leap out from the massive dataset. Asking the right questions was critical.

In business I’ve met many excellent competent data analysts who can churn out impressive charts at a great rate. Many of those charts have been completely useless in business terms. If you find someone who understands the business well enough to be able to churn out meaningful in-depth insights then you need to keep them.

Whilst A/B tests are often valuable they are by definition binary. Does the blue or the green button work best? What if a red one would be better? Multivariate testing lets you try multiple variables at the same time but then you need more time and volume to reach statistical significance. So knowing enough to ask the right question in the first place based on previous knowledge and experience can make a difference.

One of the difficulties I find with A/B tests is that everything except the variables being tested tend to be averaged in the analysis. Maybe the test shows that the blue button works best, but hidden in the data is the fact that blue works best with frequent users and green works best with infrequent users. I recall a test done many years ago by a bank who found (once they looked) that the effective colour was different in the morning from the evening. Again, these things won’t just jump out from the data – someone has to think to ask the question.

This doesn’t just apply to data. I’ve written a previous blog post on common errors in survey design. I recall one time as well when we were interviewing customers in Germany on the design of the flight selling system on ba.com. One interviewee was indicating that he thought that design A was better for a particular page than design B. My colleagues seemed to take this at face value but I wasn’t convinced – there was something that wasn’t right. In my view it was obvious that B was better and I thought that we just weren’t asking the right questions. At the end of the interview I went in to follow up. Did he prefer A or B? A was the answer. Which was better? A was the answer. Which one should we implement? A was the answer. Which one was easier to use. B! It turned out that B was easier to use but he liked the look of A. So the action resulting that we nearly didn’t get was to maintain the usability of B and combine it with the visual appeal of A. It may seem obvious in retrospect but it was a good lesson in being clear about the difference between someone ‘liking’ a design (which really doesn’t mean much), or preferring the colours, or finding one easier to use etc. What question is it that you actually want an answer to?

There is currently a debate raging on whether screen time – and how much of it – has adverse effects on children. More data is being brought to bear over the many opinions that are freely available, yet there is still no consensus. The Oxford Internet Institute has recently released a study that found that screen time had little impact on ‘teen well-being’. And the World Health Organisation came out with a report that children under two should have no sedentary screen time at all. As The Verge points out though …the guidelines are less about the risks of screen time itself, and more about the advantages of spending time doing pretty much anything else.

In a recent edition of the BBCs Tech Tent podcast the most intelligent comment I heard from a guest on the show in relation to this issue is ‘I’m not sure we’ve asked the right questions yet.’