Where data falls short
Rationale, trust and blind spots in quantitative user research.
In our line of work, we — design researchers — often get challenged on the reliability of our methods. And as someone who has been on both sides of the data science and qualitative research fence, I can appreciate this concern.
In reality, quantitative and qualitative data can and does merge beautifully to examine complex topics. This is what we’ve seen referred to recently as the convergence of big data and thick data.
In an effort to highlight the complementarity between these two, I would like to dive into the limitations of questionnaires, surveys, analytics and big data. While rarely questioned, the choice of these tools can in fact contain biases for your research. This bias can be removed through adding complementary, qualitative data to your analysis.
In the next article, we will be looking at the other side of the mirror, exploring the shortcomings of stand-alone qualitative methods.
Data fails to explain rationale.
While potentially removing the social desirability bias, quantitative research methods traditionally fail to capture the reasoning behind an observed or reported action.
Providing space for research participants to elaborate on their views and experiences allows us to gain better insight into how they think and what they value, providing possible explanations for measurable actions.
“In my own research, interviews have been an indispensable source of data. I have used them extensively and in the process have had the privilege of accessing the world of the unemployed, corporate and slum landlords, irate tenants, small business owners, age pensioners, anti-apartheid activists, sex workers, homeless people and immigrants who have experienced threatening xenophobia. It is possible to get an idea of how people see the world through the use of a survey questionnaire, observation, blogs and secondary sources, but the strength of the in-depth interview lies in its ability to create a research space in which the interviewee is able to tell their story and give the researcher a range of insights and thoughts about a particular topic. Through in-depth interviews the researcher is able to obtain an understanding of the social reality under consideration and, depending on the circumstances, collect rich data fairly rapidly.”
Lack of trust leads to unreliable data.
There is a reason for which building rapport is such a central piece of interview skills — and why it is not for quantitative data collection. In the data science field, we often accept that, with big enough samples and with randomization, our results can be generalized to the population. This assumption relies on the idea that people fill surveys with perfect openness and transparency.
When looking at vulnerable populations and sensitive topics, however, an impersonal survey may not provide the psychological safety and connection required to bring experiences to light. This is where in-depth interviews shine.
“We have to remember that no matter where they happen, research interviews are a little weird for everyone. We invite strangers to have one-sided conversations with us that can often feel quite personal. The participants just want to do a good job and make us happy, but they’re worried about doing something wrong, or worse, looking stupid. Collecting quality information from them hinges on our ability to put them at ease and quickly earn their trust. If we don’t, they’ll remain nervous, suspicious and hesitant to share their personal stories — and we’ll never get the information we’re after.”
Data comes with blind spots.
Framing the research area prior to understanding it may lead to measuring “the wrong thing”, or omit crucial elements that, once measured, would have told a different story. In this sense, quantitative research is heavily biased by the researcher’s understanding of scope, and can be imprecise in its target.
In his Field Study Handbook, Jan Chipchase introduces a series of quadrants (see below) illustrating the extent of our knowledge on any given topic. Semi-structured and narrative interviews allow interviewers to capture new themes and areas of exploration unknown to them, but deemed relevant by participants. This is something that surveys and analytics fail to do.
Qualitative Data as a Complement.
The limitations of purely quantitative data do not mean quantitative research is invalid. Rather, it can be off-target or, in the case of sensitive topics, inaccurate. When done well, it may still prove deficient. Quantitative data is descriptive in nature; it speaks of “what” — but not of “why”.
“Using data of different types can help us both to determine what interpretations of phenomena are more or less likely to be valid and to provide complementary information that illuminates different aspects of what we are studying.”
Bergman, M.M. (2008) “‘The straw men of the quantitative-qualitative divide and their influence on mixed methods research”
Quantitative data can be a starting, or an end point, as can qualitative data. We will explore this notion in the third part of this article series. In order to understand how to best combine them, however, we will first examine the shortcomings of qualitative data collection, in a next article.
