A report lands on the desk: 68% of consumers say they are willing to pay a higher price for an environmentally friendly product. Before this figure makes its way into a board presentation, it is worth asking a few questions about the study methodology – because this determines whether the result describes the market or merely the group of people who happened to click on the survey. Below is a set of checks that allow results users to assess their credibility without a background in statistics.
What exactly should you check before research findings inform a decision?
Managers rarely have time to audit a data set. However, they have the right to expect the study methodology to be described precisely enough to reconstruct the logic behind every figure. If the report does not make this possible, the problem is not the recipient’s lack of expertise – it is incomplete study documentation.
The credibility of a study can be assessed at four levels. The first is who responded – what population was targeted, how respondents were sampled, and who may have been excluded from the sample altogether. The second is what they were asked – the exact wording of the question, its order, the response options, and whether a “don’t know” option was included. The third is when they were asked – the fieldwork period and events that may have affected responses at that time. The fourth is how responses were converted into a result – weighting, the percentage base, and how missing data were handled.
A typical practical example: a report states that “73% of companies plan to increase their automation budget.” On checking the base, it turns out that the question was asked only of respondents who had previously said that automation was important to them – and they accounted for less than half of the sample. This percentage therefore cannot be interpreted directly as a result for the entire population of companies. The figure has not been falsified; it has been taken out of the context of its percentage base.
Another common case is a study of purchase intentions conducted in December and presented in March as a picture of “current consumer sentiment.” There is no error in the study methodology, but the recipient interprets the result as current because no one highlighted the fieldwork date.
How should you read a methodological note and identify gaps?
A methodological note is not a formality at the end of a report, but a guide to using its findings. It should answer questions that the recipient would otherwise have to ask by email. Its absence, or reducing it to a single sentence (“the study was conducted using CAWI on a sample of 1,000 people”), is a warning sign – such a description makes it impossible to assess anything beyond the data collection technique.
When reviewing a report, it is worth going through a list of elements whose presence in the methodological note is standard practice in rigorous research:
- Definition of the study population – who the findings are intended to describe (e.g., “people aged 18-65 who use mobile banking,” rather than “Poles”).
- Sampling method – random, quota, panel-based, with a description of the sampling frame or panel source; this makes it possible to assess the basis for any claims about the study’s representativeness.
- Sample size and subgroup sizes – including the sizes of analyzed segments, not only the total.
- Data collection technique – CAWI, CATI, CAPI, IDI, FGI, together with information about interview length and, in the case of surveys, the device used where this is relevant to data collection.
- Fieldwork period – specific data collection start and end dates.
- Weighting – whether it was used, which variables were used, and which reference data source was applied.
- Wording of key questions – the full questionnaire in an appendix or at least verbatim quotations of the questions from which the most important figures are derived.
- Client and research provider – who funded the study and who conducted it.
A result without information on sampling, the fieldwork period, and the exact wording of the question has limited value for comparisons and decision-making – it becomes a single figure without coordinates, rather than a fully described measurement. Two studies of the “same” phenomenon may differ by several percentage points solely because one asked about “considering a purchase” and the other about “planning a purchase within 3 months.”
A separate issue is measurement error, meaning the discrepancy between what a tool actually measures and what it is intended to measure. It can stem from a leading question, a scale without a neutral point, an overly long questionnaire leading to mechanical responses, or a question about behavior from a year ago that the respondent cannot recall. For sensitive topics, there is also a tendency to give socially acceptable answers – stated environmental or health-related intentions are typically higher than observed behavior. Assessing whether a tool measures what it was intended to measure is a matter of measurement validity and should be discussed in the report, rather than overlooked.
A useful practical test is to take one key figure from the report and try to write it as a complete sentence containing the population, question, base, and date. If any element has to be guessed, the methodological note is incomplete.
What errors most often undermine a study’s credibility?
Most issues with trust in findings do not result from data falsification, but from several recurring mechanisms. Issues related to systematic distortions in the sample and measurement are discussed in more detail in the material on how to detect systematic errors in quantitative research; below are those that a report recipient can identify independently by reading the document.
- Small subgroup base. A result for a segment of several dozen respondents is presented with the same level of certainty as the overall result. A difference of several percentage points between such subgroups may result from random variation, especially when sample sizes are small.
- Confusing stated intentions with behavior. “I intend to buy” measures intention, not a sales forecast. A report should name the variable for what it is.
- Leading question. A construction such as “Do you agree that the new feature makes everyday use of the app easier?” increases the share of agreement regardless of the actual assessment of the feature.
- Undescribed weighting. Data weighted to match the population structure, without specifying the weighting variables and reference source, make it impossible to assess whether the adjustment was justified.
- Selective presentation. The report includes only those breakdowns that support a coherent narrative; questions with ambiguous results disappear from the management presentation.
- Comparing waves with different methodologies. Changing the technique from CATI to CAWI, changing the panel, or rewording a question between waves means that a year-on-year difference may partly measure a change in the tool rather than a change in the market.
- Rounding and visualization. A chart with an axis starting at 60% may visually exaggerate a difference of several percentage points many times over.
A separate issue is the representativeness of research conducted using online panels. A quota sample from a non-probability panel may accurately reflect the demographic structure of the population while still differing from it in characteristics not included in quotas – digital activity, willingness to participate in surveys, or level of consumer engagement. This does not disqualify such research, but it changes how the findings should be described: precisely projecting percentages onto the entire population requires different assumptions than in a probability sample. This is discussed in more detail in the material on the difference between a probability and non-probability sample in the era of online panels.
In qualitative research, the assessment logic is different, although the question about study methodology remains the same. Representativeness and margin of error are not assessed; instead, the focus is on purposive sampling (whether participants cover key variations in experience), data saturation, the interview guide, the coding process, and whether conclusions are supported by quotations. Treating percentages from twenty in-depth interviews as population estimates is an interpretive error, not a strength of the report.
What does a step-by-step checklist for reviewing a report look like?
The sequence below makes it possible to assess a study’s methodology in a few minutes before the findings are circulated further. The order matters – each subsequent step helps refine the assessment made in the previous ones.
- Find the methodological note. If it is not included, request it before reading further.
- Check who the study describes. Compare the defined population with the population relevant to your decision. Any discrepancy limits the applicability of the findings to that decision.
- Verify the fieldwork date. Determine whether any events occurred during that period that may have changed the context of responses.
- Find the full wording of the questions behind the three most important figures in the report.
- Check the percentage bases below the charts – how many people actually answered a given question.
- Verify subgroup sizes on which conclusions about segments are based.
- Ask about weighting – which variables were used and what was the source of the population structure.
- Separate measurement from interpretation. Mark statements in the report that are the author’s conclusions rather than direct readings of the data.
- Check comparability with previous waves if the report shows changes over time.
- Establish who funded the study and whether the client has an interest in a particular result.
In Hume’s Institute projects, most disputes over findings arise not from the quality of fieldwork, but from a lack of shared understanding of definitions – when “user,” “active customer,” or “purchasing decision-maker” means something different in the questionnaire than in the client’s systems. Aligning these definitions before the study begins reduces later doubts about the study’s credibility.
It is also worth remembering that the question of “how to read research findings” has an organizational dimension, not only an analytical one. If a report is passed on in the form of three slides, those slides should include the base, date, and question wording. Otherwise, the figure begins to take on a life of its own, and after a few months no one can reconstruct where it came from.
Frequently asked questions
What should a study’s methodological note include?
At a minimum, it should include the definition of the population, the sampling method and sample size along with subgroup sizes, the data collection technique, exact fieldwork dates, information on weighting (variables and reference source), and identification of the client and research provider. In quantitative research, it is good practice to attach the full questionnaire; in qualitative research, the interview guide and recruitment criteria. The note should be detailed enough to enable an assessment of how the study was conducted and how its findings should be interpreted.
How can you check whether a study is representative?
The sample structure should be compared with the population structure on variables relevant to the research topic – not only gender, age, and place of residence. It is also essential to ask about the source of the sample: whether respondents came from a random sampling frame, an online panel, or open recruitment, as well as the refusal rate and completion rate, if available. Demographic quota alignment alone does not guarantee representativeness if the recruitment method favored a particular type of respondent.
How can you tell that a study finding has been overinterpreted?
Warning signs include causal language used with correlational data (“because,” “affects,” “causes”) and converting stated intentions into market values without specifying the assumptions. It is also questionable to treat percentages from qualitative research as population estimates, draw conclusions about segments based on several dozen responses, or use headlines that are stronger than the chart content beneath them. The test is simple: check whether the statement in the report can be derived directly from the wording of the question and the percentage base.
Request an assessment of your study methodology
If you have a report whose findings are to form the basis for a decision, Hume’s Institute can conduct an independent review of its methodology – from sampling to questionnaire design and the way data are presented. Contact Hume’s Institute to discuss the scope of such a review.