A low number of returned questionnaires may result from an incorrectly defined sample, problems with contacting respondents, or an inappropriate numerator in the fieldwork report. Response rate helps organize these situations, but only when it is calculated for a clearly defined population of units eligible for the study and interpreted together with non-response bias.
How should response rate be calculated correctly in a survey?
Response rate indicates what proportion of sampled units eligible for the study actually provided a response. In practice, however, it is not enough to divide the number of completed questionnaires by the number of messages sent. This shortcut can lead to incorrect conclusions, as contact databases often include inactive records, duplicates, nonexistent addresses, people who do not meet the study criteria, or invitations that could not be delivered.
The simplest formula is the number of complete, usable interviews divided by the number of units eligible for the study, multiplied by one hundred. The concept of “eligible” is crucial here. If a survey concerns people who used a given service during a specified period, the denominator should not include contacts belonging to people who never met this condition. In practice, it is also necessary to determine in advance how cases with unknown eligibility will be handled, such as undelivered invitations where it cannot be confirmed whether they belonged to the study population.
In research projects, it is worth separating record statuses from the outset. This makes it possible not only to calculate the response rate, but also to determine at which stage of the process a problem occurs. Statuses should distinguish completed cases from those that could not be completed for technical or methodological reasons.
- complete response – the respondent completed the questionnaire, and their data meet the quality criteria adopted for the project;
- partial response – the questionnaire was started but discontinued before the point considered sufficient for analysis;
- refusal – the person had the opportunity to participate but consciously chose not to take part;
- no contact – contact could not be established or no response was obtained; simply not opening a message does not always make it possible to identify such a case unambiguously;
- ineligible – contact was established, but the person did not meet the recruitment criteria;
- case ineligible for contact – the record concerned a unit outside the study population or the contact information was permanently invalid; duplicates should be identified and removed according to the adopted procedure.
In CAWI studies, distinguishing between these categories is particularly important. The number of messages sent by an email system does not necessarily correspond to the number of people who had a genuine opportunity to respond. Messages may bounce, end up in the spam folder, be sent to an inactive address, or reach a mailbox shared by several people. Therefore, the fieldwork report should show both the number of messages sent and the number of delivered invitations, survey starts, and completed questionnaires. Open-rate data should be interpreted cautiously, as they may be incomplete due to email programs blocking tracking mechanisms.
It is also worth distinguishing response rate from completion rate. The completion rate answers the question of how many respondents who started the form reached the end. Response rate, in contrast, describes the relationship between completed interviews and the eligible sample. It is therefore possible to have a high completion rate and a low response rate when most people who start the form complete it, but the invitation itself attracts few participants.
The definition used in the report should be stated explicitly. This allows the recipient of the findings to know whether the denominator includes all units selected for the sample, only confirmed eligible cases, or also cases with unknown eligibility. The number of messages sent or delivered can be reported as separate operational metrics, but should not automatically be equated with response rate. Without this information, comparing response rates across study waves, sample segments, or different recruitment channels may be misleading.
How can a low response rate be diagnosed and non-response bias reduced?
A low response rate does not in itself indicate a single cause. It may result from the quality of the contact database, the invitation approach, the timing of fieldwork, questionnaire length, an overly broad selection of recipients, or technical barriers. Analysis should therefore begin not with the general question, “Why do people not respond?”, but with a review of the fieldwork funnel and record statuses.
First, the proportion of invitations that were delivered should be checked. If the problem occurs at this stage, it does not concern respondent behavior, but rather the condition of the contact data or the sending configuration. Next, open rates should be analyzed, with caution in their interpretation, as well as clicks through to the survey. A low proportion of visits to the form may indicate that the subject line, sender, or invitation content does not allow recipients to quickly identify the purpose of the contact. The next stage is survey starts and completions. A drop between these points requires a review of the questionnaire design, skip logic, technical messages, and devices used by respondents.
In CAWI studies, it is worth checking whether the form works properly on mobile devices. This is not merely a matter of visual appearance. Questions with long answer lists, invisible next buttons, required fields without an error message, or overly complex matrices can increase the number of discontinued questionnaires. Such issues primarily affect the completion rate and may indirectly reduce the final response rate.
More important than the response rate itself is whether people who did not respond differ from those who did in terms of characteristics relevant to the study findings. It is this type of difference that can create non-response bias. If nonparticipation is not systematically related to the characteristics being studied, a lower response rate does not necessarily distort estimates. However, if certain groups systematically ignore the invitation more often, the resulting sample may not reflect the population structure.
Assessing non-response bias requires auxiliary data. These may include known characteristics from a customer database, participant register, contact history, or operational data available before the survey begins. For example, the profile of respondents can be compared with the profile of the entire invited population by customer segment, contact channel, relationship tenure, region, or other legally and methodologically justified variables.
An analysis of responses after successive reminders is also useful. People who respond only after being contacted again may be more similar to nonrespondents than respondents who react immediately. This is not evidence in itself, but it may signal that subsequent contact attempts change the composition of the completed sample. The report should therefore present not only the overall completion rate, but also the response structure by timing of participation.
Measures to reduce low response rates should follow from the diagnosis. Before launching the study, it is essential to ensure clear eligibility criteria, up-to-date contact details, and an understandable invitation. Recipients should be informed who is conducting the study, what it concerns, how much time it requires, and how the data will be used. In B2B relationships, it is also important that the invitation reaches a person with the knowledge needed to provide an answer.
During fieldwork, results should be monitored by segment, channel, and contact wave. This makes it possible to distinguish a situation in which the response rate is uniformly low from one in which participation from a specific group is disappearing. In the latter case, the risk is not only the number of questionnaires, but also the unbalanced composition of the sample. Depending on the auxiliary data available, it may then be possible to make additional contact with the underrepresented group, change the outreach channel, or apply a weighting adjustment at the analysis stage.
It is important for reminders to be planned in the fieldwork protocol. The protocol should specify who they are sent to, after how much time, how many contacts are permitted, and how their effect will be reported. This ensures that efforts to increase the response rate are not random and that every decision can be reproduced and evaluated methodologically.
What most often distorts the interpretation of response rate?
The most common mistake is treating response rate as a universal measure of study quality. Response rate is an important fieldwork parameter, but it does not independently describe representativeness, question validity, or data quality. Survey results may also be biased when the response rate is relatively high, for example because of coverage error, imprecise questions, incorrect respondent screening, or improper data weighting.
Another pitfall is comparing rates calculated using different methods. One contractor may include all units selected for the sample in the denominator, another only confirmed eligible cases, and a third may apply a rule for estimating the eligibility of cases with unknown status. Each of these metrics may be useful, but each answers a different question. The number of records in the database or delivered messages should be treated as separate operational metrics. Without a definition of the numerator and denominator, comparing completion rates across projects has no analytical value.
It is also a mistake to include all started questionnaires in the final number of responses. Partial interviews can sometimes be used if the respondent completed key blocks of questions and the analysis rules were established in advance. However, response rate should not be improved by automatically treating every started questionnaire as a complete interview. This practice blurs the distinction between contact, participation, and the provision of complete data.
High email open rates should be interpreted carefully. Opening a message does not mean reading the invitation, clicking through to the form does not mean starting the survey, and starting it does not mean providing analytically useful responses. In addition, open-rate measurement may be incomplete or distorted by email program settings. Each stage of the funnel measures different behavior. Only examining these stages together shows whether low response rates result from deliverability, response to the invitation, or drop-off during the form.
Another limitation concerns analytical weights. Weighting can reduce non-response bias when reliable data on the population structure and appropriate auxiliary variables are available. However, it is not an automatic solution to every problem. If nonrespondents differ from respondents in terms of characteristics not included in the auxiliary data, weighting adjustments will not fully eliminate the risk of error.
Cases excluded during quality control should not be concealed either. A survey may include responses that are too fast, inconsistent, duplicated, or indicative of a lack of engagement. The report should show how many interviews were considered complete, how many were rejected after quality control, and how this decision affected the final response rate. Transparency of the procedure is more important than presenting a single metric without context.
How should response rate monitoring be prepared before fieldwork begins?
Monitoring the response rate should be part of the research design, rather than an activity carried out only after the survey has closed. At the preparation stage, the study population, sample source, unit of analysis, and eligibility rules should already be described. Response rate is calculated differently for individual consumers, business contacts, and surveys addressed to users of a specific service.
A good solution is to prepare a status table before fieldwork begins. The table should indicate which statuses are included in the numerator, which in the denominator, and which are excluded as ineligible cases. It should also describe how cases with unknown eligibility will be handled. This ensures that the fieldwork team, analytical team, and the person receiving the findings all use the same definition.
Before launching the study, it is worth agreeing on the following monitoring elements:
- the definition of a complete and partial response, as well as the minimum scope of data required for analysis;
- rules for identifying duplicates, invalid contacts, and people who do not meet the criteria;
- the method for reporting deliverability, survey visits, starts, completions, and quality-related exclusions;
- the frequency of response rate monitoring during fieldwork;
- auxiliary variables that will enable an assessment of non-response bias;
- rules for reminders, additional contact attempts, and documenting changes in the process.
Monitoring prepared in this way makes it possible to distinguish between two issues: fieldwork efficiency and the quality of the achieved sample. The first indicates whether the survey reaches invited individuals and whether the form is completed. The second answers the question of whether respondents are similar to the population that the findings are intended to describe. Response rate is an important fieldwork metric and may signal the need to analyze sample quality, but it does not replace an assessment of non-response bias.
Frequently asked questions
How is response rate calculated in a survey?
Response rate is calculated as the number of complete, accepted interviews divided by the number of units eligible for the study, multiplied by one hundred. The method for handling cases with unknown eligibility must be established beforehand. The number of invitations sent or delivered should be reported separately as an operational metric. The definition should be included in the methodological report.
Does a low response rate mean that the findings are unreliable?
A low response rate does not automatically determine that the findings are unreliable. The key issue is whether nonrespondents differ systematically from respondents in terms of characteristics related to the topic being studied. Assessment requires an analysis of non-response bias and, where possible, comparison of the sample with population data.
How can the response rate in a survey be increased?
First, it is necessary to determine at which stage of the funnel respondent participation declines: during invitation delivery, when clicking through to the form, at survey start, or at survey completion. The quality of contact details, clarity of the message, operation of the form, and reminder process can then be improved. Every change should be documented in order to assess its impact on the response rate and sample composition.
Ask about conducting a survey with sample quality control
Hume’s Institute helps plan survey research, monitor response rates, and assess the risk of non-response bias based on transparent fieldwork procedures and data analysis.