Quasi-experiment

A quasi-experiment is a research design used to assess the effects of an action, campaign, or business change when it is not possible to randomly assign individuals, companies, or areas to test and control groups. A well-designed quasi-experimental design can support causal inference, but it requires more cautious interpretation than a randomized experiment.

What is a quasi-experiment?

A quasi-experiment is a research method that analyzes the impact of a specific intervention on an outcome even though participants were not randomly assigned to study conditions. An intervention may include an advertising campaign, a price change, a new product display, the implementation of a loyalty program, a website modification, or a brand’s entry into a selected regional market.

The basic logic of a quasi-experimental design is similar to that of an experiment. The situation after the change is introduced is compared with a situation that serves as a reference point. The difference lies in how comparison groups are created. In a true experiment, participants are randomly assigned to experimental and control groups. In a quasi-experiment, groups usually result from real operational conditions, business decisions, geographic location, implementation timing, or pre-existing audience characteristics.

In market research, a quasi-experiment is used to answer whether an observed change in metrics may have been caused by a specific marketing or commercial activity. The analysis may concern sales, market share, brand awareness, purchase intention, lead volume, conversion, customer retention, or consumer-attitude indicators.

Most often, a quasi-experiment is based on one of the following designs:

• measurement before and after an intervention in the same population or location,
• comparison of an area exposed to an intervention with a similar area not exposed to it,
• comparison of groups that naturally differ in their exposure to an intervention,
• phased implementation of a change across channels, regions, or customer segments,
• analysis of historical data to assess whether a meaningful deviation from the previous trend occurred after the intervention.

Methodologically, a quasi-experiment is an attempt to recreate the conditions of a counterfactual comparison: what would probably have happened without the intervention? Because this alternative scenario cannot be observed directly, it is estimated using control groups, pre-intervention data, or statistical models.

Using quasi-experiments in practice

Quasi-experiments are used primarily when a full experiment would be impossible, too costly, unethical, or disruptive to an organization’s ongoing operations. This is especially relevant in projects where implementation decisions are made by a client, sales network, platform operator, or local operational teams, while the researcher does not control the random assignment of recipients.

In marketing and consumer research, a quasi-experimental design can be used to assess the effectiveness of a campaign conducted in only part of a country. Changes in results in regions exposed to the communication are compared with changes in control regions, while accounting for seasonality, differences in sales potential, and previous trends. Such a design is sometimes described as a regional or geographic test.

In retail, a quasi-experiment can assess the impact of a new shelf display, point-of-sale materials, a price promotion, or an assortment change. If a solution is implemented in selected stores, their results can be compared with those of similar stores that did not receive the change. It is important that comparison stores are similar in terms of format, location, customer mix, and sales history.

In the B2B sector, a quasi-experiment can support the evaluation of a lead-generation program, new customer-service standards, or account-based marketing activities. For example, the effects of a program can be analyzed by comparing companies exposed to it with similar companies that were not exposed. Differences in customer size, industry, stage of the commercial relationship, and purchasing potential should also be controlled for.

Quasi-experiments are also used in digital research when a standard A/B test is unavailable or does not cover the full business decision. This may include implementing a new checkout process only for mobile-app users, launching a feature in selected countries, or changing communication policy in one channel.

In mixed-methods projects, quantitative results from a quasi-experiment can be supplemented with qualitative research. In-depth interviews, observations, customer-journey analysis, or workshops with sales teams can help explain the mechanism behind an observed change. Qualitative data do not replace methodological controls, but they may indicate why an intervention worked differently across segments or locations.

Quasi-experiments and related methods

A quasi-experiment is one of the methods used to investigate cause-and-effect relationships, but it does not provide as strong a basis for causal inference as a randomized experiment. This difference has practical importance: an increase in sales after a campaign does not necessarily mean that the campaign alone caused the increase. The result may also have been affected by competitors’ actions, price changes, seasonality, product availability, or differences in audience composition.

In the comparison of a quasi-experiment versus a true experiment, the key criterion is random assignment. A true experiment assumes that participants are randomly assigned to study conditions. This means that groups should be comparable with respect to both measured and unmeasured characteristics, on average. A quasi-experiment does not provide the same protection against selection bias, so it requires additional analytical procedures and more conditional conclusions.

An A/B test is usually a specific form of a true experiment, provided that users or respondents are randomly assigned to variant A or B. However, not every comparative test is an A/B test in the methodological sense. If variant B is available only in one region, to one customer group, or after a specific date, and assignment is not random, the study is quasi-experimental.

Quasi-experimental design is often combined with statistical tools intended to reduce the influence of differences between groups. Common approaches include:

• difference-in-differences, which compares the change in the intervention group with the change in the control group and relies, among other things, on a parallel-trends assumption,
• matching, which selects comparison units with similar observed characteristics,
• regression with controls for confounding variables,
• time-series analysis, used with data collected across multiple periods,
• synthetic control, which creates a synthetic control group from data on multiple comparison units.

The choice of technique depends on the available data, the way the intervention was implemented, and the risk of confounding factors. Even an advanced statistical method cannot compensate for a weak research design or account for all unobserved differences between groups. The greatest value comes from combining an appropriately selected comparison group, pre-intervention data, and transparently described assumptions.

Limitations and conditions for the credibility of a quasi-experiment

The most important limitation of a quasi-experiment is the lack of full control over who enters the group exposed to the intervention. For example, if a campaign is directed at regions with higher purchasing potential, a simple post-campaign difference in sales may partly result from pre-existing differences between regions rather than from the campaign itself.

The credibility of a quasi-experimental design is increased by the following actions:

• defining the study outcome and success criteria before analyzing the data,
• selecting a control group using historical data and operational characteristics,
• checking whether groups showed similar trends before the intervention, while recognizing that similar pre-intervention trends alone do not prove the parallel-trends assumption,
• accounting for seasonality, promotions, price changes, availability, and other marketing activities,
• assessing possible spillovers, such as exposure of the control group to the intervention,
• analyzing results by segment when the intervention may have had uneven effects,
• clearly communicating uncertainty and alternative explanations.

Quasi-experiments are particularly useful in market research because they make it possible to analyze real business decisions and operating conditions rather than only behavior in a controlled environment. Their value nevertheless depends on data quality, the validity of the comparison, and discipline in interpretation. Conclusions should indicate whether an intervention is a plausible cause of a change rather than presenting an observational relationship as indisputable proof of causality.