When a respondent in an interview recounts what they ate for breakfast three days ago or why they chose a particular coffee brand in a store, they provide rationalizations rather than facts. Diary studies and ESM address this problem at its source – they record behavior, emotion, or a decision at the moment it occurs or as close to that moment as possible, before memory has time to process and simplify it.
What are diary studies and ESM, and when should they be used?
The experience sampling method (ESM) is a measurement technique in which respondents repeatedly answer short questions throughout the day about what they are currently doing, feeling, or thinking. The prompt to complete a micro-survey arrives at random or predefined times – most often as a push notification in a mobile app. Traditional consumer diaries work similarly but give respondents greater flexibility: they record events on their own initiative (for example, every instance of product use) or summarize a selected area of their life once a day.
What both approaches have in common is tracking behavior close to real time – recording an event in its natural context, without pulling the respondent into a focus group facility. This distinguishes diary studies and ESM from retrospective interviews, in which respondents reconstruct the past through the lens of their current mood and knowledge.
The method works well wherever situational context, frequency of behavior, or the dynamics of emotions matter. Typical applications include:
- usage journeys for everyday consumer products (beverages, snacks, cosmetics) – when, where, with whom, and in what mood they are used,
- service experiences (mobile banking, streaming, transportation) – the moments when frustration arises and those when satisfaction occurs,
- advertising exposure and media usage research – recording contact with a message when it occurs,
- customer journeys in categories with long decision cycles – furniture, home appliances, insurance,
- employee well-being and B2B behavior research – when specific professional needs arise during the day.
In practice, mobile research has become a very common way of conducting ESM today. The respondent’s smartphone serves as a measurement tool: it sends notifications and collects text responses, photos, short videos, and, in some projects, metadata as well (for example, response time, location – with the respondent’s consent, and time of day). This provides quantitative and qualitative material that cannot be replicated in a traditional interview.
How to design an ESM diary study step by step?
Designing diary studies and ESM begins with the research question, not the tool. The signal schedule for a study of impulsive grocery purchases will differ from one designed to track satisfaction with a banking app used once a week. Key design decisions cover several layers.
Sampling schedule. There are three basic types: signal-contingent (signals at random times), interval-contingent (set times of day), and event-contingent (the respondent responds independently when an event occurs). The choice depends on the frequency of the behavior – infrequent events often require an event-contingent approach, because random signals would be unlikely to capture the moment of product use.
Study duration and number of signals per day. Consumer diaries typically run from several days to three weeks. A period that is too short will not capture a pattern, while one that is too long leads to participant fatigue, meaning a decline in response quality toward the end of the study. The number of daily signals usually ranges from two to seven – with a greater burden, respondents are more likely to start ignoring notifications.
Length of an individual micro-survey. The rule here is simple: the shorter, the better. Completing the survey should not require interrupting the current activity for more than one or two minutes. Questions must be clear, preferably closed-ended, with an optional open-ended field or space for a photo.
Recruitment and onboarding. ESM respondents work more intensively than respondents in a traditional survey, so time should be devoted to explaining the rules, testing the app, and setting an incentive appropriate to the effort required. Good onboarding can significantly reduce drop-out and improve data completeness.
Data validation and cleaning. Responses provided with a substantial delay after a signal (for example, more than 30 minutes) lose the value of near-real-time measurement and are usually excluded from the main analysis or coded separately. Likewise, a series of identical responses within a short period may suggest straight-lining and requires separate verification.
As methodological literature shows, respondent memory is systematically unreliable – data collected at the time of an event or shortly afterward can be more accurate than data reconstructed many hours or days later, because the memory process filters out emotions, context, and minor yet important behavioral details. In practice, the same person may describe their shopping habits in an in-depth interview very differently from what emerges from their own diary entries from the same week – not because they are being dishonest, but because retrospective narratives organize reality more than it actually was.
It is worth remembering that ESM does not replace in-depth interviews – it complements them. The greatest value comes from the following sequence: first, the diary records what actually happens, and then the in-depth interview makes it possible to understand why. This mixed-methods approach is now widely used in UX research, FMCG, and service categories.
What mistakes should be avoided in diary studies and ESM?
Although mobile research and ESM provide uniquely valuable data, poorly designed studies can produce material that appears rich but is analytically useless. The most common pitfalls concern four areas.
Measurement reactivity. Keeping a diary can itself change behavior. A respondent who has to report their consumption of sweets every day will often reduce it after a few days – not because the study requires it, but because increased attention modifies the habit. The project should consciously account for the possible reactivity effect, and results from the first days of the study should be interpreted carefully.
Too many questions. The temptation to use every micro-survey to ask a dozen or more questions leads to drop-out. Each additional question can reduce the compliance rate, meaning the proportion of completed signals. It is better to ask three questions seven times a day than fifteen questions twice a day.
An unsuitable sample. The method requires respondents who are comfortable using smartphones, motivated, and consistent. A self-selected sample may differ systematically from the population – such respondents are often more organized and more engaged with the category. For studies intended to have population-level relevance, quota sampling and control of this tendency are necessary.
Omitting the qualitative layer. Raw quantitative ESM data (when, where, how often) without the “why” layer remain descriptive rather than explanatory. This is why projects that combine a diary with a follow-up in-depth interview or a short video debrief at the end of the week have an advantage.
Compared with a retrospective interview, focus group, or CAWI survey, diary studies and ESM are usually more expensive per respondent and require longer fieldwork. In return, however, they provide something other methods cannot: a record of behavior in its natural context, with less influence from memory errors than retrospective self-reports. They are therefore chosen wherever precision regarding timing and frequency has methodological significance rather than merely illustrative value.
When should you choose diaries, and when a traditional survey?
The decision between consumer diaries and a traditional survey or interview depends on several criteria that are worth verifying at the briefing stage:
- Is the behavior frequent and repetitive? If so (coffee consumption, scrolling social media, public transportation) – ESM makes sense. If the event is one-off or very rare (for example, buying an apartment) – an in-depth interview will work better, and sometimes an event-contingent diary.
- Does situational context matter? If the same product is used differently at work and at home, alone and in a group – a diary will capture this, while a survey will not.
- Are emotions an important variable? Emotions reconstructed from memory are flattened. If the study concerns satisfaction, frustration, or engagement – real-time measurement may be particularly valuable.
- Is the research team ready for fieldwork spread over time? ESM requires active monitoring of compliance and a quick response to declining respondent engagement. This is a different rhythm of work from a one-time survey.
- Do the budget and schedule allow for fieldwork lasting several or more than a dozen days? If a decision must be made within a week – it is better to use faster techniques.
With a well-matched design, the experience sampling method provides material whose depth explains both the behavioral pattern and its variability over time – two dimensions that are difficult to access simultaneously through a one-time measurement.
Frequently asked questions
What does the ESM method involve?
The experience sampling method involves repeatedly surveying respondents briefly throughout the day, most often through a mobile app that sends push notifications. Respondents answer several questions about what they are currently doing, feeling, or thinking, allowing the researcher to obtain a record of behavior in its natural context rather than only a reconstruction from memory. Measurement typically lasts from several days to three weeks.
When does a consumer diary provide better data than an interview?
Wherever frequency, context, and the dynamics of emotions matter. A diary records moments of product use, exposure to a message, or contact with a brand closer to when they occur, whereas an interview provides a post hoc declaration. However, the best approach is to combine both methods: the diary shows what is happening, while the in-depth interview explains why.
How can data be collected technically using ESM?
The standard is to use dedicated mobile apps or research platforms that support push notifications, micro-surveys, photo and video attachments, and real-time compliance monitoring. Smaller projects are sometimes conducted via SMS or messaging apps, but this means losing some advantages, such as easy monitoring of response times or automated reminders. Regardless of the tool, a simple survey logic and a short completion time for each signal are essential.
Ask about a project using diaries or the ESM method
If your research question concerns behaviors that unfold over time, emotions at specific moments, or the real context of product use – contact the research team to discuss a diary study or ESM project tailored to your category.