A researcher writes a task asking participants to “archive last month’s project”, then discovers in the first session that half the audience uses “archive” to mean download a copy. The prototype may work exactly as intended while the study measures something different from what the team planned. A pilot creates an opportunity to find that mismatch before full fieldwork begins.
A pilot study is a smaller run used to assess and refine a planned study’s feasibility, procedures or instruments. In product research, it often examines recruitment, instructions, timing, recording and whether the resulting material can support the intended analysis.
Rehearse the whole journey
Include the invitation, screener, participation arrangements and technical setup as well as the main questions. A survey can be clear on a researcher’s laptop but difficult to complete on a phone, while a remote test can fail before the first task because participants cannot open the prototype.
Nielsen Norman Group’s guide to pilot testing describes its role in preparing a study. Participants similar to the intended audience are especially useful for detecting misunderstandings that colleagues familiar with the product may overlook.
Take the pilot material through the planned analysis. If success cannot be scored consistently, or an answer cannot be assigned to the categories the team needs, revise the instrument before collecting more of the same problem.
Decide what would make you change the study
Set practical questions for the pilot: can eligible people be recruited, do they understand the task, does the session fit the available time and are the relevant interactions captured? For a more complex programme, define feasibility criteria appropriate to its risks and resources.
Record what changed and why. If an ambiguous task scenario is rewritten, check that the new wording no longer supplies the route or leaves participants guessing which goal matters. Significant changes may justify another rehearsal.
A small pilot can reveal problems without establishing that none remain. Rare circumstances and access needs may not appear in a handful of sessions, so its coverage should be described honestly.
Keep feasibility evidence separate from product conclusions
A preference expressed by a few pilot participants is not a dependable basis for declaring a winning design. Avoid tuning the study to produce the preferred substantive result after seeing early answers.
Whether pilot data can be included in the main analysis depends on the design, eligibility, procedures and any changes made. Do not merge materially different versions without accounting for those differences. In a confirmatory study, make that decision prospectively where possible and document it.
The pilot’s practical outcome is a study ready to answer its intended question under clearer conditions. Good questionnaire design and preparation improve that readiness, while the full study still needs its own recruitment, analysis and interpretation.