A team wants to test an assistant that can turn a rambling request into a sensible plan. Building the underlying system may take months, while showing a slide of its proposed answer reveals little about what people would actually ask. Wizard of Oz testing offers an intermediate step: a person supplies some or all of the responses through an interface that presents them as the system’s behaviour.
The method uses a human operator, sometimes called the wizard, to simulate a capability during research. It can help explore an interaction before the automation exists, including what people request, what they expect in return and how they respond when the experience goes wrong. The human’s performance must remain distinguishable from evidence about what the eventual technology can deliver.
Define the system the operator is allowed to represent
Imagine a fictional meal-planning assistant. A participant asks for dinners that fit a budget, use leftover ingredients and avoid a household allergy. A skilled researcher could produce an exceptionally thoughtful response, but that would say little about a future system unless the study has defined which information and capabilities it is allowed to use.
Write the operating rules before sessions. Decide what inputs the assistant accepts, what information it can access, what response time it should approximate and how it handles an unsupported request. Prepare examples and rehearse the handoff between the participant-facing interface and the operator.
Nielsen Norman Group’s guide to the Wizard of Oz method describes using a human-controlled simulation to investigate an experience without implementing all its functionality. The simulation is most informative when the behaviour being represented is explicit enough for the team to interpret the results.
Test an experience, not an operator’s ingenuity
Give participants realistic task scenarios, leaving room for requests the team did not anticipate. Record those requests as well as the responses. If the operator improvises beyond the agreed rules, preserve that deviation in the record instead of silently treating it as normal system behaviour.
The meal-planning example might reveal that people want to revise one ingredient without regenerating the whole week. That is a useful interaction requirement. Satisfaction with a carefully crafted plan, however, would not demonstrate that an automated model can produce equally appropriate recommendations.
Include plausible limitations where they are relevant to the question. Testing only ideal responses can conceal the need for correction, refusal, clarification or recovery. A separate prototype test can then examine how people handle those states once the team has designed them.
Plan consent and debriefing before the simulation
Participants may not know exactly which parts of the prototype a person controls. That makes disclosure a substantive research-design decision. Prefer openness about the simulation where it still allows the question to be studied, and avoid unnecessary concealment merely to make the experience more convincing.
Where any information is withheld, the research plan needs an appropriate ethical review, clear limits and a considered debrief. Participants should understand what happened and how their data will be handled. Informed consent is not satisfied by treating the eventual reveal as an entertaining surprise.
Keep the activity safely within research. In the meal example, generated plans should not be treated as verified allergy advice or acted on as though the simulated service were reliable. The study can examine expectations using suitable scenarios without asking anyone to depend on an unvalidated recommendation.
Carry the findings into the buildable version
Separate findings about requests, interface understanding and recovery from findings that depend on response quality. Document the operator’s involvement, delays and deviations so that later readers can see which experience participants encountered.
The next research round should examine the actual capability when it becomes available. People may formulate the same requests yet react differently to real delays, inconsistent answers or limits on available data. Wizard of Oz testing helps the team discover what deserves building and testing; it cannot certify the system that eventually takes the operator’s place.