Write a Product Experiment Brief Before Building the Test

An experiment should help resolve a decision. If every possible result leads to shipping the same feature, the work may be a launch rehearsal rather than an experiment.

Start by writing the uncertain belief and the action that depends on it. Keep the first test small enough that a surprising result is affordable.

Name one uncertainty

Suppose a team believes new users abandon setup because importing existing data is difficult. That is different from believing they do not understand the product’s value. A single test should not pretend to settle both at once.

Write the belief in observable terms. “Users need easier onboarding” is broad. “Users with an existing spreadsheet cannot complete the first import without help” points to a specific workflow.

Choose a method that can reveal the problem

For that example, observe a small set of relevant users attempting the import with their normal data, subject to appropriate privacy handling. Record where they stop, what they misunderstand, and what assistance changes the outcome.

This can reveal usability problems. It does not automatically estimate how common the problem is across the entire market. Match the conclusion to the method.

Define the observation

Specify what you will record: completion, time spent at a particular step, errors encountered, or the user’s explanation after the attempt. Avoid collecting a large dashboard of unrelated numbers because they are available.

Keep success and failure interpretable. If a participant receives help halfway through, record the intervention rather than counting the session as an unqualified completion.

Write the decision rule

Before running the test, state what kinds of results would lead to improving the importer, changing the explanation, or investigating a different cause. Numerical thresholds can be useful when justified, but do not invent them to make the brief look scientific.

Include an inconclusive outcome. Missing data, an unsuitable participant group, or a broken prototype can prevent the test from answering the question.

Preserve the result and the limit

Summarize the observation, interpretation, decision, and remaining uncertainty. Keep quotes and identifiable information only with appropriate consent and storage controls.

A useful experiment brief makes it possible to disagree about the interpretation while sharing the same evidence. It also prevents a team from quietly changing the question after seeing a result it did not expect.