The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)

Kohavi asks what the team is optimizing for and calls the combined evaluation an overall evaluation criterion. A result cannot be called good merely because a convenient number went up. Establish the intended outcome and the constraints that make the improvement acceptable.

He warns that a short-term business metric can improve while the user experience worsens. Countervailing measures or explicit constraints help make the trade-off visible. Check the side effects rather than treating the primary metric as the entire product.

Kohavi describes considering whether a customer remains happy after the initial conversion. The later outcome may not be available when the first result appears. Recognize that timing limitation instead of assuming an immediate action proves a lasting benefit.

He says not every domain is suitable for A/B testing and that enough experimental units are needed. A one-time decision is different from a repeatable product interaction. His discussion does not supply a universal sample-size rule for every metric or design.

Kohavi highlights sample-ratio mismatch as a warning that experiment data may not reflect the intended allocation. The warning needs diagnosis before relying on the result. A polished scorecard does not establish that the experiment was valid.

He describes people presenting results despite a mismatch warning. The lesson is that an unresolved validity problem must remain visible when the result is shared. Hiding the warning behind a separate screen can let an unreliable conclusion look ready to use.

Kohavi uses Twyman's law to encourage scrutiny of unusually large or unexpected changes. Surprising results can be real, but they need checking rather than automatic celebration. Replication and investigation can help distinguish a useful discovery from a flaw.