Embrace Your Ignorance

摘要

Microsoft's Ronny Kohavi, a pioneer in online experimentation since his days as director of data mining and personalization at Amazon.com, likes to challenge his Internet-savvy audiences with a little experiment. Kohavi, a Microsoft distinguished engineer and general manager for analysis and experimentation, shows screenshots of simple but real A/B experiments that Microsoft ran to test different Web design options. Some visitors saw the A version, some the B version - and Microsoft tracked which version yielded better results. Kohavi then challenges his audience to predict the real-world outcomes of the test: Did Option A win? Did Option B win? Or were the results a statistical dead heat?