by | Released September 2015 | ISBN: 9781491934777

Deciding whether or not to launch a new product or feature is a resource management bet for any Internet business. Conducting rigorous online A/B tests flattens the risk. Drawing on her experience at Airbnb, data scientist Lisa Qian offers a practical ten-step guide to designing and executing statistically sound A/B tests. Discover best practices for defining test goals and hypotheses Learn to identify controls, treatments, key metrics, and data collection needs Understand the role of appropriate logging in data collection Determine how to frame your tests (size of difference detection, visitor sample size, etc.) Master the importance of testing for systematic biases Run power tests to determine how much data to collect Learn how experimenting on logged out users can introduce bias Understand when cannibalization is an issue and how to deal with it Review accepted A/B testing tools (Google Analytics, Vanity, Unbounce, among others)Lisa Qian focuses on search and discovery at Airbnb. She has a PhD in Applied Physics from Stanford University. Show and hide more Publisher resources View/Submit Errata

1. Overview of the Course 00:02:15
2. Why Should You Run A/B Tests? 00:04:18
3. The 10 Steps and An Overview of Case Studies 00:03:01
4. Case Study 1: Red vs. Green Button 00:22:16
5. Case Study 2: Testing a New Landing Page 00:16:11
6. Case Study 3: Price Recommendations on an Online Marketplace 00:11:46
7. Summary: Setting Up an A/B Test 00:05:16
8. Some of Your Options 00:04:53
9. Scaling A/B Testing and Developing a Culture of Experimentation 00:06:53
Show and hide more