If you are running a testing program, then you’ve more than likely had to think about what should go into a test you are running. This could include the problem statement, your hypothesis, how many variants, how different these variants will be, what your measures of success will be, screen captures, and more. It’s important to create a doc or some sort of accessible page/application for your teams to be able to reference this information. This helps to foster and open and collaborative culture of optimization. It will also help you as you look back to understand what your test objectives were and how the test did compared to those objectives. I track all of this via a Google Doc for each test I run. I used the same template for
Continue reading...
Setting up a Google Analytics Content Experiment is easy! Follow this four-step process and you’ll be on your way to running your first test. To start, first go to the ‘Experiments’ section of Google Analytics and click on ‘Create Experiment’. Step 1: Setup the test Advanced: if you are working with a high volume page and want to analyze more than one goal at a time, you can set up a ‘fake goal’ so that the test will not optimize towards a single winner. Use a ‘fake goal’ to run the test longer than 2 weeks: Multi-armed bandit: Content Experiments uses a traffic splitting method called Multi-armed bandit (MAB) which essentially weights the traffic towards the variation(s) that appear to be winning, away from losing variations. In theory, this could
Continue reading...
In part 5 of this 5-part blog series about ‘Building a Culture of Optimization’ I’m going to talk about the importance of sharing your wins and bringing your organization along with you. You can see part 1, part 2, part 3 & part 4 here. Part 5: So you’ve found a big win. Now what? Ensure you’ve double triple checked your results! Are they statistically confident? Did you control for external variables? Why is this important? A personal example… I ran a test where we found significant uplift over our control from a couple of test variations, but one version stood out as the clear winner. After closing the test, reviewing and analyzing the data, I communicated the results and recommendation to launch the winner to the rest of my organization. Most people
Continue reading...
In part 4 of this 5-part blog series about ‘Building a Culture of Optimization’ I’m going to talk about evangelizing your process within your organization. You can see part 1, part 2, & part 3 here. Part 4: Evangelize the process Process is important. Process leads to consistency, repeatability, and authority in a testing program. Sharing that process and getting others in your organization bought in and supportive is even more important. One source of truth One of the best ways to make your optimization program better known within your organization is to evangelize it via a widely accessible & visible roadmap. Here’s an example roadmap that I use within my organization: I host this roadmap in a Google doc that is accessible to everyone in my organization, from analysts
Continue reading...
In part 3 of this 5-part blog series about ‘Building a Culture of Optimization’ I’m going to talk about the importance of bringing your organization up to speed on the math behind the tests. You can look back and see part 1 on the basics and part 2 on good test design. Part 3: Know the Math! Give your peers a short stats lesson (but keep it light)! What does someone running an A/B or MVT test need to know about math? How detailed should they be? Here is what I tell all of my coworkers: Statistical Confidence = confidence in a repeated result The confidence level, or statistical significance indicate how likely it is that a test experience’s success was not due to chance. A higher confidence indicates that: – the
Continue reading...