Lately, I’ve been meeting with founders and CTOs regarding the challenges of a/b testing on iOS, and I’ve found that I’m repeating myself a lot.
Releasing frequent updates and running tests on iOS aren’t as easy as on the web: you have to push to the App Store and wait for your users to update before they even receive the experiment.
The Problem
The issues regarding testing on iOS mainly involve the following:
- The App Store’s slow review process.
- Users not updating frequently enough to always be on the latest version (which is also a problem if you’re multi-platform).
- Testing features across multiple platforms and being in sync.
I believe these are just small setbacks and shouldn’t stop you from moving fast and testing your hypotheses.
The Solution
At Frank & Oak, we try to follow a system to work against these problems and move fast. We’ve been refining the system for a while. When we started using it, we had only one developer, a part-time designer, minimal back-end resources, and a product manager (me!) working on the product. So, it should be possible to implement with any team size.
The system consists of building everything as a test and releasing every 2-3 weeks. Here it is:
Build Everything as a Test
Early on, we made the decision that everything we build should be a test (with the obvious exception of bug fixes).
We effectively instigated this rule after mistakes we made re-launching our website in early 2013. This launch dramatically decreased our signup and purchase rates, but due to the fact that it was not run as a test, we couldn’t tell why it decreased our performance. We were blind.
The move to running everything as a test is definitely extreme on our end. But, since we’ve put this system in place, we know how we’ve improved (or failed to improve) each metric we track. Better yet, we can attribute changes in metrics to specific features we’ve released. For example, I know that the conversion rate on our iOS apps has increased by about 60-70% since last year, and I can easily list all the features and improvements we’ve built to make that happen.
Generally speaking, I recommend this for almost every product. Everything you build alters the experience for your users, which could have drastic effects on the performance of your app and sales of your product, positively or negatively.
We use Taplytics’s mobile A/B testing platform to help distribute the different versions we build to our users. It’s very simple and also multi-platform. I highly recommend it.
This enables us to push features to iOS. If Web or Android are behind, we can make sure that multi-platform users don’t have access to them until everyone else is ready. Being multi-platform shouldn’t slow you down.
Release Every 2-3 Weeks
We try to push a build to the App Store every 2-3 weeks. This has caused a few interesting side-effects for us:
1. It has decreased review time for the App Store.
On average, our app is reviewed and published in 1-3 days. We’ve noticed that the more regular and consistent our releases have become, the quicker the review process has been.
This could be due to Apple prioritizing apps differently when they’re following a schedule, or the review team getting used to our app, or some other factor that we haven’t thought about.
In any case, this has enabled us to keep pushing releases at the same rate and not worry too much about how long it will take to get on the store.
2. It has trained our users to update the app quickly.
About 90+% of our active users download the latest update in 2-4 days. This has been a great side effect of having frequent updates, giving users a nudge to open the app and keep up-to-date with the changes.
We make sure to notify our users that a new version of the app is available for them by displaying an overlay in the app. It has definitely helped accelerate the adoption of the latest versions.
3. It has trained us to simplify features and provide a better user experience.
This approach has forced us to be proactive with designing upcoming features, as well as prioritizing back-end resources, to make sure we have everything ready when we start building for the next release.
The benefits outweigh the challenges that come with releasing so frequently and make us better at focusing on the right things to build.
Conclusion
By testing every feature, you can measure and understand the impact of everything you’ve built. This will keep the focus on improving the experience for your users and moving the business forward as a whole.
You can also move fast by organizing your team and training your users to get used to the speed. However, such processes and philosophies will not work unless you’re truly committed to them throughout your organization.
We’ve spent a lot of time building a culture of experimentation, to the point that every person involved thinks about the metric each project is working to move, be it the developers, the designers, or the customer service agents.
I encourage you to do the same.
About the Author: Nima Gardideh is the Product Manager, Mobile, at Frank and Oak. You can follow him @ngardideh or subscribe to his mailing list for more posts about mobile, experimentation, and product management here.
Comments (2)