The DevOps Handbook – The Value of A/B Testing

Podcast: Play in new window | Download

Subscribe: Apple Podcasts | Spotify | TuneIn | RSS

We wrap up the second way from The DevOps Handbook, while Joe has a mystery episode, Michael doesn’t like ketchup, and Allen has a Costco problem.

These show notes, in all of their full sized digital glory, can be found at https://www.codingblocks.net/episode142, where you can join the conversation, for those using their podcast player to read this.

Sponsors

• Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after your first dashboard.

Survey Says

News

• We appreciate the new reviews, thank you!
  • iTunes: Jynx_Protocol
  • Stitcher: IllKeepItEasyForYouMike, KingJArthur

Treating Features as Experiments

Integrate Hypothesis Driven Development and A/B Testing

“The most inefficient way to test a business model or product idea is to build the complete product to see whether the predicted demand actually exists.”

Jez Humble

• Constantly ask should we build it and why? A/B testing will allow us to know if an idea is worthwhile because allows for fast-feedback on what’s working.
• Doing these experiments during peak season can allow you to out-experiment the competition.
  • But this is only possible if you can deploy quickly and reliably.
    • This allows A/B testing to support rapid, high-velocity experiments.
• A/B test is also known as “split testing”.
• A/B testing is where one group sees one version of a page or feature and the other group sees another version.
• Study from Microsoft found that only about 1/3 of features actually improved the key metric they were trying to move!
  • The important takeaway? Without measuring the impact of features, you don’t know if you’re adding value or decreasing it while increasing complexity.

Integrate A/B Testing Into Releases

• Effective A/B testing is only possible with the ability to do production releases quickly and easily.
• Using feature flags allow you to delivery multiple versions of the application without requiring separate hardware to be deployed to.
  • This requires meaningful telemetry at every level of the stack to understand how the application is being used.
• Etsy open-sourced their Feature API, used for online ramp-ups and throttling exposure to features.
  • Optimizely and Google Analytics offer similar features.

Integrating A/B Testing into Feature Planning

• Tie feature changes to actual business goals, i.e. the business has a hypothesis and an expected result and A/B testing allows the business owner to experiment.
• The ability to deploy quickly and reliably is what enables these experiments.

Create Processes to Increase Quality

• Eliminate the need for “approvals” from those not closely tied to the code being deployed.
• Development, Operations and InfoSec should constantly be collaborating.

The Dangers of Change Approval Process

• Bad deployments are often attributed to:
  • Not enough approval processes in place, or
  • Not good enough testing processes in place
• The findings of this is that often, command-and-control environments usually raise the likelihood of bad deployments.

Beware of “Overly Controlling Changes”

• Traditional change controls can lead to:
  • Longer lead times, and/or
  • Reducing the “strength and immediacy” of the deployment process.
• Adding the traditional controls add more “friction” to the deployment process, by:
  • Multiplying the number of steps in the approval process,
  • Increasing batch sizes (size of deployments), and/or
  • Increasing deployment lead times.
• People closest to the items know the most about them.
  • Requiring people further from the problem to do approvals reduces the likelihood of success.
  • As the distance between the person doing the work and the person approving the work increases, so does the likeliness of failure.
• Organizations that rely on change approvals often have worse stability and throughput in their IT systems.
• The takeaway is that peer reviews are much more effective than outside approvals.

Enable Coordination and Scheduling of Changes

• The more loosely coupled our architecture, the less we have to communicate between teams.
  • This allows teams to make changes in a much more autonomous way.
• This doesn’t mean that communication isn’t necessary, sometimes you HAVE to speak to someone.
  • Especially true when overarching infrastructure changes are necessary.

Enable Peer Review of Changes

• Those who are familiar with the systems are better to review the changes.
• Smaller changes are much better.
• The size of a change is not linear with the risk of the change. As the size of a change increases, the risk goes up by way more than a factor of one,
• Prefer short lived branches.
• “Scarier” changes may require more than just one reviewer.

Potential Dangers of Doing More Manual Testing and Change Freezes

• The more manual testing you do, the slower you are to release.
• The larger the batch sizes, the slower you are to release.

Enable Pair Programing to Improve all our Changes

“I can’t help wondering if pair programming is nothing more than code review on steroids.”

Jeff Atwood

• Pair programming forces communication that may never have happened.
• Pair programming brings many more design alternatives to life.
• It also reduces bottlenecks of code reviews.

Evaluating the Effectiveness of Pull Request Processes

• Look at production outages and tie them back to the peer reviews.
• The pull request should have good information about what the context of the change is:
  • Sufficient detail on why the change is being made,
  • How the change was made, and
  • Any risks associated with it.

Fearlessly Cut Bureaucratic Processes

• The goal should be to reduce the amount of outside approvals, meetings, and signoffs that need to happen to deploy the application.

Resources We Like

• The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations (Amazon)
• The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win (Amazon)
• The Unicorn Project: A Novel about Developers, Digital Disruption, and Thriving in the Age of Data (Amazon)
• Multivariate Testing vs A/B Testing (optimizely.com)
• Esty’s Feature API (GitHub)
• Pair Programming vs. Code Reviews (blog.codinghorror.com)

Tip of the Week

• Allen’s love of all things Costco has been extended to their credit card. Of course it has. (Costco)
• Doom Emacs – A configuration framework tailored for those that want a faster, stable environment with less framework in their frameworks. (GitHub)
• Create (and commit!) an EditorConfig based on your current project with IntelliCode in Visual Studio. (docs.microsoft.com)
• Allen’s less trick: Is it possible to keep the output of less on the screen after quitting? (Stack Overflow)
• Head in the Clouds podcast (libsyn.com)
• GitHub CLI (!!!) (github.blog)
• Speed up your k8s development on Windows with WSL: Developing for Docker + Kubernetes with Windows WSL (Medium)
• Configure bash completion for your kubectl commands: kubectl Cheat Sheet (kubernetes.io)
  • We previously discussed this cheat sheet during episode 107, but didn’t highlight the bash completion at that time.