A skeptic’s guide to continuous delivery, Part 1: The business case

This is the first post in our five-part series from guest blogger J. Paul Reed – build engineer, automation enthusiast, and host of The Ship Show podcast.

Software development is moving towards continuity. The recent emphasis on continuous integration, built-in testing, constant monitoring, and feedback telemetry all point toward an overall trend in the software industry: increasing the ability to react. To become real-time businesses.

As organizations explore what these changes mean for them, they invariably discover continuous delivery, commonly known as CD. High-performing organizations like Netflix and Etsy serve as poster children for this new technology, and with their success in the marketplace facilitated by their CD implementations, it’s easy to see why CD is so seductive.

This article – the first in a series on evolving your organization’s software development and release practices toward continuous delivery – takes a look at the business case for making this switch. I’ll pull back the curtain and discuss the work that lies ahead, as well as the decisions and trade-offs needed to push bits out of the building using CD methodologies.

The business case

The benefits CD offers are pretty obvious. In the best of implementations, CD allows for:

1. Faster reaction times – The most obvious benefit of CD is that it allows quick reactions to stimuli, both external and internal – whether it’s a market segment that takes a weird left turn, morphing that P5 feature into a company-saving imperative, or a researcher that uses your flagship application as an example of what-not-to-do at the security conference du jour. What business doesn’t want that?
2. Reduced risk – For many organizations, there’s a non-trivial amount of pageantry around shipping a release. And why shouldn’t there be? New features are finally out to customers. Bugs are fixed. Everyone’s happy, right? The dark secret in many organizations is that shipping a release takes a huge amount of effort on the part of the QA and build/release teams. Under a continuous delivery model, software is “released” constantly (though not necessarily to customers). Therefore, the ceremony and the risk around releasing is reduced. If you’re relying on your release infrastructure daily, you will notice (and resolve!) its deficiencies much quicker than if the process is only undergone once every few weeks or months.
3. Exposed inefficiencies and costs – Whether it’s the latest web service or classic ISO images, every software organization incurs a cost to ship the bits out the door. Often times, these costs aren’t entirely clear to the business. There can be all sorts of forces that obscure how much it really costs to ship a release. Continuous delivery makes this process apparent not only to the business itself, but to the decision makers. When constructing a pipeline, it will become clear where humans are involved, what bottlenecks exist, and what low-hanging automation fruit there is to pick. And once constructed, the pipeline creates a virtuous cycle. Clear incentives for a healthy software delivery dynamic now exist, replacing a nebulous discontent with how long and costly it is to release.
4. Flexible release options – Moving to a continuous delivery model requires the build-out of infrastructure, both in an operational context and a software architecture context. But once that’s done, it provides the business with more flexibility in how it delivers features and fixes. Specific sets of features can be released to specific customers, or released to a subset of customers, to ensure they function and scale as designed. Features can be tested and developed, but left dormant in the product, baking for multiple releases. Your marketing department wants that “big splash” at the yearly industry convention? With continuous delivery, it’s not only possible, it can become a trivial request.

Continuous delivery is not a unicorn

Given all the business benefits, continuous delivery may seem almost magical. It’s not.

It’s just a shift in thinking about how to design, develop, and deliver software, followed by focused investment on the initiatives required to implement it. Where many organizations falter with their CD transformation is this first step: defining and committing to that investment.

Because CD requires a large overhaul in technical processes, operational culture, and organizational thinking, it can often feel like there’s a large hurdle to getting started. The fact that it requires a hefty amount of change and build-out of areas of a company’s software delivery infrastructure that may have been neglected over the years can make it an even more bitter pill to swallow.

The largest initial investment is an agreement that the transition toward CD is a worthy business goal. An important one. One which is not attainable without the participation of all, and one which will require some serious trade-offs to be made during that transition period.

“Let’s spend the next few months working on automated testing and build/release infrastructure, and redesigning how our application is written. We can postpone our feature development,” said no product manager ever. Their outlook makes sense given their responsibilities to react to an ever-shifting market and the pressures to outmaneuver the competition.

But many continuous delivery transformations have been derailed when a product manager convinces the business that a “killer feature” must be the top priority and “all hands must be on deck” to make sure it gets implemented, tested, and shipped on schedule. Often, this is accompanied by promises to get the continuous delivery project back on track once it ships. As with all complex initiatives that require heavy initial investment, we know how this movie ends. (The especially ironic subtext is that often times, the focused investments were just starting to bear fruit in their various areas when the plug was pulled; their benefits simply weren’t yet evident.)

This isn’t to say that all feature development must cease and the product can’t move forward. Rather, all areas of the business – product management, engineering, and engineering support (QA, ops, build/release) – must buy into the long-term benefits CD will bring. That part isn’t hard. But real discipline is required to keep it front-of-mind to ensure the initiative doesn’t stagnate or get hauled out to the proverbial scrap yard altogether.

And let’s be real: There are other areas that will need investment in your pursuit of continuous delivery:

• Automated testing – This includes not only a focus on unit tests, but integration and system tests. They need to be driven in an automated fashion, by a system like Bamboo. One of the hugest investments a company I worked with made was requiring an associated unit (and, where it made sense, integration) test for every bug – (every bug!) – in a particularly important software component. It took 18 months to start seeing the benefits. But by then not only had automated testing become old hat for the teams, its value became more and more obvious with every (shorter and shorter) release cycle.
• Application architectural changes – Moving toward a service-oriented architecture is a must. If your software isn’t SaaS, you’ll need to work toward a component architecture with the necessary versioning and release engineering for the components to be individually shipped. So-called “feature flagging” and “canary releases” can then make use of these componentized services/modules. This is especially difficult for organizations with established products, but it can be done. And it pays for itself by decoupling these complex systems so their interfaces are better understood, which allows teams to release their components on their own cycles.
• Build/test farm – Each checkin for the Firefox web browser triggers thousands of tests, requiring over 200 hours of CPU compute time – to say nothing of the build time. It took a notable hardware (and, in some cases, cloud) investment to be able to support this for each commit. Your current build/test farm will need enough capacity to support CD.
• Focused staffing – Depending on the structure of your organization, getting traction on CD transformation may require hiring some extra heads, or a consultant or two whose sole focus is the infrastructure work required to move to CD. If current engineers can’t be insulated enough to make the needle move, this allows the organization to move forward, and as their work comes together, help the team transition to the Brave New CD World.
• Emphasis on culture – CD fundamentally requires some amount of cultural rewiring because the core structure of CD is a delivery pipeline through which changes flow, not a set of human-controlled gates that must be passed. This structure’s very nature can help to start chipping away at organizational silos. But unless the people in those silos are prepared for that, it can feel like their silo is coming down on top of their heads. (And they’ll understandably fight to do everything to keep it from doing so!)

Starting your CD journey

If all of this sounds a bit discouraging, take heart. It’s a lot of work. It requires change from teams across the organization. It requires focused investment in tooling, hardware, and people. All of that takes time. But all of it is possible.

The first step is acknowledging that trying to transform to continuous delivery “on the cheap,” as part of someone’s “20% time”, or “in our slack time between sprints and releases” is a recipe for spending time (and therefore money) on failure.

But viewing it as an investment that will pay future dividends in making the business able to react – in real time – to the dynamic world that is the software industry, which should help offset the cost of that investment on the balance sheet. And if you factor in the employees that won’t burn out due to “midnight oil” releases, and people who get to work on making their own worlds better instead of propping up a building that’s constantly falling apart, the value becomes obvious.

In the next few articles, we’ll talk a bit more about the specifics of that work, and how you can get your own organization’s continuous delivery journey jump started. Stay tuned!

 

Editor’s note: There’s much more to CD that just tooling, as we’ve seen here. But great tooling doesn’t hurt! Check out Atlassian Bamboo–the first world-class continuous delivery tool on the market.