継続的デリバリー

Your guide to continuous delivery


What is continuous delivery?

Continuous delivery is an approach where teams release quality products frequently and predictably from source code repository to production in an automated fashion. A continuous delivery pipeline could have manual gates where the business team makes the final call to release the product to customers.

Check out the diagram below. It illustrates how developers write code on their laptops and commit those changes to a source code repository, like Bitbucket. By code, we mean features, tests for those features, and infrastructure that will be used to deploy and maintain those features. Bitbucket Pipelines can ship the product from test to staging to production, and help customers get their hands on those new features.

Continuous delivery pipeline

継続的インテグレーション

For us to do continuous delivery well, we have to first get good at continuous integration.

Continuous integration is a technique where everyone commits every day, and code is integrated early and often. Continuous integration, as the name suggests, is a method used by scrum teams to confirm their code continually integrates.

When that integration fails, a broken pipeline forces a conversation to happen (one that should have ideally happened earlier). A guiding principle of the continuous paradigm is the idea of fixing broken builds with urgency. This keeps outstanding integration errors to a bare minimum, and prevents technical debt from piling up.

So how does a scrum team do continuous integration? Assume for a minute that our friends from Disney World have come over to lend a hand.

  • Mickey Mouse writes an application and some tests to ensure that basic functionality is working. Mickey makes sure that his changes do not break the pipeline.
  • Donald Duck writes some advanced tests to detect security vulnerabilities and to identify performance bottlenecks. Donald makes sure that the pipeline continues to pass.
  • Minnie Mouse makes sure that an environment in the cloud is ready where the team can deploy and test the application. Minnie creates an account on AWS (Amazon Web Services), creates an S3 bucket, and enables versioning on that bucket.
  • Goofy writes a deployment script that leverages the AWS CLI (Command Line Interface) to publish versioned artifacts to the S3 bucket that Minnie created. Together, Minnie and Goofy test that the pipeline can successfully deploy their application.
  • In the meantime, Scooby Doo integrates the pipeline with a static code analyzer to identify violations of coding best practices, and fixes the errors that were reported.

And so on. Quite the team! They constantly collaborate to make sure that their changes do not break the pipelines.

As a scrum team member, I remember teaching our teams to fix broken builds within the hour. However, they say humans respond to only threats and opportunities. So, we dialed the “education” one notch up by shining the spotlight!

  • We displayed names and mugshots of potential culprits on large-screen TVs in office areas with high foot-traffic.
  • Flashing red lights in the hallways helped draw attention.
  • We sounded sirens that penetrate noise-canceling headphones.
  • We charged a dollar every time a build broke (and later on threw beer bashes with that booty).
  • We tracked MTTR (mean time to resolve) broken builds per team per week, and factored in those metrics during performance evaluations.

These measures helped maintain decorum between teams who relied on each other’s pipelines being operational. Broken pipelines amount to a negative RoI (Return on Investment), and can lead to friction and long meetings, which we would like to avoid.

 

Continuous testing

Continuous testing is a technique where automated tests are integrated with the software delivery pipeline, and validate every change that flows through it.

 

continuous testing

 

Continuous testing is at the heart of continuous integration, delivery, and deployment, and is the secret sauce without which the entire continuous paradigm would crumble to pieces. And contrary to common belief, the architecture of the product that’s flowing through the pipeline has a lot to do with it.

The product architecture is a key factor that determines:

  • The anatomy of the continuous delivery/deployment pipeline.
  • The different phases of the pipeline and what artifacts are produced in each phase.
  • And last but not least, the different types of tests that will be performed in each phase of the pipeline to validate the artifacts produced in that phase.

A monolithic product architecture, or a Big Ball of Mud, can result in a Big Ball of Tests. We recommend investing into microservices such that independently deployable artifacts can flow through pipelines without needing a highly integrated environment for certification. Also, independently deployable artifacts enable faster teams to not get bogged down by slower teams. For more details, see what microservices can bring to DevOps.

Value of continuous delivery

The software delivery pipeline is a product in its own right, and is a priority for business, otherwise we should not send our revenue-generating products through it.

Continuous delivery adds value in three ways. It improves velocity, productivity, and sustainability.

ベロシティ

Velocity means responsible speed and not suicidal speed. Pipelines are meant to ship quality products to our customers. Unless teams are disciplined, pipelines can shoot faulty code to Production, only faster! Automated software delivery pipelines help organizations respond to market changes better.

Productivity

A spike in productivity results when tedious tasks, like submitting a change request for every change that goes to Production, can be performed by pipelines instead of humans. This lets scrum teams focus on products that wow the world, instead of draining their energy on logistics. And that can make team members happier, more engaged in their work, and want to stay on the team longer.

Sustainability

Sustainability is key for all businesses, not just tech. “Software is eating the world” is no longer true — software has already consumed the world! Every company at the end of the day, whether in healthcare, finance, retail, or some other domain, is using technology to differentiate, and to outmaneuver their competition. Automation helps reduce/eliminate manual tasks that are error-prone and repetitive, thus positioning the business to innovate better and faster to meet their customers' needs.

When and who should do CD?

When is a good time to adopt continuous delivery? It was yesterday.

Teams need a single prioritized backlog where:

  • Continuous delivery has been embraced, instead of being relegated to the background
  • The acceptance criteria of user stories explicitly mention automated software delivery approaches instead of manual, and
  • The Sprint DoD (Definition of Done) prevents teams from finishing Sprints where the product shipped manually.

 

 

Continuous delivery is the right thing to do, and occasionally require champions to jumpstart the transformation. Eventually, when designed right, continuous delivery pipelines pay for themselves and organizations, on hindsight, are glad they decided to bite the bullet.

So, who is involved?

Some organizations put inexperienced people to design and implement continuous delivery pipelines, and learnt the hard way that there were deep intricacies involved. Appointing junior members sends the wrong signal to teams, and implies that continuous delivery has a low priority. We strongly recommend putting a senior architect in charge, who has a deep appreciation for technology and business.

 

Beyond continuous delivery

Irrespective of where we are in our journey of continuous everything (integration | testing | delivery | deployment | analytics), it is neither a checklist nor a destination, and continuous improvement is at the heart of it.

Sooner or later, everyone in the organization gets a call when continuous delivery pipelines are being constructed. Executives, Engineering, Product Management, Governance, Risk, Compliance, InfoSec, Operations, Legal and whatever you have. Pipelines gnaw through silos and break down walls. All of us are part of this transformation, one way or the other, and continuous everyone is the new normal!