In another post, we talked about blue-green deployments. In this one, we will learn about another implementation strategy, which shares the same advantages but is less risky and often leads to better insights.
Continuous integration has changed the way we develop software. But an IC environment is different from production, and synthetic testing isn’t always enough to reveal problems. Some problems only appear when they reach production, and by that time, the damage is already done. The Canarian deployments allow us to test the waters before jumping.
What is Canary Deployment
? In software engineering, canary deployment is the practice of making phased releases. We first deploy a software update to a small portion of users, so they can test it and provide feedback. Once the change is accepted, the update is extended to all other users.
Canary implementations show us how users interact with real-world application changes. As in blue-green deployments, the Canarian strategy offers zero-downtime upgrades and easy rollbacks. Unlike blue-green, Canarian implementations are softer and failures have limited impact.
Versions vs. implementations
A Canarian version is an early compilation of an application. Splitting the stable and development branches is a widespread strategy in the open source world. Many projects use an even and odd numbering scheme to separate the stable version from the unstable version.
Often, companies publish Canarian versions of their products, hoping that tech-savvy or advanced users will want to download and try them out. Examples of companies that canarying their applications are Mozilla and its nightly and beta versions of Firefox, and Google, with its Canarian launch channel for Chrome.
In a Canary deployment, on the other hand, we install the update on our systems and divide users into two groups. A small percentage of them will go to the canary while the rest will stay in the previous version, as a control.
Later, once we have taken the time to evaluate the canary version, we can decide to migrate the rest of the users to the canary or revert everyone to the previous version.
Read more: What’s the difference between continuous delivery and continuous deployment?
How they work
Canarian implementations As we have seen, Canarian implementations involve running two versions of the application simultaneously. We will call the old version “the stable” and the new one “the canary”. We have two ways to deploy the upgrade: continuous deployments and side-by-side deployments.
Let’s see how they work.
In
a
continuous deployment, we install changes in waves or stages, a few machines at a time. The others continue to run the stable version. This is the easiest way to make a Canarian implementation.
As soon as the canary runs on a server, some users start seeing updates.
Deployment While this is happening, we are looking at how the updated machines are performing. We check for bugs and performance issues, and listen to user feedback.
As we rely on the canary, we continue to install it on the rest of the machines, until they are all running the latest version.
If we detect a failure or get disappointing results, we can undo the change by reverting the upgraded servers to their initial state.
Implementations
In parallel The side-by-side strategy has much in common with blue-green implementations. Instead of upgrading the machines in stages, we created a completely new duplicate environment and installed the Canarian version there.
Assume that your application runs on multiple computers or containers, some services, and a database.
To deploy, we clone the hardware resources and install the updates. Once the canary runs in the new environment, we show it to a portion of the user base. This typically occurs through a router, load balancer, reverse proxy, or some other application business logic.
version
As in continuous deployments, we monitor the canary as we gradually migrate more and more users out of the control version. The process continues until we detect a problem or all users are in the canary.
After the deployment is complete, we delete the control environment to free up resources. The Canarian version is now the new stable.
Benefits of
Canary implementations
Why go to the trouble of implementing a Canarian strategy? The benefits are many
: A/B testing
- : we can use the canary to do A/B testing. In other words, we present two alternatives to users and see which one gets better reception.
- Capacity testing: It is impossible to test the capacity of a large production environment. With Canarian deployments, capacity testing is integrated. Any performance issues we have in our system will start to arise as we slowly migrate users to the canary.
- Feedback: We receive invaluable information from real users.
- No cold starts: New systems can take a while to get up and running. Canary deployments slowly increase momentum to avoid slow cold starting.
- Like blue-green deployments, a Canarian deployment generates no downtime
- Easy rollback: If something goes wrong, we can easily revert to the previous version.
No downtime:
.
The
first Canary Islands The
idea of using a canary as an early warning system has a long history. Long before Google or Netflix used them, coal miners carried real canaries to find gas leaks. When these birds, more susceptible than humans to odorless fumes, retreated, it was time to get out of the mine.
Unsurprisingly, as technology advanced, things got better. Today, cloud technologies make things less dangerous for wildlife and much more practical
:
- Canarian launches: As long as we have some way to update the software remotely, we can do Canarian launches. App stores are a great example of this. Both Google Play and Apple’s App Store support staged releases. This feature allows us to send updates in waves, to a certain percentage of users at a time.
- Rolling canaries: we have numerous tools such as AWS CodeDeploy, Chef, Puppet or Docker to help us make continuous updates.
- Canaries side by side: the cloud allows us to create and disassemble hardware and services on demand. We have tools like Terraform, Ansible, or AWS CloudFormation to define infrastructure using code.
- CI/CD: When we add continuous delivery and deployment to the mix, we get one of the most effective patterns for submitting code.
Kubernetes
Kubernetes deserves a separate section because it is specifically designed to deploy complex distributed systems. Kubernetes is a popular orchestration platform that has built-in tools for continuous and parallel deployments. This platform is ideal for Canarian deployments, as it allows us to deploy, update and scale applications using only code.
If you want to learn about Kubernetes and see a step-by-step Canarian deployment in action, download our free CI/CD eBook with Docker Kubernetes.
Planning
the canary
We have to take into account a few things when planning a
Canarian deployment: Stages: how many users we send to the
- canary at the beginning, and in how many stages.
- Duration: How long do we plan to run the canary? Canary deployments typically run for minutes or hours. The Canary versions are different, as we must wait for enough clients to be updated before we can evaluate the results. This can happen over several days or even weeks.
- Metrics: What metrics we need to record to analyze progress, including app performance and bug reports. Well-chosen parameters are essential for a successful Canarian deployment.
- Evaluation: what criteria we will use to determine the canary is a success.
For starters, we could try the logarithmic approach, which uses steps from 1-10-100. That is, start by sending 1% of the user base to the canary to make sure there are no serious problems. Then, we increase the canary to 10% to see how it behaves under load. Finally, we migrated 100% of the user base and completed the process.
We can have as many stages as we need. Perhaps two stages of 2-100 will suffice, while some may prefer gradual steps of 10-20-50-100.
<img src="https://wpblog.semaphoreci.com/wp-content/uploads/2020/08/canary-stages.png" alt="Various
Strategies for migrating
users Once we have
planned stages, we have to define how we will choose the users who participate in the
canary:
- Randomly: we can send a percentage of users to the canary by chance.
- By region: deploy the canary one geographical area at a time. For example, we could choose a nightly tracking strategy and launch overnight in each region, when there are fewer users online.
- Early adoption program: Giving users the opportunity to opt in (or opt out) of the canary program could lead to the best results. Early adopters are more likely to offer quality feedback.
- Dogfooding: This term relates to the saying “eat your own dog food” and involves releasing the canary to internal users and employees first.
The best selection criteria can combine different strategies. Facebook, for example, uses multiple canaries. The first is deployed internally and goes to your employees. Then a second wave goes to a small portion of users. Later, they gradually increase it to the entire population.
Disadvantages of Canarian deployments
Nothing is perfect. And a Canarian deployment is no exception. Let’s look at the other side of the Canarian deployments.
- Frustration: The first group to use the canary will find the worst mistakes. In addition, some users may be discouraged to learn that they were used as guinea pigs. If you’re concerned about this, consider starting an optional program (call it something like an “early adopter” or “insider program” for freshness).
- Costs: The cost of parallel deployments is higher because we need additional infrastructure. Use the cloud to your advantage; Create and eliminate resources on demand to keep costs down.
- Complexity: Canarian implementations share the same complexities as blue-green implementations. Have many production machines, migrate users and monitor the new system; These are complicated tasks. Avoid making them by hand at all costs. Always automate the deployment process using a CI/CD platform like Semaphore.
- Time: Setting up a healthy Canary deployment pipeline takes time and effort. On the plus side, once we get it right, we can make deployments more frequent and secure.
- : Entire books have been written on how to make changes to the database schema. The problem is that the database must work simultaneously with the canary and control versions during deployment. So if we have breakup scheme changes, we’re in trouble. We need to maintain backward compatibility as we make changes, which adds another layer of complexity.
Databases
The Canary Islands are not for
everyone
Unfortunately, not everyone will be able to use Canarian implementations. There are some cases in which we will not be able to use the strategy:
When
- we work in environments that will not allow any type of continuous deployments, such as in the medical or aerospace industries
- it comes to mission-critical or life support systems, such as applications for power grids or nuclear reactors. When
- a failure has disastrous economic consequences, as in financial systems.
- When we need to make break schema changes to the database or storage backend.
- When we depend on software installed on the user’s computers and have no way to update it remotely.
. When
Canary vs. Blue-Green
We have discussed the benefits and disadvantages of blue-green implementations in a previous post. Now that we’ve learned about canaries, the million-dollar question is: which is better?
There are many variables to consider and never a single answer that works for everyone. But as a general rule, we should consider blue-green binary implementations if any of these are true:
- we trust the new version. We have tested the code thoroughly, and estimate that the probability of failure is quite low.
- We are working on small safe iterations.
- We need to change all users at once.
Canary is probably a better choice when:
- We are not 100% sure of the new version, we believe we have a low to decent probability of failure
- are concerned about performance or scaling issues
- We’re doing a major update, like adding a brilliant new or experimental feature
- We agree with a slow rollout.
- Our application depends on a legacy or third-party system that we cannot replicate in a test environment. In this case, testing is only possible in live production systems.
. We
.
.
The
real test comes when people start using our software. Canary deployments allow us to do controlled tests with real users. When we combine this strategy with a fast CI/CD workflow, we get a productive and feature-rich release cycle.
Check out these resources to learn more about deployments.
- Download our free eBook: CI/CD with Docker and
- about blue-green deployments
- Read the step-by-step guide to continuous deployment on Kubernetes.
Kubernetes. Read
.
Thanks for reading!