Throughout the blogs on this website, DevOps terminology often bleeds through the cloud concepts we discuss because I work as a DevOps Engineer. A friend recently tagged me in the video below that beautifully captures my life 😀

The video hit over 2 million views on LinkedIn in like a week because I think it resonates so well with all of us involved with DevOps. A major component of DevOps is building DevOps Pipelines – and as this video illustrates, it doesn’t just happen – and it often isn’t pretty. Today we are going to discuss what Pipelines are and how unbelievably useful they can be!

What is a DevOps Pipeline

When you ponder on the word ‘pipelines’, you often think of how either oil or water flows through metal pipes. You can imagine all the complex turns pipes make as they split and join together and move some liquid from a source to the end goal. Kind of like oil pipelines that transport oil around ginormous facilities or plumping to get water to your apartment in the city. In DevOps, we also build these pipelines that primarily allow us to transport software from the ‘source files’ all the way to working applications that users can interact with.

There are several vendors that offer tools with which these pipelines are run. I have mostly used Jenkins in my work as well as Azure DevOps. A lot of the work is done by making sure everything can talk to each other by correct configuration, but most importantly by writing good Scripts. Well-written scripts are at the centre of DevOps pipelines.

These tools, using Groovy (Java) or YAML, allow you to provide step-by-step instructions for each stage of the pipeline. Jenkins or Azure DevOps will then make sure to precisely execute these steps whenever the pipeline is triggered. This way, pipelines allow for continuous integration and continuous delivery of code – an essential part of working DevOps. 

Pipelines allow for automatic build and deployments in a consistent manner, saving valuable time and allowing new code to be of value to end-users quicker. Let’s look at these build and deployment pipelines.

Build Pipelines

Build pipelines take care of the first part of the process. Often a group of developers has source code sitting in a folder, often a GitHub repository (collective platform for version control and storing source code), which needs to be put together (compiled) into a single artifact such as a Docker Image so it’s ready to run.

When I used to start I always wondered why there needed to be a whole pipeline set up to accomplish this. Isn’t once enough? Well, this is because software development is an ongoing process. Once developers have created a working application, there are always new things that need change. 

There is code that needs updating, bugs to fix, and features to add on a continual basis. It’s not uncommon for large companies like Netflix and Amazon to have hundreds if not thousands of little changes made to the source code and deployed per day. This would be a horribly painful process to have to do manually for every little change. 

There are several forms of these build pipelines, but the most common ones make sure that once code changes are made to the application’s source code, a new version of the application image is created for use. This new image, or artifact, is then published to cloud platforms that can hold the different versions of these images such as Dockerhub or Azure Artifacts. 

Deploy Pipelines

Deploy pipelines are responsible for the second part of the process. They keep an eye on any new images/artifacts/packages that are deployed in Azure Artifacts or Dockerhub by the build pipelines. When a new version is available, this pipeline is responsible for picking up the image and making sure that it is pushed to the correct servers, configured correctly and made available to be used by end-users.

As each new source code change can lead to a new artifact being published, it also means there are constantly deployment pipelines running to keep the environments running updated with the latest available images.

Get started with DevOps Pipelines

Pipelines are really fun to work on, but they are so valuable because they allow new code to be available faster. From a business perspective, good pipelines ensure quicker time to market which can provide a competitive advantage. DevOps pipelines truly are essential for modern-day software development. To get started, I suggest reading up on Azure DevOps pipelines. You can always reach out to me for further information as well. This blog is littered with several elements essential to running DevOps Pipelines such as Infrastructure of Code and Containerization – I would check those out as well if you are interested!