Following up on our v0.3 release, we wanted to take a deeper dive into some of the advancements and features we rolled out as part of this exciting release. In this first entry in our series of v0.3 deep dives, we’ll explore more of the what, how, and why we undertook a major refactoring effort to move all of the cloud provider functionality out of the core Crossplane code base and into their own new repositories.
What’s a Stack?
First, let’s review more about what a Crossplane Stack actually is. The main purpose of a Stack is to extend Crossplane’s multicloud control plane with new functionality. When you first start up a Crossplane instance, it contains reusable logic for orchestration, provisioning, and scheduling of resources across clouds in a general way. In order to add the knowledge of how to manage resources in specific environments, such as AWS or GCP, you can install cloud- specific stacks that extend Crossplane’s general knowledge with the new environment-specific knowledge contained in those stacks.
Stacks are modular and you can install as many as you like to compose applications with your choice of cloud services. With the big three cloud provider stacks moving out-of-tree, it’s now easier to add support for more clouds and cloud services to Crossplane following this newly established pattern.
Crossplane Stacks so far have fallen into two high level categories: infrastructure and application. Infrastructure stacks will include controllers, types, and logic for managing all the resources and services for an entire cloud provider, such as Microsoft Azure. For example, after installing the Azure stack into Crossplane, it will then be able to provision and manage all sorts of high level managed services in Azure such as Azure Kubernetes Service (AKS) and Azure Cache for Redis, as well as lower level Azure resources such as Azure Virtual Networks.
Application stacks, on the other hand, will teach Crossplane about managing the resources of a single application. For instance, after installing the WordPress stack, your Crossplane instance will have all the necessary domain expertise to deploy and manage not only the WordPress application, but also the platform services that it depends on, such as a managed MySQL database. Application stacks contain all the knowledge to completely automate this for you.
Putting these two categories of stacks together provides the foundation necessary to achieve application portability. Or, in other words, easily and smoothly deploy an application to different cloud providers and environments without having to modify the application itself. Install the AWS stack in order for Crossplane to be able to deploy your applications to AWS. Now you want to deploy that same application to GCP? Just go ahead and install the GCP stack too.
Moving Out
Prior to the v0.3 release, all the cloud provider logic and source code lived all lumped together in the main Crossplane repository. When Crossplane was installed, the full set of controllers that manage the resources supported by all the cloud providers were installed as well. This means that your control plane would have controllers running and resources being consumed that you may very well never want to use.
During the development of v0.3, we underwent a refactoring effort to move all the cloud providers out-of-tree and into their own brand new code repositories. This is a common pattern with numerous benefits that have been demonstrated in other projects such as upstream Kubernetes.
Now when you install Crossplane v0.3, you’re given only the base control plane with its general orchestration and scheduling logic. In order to start deploying applications and resources in the cloud provider of your choice, you simply need to install the stack for that cloud provider.
This approach has a number of advantages beyond giving you only the functionality you actually need, reducing the surface area of running code, and consuming less resources within the control plane. Additionally, this approach increases the flexibility that each cloud provider has for releasing and shipping updates for their stack. When everything was bundled into a single repository, it was impossible to release scoped updates that only affected a single cloud provider. Now, with all the cloud providers completely decoupled from each other and from core Crossplane, they can each ship their own independent releases on their own schedules.
This also allows a stronger sense of ownership and influence by each cloud provider on their individual stack. They can make more independent choices about how to implement features, coding style, build system, etc. since there is no dependency or coupling between any of these cloud provider stacks. When each stack’s code base is owned by the same teams that provide the managed services, it's easier to continuously update them with new features as they become available.
Preserving History
For all you git nerds out there, you may appreciate the process we took to preserve the commit history while moving a lot of code to new repositories. If just the thought of git is already giving you a headache, feel free to skip ahead to the next section!
It was an explicit goal of this migration to have the entire history of all updates to the codebase after it was migrated to a new blank repository. Commit history contains valuable information for troubleshooting and understanding how a portion of the codebase evolved through a number of revisions to its current form. Have you ever used git blame before? It’s a fantastic way to find out who to go ask when you don’t understand something about a particular line or section of code.
git provides a number of (potentially dangerous) ways to manipulate the code base and its history. To solve the problem of moving code and migrating its history at the same time, we utilized the very powerful git filter-branch command. This command essentially enables you to perform an arbitrary transformation on the source code tree, such as removing a bunch of directories, then completely rewrite the tree’s history to only include information on whatever still remains.
For example, the command below completely removes the aws, azure, and gcp directories from the tree, then rewrites the history as if they never existed. After this operation is completed, browsing the history will only show commits that are relevant to what is left in the repository.
For example, the command below completely removes the aws, azure, and gcp directories from the tree, then rewrites the history as if they never existed. After this operation is completed, browsing the history will only show commits that are relevant to what is left in the repository.
git filter-branch -f \
--tree-filter 'find . \( -path "./aws" -o \
-path "./azure" -o \
-path "./gcp" \) \
-exec rm -fr {} +' \
--prune-empty \
HEAD
Similar to what was shown above to remove all cloud provider code from the main Crossplane repository, the inverse operation can be performed that starts with the Crossplane code base and then removes everything that is not the cloud provider of concern. After git removes code and rewrites history, you simply push it to a new repository, and voila! A brand new repository now exists with only the code of the cloud provider you have chosen, but also all of its very important history.
More v0.3 Deep Dives to Come
As mentioned previously, this is just the first v0.3 deep dive in a series that we’ll continue rolling out over the coming weeks. We’re very excited to share more details about all the awesome work that went into the v0.3 release from the Crossplane community, and hopefully we’ll inspire some readers to join us and make contributions of their own!
We’re excited to see additional Stacks being created by the community. If you don’t see a stack for the cloud or cloud service you need please drop us a line on Slack or submit a GitHub issue. If you’re interested in extending or building a stack, be sure to checkout our new developer guide, and reach out with any questions or suggestions, we’re here to help!
There are many different ways to get involved in the Crossplane project, both from the user side and the developer side. Please join us in helping the project continue to grow on its way beyond the v0.3 milestone as we move from alpha to beta over the coming months!