Kubernetes has demonstrated the power of a well architected control plane with a great API. The industry is beginning to notice that this control plane can be used to do much more than orchestrate containers, and are increasingly looking to use the Kubernetes control plane to manage all of their infrastructure.
Several “cloud provider infrastructure” addons exist for Kubernetes. These addons provide a Kubernetes interface to a cloud provider’s infrastructure - databases, queues, etc. The three major clouds each maintain their own - Google Config Connector, Azure Service Operator, and Amazon Controllers for Kubernetes. Each exposes their respective cloud’s control plane APIs as custom resources, allowing Kubernetes users to manage an RDS instance (for example) using the same tools they would use to manage a
Deployment or a
Crossplane is often compared to the various cloud provider infrastructure addons. There are certainly similarities - it is also a Kubernetes addon, and it also exposes all of the major cloud provider control plane APIs as custom resources. Crossplane even shares code with some of these addons. Where Crossplane differs is in how we expose cloud provider APIs.
The Crossplane community believes that the typical developer using Kubernetes to deploy their application shouldn’t have to deal with low level infrastructure APIs.
Drawing on our experiences as platform builders, SREs, and application developers we’ve designed Crossplane as a toolkit to build your own custom resources on top of any API - often those of the cloud providers. We think this approach is critical to enable usable self-service infrastructure in Kubernetes.
In this post we’ll demonstrate that seemingly simple tasks like spinning up a new database in the cloud for your applications to consume can often be more complicated than it would at first seem, and how Crossplane is designed to help tame that complexity.
Crossplane today consists of three things:
- Providers extend Crossplane with custom resources that can be used to declaratively configure a system. The AWS provider for example, adds custom resources for AWS services like RDS and S3. We call these ‘managed resources’. Managed resources match the APIs of the system they represent as closely as possible, but they’re also opinionated. Common functionality like status conditions and references work the same no matter which provider you’re using - all managed resources comply with the Crossplane Resource Model, or XRM.
- Composition allows a platform team to define new custom resources that are composed of managed resources. We call these composite resources, or XRs. An XR typically groups together a handful of managed resources into one logical resource, exposing only the settings that the platform team deems useful and deferring the rest to an API-server-side template we call a ‘Composition’.
- Packages allow a platform team to quickly package, share, and declaratively install new kinds of composite resources and the providers they build on.
Despite the name, “provider” doesn’t necessarily mean “cloud provider”. Crossplane has providers that add support for managing databases on a SQL server, managing Helm releases, and ordering pizza.
When we founded the Crossplane project we started at the managed resource layer. We focused on the resources we thought would be the most useful to application developers - things like cloud databases, caches, and storage buckets. We quickly heard from early adopters that being able to spin up these resources alone wasn’t enough. An RDS instance might also need a security group or a subnet group to be reachable. GCP Cloud SQL instances and Azure SQL servers face similar issues. This pattern permeates cloud infrastructure; another example we see often is folks wanting to create an IAM policy for each DynamoDB table they create.
Let’s dig into the example of an application developer requesting an SQL database for their application to use. An experience you initially hope will look like this:
apiVersion: database.aws.crossplane.io/v1beta1 kind: RDSInstance metadata: name: example-rds spec: forProvider: dbInstanceClass: db.t3.medium engine: mysql allocatedStorage: 20
Can easily end up looking like this:
apiVersion: database.aws.crossplane.io/v1beta1 kind: RDSInstance metadata: name: example-rds spec: forProvider: region: "us-west-2" dbInstanceClass: "db.t3.medium" engine: mysql engineVersion: "5.7" dbSubnetGroupName: external-subnet-group vpcSecurityGroupIDRef: name: example-sg masterUsername: cooladmin skipFinalSnapshotBeforeDeletion: false publiclyAccessible: true allocatedStorage: 20 autoMinorVersionUpgrade: true backupRetentionPeriod: 30 caCertificateIdentifier: "rds-ca-2019" copyTagsToSnapshot: true deletionProtection: true enableIAMDatabaseAuthentication: false enablePerformanceInsights: true performanceInsightsRetentionPeriod: 7 finalDBSnapshotIdentifier: example-rds-snapshot licenseModel: general-public-license multiAZ: true port: 3306 preferredBackupWindow: "06:15-06:45" preferredMaintenanceWindow: "sat:09:21-sat:09:51" storageEncrypted: false storageType: "gp2" writeConnectionSecretToRef: namespace: app-team-a name: example-rds --- apiVersion: ec2.aws.crossplane.io/v1beta1 kind: SecurityGroup metadata: name: example-sg spec: forProvider: region: us-west-2 vpcId: externally-managed-vpc groupName: crossplane-getting-started description: Allow access to MySQL ingress: - fromPort: 3306 toPort: 3306 ipProtocol: tcp ipRanges: - cidrIp: "10.0.10.0/0" description: Production --- apiVersion: mysql.sql.crossplane.io/v1alpha1 kind: ProviderConfig metadata: name: example-rds spec: credentials: source: MySQLConnectionSecret connectionSecretRef: namespace: platform-infra name: example-rds --- apiVersion: mysql.sql.crossplane.io/v1alpha1 kind: Database metadata: name: example spec: providerConfigRef: name: example-rds --- apiVersion: mysql.sql.crossplane.io/v1alpha1 kind: User metadata: name: example-user spec: providerConfigRef: name: example-rds writeConnectionSecretToRef: namespace: app-team-a name: example-rds-user --- apiVersion: mysql.sql.crossplane.io/v1alpha1 kind: Grant metadata: name: example spec: providerConfigRef: name: example-rds forProvider: privileges: - SELECT - INSERT - DELETE - UPDATE - EXECUTE userRef: name: example-user databaseRef: name: example
You might notice that we create a database and a user in the above example - something the RDS API does not support natively.
Sometimes spinning up practically functional infrastructure requires multiple providers - even if you only use one cloud. In this case we use the SQL provider to finish what we started with the AWS provider.
This will be an imposing amount of configuration for some application developers. They want to be able to self service their infrastructure needs, but shouldn’t need to be experts in the features and functionality of cloud provider APIs to do so. This is where Composition comes in - it allows a platform team to offer their application developers an experience much closer to the one they originally envisioned. Using Composition the application developer experience can look like this:
apiVersion: example.org/v1 kind: ExampleCoDatabase metadata: name: example spec: parameters: storageGB: 20 username: example-user compositionSelector: matchLabels: engine: mysql class: production writeConnectionSecretToRef: name: example-database-credentials
This is a composite resource - an XR.
When an application developer creates, updates, or deletes this XR Crossplane can create, update, or delete the more verbose set of managed resources from the previous example.
Composition allows the infrastructure experts - the platform team - to determine what settings their application developers need, and how to use those settings to produce an RDS instance, security group, database, etc. In this example the application developer can influence the size of the database but not its backup settings, which are enforced by the platform team.
You may wonder why a platform team would use Composition rather than a familiar, existing tool like Helm or Kustomize. While it’s true that there are similarities, we feel that it’s better to frame your organisation's opinions at the API level, rather than via client-side tooling. For example when you use Crossplane to expose the above purpose-built
- Any REST client (from kubectl to curl to Python) can create an
ExampleCoDatabasewith a single API call. This fosters automation and eases integration with other systems.
- Policy is enforced. RBAC can ensure at the API level that application developers may influence only the fields exposed by an
ExampleCoDatabaseXR. The platform team can ensure application developers have RBAC access to configure the settings they need, and nothing else.
- RBAC is framed around your desired abstractions. Access is granted “to create an ExampleCo database”, not “to create an RDS instance, a security group, etc etc”.
One novel design folks notice when they start using Crossplane is that most Crossplane custom resources are cluster scoped - they exist above the scope of any Kubernetes namespace. While this can seem odd at first, it’s another decision that was informed by real world scenarios.
Take the above
ExampleCoDatabase XR. Assume that several teams of application developers, each with their own namespace, want to create their own
ExampleCoDatabases. Now imagine the platform team also wants to use Crossplane to manage the VPC network to which these databases should be attached. We have several resources spread across several different namespaces wanting to share a single resource.
In Kubernetes one resource specifies that it uses another with a reference. However API conventions do not allow an object in one namespace to reference an object in another. We can’t simply create a ‘platform-infrastructure’ namespace in which to create our VPC and have each
ExampleCoDatabase reference it - doing so would violate isolation boundaries. When a resource is shared by other resources spread across many namespaces the shared resource must be cluster scoped. Many (namespaced) pods consuming a (cluster scoped) node is the quintessential example.
Platform teams can use Crossplane to create new kinds of composite resources, so we can’t know ahead of time whether an XR will be ‘node-like’ (shared by resources spread across many namespaces) or not. To accommodate this while keeping our architecture as conceptually and referentially simple as possible, we:
- Make all managed resources cluster scoped.
- Make all composite resources cluster scoped.
- Allow platform teams to offer a namespaced ‘claim’ for any composite resource.
A composite resource claim is a namespaced proxy for an XR. It has the same schema as the XR it represents. When you create, update, or delete a claim, a corresponding XR is created, updated, or deleted accordingly.
In the example above the platform team would offer a claim when defining their
ExampleCoDatabase XR, thus allowing application developers to create an
ExampleCoDatabase in the same namespace in which they create the application deployment that will consume it. Conversely, the platform team would not offer a claim for the VPC. This approach makes references predictable and easy to follow, as compared to the alternative of eschewing claims and allowing some XRs (and by extension some managed resources) to be cluster scoped while others were namespaced.
This post has highlighted some of the subtleties we believe modern enterprises will face as they seek to enable their developers to self service their infrastructure needs. The Crossplane maintainers are supremely thankful for the community of platform builders and application developers who have helped us uncover such subtleties and evolve our architecture accordingly since the project was founded in 2018. We think Crossplane is a great way for platform teams to empower the developers they support to self-service their infrastructure needs. Check out our guide to getting started with Crossplane if you’d like to try it for yourself, and reach out to us on Slack if you have any questions or feedback about Crossplane.