Crossplane vs Cloud Provider Infrastructure Addons

Kubernetes has demonstrated the power of a well architected control plane with a great API. The industry is beginning to notice that this control plane can be used to do much more than orchestrate containers, and are increasingly looking to use the Kubernetes control plane to manage all of their infrastructure.

Several “cloud provider infrastructure” addons exist for Kubernetes. These addons provide a Kubernetes interface to a cloud provider’s infrastructure - databases, queues, etc. The three major clouds each maintain their own - Google Config Connector, Azure Service Operator, and Amazon Controllers for Kubernetes. Each exposes their respective cloud’s control plane APIs as custom resources, allowing Kubernetes users to manage an RDS instance (for example) using the same tools they would use to manage a Deployment or a ConfigMap.

Crossplane is often compared to the various cloud provider infrastructure addons. There are certainly similarities - it is also a Kubernetes addon, and it also exposes all of the major cloud provider control plane APIs as custom resources. Crossplane even shares code with some of these addons. Where Crossplane differs is in how we expose cloud provider APIs.

The Crossplane community believes that the typical developer using Kubernetes to deploy their application shouldn’t have to deal with low level infrastructure APIs.

Drawing on our experiences as platform builders, SREs, and application developers we’ve designed Crossplane as a toolkit to build your own custom resources on top of any API - often those of the cloud providers. We think this approach is critical to enable usable self-service infrastructure in Kubernetes.

In this post we’ll demonstrate that seemingly simple tasks like spinning up a new database in the cloud for your applications to consume can often be more complicated than it would at first seem, and how Crossplane is designed to help tame that complexity.

Crossplane today consists of three things:

1. Providers extend Crossplane with custom resources that can be used to declaratively configure a system. The AWS provider for example, adds custom resources for AWS services like RDS and S3. We call these ‘managed resources’. Managed resources match the APIs of the system they represent as closely as possible, but they’re also opinionated. Common functionality like status conditions and references work the same no matter which provider you’re using - all managed resources comply with the Crossplane Resource Model, or XRM.
2. Composition allows a platform team to define new custom resources that are composed of managed resources. We call these composite resources, or XRs. An XR typically groups together a handful of managed resources into one logical resource, exposing only the settings that the platform team deems useful and deferring the rest to an API-server-side template we call a ‘Composition’.
3. Packages allow a platform team to quickly package, share, and declaratively install new kinds of composite resources and the providers they build on.

Despite the name, “provider” doesn’t necessarily mean “cloud provider”. Crossplane has providers that add support for managing databases on a SQL server, managing Helm releases, and ordering pizza.

When we founded the Crossplane project we started at the managed resource layer. We focused on the resources we thought would be the most useful to application developers - things like cloud databases, caches, and storage buckets. We quickly heard from early adopters that being able to spin up these resources alone wasn’t enough. An RDS instance might also need a security group or a subnet group to be reachable. GCP Cloud SQL instances and Azure SQL servers face similar issues. This pattern permeates cloud infrastructure; another example we see often is folks wanting to create an IAM policy for each DynamoDB table they create.

Let’s dig into the example of an application developer requesting an SQL database for their application to use. An experience you initially hope will look like this:

apiVersion: database.aws.crossplane.io/v1beta1
kind: RDSInstance
metadata:
  name: example-rds
spec:
  forProvider:
    dbInstanceClass: db.t3.medium
    engine: mysql
    allocatedStorage: 20

Can easily end up looking like this:

apiVersion: database.aws.crossplane.io/v1beta1
kind: RDSInstance
metadata:
  name: example-rds
spec:
  forProvider:
    region: "us-west-2"
    dbInstanceClass: "db.t3.medium"
    engine: mysql
    engineVersion: "5.7"
    dbSubnetGroupName: external-subnet-group
    vpcSecurityGroupIDRef:
      name: example-sg
    masterUsername: cooladmin
    skipFinalSnapshotBeforeDeletion: false
    publiclyAccessible: true
    allocatedStorage: 20
    autoMinorVersionUpgrade: true
    backupRetentionPeriod: 30
    caCertificateIdentifier: "rds-ca-2019"
    copyTagsToSnapshot: true
    deletionProtection: true
    enableIAMDatabaseAuthentication: false
    enablePerformanceInsights: true
    performanceInsightsRetentionPeriod: 7
    finalDBSnapshotIdentifier: example-rds-snapshot
    licenseModel: general-public-license
    multiAZ: true
    port: 3306
    preferredBackupWindow: "06:15-06:45"
    preferredMaintenanceWindow: "sat:09:21-sat:09:51"
    storageEncrypted: false
    storageType: "gp2"
  writeConnectionSecretToRef:
    namespace: app-team-a
    name: example-rds
---
apiVersion: ec2.aws.crossplane.io/v1beta1
kind: SecurityGroup
metadata:
  name: example-sg
spec:
  forProvider:
    region: us-west-2
    vpcId: externally-managed-vpc
    groupName: crossplane-getting-started
    description: Allow access to MySQL
    ingress:
      - fromPort: 3306
        toPort: 3306
        ipProtocol: tcp
        ipRanges:
          - cidrIp: "10.0.10.0/0"
            description: Production
---
apiVersion: mysql.sql.crossplane.io/v1alpha1
kind: ProviderConfig
metadata:
  name: example-rds
spec:
  credentials:
    source: MySQLConnectionSecret
    connectionSecretRef:
      namespace: platform-infra
      name: example-rds
---
apiVersion: mysql.sql.crossplane.io/v1alpha1
kind: Database
metadata:
  name: example
spec:
  providerConfigRef:
    name: example-rds
---
apiVersion: mysql.sql.crossplane.io/v1alpha1
kind: User
metadata:
  name: example-user
spec:
  providerConfigRef:
    name: example-rds
  writeConnectionSecretToRef:
    namespace: app-team-a
    name: example-rds-user
---
apiVersion: mysql.sql.crossplane.io/v1alpha1
kind: Grant
metadata:
  name: example
spec:
  providerConfigRef:
    name: example-rds
  forProvider:
    privileges:
    - SELECT
    - INSERT
    - DELETE
    - UPDATE
    - EXECUTE
    userRef:
      name: example-user
    databaseRef:
      name: example

You might notice that we create a database and a user in the above example - something the RDS API does not support natively.

Sometimes spinning up practically functional infrastructure requires multiple providers - even if you only use one cloud. In this case we use the SQL provider to finish what we started with the AWS provider.

This will be an imposing amount of configuration for some application developers. They want to be able to self service their infrastructure needs, but shouldn’t need to be experts in the features and functionality of cloud provider APIs to do so. This is where Composition comes in - it allows a platform team to offer their application developers an experience much closer to the one they originally envisioned. Using Composition the application developer experience can look like this:

apiVersion: example.org/v1
kind: ExampleCoDatabase
metadata:
  name: example
spec:
  parameters:
    storageGB: 20
    username: example-user
  compositionSelector:
    matchLabels:
      engine: mysql
      class: production
  writeConnectionSecretToRef:
    name: example-database-credentials 

This is a composite resource - an XR.

When an application developer creates, updates, or deletes this XR Crossplane can create, update, or delete the more verbose set of managed resources from the previous example.

Composition allows the infrastructure experts - the platform team - to determine what settings their application developers need, and how to use those settings to produce an RDS instance, security group, database, etc. In this example the application developer can influence the size of the database but not its backup settings, which are enforced by the platform team.

You may wonder why a platform team would use Composition rather than a familiar, existing tool like Helm or Kustomize. While it’s true that there are similarities, we feel that it’s better to frame your organisation's opinions at the API level, rather than via client-side tooling. For example when you use Crossplane to expose the above purpose-built ExampleCoDatabase API:

• Any REST client (from kubectl to curl to Python) can create an ExampleCoDatabase with a single API call. This fosters automation and eases integration with other systems.
• Policy is enforced. RBAC can ensure at the API level that application developers may influence only the fields exposed by an ExampleCoDatabase XR. The platform team can ensure application developers have RBAC access to configure the settings they need, and nothing else.
• RBAC is framed around your desired abstractions. Access is granted “to create an ExampleCo database”, not “to create an RDS instance, a security group, etc etc”.

One novel design folks notice when they start using Crossplane is that most Crossplane custom resources are cluster scoped - they exist above the scope of any Kubernetes namespace. While this can seem odd at first, it’s another decision that was informed by real world scenarios.

Take the above ExampleCoDatabase XR. Assume that several teams of application developers, each with their own namespace, want to create their own ExampleCoDatabases. Now imagine the platform team also wants to use Crossplane to manage the VPC network to which these databases should be attached. We have several resources spread across several different namespaces wanting to share a single resource.

In Kubernetes one resource specifies that it uses another with a reference. However API conventions do not allow an object in one namespace to reference an object in another. We can’t simply create a ‘platform-infrastructure’ namespace in which to create our VPC and have each ExampleCoDatabase reference it - doing so would violate isolation boundaries. When a resource is shared by other resources spread across many namespaces the shared resource must be cluster scoped. Many (namespaced) pods consuming a (cluster scoped) node is the quintessential example.

Platform teams can use Crossplane to create new kinds of composite resources, so we can’t know ahead of time whether an XR will be ‘node-like’ (shared by resources spread across many namespaces) or not. To accommodate this while keeping our architecture as conceptually and referentially simple as possible, we:

• Make all managed resources cluster scoped.
• Make all composite resources cluster scoped.
• Allow platform teams to offer a namespaced ‘claim’ for any composite resource.

A composite resource claim is a namespaced proxy for an XR. It has the same schema as the XR it represents. When you create, update, or delete a claim, a corresponding XR is created, updated, or deleted accordingly.

In the example above the platform team would offer a claim when defining their ExampleCoDatabase XR, thus allowing application developers to create an ExampleCoDatabase in the same namespace in which they create the application deployment that will consume it. Conversely, the platform team would not offer a claim for the VPC. This approach makes references predictable and easy to follow, as compared to the alternative of eschewing claims and allowing some XRs (and by extension some managed resources) to be cluster scoped while others were namespaced.

This post has highlighted some of the subtleties we believe modern enterprises will face as they seek to enable their developers to self service their infrastructure needs. The Crossplane maintainers are supremely thankful for the community of platform builders and application developers who have helped us uncover such subtleties and evolve our architecture accordingly since the project was founded in 2018. We think Crossplane is a great way for platform teams to empower the developers they support to self-service their infrastructure needs. Check out our guide to getting started with Crossplane if you’d like to try it for yourself, and reach out to us on Slack if you have any questions or feedback about Crossplane.