Insights into how we designed and built Crossplane Services to support secure connectivity, starting with a solid foundation of cloud-specific, high-fidelity resources, and then layering a PVC-style claims/classes model on top. Resource classes now include secure connectivity options and new networking and security resources can be configured from kubectl. This allows complete environments (GKE, networks, subnets, and managed service classes) to be defined using kubectl, so apps can securely consume the cloud services they need.
Continuing our series on cloud networking & security we explore Crossplane’s newfound ability to enable secure network connectivity between cloud services.
When building Crossplane Services, we started with the services we hoped would be most useful to the most people, such as SQL databases, Kubernetes clusters, storage buckets, and Redis caches. A common theme in the feedback we received from early adopters was that they wanted Crossplane to allow them to use kubectl to connect their cloud services securely, in the same way it allowed them to use kubectl to deploy their cloud services. It was frustrating to have to leave the Crossplane ecosystem to setup things like Virtual Private Cloud (VPC) networks, peerings, and subnets before you could start deploying cloud services like SQL databases with Crossplane. We're big believers in listening to our community, so we made improving this a priority for v0.3.
Building a Solid Foundation
It can be tempting when building for a multicloud world to start from the abstract and work backward to the details. Crossplane aims to make deploying cloud services a portable experience, but we've learned through trial and error to first build a solid foundation. This ensures we can build a great abstraction once we've explored the problem space.
Fortunately, Crossplane employs a layered architecture. The lowest layer consists of managed resources. Managed resources are high fidelity representations of the API resources that make up a cloud service. They're not portable across clouds. A CloudSQLInstance is an example of a managed resource - it's relevant only to the Google Cloud Platform (GCP) and exposes all of the nitty gritty configuration details of a CloudSQL instance. Resource claims and classes are the next layer up. Resource claims like MySQLInstance build a portable abstraction atop managed resources. This layered approach means we can build the foundation of network connectivity resources using managed resources.
In Crossplane v0.3 we started with a common use case - deploying an SQL database and a Kubernetes cluster whose pods may securely access said database. We frequently imagine two personas using Crossplane - the infrastructure operator and the application operator. The infrastructure operator is opinionated about cloud specific details - they care how many IOPS a production database has, and what VPC network it's in. The application operator on the other hand just wants to create a production database to use with their application. They trust the infrastructure operator to define a class of database that meets their production needs. In our use case the infrastructure administrator would set up all the connectivity plumbing - provisioning VPC networks etc - using managed resources, then expose two resource classes configured to use it. One for the SQL database and another for the Kubernetes cluster. This enables application operators to simply make a resource claim for their SQL database and a resource claim for their Kubernetes cluster and trust that the resulting managed resources will be securely connected.
Putting it into Action on Google Cloud Platform
Let's walk through a concrete example in which an infrastructure operator wants to configure Crossplane such that an application operator who requests a SQL database and a Kubernetes cluster will have their need satisfied by a dynamically provisioned Kubernetes Engine (GKE) cluster and a Cloud SQL instance. The infrastructure operator could just as easily configure Crossplane to satisfy such requests using an Azure Kubernetes Service cluster and SQL Database, or an Amazon Elastic Kubernetes Service cluster and RDS database, but we'll focus on GCP for this example.
The simplest way to enable secure connectivity from a GKE cluster to a Cloud SQL instance is to use private IP addresses. When using private IP addresses the Cloud SQL instance (which always runs in a Google-managed VPC network) uses private services access to peer with the VPC network in which your GKE cluster's pods run. The steps to configure this from scratch are:
- Create a VPC network and subnetwork for your GKE cluster.
- Create a Service Networking connection to peer the VPC network you created in step 1 to the VPC in which private Cloud SQL instances are created.
- Create a VPC native GKE cluster in the VPC from step 1.
- Create a Cloud SQL instance with a private IP peered to the VPC from step 1.
The Infrastructure Operator: Paving the Way
This process looks very similar for a Crossplane infrastructure operator, except that they won't create an actual GKE cluster and Cloud SQL instance in steps three and four, but will instead create resource classes that teach Crossplane how to dynamically provision a GKE cluster and a Cloud SQL instance when an application operator needs a Kubernetes cluster and SQL database.
First, let's create the VPC network and subnet. Crossplane models these as high fidelity managed resources. High fidelity is our way of saying "the YAML closely matches the underlying cloud provider API". This means Crossplane lets you configure anything you could configure in the GCP console or via gcloud:
---
apiVersion: compute.gcp.crossplane.io/v1alpha2
kind: Network
metadata:
name: example
namespace: gcp
spec:
providerRef:
name: example
namespace: gcp
name: private
autoCreateSubnetworks: false
---
apiVersion: compute.gcp.crossplane.io/v1alpha2
kind: Subnetwork
metadata:
name: example
namespace: gcp
spec:
providerRef:
name: example
namespace: gcp
name: gke
region: us-central1
ipCidrRange: 192.168.0.0/24
privateIpGoogleAccess: true
secondaryIpRanges:
- rangeName: pods
ipCidrRange: 10.0.0.0/8
- rangeName: services
ipCidrRange: 172.16.0.0/16
network: projects/example-project/global/networks/privateIn the above example we create a VPC network, then create a subnetwork in which the GKE cluster will be deployed. Nodes will have their IP addresses allocated from the 192.168.0.0/24 range, while pods will have their IP addresses allocated from the 10.0.0.0/8 range. Note that the subnetwork references the network via its .spec.network field. These resources are automatically created by Crossplane as soon as they're submitted via kubectl.
Next we'll enable private services access to Cloud SQL instances from the VPC we've just created. This requires allocating an IP range known as a "global address" that will be used by private Cloud SQL instances. Again, we use high fidelity managed resources:
---
apiVersion: compute.gcp.crossplane.io/v1alpha2
kind: GlobalAddress
metadata:
name: example
namespace: gcp
spec:
providerRef:
name: example
namespace: gcp
name: example-range
purpose: VPC_PEERING
addressType: INTERNAL
prefixLength: 16
network: projects/example-project/global/networks/private
---
apiVersion: servicenetworking.gcp.crossplane.io/v1alpha2
kind: Connection
metadata:
name: example
namespace: gcp
spec:
providerRef:
name: example
namespace: gcp
parent: services/servicenetworking.googleapis.com
network: projects/example-project/global/networks/private
reservedPeeringRanges:
- example-rangeHere we've reserved an arbitrary /16 range of internal IP addresses for VPC peering purposes within the example-network VPC network that we created earlier. We then use that IP range to establish a service networking connection to our VPC network.
Now we start to get into Crossplane's meat and potatoes! Dynamic provisioning. Unlike the previous step we're not going to create managed resources, but instead create resource classes. Resource classes - a class of resource - are a template used to create managed resources on demand when an application operator needs one. They're high fidelity, just like managed resources. Let's teach Crossplane how to create GKE clusters and Cloud SQL instances that are connected using the network connectivity primitives we created earlier:
---
apiVersion: database.gcp.crossplane.io/v1alpha2
kind: CloudsqlInstanceClass
metadata:
name: prod-secure
namespace: gcp
specTemplate:
providerRef:
name: example
namespace: gcp
databaseVersion: POSTGRES_9_6
tier: db-custom-1-3840
region: us-central1
storageType: PD_SSD
storageGB: 100
privateNetwork: projects/example-project/global/networks/example-network
---
apiVersion: compute.gcp.crossplane.io/v1alpha2
kind: GKEClusterClass
metadata:
name: prod-secure
namespace: gcp
specTemplate:
providerRef:
name: example
namespace: gcp
machineType: n1-standard-4
numNodes: 3
zone: us-central1-b
network: projects/example-project/global/networks/private
subnetwork: projects/example-project/regions/us-central1/subnetworks/gke
enableIPAlias: true
clusterSecondaryRangeName: pods
servicesSecondaryRangeName: servicesOnce the above resource classes are submitted via kubectl Crossplane knows how to create GKE clusters and Cloud SQL instances that are privately connected. The key here is that both resource classes will dynamically provision managed resources connected via the private VPC network - Cloud SQL instances will have their privateNetwork field set to enable private IP access, and GKE clusters will have their nodes and pods created in the gke subnetwork of the private VPC network.
The last thing the infrastructure operator must do is 'publish' these cloud provider specific resource classes as portable resource classes. This allows resource claim authors - application operators - to easily discover the resource classes they can use. Let's assume the application operators in this case are the ACME team, who deploy their applications into the acme-team namespace:
---
apiVersion: database.crossplane.io/v1alpha1
kind: PostgreSQLInstanceClass
metadata:
name: prod-secure
namespace: acme-team
classRef:
kind: CloudsqlInstanceClass
apiVersion: database.gcp.crossplane.io/v1alpha2
name: standard-cloudsql
namespace: gcp
---
apiVersion: compute.crossplane.io/v1alpha1
kind: KubernetesClusterClass
metadata:
name: prod-secure
namespace: acme-team
classRef:
kind: GKEClusterClass
apiVersion: compute.gcp.crossplane.io/v1alpha2
name: standard-gke
namespace: gcpThat's it for the infrastructure operator! They've taught Crossplane how to create GKE clusters and Cloud SQL instances that will be securely connected to each other. Now it's time to see how the application operator can benefit from their hard work.
The Application Operator: Deploy secure DBaaS
Frequently an application operator will be deploying an application they write - perhaps their organization's billing service. Crossplane frees up application owners from having to sweat the details about their infrastructure, allowing them to focus on adding value by building and running their applications. In this example the application owner is going to use Crossplane to:
- Create a SQL database and a Kubernetes cluster.
- Deploy their
acme-billingservice to the Kubernetes cluster as a Crossplane Workload, and have it connect to the SQL database.
When deploying an app, resource claims are used to request an abstract resource like a "PostgreSQL instance" or a "storage bucket" rather than cloud provider specific implementations like a Cloud SQL instance. The application operator requests their resources using the following claims:
---
apiVersion: compute.crossplane.io/v1alpha1
kind: KubernetesCluster
metadata:
name: billing-cluster
namespace: acme-team
labels:
workload: billing-service
spec:
classRef:
name: prod-secure
writeConnectionSecretToRef:
name: billing-cluster
---
apiVersion: database.crossplane.io/v1alpha1
kind: PostgreSQLInstance
metadata:
name: billing-database
namespace: acme-team
spec:
classRef:
name: prod-secure
writeConnectionSecretToRef:
name: billing-databaseIn the above example the application operator has requested a "prod-secure" class Kubernetes cluster and PostgreSQL instance. They don't need to specify which cloud they run on, what VPC they should use, or what machine type they need. Crossplane will immediately provision a GKECluster and a CloudSQLInstance managed resource using the resource classes the infrastructure operator configured earlier. These managed resources will bind to the resource claims that triggered their provisioning when they come online, signaling they are ready for use.
Now the app operator can deploy their billing service workload:
---
apiVersion: workload.crossplane.io/v1alpha1
kind: KubernetesApplication
metadata:
name: billing-service
namespace: acme-team
spec:
resourceSelector:
matchLabels:
workload: billing-service
clusterSelector:
matchLabels:
workload: billing-service
resourceTemplates:
- metadata:
name: billing-service
labels:
workload: billing-service
spec:
# Secrets are propagated to the KubernetesCluster's namespace
# as {resource-template-name}-{secret-name}
secrets:
- name: billing-database
template:
apiVersion: apps/v1
kind: Deployment
metadata:
name: billing-service
labels:
app: billing-service
spec:
selector:
matchLabels:
app: billing-service
template:
metadata:
labels:
app: billing-service
spec:
containers:
- name: billing
image: acme.example.org/billing-service:v0.4.2
env:
- name: DB_HOST
valueFrom:
secretKeyRef:
name: billing-service-billing-database
key: endpoint
- name: DB_USER
valueFrom:
secretKeyRef:
name: billing-service-billing-database
key: username
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: billing-service-billing-database
key: passwordThat's it! The billing service workload is deployed to the KubernetesCluster the application operator created earlier because it matches its cluster selector labels. It will propagate the "connection secret" published by the PostgreSQLInstance that contains the instance's address and credentials to that cluster for the templated deployment to use. More complex workloads could also include a service, ingress, or anything else Kubernetes supports.
What's Next?
In this post we've explored how an infrastructure operator can use resource classes to teach Crossplane how to satisfy an application owner's need for infrastructure by dynamically provisioning securely connected managed resources.
Over the next few Crossplane releases we hope to add support to securely connect more kinds of resources - it's easier than ever for the community to add support for new managed services thanks to out-of-tree stacks and our new developer guide! We'll also be ensuring all Crossplane resources support GitOps, by allowing resources like Subnetwork to refer to resources like Network by their Kubernetes resource name, rather than their 'real name' in the cloud which can sometimes be non-deterministic.
There are many different ways to get involved in the Crossplane project, both from the user side and the developer side. Keep an eye out for more deep dives, and let us know what resources you'd like to be able to provision and manage with Crossplane!
Join the open cloud movement to help level the playing field for everyone!