PostgreSQL across clouds and on-premises with Crossplane and Rook

To cap off an amazing 2018, we introduced Crossplane, an open source multicloud control plane, with its initial v0.1 release right before Kubecon Seattle. We've been thrilled with the reception of the project and the response from the community that is starting to build around it. We are very enthusiastic about collaborating with everyone and continuing to execute on the Crossplane vision of creating a more open cloud computing platform.

We've been hard at work over the holidays and we're ready to share the new support for PostgreSQL that is now available in builds of Crossplane from the master branch.

PostgreSQL is a very popular open source relational database. It's been in active development for over 30 years and has achieved a very high level of reliability and performance, as well as a very robust feature set. If your applications make use of PostgreSQL, you'll now be able to use Crossplane to deploy both your application and its PostgreSQL database to many of the major cloud providers as well on on-premises. We'll get more into how you can start using this new support, but let's first explore a bit of background information on these resources will be provisioned.

Dynamic Provisioning

Dynamic provisioning is a very useful feature that was first introduced for storage volumes in Kubernetes 1.6. Essentially, this concept allows a resource to be created on-demand at deployment time when it is actually needed by a consuming application. The environment specific details and policies of how the resource is provisioned and configured are specified by the administrator in a "blueprint", which frees the application developer from having to worry about any of these details. The application simply needs to express its general need for the resource and at deployment time the resource will be created on-demand in the correct cloud provider or even on-premises. This makes applications much more portable as they can now be written a single way and be able to run in many environments.

Crossplane builds on this concept for many different types of resources (databases, clusters, buckets, etc.), enabling workload portability of complicated applications. Further information about these capabilities can be found in the Crossplane documentation on resource claims and classes.

Using Crossplane to Provision PostgreSQL

Let's start exploring how your applications can use Crossplane to dynamically (on-demand) provision PostgreSQL databases. Specifically, we are going to look at how you can use some simple resource classes and resource claims to deploy PostgreSQL in Google Cloud SQL and also on-premises with the Rook CockroachDB operator. While we will be using Google Cloud SQL in this example, the same could easily be done for AWS and Azure.

PostgreSQL in Google Cloud SQL

Starting with a cloud example, let's look at what the application developer needs in order to provision a PostgreSQL database and also connect it to their application. Below is an example "resource claim" that requests a PostgreSQL instance using version 9.6:

apiVersion: storage.crossplane.io/v1alpha1
kind: PostgreSQLInstance
metadata:
  name: cloud-postgresql-claim
spec:
  classReference:
    name: cloud-postgresql
  engineVersion: "9.6"

Note that the details of how this general need for PostgreSQL will be fulfilled are hidden from the developer beyond the "resource class" named cloud-postgresql. The administrator will have created this "blueprint" that contains all the configuration and policy needed to completely provision the PostgreSQL instance, in this case using Google Cloud SQL:

apiVersion: core.crossplane.io/v1alpha1
kind: ResourceClass
metadata:
  name: cloud-postgresql
parameters:
  tier: db-custom-1-3840
  region: us-west2
  storageType: PD_SSD
  storageGB: "10"
provisioner: cloudsqlinstance.database.gcp.crossplane.io/v1alpha1
providerRef:
  name: gcp-sql-provider
reclaimPolicy: Delete

The last step is to wire up the application to access the database with proper credentials. During the provisioning process, Crossplane automatically stored this information into a secret object. Therefore, we simply need to access the fields from the secret in our application manifest, for example through environment variables, as demonstrated below:

env:
  - name: POSTGRESQL_HOST
    valueFrom:
      secretKeyRef:
        name: cloud-postgresql-claim
        key: endpoint
  - name: POSTGRESQL_USER
    valueFrom:
      secretKeyRef:
        name: cloud-postgresql-claim
        key: username
  - name: POSTGRESQL_PASSWORD
    valueFrom:
      secretKeyRef:
        name: cloud-postgresql-claim
        key: password

That's all that is needed for the developer to on-demand get an instance of PostgreSQL provisioned and connected to their application. Their application didn't have to worry about what particular cloud provider it was being deployed to, the same application works without any changes.

PostgreSQL On-Premises with Rook CockroachDB Operator

Similar to the above example that demonstrated using PostgreSQL in a cloud provider, we can do the exact same thing in a local environment using the Rook CockroachDB operator. The exact same application, without any changes, would be able to continue expressing its general need for a PostgreSQL database, and the Crossplane machinery will dynamically provision CockroachDB on-premises and connect the application to it. It's that simple!

When the administrator creates a "resource class" blueprint that provides the specific details of how the PostgreSQL database will be provisioned, they would simply fill in information needed for Rook CockroachDB instead of a cloud provider service:

apiVersion: core.crossplane.io/v1alpha1
kind: ResourceClass
metadata:
  name: local-postgresql
parameters:
  nodeCount: "3"
  storageSize: "10Gi"
  cachePercent: "25"
  maxSQLMemoryPercent: "25"
provisioner: cluster.cockroachdb.rook.io/v1alpha1
providerRef:
  name: ""
reclaimPolicy: Delete

Crossplane will use the information in this "resource class" to dynamically provision a CockroachDB PostgreSQL instance for your application, making the application truly portable not only across cloud providers but in local environments as well. Note that this on-premises support with the Rook CockroachDB operator is very early and currently in a demo state where it must be running in the same cluster as Crossplane. As this support is developed further, Crossplane will be able to deploy CockroachDB to all clusters in your multi-cluster environment.

See it in Action

We demonstrated this new dynamic provisioning behavior across cloud providers and on-premises live on stage at Kubecon Seattle. The full talk and demo are available on Youtube for you to see it all in action!

What's Next

The Crossplane project has just gotten off the ground and we are carrying this awesome momentum into 2019. Crossplane is a community driven effort, so we encourage you to get involved and help us build this exciting open source multicloud control plane. In upcoming releases, we want to expand the coverage and support with more cloud providers, managed services, local services, and a whole lot more features. We'd love to see you join the community and get involved!