title: "Crossplane in Production: Building a Self-Service Internal Cloud Platform with Kubernetes-Native Infrastructure APIs" date: 2026-09-21 tags: [Crossplane, Platform Engineering, Kubernetes, GitOps, Internal Developer Platform, AWS, Multi-Cloud, CNCF, ArgoCD, FinOps] cover: ../images/crossplane-internal-cloud-platform-cover.png
Crossplane in Production: Building a Self-Service Internal Cloud Platform with Kubernetes-Native Infrastructure APIs
1. Topic of the Day
Every platform team eventually hits the same wall: infrastructure-as-code tooling built around a plan/apply batch model doesn't compose into a self-service platform without a mountain of glue. Terraform (and OpenTofu) describe desired state well, but they don't reconcile it — nothing is watching a deployed resource to notice it drifted, nothing exposes "give me a database" as an API a developer can call, and every provisioning request still routes through a CI pipeline, a state-file lock, and usually a human review. Crossplane exists to close that gap by turning cloud infrastructure into Kubernetes-native APIs: instead of a Terraform module and a terraform apply, a team gets a Custom Resource Definition — an XRDatabaseInstance, say — that a developer can kubectl apply (or, more realistically, request through a Backstage template or a GitOps PR) and get back a real RDS instance, provisioned and continuously reconciled by a controller running inside the cluster, exactly the way Kubernetes reconciles a Deployment against its desired replica count.
This is a deliberate architectural bet, not a Terraform replacement for its own sake: it takes Kubernetes's core primitive — a control loop watching desired state in etcd and continuously driving actual state toward it — and applies it to VPCs, managed databases, IAM roles, and DNS zones instead of just Pods and Services. That bet paid off enough that Crossplane graduated to CNCF's highest maturity tier on October 28, 2025, joining a small group of projects (Kubernetes, Prometheus, Envoy, and a handful of others) that have cleared CNCF's graduation bar for security audits, governance, and adoption evidence. It now sits in the top 10% of all CNCF projects by contributor count, with more than 3,000 contributors from 450+ organizations, and IBM formally joined the Crossplane community's governance alongside Red Hat's existing work integrating it with OpenShift — signal that this isn't a niche tool, it's becoming a default layer in how large organizations build internal cloud platforms.
In production, Crossplane exists specifically where platform engineering meets infrastructure provisioning: a platform team defines a small number of opinionated, paved-road APIs (Compositions and XRDs) that abstract away the 40 raw AWS/Azure/GCP resources behind "a production-ready Postgres instance," and application teams consume those APIs the same way they consume any other Kubernetes object — through kubectl, through GitOps, through a Backstage software template — without needing to understand VPC peering, subnet groups, or parameter groups. Deutsche Telekom, Grupo Boticário, and Autodesk are among the enterprises now running Crossplane in production for exactly this reason: it turns "provisioning infrastructure" from a ticket-driven, multi-day process into a self-service, API-driven one, without giving every developer raw cloud-console or cloud-API access.
2. Real Business Problem
Scenario: A 400-engineer SaaS company has grown its cloud footprint across AWS and Azure through six years of organic Terraform adoption. There are 140 Terraform root modules, spread across 30 repositories, with state files in four different S3 backends (two of which were set up before the team standardized on remote state with locking). A platform team of five people is now the de facto approval gate for every infrastructure change company-wide, because Terraform's plan/apply model has no natural way to expose "safe, pre-approved self-service" without either (a) giving every team terraform apply rights against shared state, which is a blast-radius nightmare, or (b) routing everything through the platform team's PR review, which is now the #1 complaint in engineering satisfaction surveys.
The investigation that follows the complaint surfaces the actual shape of the problem:
- Provisioning lead time for "create me a new Postgres instance for this microservice" averages 3.2 business days — not because the underlying Terraform apply takes long (it's 8 minutes), but because it requires a PR against a shared repo, platform-team review, a scheduled apply window (to avoid state-lock contention with other in-flight changes), and manual follow-up to wire the connection secret into the requesting team's application config.
- State drift is discovered reactively, not proactively. Three separate incidents in the last two quarters involved a resource that had been modified out-of-band (a security team change to a security group, a manual RDS parameter tweak during an incident) that Terraform's next
planflagged as "will be reverted" — and in one case, an on-call engineer ranapplywithout reading the plan output carefully enough, silently reverting the security team's emergency lockdown. - State-file lock contention is now a weekly occurrence. With 30 repositories but only 4 backend buckets, concurrent applies from different teams routinely queue behind each other's DynamoDB locks, and one stuck lock (from a crashed CI runner mid-apply) required manual
force-unlockintervention twice in the last month — each time with real risk of corrupting concurrent state. - No consistent tagging or cost attribution. Because modules were written independently by different teams at different times, tagging conventions drifted; FinOps cannot reliably attribute 20% of cloud spend to an owning team, and a recent cost-optimization push stalled because nobody could confidently say which Terraform module, if any, still manages a given orphaned resource.
- Multi-cloud is two disconnected worlds. The Azure modules and AWS modules were written by different sub-teams with different conventions, different variable-naming schemes, and no shared abstraction — a team building a workload that needs both an AWS S3 bucket and an Azure Blob container consumes two entirely different interfaces with no common "storage bucket" concept.
- Terraform Cloud/Enterprise licensing cost is a growing line item, and a self-managed OSS alternative that still provides state management, RBAC, and a policy layer without a per-seat or per-workspace commercial license is now an explicit line item in a board-level cost review.
Leadership's mandate: cut median provisioning lead time to under 30 minutes for pre-approved resource types, eliminate reactive drift discovery, unify the AWS/Azure provisioning experience behind one API surface, and do it without adopting a proprietary commercial IaC platform — using the Kubernetes clusters and GitOps tooling (ArgoCD) the platform team already operates and is expert in.
3. Production Architecture
Control-plane cluster, deliberately separate from workload clusters. Crossplane runs on its own dedicated "hub" EKS cluster rather than colocated with any single workload cluster — a small but important decision: if Crossplane ran inside a workload cluster and that cluster had an incident requiring a rebuild, the very system responsible for provisioning its replacement would be down with it. The hub cluster runs Crossplane core, the provider pods (provider-aws-rds, provider-aws-ec2, provider-aws-iam, provider-azure-* — installed as the post-v1.14 provider families, split per-service specifically to avoid the multi-hundred-MB monolithic provider binary and CRD-count explosion that made early Crossplane installs slow to start and heavy on API server watch load), and the Composition Function pods that implement the platform's actual business logic.
The developer-facing surface: Claims, not raw Managed Resources. Application teams never touch a RDSInstance or AzureSQLDatabase managed resource directly. They interact with a small, curated set of Composite Resource Definitions (XRDs) the platform team owns — XPostgreSQLInstance, XRedisCache, XStorageBucket — each exposing a handful of meaningful fields (size: small|medium|large, environment: dev|staging|prod, region) instead of the 40+ raw fields the underlying cloud resource actually has. In Crossplane v2, these Claims and their resulting Composite Resources (XRs) are namespaced by default — a deliberate architecture shift from v1, where XRs were cluster-scoped and required a separate Claim object purely to get namespace-level RBAC isolation. Namespaced XRs in v2 mean a team's database request lives in that team's own namespace, standard Kubernetes RBAC (RoleBinding scoped to that namespace) is sufficient to control who can request what, and the platform team no longer needs the Claim/XR indirection layer purely as an isolation mechanism.
Request path. A developer doesn't run kubectl apply directly against production, in this architecture — they open a PR against a platform-requests GitOps repo (or, more commonly, generate that PR from a Backstage software template that renders the XR manifest from a short form: service name, size, environment), ArgoCD picks it up post-merge and applies it into the developer's namespace on the hub cluster. Crossplane's controller sees the new XR, resolves it against the matching Composition, and the Composition's function pipeline executes: a function-patch-and-transform or function-go-templating step maps the abstract size: medium into concrete db.r6g.xlarge, applies the org's mandatory tagging (cost-center, owning-team, managed-by: crossplane) derived from the requesting namespace's labels, and emits the underlying RDSInstance and DBSubnetGroup managed resources. Crossplane's core reconciler then drives those toward existence via provider-aws-rds, continuously — not once at apply time, the way Terraform does, but as an ongoing control loop that re-asserts desired state on every reconcile interval and on every watched-resource change.
Policy enforcement before reconciliation, not after. Kyverno (already the org's admission-control layer, covered in an earlier post) validates every incoming XR against platform guardrails before Crossplane ever calls a cloud API: instance sizes above large are blocked outside an explicit finance-approved: "true" annotation, regions are restricted to an approved allow-list per compliance zone, and environment: prod XRs require the requesting namespace to carry a production-workload label set only by the platform team. This closes the exact gap Terraform's model structurally can't: in Terraform, "policy" is a terraform plan scan (Sentinel, OPA via Conftest) that runs before apply but is disconnected from ongoing state — nothing stops someone from hand-editing a resource afterward. Kyverno plus Crossplane's continuous reconciliation means an out-of-band change to a Crossplane-managed resource gets silently reverted on the next reconcile (the drift-correction behavior that directly targets the security-group-revert incident from Section 2 — except now it's the expected, monitored behavior instead of a surprise).
Secrets and cross-namespace consumption. Crossplane writes each provisioned resource's connection details (host, port, generated credentials) as a Kubernetes Secret in the same namespace as the Claim. External Secrets Operator (already deployed for other secret sources) is configured with a PushSecret-style flow to replicate that connection secret into the actual application's runtime namespace or into the org's HashiCorp Vault, closing the loop from "infrastructure provisioned" to "application can consume it" without a manual handoff — directly targeting the "3.2-day lead time is mostly manual wiring" finding from Section 2.
Multi-cloud unification. The XStorageBucket XRD is the concrete answer to the AWS/Azure split in Section 2: one Composition selects provider-aws-s3 when the requesting environment's cloud label is aws, and a second Composition selects provider-azure-storage when it's azure — both satisfy the same XRD, so from the developer's perspective, requesting a storage bucket is identical regardless of which cloud the target environment runs on. The platform team owns the cloud-specific complexity once, centrally, instead of every consuming team learning two clouds' APIs.
Why this design, and the trade-offs. The alternative — keeping Terraform but wrapping it in a Backstage-driven, CI-triggered self-service layer (Terraform Cloud's run-tasks model, or a homegrown "Atlantis-plus-approval-bot" setup) — was seriously evaluated and rejected specifically because it doesn't solve continuous drift correction or eliminate state-file lock contention; it only makes the request path self-service while leaving Terraform's core batch-apply, file-based-state mechanics untouched. Crossplane's control-loop model solves drift and lock contention structurally (state lives as Kubernetes object status in etcd, which handles concurrent writes natively via optimistic concurrency, not file locks), at the cost of a real migration effort and a new piece of platform infrastructure (the hub cluster itself) that now needs its own operational rigor — HA, backup, and upgrade discipline, covered in Sections 5 and 6. As the platform scales past a few hundred XRs, the natural evolution is splitting the single hub cluster into a hub-and-spoke model (a hub cluster running Crossplane's control plane, reconciling against multiple regional "spoke" identities) purely to keep blast radius and API server load bounded — not because a single hub can't scale further today, but because operational isolation matters more than raw capacity once the platform is genuinely business-critical.
4. Solution Design
Design decision: Composition Functions (Pipeline mode) exclusively, no native Patch-and-Transform. Crossplane deprecated the original, YAML-only "Resources" mode of Composition (native inline patch-and-transform) back in v1.17, replacing it with a mode: Pipeline Composition that chains WASM/gRPC Composition Functions — function-patch-and-transform for the parts that are genuinely declarative mapping, function-go-templating or function-kcl for anything needing real conditional logic (sizing tiers, environment-dependent defaults, computed tag values). This was a deliberate choice over keeping any legacy native compositions: Pipeline-mode functions are independently testable Go/KCL code with real unit tests, versionable and releasable outside the Crossplane core upgrade cycle, and — critically for a 400-engineer org — reviewable by the platform team the same way any other service code is reviewed, rather than being sprawling, hard-to-diff YAML patch lists that were the single biggest source of "how does this actually behave" ambiguity in the org's earlier prototype.
Alternative considered and rejected: Upbound's commercial control-plane offering. Upbound (Crossplane's originating vendor) offers a managed, multi-tenant control-plane product (Upbound Cloud) with built-in cost visibility and a curated Composition marketplace. Evaluated and deferred — not rejected outright, flagged as a re-evaluation candidate in 12 months — specifically because the org's mandate explicitly ruled out new proprietary per-seat IaC licensing (Section 2), and the self-hosted OSS Crossplane-on-EKS path, while requiring more in-house operational investment, has a materially lower and more predictable cost profile for an org already running EKS at scale with existing platform-engineering headcount.
Alternative considered and rejected: staying on Terraform with a policy-as-code and Atlantis-driven self-service wrapper. Rejected as a full solution (see Section 3) because it doesn't touch the structural state-file/plan-apply limitations, though the org kept Terraform in place for resources genuinely outside Crossplane's current provider coverage (a small number of legacy on-prem network appliances configured via a vendor-specific Terraform provider with no Crossplane equivalent) — a pragmatic coexistence rather than a rip-and-replace, migrating the highest-volume, highest-friction resource types (databases, storage, IAM roles, VPC/networking primitives) first.
Scalability considerations. CRD and API-server watch load scale with the number of distinct managed-resource types installed, not primarily with the number of resource instances — this is why provider families (splitting provider-aws into provider-aws-rds, provider-aws-ec2, etc., each installing only the CRDs it needs) matters at scale: the org installs nine provider-family packages instead of the single monolithic provider-aws, cutting API-server CRD count from several thousand to under 400. Composition Function execution adds a small but real per-reconcile latency cost (each pipeline step is an out-of-process gRPC call); the platform team benchmarked reconcile latency at roughly 200-400ms per XR under normal load, acceptable given XRs reconcile every few minutes, not on every user request.
Cost implications. Removing the Terraform Cloud/Enterprise licensing line item was the most direct win, but the larger cost impact is FinOps-driven: mandatory, Composition-enforced tagging means cost attribution went from "20% unattributed" to effectively 100% attributed within one billing cycle, because tags are now generated by platform code rather than hand-written per module. The hub cluster's own compute footprint (a modest 3-node control-plane-grade node pool plus provider pods) is a rounding error against what it replaces.
Security implications. Centralizing provisioning logic in reviewed, tested Composition Functions rather than 140 independently-authored Terraform modules meaningfully narrows the org's IAM surface: the hub cluster's IRSA roles for each provider pod are scoped tightly to exactly the AWS actions that Composition actually uses, versus the historical pattern of Terraform CI runners often carrying broad, rarely-audited PowerUserAccess-adjacent roles because nobody wanted to debug a missing-permission failure mid-pipeline.
Performance implications. Provisioning lead time for the pre-approved resource types dropped from a 3.2-day median to under 20 minutes end-to-end (PR merge to usable connection secret) — inside the sub-30-minute mandate — with the bulk of remaining time being the underlying cloud API's own provisioning latency (RDS instance creation genuinely takes 8-12 minutes) rather than any platform overhead.
5. Deep Technical Walkthrough
The reconciliation loop, precisely. Crossplane's core is a set of Kubernetes controllers, structurally identical to any other controller-runtime-based operator: a Composite Resource Controller watches XR objects, a Composition Controller watches Compositions, and — for each XR — a dedicated reconcile loop runs on a fixed interval (default poll interval, configurable per-provider, typically 1 minute for fast-changing resources, up to 10 for stable ones) and on any watched-object change event. On each reconcile, the controller: (1) fetches the XR's current spec, (2) resolves the matching Composition, (3) invokes the Composition Function pipeline — passing the XR plus any "extra resources" the pipeline requested (e.g., an existing VPC lookup via function-extra-resources) — which returns a desired-state list of Managed Resources, (4) diffs that desired list against the actual Managed Resource objects already owned by this XR, and (5) creates, updates, or (on deletion) tears down the difference. This is the same reconcile-and-converge pattern as a Deployment controller reconciling ReplicaSets — it is not a plan/apply transaction, there is no single point where "the plan" is computed once and executed; the loop simply keeps running.
Managed Resource lifecycle and external-name. Each Managed Resource (an RDSInstance, say) carries a crossplane.io/external-name annotation that is the durable link between the Kubernetes object and the actual cloud resource (the RDS instance identifier). This annotation, not any separate state file, is Crossplane's state — it's why kubectl get rdsinstance -o yaml shows you both the desired spec and, in .status, the observed real-world state, reconciled continuously rather than snapshotted at last-apply time. Import of a pre-existing, unmanaged cloud resource is done by creating the Managed Resource object with the external-name annotation pre-set and no other changes — Crossplane's next reconcile observes the existing resource matches and adopts it, rather than trying to create a duplicate — the direct analog to terraform import, but requiring no state-file surgery.
Composition Function pipeline execution in detail. Each function in a Pipeline-mode Composition runs as a separate, addressable component — either a long-running gRPC server pod (the common pattern for org-authored functions like function-go-templating) or a short-lived process invoked per-reconcile. The Composition Controller calls each function in the pipeline's declared order, passing the accumulated "desired resources" state forward — function N sees the output of function N-1, can add, modify, or (via function-extra-resources) request additional context objects to inform its logic before returning its own contribution to the desired-state list. This composability is what lets the platform team layer standard tagging logic, environment-specific sizing, and cloud-specific resource selection as independently maintained, independently versioned steps rather than one monolithic template.
Control-plane and data-plane interaction. The "control plane" here is the Crossplane hub cluster's own API server plus its controllers; the "data plane" is the actual cloud provider APIs (AWS RDS, EC2, IAM control planes) that provider pods call via the AWS/Azure Go SDKs, authenticated through IRSA-style workload identity (the same zero-trust credential pattern covered in an earlier post on AWS IRSA and Azure Workload Identity) rather than long-lived static credentials mounted as Secrets. A provider pod crash or restart is a non-event for correctness — on restart, it simply re-establishes watches and the next reconcile proceeds normally, because no in-flight "plan" state lives only in that pod's memory.
Failure scenarios and recovery mechanisms.
- A Composition Function pod crashes mid-pipeline: the reconcile for that XR fails and requeues with exponential backoff (standard controller-runtime behavior); no partial or corrupted state results, because the function pipeline's output isn't applied until it completes successfully — this is meaningfully safer than a Terraform apply interrupted mid-execution, which can leave state and reality inconsistent.
- A cloud API rate-limits the provider during a burst of new XRs (e.g., a bulk migration event): the provider's controller-runtime client backs off and retries per-resource; other unrelated XRs' reconciles are unaffected, since each is an independent watch-and-reconcile loop rather than a shared batch operation.
- Out-of-band manual change to a Crossplane-managed cloud resource: the next reconcile detects the drift between desired (from the Composition) and observed (from the live cloud API read) state and reverts it — by design. When this behavior is unwanted temporarily (an approved emergency change), the platform uses Crossplane's
crossplane.io/paused: "true"annotation on the specific Managed Resource to suspend reconciliation, explicitly and audibly, rather than fighting the control loop. - Hub cluster itself is lost: this is the scenario Section 6's DR drill targets directly — since all Crossplane state is Kubernetes objects, a Velero-restored hub cluster (from the same DR architecture covered in an earlier post) recovers the entire provisioning control plane; the actual cloud resources it manages were never inside the hub cluster and are untouched by the hub's own outage, so a hub-cluster incident does not, by itself, threaten already-provisioned infrastructure — only new provisioning and drift correction pause until the hub is restored.
Performance bottlenecks and scaling behavior. At a few hundred XRs, the binding constraint the platform team observed was Composition Function pipeline latency under concurrent reconciles — the org's function-go-templating pods, initially deployed as a single replica, became a serialization point during a bulk-onboarding event (40 new services requesting databases in one sprint). Scaling the function pods horizontally (they're stateless, safe to run 3+ replicas behind the function gRPC service) removed this bottleneck; the next anticipated constraint, not yet hit, is API server watch load as managed-resource-instance count grows into the tens of thousands, which the org is preemptively addressing by planning the hub-and-spoke split mentioned in Section 3 before it becomes an incident rather than after.
6. Production Troubleshooting
Incident: a batch of XPostgreSQLInstance claims gets stuck in Ready: False, blocking a product launch.
Symptoms. Twelve newly created Claims across four teams' namespaces show SYNCED: True, READY: False when checked via kubectl get claim -A, well past the expected provisioning window; the teams' Backstage-generated PRs merged cleanly and ArgoCD shows the Claims as Synced and Healthy at the GitOps layer — the failure is happening entirely inside Crossplane's own reconciliation, invisible to ArgoCD.
Commands and signals to pull first.
kubectl describe xpostgresqlinstance <name> -n <namespace>— theEventsandConditionssections surface the specific reason a composite resource isn't ready, almost always faster than digging through controller logs first.kubectl get managed(a Crossplane-provided aggregate view across all Managed Resource types) filtered to the affected namespace's owned resources, to see which underlyingRDSInstanceorDBSubnetGroupis the actual blocker.kubectl logs -n crossplane-system deploy/provider-aws-rds -f— provider pod logs show the raw AWS API error, which is frequently more informative than anything surfaced up through XR status conditions.kubectl logs -n crossplane-system deploy/function-go-templating -f— checked specifically because the templating function pipeline is where org-specific logic lives, and a bug there can produce a syntactically valid but semantically wrong desired-state object that then fails cloud-side validation.- Crossplane's own Prometheus metrics:
crossplane_reconcile_duration_secondsandcrossplane_managed_resource_ready— a spike in reconcile duration paired withready=0across a specific Managed Resource kind (not scattered across kinds) points at a systemic issue with that resource type or the provider serving it, not twelve unrelated one-off failures.
Root cause analysis, step by step. kubectl describe on the affected XRs shows a consistent condition: ReconcileError: cannot create RDSInstance: DBSubnetGroupNotFoundFault. Tracing into kubectl get managed, all twelve XRs' underlying DBSubnetGroup Managed Resources are themselves stuck, and their events show rate exceeded from the AWS RDS API. The batch of twelve had all been merged and synced by ArgoCD within the same two-minute window (a coincidental product-launch crunch across four teams), and each XPostgreSQLInstance's Composition creates its own dedicated DBSubnetGroup rather than reusing a shared one per-VPC — a design gap from the initial Composition authoring, not previously exposed because provisioning volume had never before been this concentrated in time.
Configuration changes made. Short-term: manually staggered the stuck reconciles by adding a brief crossplane.io/paused annotation to eight of the twelve, unpausing four at a time, to get the launch-blocking ones through immediately without further amplifying the rate-limit. Root-fix: reworked the XPostgreSQLInstance Composition to use function-extra-resources to look up and reuse an existing per-VPC-per-environment DBSubnetGroup rather than creating a new one per database instance, eliminating both the rate-limit exposure and a latent resource-sprawl problem (dozens of near-identical subnet groups that nobody was cleaning up).
Verification. Re-ran a synthetic burst test (20 concurrent XPostgreSQLInstance Claims against a non-production namespace) after the Composition fix; all twenty reached Ready: True within the expected window with zero DBSubnetGroupNotFoundFault events, and AWS RDS API call volume for CreateDBSubnetGroup dropped to near-zero (reusing existing groups, only new VPC/environment combinations trigger creation) confirming the fix addressed the actual cause rather than just the symptom.
7. Hands-on Lab
A local, provider-agnostic version of the core Crossplane mechanics — Composite Resource, XRD, Pipeline-mode Composition — runnable on kind, using the provider-kubernetes provider (no real cloud account required) to keep the lab self-contained while exercising the exact same reconciliation model used against AWS/Azure in production.
Setup.
kind create cluster --name crossplane-lab
helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update
helm install crossplane crossplane-stable/crossplane \
--namespace crossplane-system --create-namespace --wait
kubectl create -f - <<EOF
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
name: provider-kubernetes
spec:
package: xpkg.upbound.io/crossplane-contrib/provider-kubernetes:v0.15.0
EOF
kubectl wait provider.pkg.crossplane.io/provider-kubernetes --for=condition=Healthy --timeout=180s
Define the platform API: an XRD for a self-service "AppNamespace" abstraction.
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
name: xappnamespaces.platform.example.org
spec:
group: platform.example.org
names:
kind: XAppNamespace
plural: xappnamespaces
claimNames:
kind: AppNamespace
plural: appnamespaces
versions:
- name: v1alpha1
served: true
referenceable: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
teamName: { type: string }
quotaCpu: { type: string, default: "4" }
A Pipeline-mode Composition implementing it with function-patch-and-transform.
apiVersion: pkg.crossplane.io/v1
kind: Function
metadata:
name: function-patch-and-transform
spec:
package: xpkg.upbound.io/crossplane-contrib/function-patch-and-transform:v0.7.0
---
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
name: appnamespace-basic
spec:
compositeTypeRef:
apiVersion: platform.example.org/v1alpha1
kind: XAppNamespace
mode: Pipeline
pipeline:
- step: patch-and-transform
functionRef: { name: function-patch-and-transform }
input:
apiVersion: pt.fn.crossplane.io/v1beta1
kind: Resources
resources:
- name: namespace
base:
apiVersion: kubernetes.crossplane.io/v1alpha2
kind: Object
spec:
forProvider:
manifest:
apiVersion: v1
kind: Namespace
patches:
- type: FromCompositeFieldPath
fromFieldPath: spec.teamName
toFieldPath: spec.forProvider.manifest.metadata.name
Apply and observe the control loop.
kubectl apply -f xrd.yaml -f composition.yaml
kubectl apply -f - <<EOF
apiVersion: platform.example.org/v1alpha1
kind: AppNamespace
metadata:
name: checkout-team
namespace: default
spec:
teamName: checkout-team-ns
EOF
kubectl get appnamespace -n default -w
kubectl get xappnamespace
kubectl get namespace checkout-team-ns
Validation. Confirm the Namespace object was created purely as a side effect of the AppNamespace claim (kubectl get namespace checkout-team-ns succeeds with no direct kubectl create namespace ever run), then delete the underlying Namespace manually (kubectl delete namespace checkout-team-ns) and watch Crossplane's reconcile loop recreate it within the next poll interval — the drift-correction behavior from Section 5, observable directly in a five-minute lab.
Cleanup.
kubectl delete appnamespace checkout-team -n default
kind delete cluster --name crossplane-lab
8. Production Case Study
Deutsche Telekom's platform engineering org is one of the most-cited large-enterprise Crossplane adopters, using it to give internal teams self-service access to infrastructure across the company's own private and public cloud footprint without every team needing direct cloud-account access — the same "curated XRD as the only door in" pattern this post's Section 3 architecture implements, at a scale spanning many internal business units with very different infrastructure maturity levels.
Autodesk and Grupo Boticário's production Crossplane deployments are frequently referenced (including by CNCF and Upbound) as validating the graduation-level maturity bar Crossplane cleared in October 2025 — both run it as the provisioning layer under real customer-facing production workloads, not just internal tooling, which is precisely the bar CNCF's graduation criteria (security audit, adoption breadth, governance maturity) are designed to test for.
IBM's decision to formally join the Crossplane community's governance, announced alongside the CNCF graduation, mirrors a pattern the industry has seen before with Kubernetes itself: a major enterprise vendor investing engineering effort directly into a CNCF-graduated control-plane project because it has become foundational enough to justify first-party investment rather than building a competing proprietary layer — Red Hat's parallel, longer-running work integrating Crossplane concepts into OpenShift's own platform story reinforces the same signal from a second major vendor.
The broader "Kubernetes as the universal control plane" thesis that Crossplane embodies is the same thesis behind Cluster API (declarative cluster lifecycle) and increasingly behind AI infrastructure tooling (KServe, Kubeflow) covered in earlier posts — the pattern of "define a CRD, write a controller, let Kubernetes's reconciliation model handle convergence and self-healing" keeps winning over bespoke, batch-oriented tooling for exactly the reasons Section 3's architecture chose it over Terraform: continuous correction beats periodic re-assertion for anything that needs to stay correct between human-triggered runs.
9. Architecture Review
Strengths. The namespaced-XR model in Crossplane v2 gives the platform natural, Kubernetes-native multi-tenancy without a bespoke authorization layer — RBAC that already exists for every other namespaced resource in the cluster applies identically to infrastructure requests, which is a meaningfully lower operational burden than the v1-era Claim/XR indirection this design would have otherwise needed purely for isolation. Continuous drift correction, enforced structurally rather than by policy convention, directly closes the exact incident class (out-of-band changes silently reverted by the next Terraform apply, discovered only after the fact) that motivated the redesign.
Weaknesses. The hub-cluster-as-single-control-plane design, while operationally simpler than a distributed alternative, means the entire org's provisioning capability has one control plane's availability as its ceiling — Section 5's DR analysis shows already-provisioned infrastructure survives a hub outage, but new provisioning and drift correction are fully blocked until it's restored, which is an acceptable but real single point of degraded capability that a larger org should track as an SLO in its own right, not assume away. The platform also still depends on the org's own Composition Function code for anything beyond straightforward field mapping — this is deliberately more maintainable than sprawling Terraform modules, but it does mean platform-team engineering capacity is now a harder dependency for onboarding new resource types than "any team can write their own Terraform module," a trade-off worth being explicit about with stakeholders expecting infinite self-service flexibility.
What fails first at 10x scale. API server watch load and CRD count are the first ceiling, as flagged in Section 5 — at 10x the current managed-resource-instance count, even with provider-family CRD splitting, the single hub cluster's API server becomes the binding constraint, pushing the org toward the hub-and-spoke split earlier than the "someday" timeline currently planned. Composition Function pipeline throughput during burst events (Section 6's incident) would recur more severely without the subnet-group-reuse fix and function-pod horizontal scaling already applied — worth load-testing proactively rather than waiting for the next organic burst to find the next such gap.
What changes at 100 million users / global scale. At genuinely global scale, a single hub cluster (even split hub-and-spoke) becomes a latency and blast-radius concern for geographically distributed platform consumers — the likely evolution is regional hub clusters, each owning provisioning for its region's workloads, federated through a shared Composition/XRD source (GitOps-synced identically to every hub) so the API stays consistent globally even as the control-plane execution is regionalized; this mirrors the multi-region control-plane patterns already covered for Kubernetes API server scaling and DNS in earlier posts.
What would be redesigned. Per-resource-type SLOs for provisioning latency and drift-correction time, tracked and alerted on independently rather than as one blended "Crossplane health" metric — Section 6's incident showed that a systemic issue with one Managed Resource kind (subnet groups) can masquerade as twelve unrelated failures until someone thinks to aggregate by kind; building that aggregation into the platform's default dashboards from day one, rather than discovering the need for it during an incident, is the most concrete near-term improvement.
10. Best Practices
Reliability. Run the Crossplane hub cluster with the same operational rigor as any tier-0 control-plane system — multi-AZ node placement, HA controller-manager replicas, and its own Velero-backed DR posture — because it has quietly become the org's provisioning single point of failure even though it's "just infrastructure tooling." Scalability. Adopt provider families over monolithic providers from the start, and design Compositions to reuse shared infrastructure (subnet groups, VPCs) rather than creating dedicated copies per XR, before burst-provisioning events expose the gap the hard way. Observability. Instrument and alert on reconcile duration and readiness per Managed Resource kind, not just an aggregate Crossplane health signal — kind-level granularity is what turns a confusing multi-XR incident into an obvious root cause. Security. Scope each provider pod's cloud IAM role tightly to the actions its Compositions actually use, and treat the hub cluster's own RBAC and namespace isolation as seriously as any multi-tenant application platform, since a compromised hub cluster now has provisioning authority across every cloud account it's federated with. Cost Optimization. Bake tagging and cost-attribution logic directly into Composition Functions rather than relying on submitter discipline — this is the single highest-leverage FinOps change available in a Crossplane-based platform. Performance. Scale stateless Composition Function pods horizontally ahead of expected burst events (product launches, migrations) rather than reactively. Maintainability. Keep Compositions and Functions in version-controlled repositories with real CI (unit tests for function-go-templating/function-kcl logic, not just YAML lint), reviewed with the same rigor as any other production service code. Operational Excellence. Treat XRD/Composition changes as API changes with real versioning and deprecation policy (Crossplane's Composition Revisions feature exists specifically to let in-flight XRs continue reconciling against an older Composition version during a rollout) — breaking the platform API out from under existing consumers is exactly the kind of trust-destroying incident that kills self-service platform adoption.
11. Common Production Mistakes
Exposing raw Managed Resources directly to application teams instead of curated XRDs defeats the entire point of the platform-API abstraction and reproduces Terraform's "everyone needs to understand every cloud provider field" problem inside Kubernetes instead of outside it. Treating Composition authoring as a one-time YAML exercise rather than versioned, tested platform code is how orgs end up with the sprawling, undebuggable native patch-and-transform compositions Crossplane itself deprecated — Pipeline-mode functions exist specifically so this logic can be written, tested, and reviewed like real software. Running the Crossplane hub cluster without independent DR and HA planning, on the assumption that "it's just controlling infrastructure, the infrastructure itself is fine," ignores that new provisioning and drift correction stop entirely during a hub outage — a real, measurable capability loss that deserves its own SLO. Letting every team's Compositions create fully independent, non-shared supporting infrastructure (subnet groups, VPCs, IAM roles) rather than designing for reuse is a direct path to the rate-limiting and resource-sprawl incident in Section 6, and it compounds silently until a burst event exposes it. Finally, skipping admission-policy enforcement on XR fields (instance sizing, region allow-lists, environment gating) on the assumption that "the Composition already handles defaults" conflates defaults with guardrails — a Composition's defaults are what happens when a field is omitted, not a limit on what a well-formed but malicious or mistaken request can specify.
12. Interview Preparation
"How does Crossplane's model fundamentally differ from Terraform's, beyond 'it uses Kubernetes'?" A strong answer centers on the reconciliation model: Terraform computes a plan and executes it once, at apply time, with state captured in a file that must be explicitly re-read (plan) to detect drift; Crossplane runs a continuous control loop, the same pattern as any Kubernetes controller, that keeps re-asserting desired state against observed state indefinitely, with "state" being the live Managed Resource object's own status, not a separate artifact. This is why drift correction is automatic in Crossplane and requires an explicit, separately-scheduled step in Terraform.
"Walk me through what happens when you delete a Composite Resource — does the underlying cloud infrastructure get deleted too?" Depends on the Composition's and Managed Resource's deletion policy (spec.deletionPolicy, default Delete). A strong answer explains that setting it to Orphan leaves the real cloud resource intact while removing Crossplane's management of it — useful for a controlled handoff or a deliberate "stop managing but don't destroy" operation — and that this is analogous to, but safer than, terraform state rm, since the object and its external-name linkage still exist and can be re-adopted later rather than being fully forgotten.
"How would you design multi-tenancy for a Crossplane-based internal platform, and how did that change in v2?" Points to the v1-era pattern (cluster-scoped XRs requiring the separate namespaced Claim object purely for RBAC isolation) versus v2's namespaced XRs by default, which let standard Kubernetes RBAC apply directly without that indirection — a strong candidate can articulate why v1 needed Claims in the first place (XRs were cluster-scoped, so a namespace-scoped proxy object was the only way to get per-team isolation) rather than just naming the versions.
"Your Composition Function is producing correct-looking YAML, but the underlying cloud resource never reaches Ready. How do you debug this?" Points toward separating "Crossplane thinks the request is well-formed" from "the cloud API accepted it" — kubectl describe on the Managed Resource surfaces the actual cloud-provider error (often an IAM permission gap, a quota limit, or a malformed field the Composition's function logic produced but didn't validate), and a thorough answer distinguishes this from a Composition Function pipeline error (which shows up as the XR failing to even produce the expected Managed Resource in the first place) — two different failure classes requiring different debugging starting points.
"When would you choose to keep using Terraform/OpenTofu instead of migrating a resource type to Crossplane?" A well-reasoned answer avoids "always migrate everything" as a reflexive default: resource types with no mature Crossplane provider, low-volume/rarely-changed infrastructure where self-service isn't a real pain point, or infrastructure genuinely better modeled as a one-time bootstrap (initial account/organization setup) rather than a continuously-reconciled API are all reasonable cases to leave on Terraform — the decision criterion is whether continuous reconciliation and self-service API access provide real value for that resource type, not ideological preference for one tool.
13. Latest Industry Updates
Crossplane's CNCF Graduation (October 28, 2025) is the single most consequential recent milestone — it cleared third-party security audits, demonstrated vendor-neutral governance, and moved its release/artifact infrastructure onto CNCF-operated systems, all of which are the concrete, verifiable signals a platform team should point to when justifying a Crossplane adoption to a risk-averse security or compliance review, rather than relying on general "it's popular" arguments.
IBM joining the Crossplane community's governance, announced alongside graduation, is worth watching for its downstream effects on provider coverage and enterprise-hybrid-cloud integration — IBM and Red Hat's parallel OpenShift-integration work suggests Crossplane's role in hybrid/multi-cloud enterprise platforms is deepening rather than staying a pure-public-cloud pattern.
Crossplane v2's namespaced Composite Resources and Managed Resources, now GA as of the v2.x line (latest v2.4 at time of writing), is the biggest architecture-relevant change for any team currently running v1 — the migration guide's tooling (crossplane beta convert pipeline-composition for Compositions still on deprecated native patch-and-transform, plus the documented Claim-to-namespaced-XR migration path) should be read closely before attempting the jump, since it changes RBAC design assumptions, not just API versions.
Crossplane v2's "Operations" feature — function pipelines for operational, non-provisioning tasks like certificate rotation, scheduled maintenance windows, and responding to specific resource-change events — is a meaningful expansion beyond "Crossplane provisions things," worth evaluating for any team currently running separate CronJobs or Lambda functions for day-2 infrastructure operations that could instead live in the same reviewed, versioned Composition-Function codebase as provisioning logic.
Provider-family adoption continuing to expand across AWS, Azure, and GCP providers is directly relevant to any team still running the older monolithic provider packages — the CRD-count and API-server-load reduction described in Section 5 is a "upgrade this before it becomes a scaling incident" item, not a nice-to-have.
14. Summary & Cheat Sheet
Key concepts. Crossplane turns cloud infrastructure into Kubernetes-native, continuously-reconciled APIs — XRDs define the platform's abstract, curated interface; Compositions (Pipeline mode, via Composition Functions) implement how that interface maps to real Managed Resources; Managed Resources are the actual cloud-provider-backed objects, linked to real infrastructure via external-name, with no separate state file. v2 makes Composite Resources namespaced by default, giving native Kubernetes RBAC-based multi-tenancy without the v1-era Claim indirection.
Architecture. GitOps request (Backstage template → PR → ArgoCD sync) → namespaced XR on a dedicated hub cluster → Kyverno policy validation → Composition Function pipeline (tagging, sizing, cloud selection) → Managed Resources reconciled continuously against real cloud APIs via provider pods using workload-identity credentials → connection Secrets replicated to consuming namespaces via External Secrets Operator.
Commands worth memorizing.
kubectl get claim -A
kubectl get managed
kubectl describe xrd <name>
kubectl describe composition <name>
kubectl describe <xr-kind> <name> -n <namespace>
crossplane beta convert pipeline-composition old.yaml -o new.yaml
kubectl annotate <managed-resource> crossplane.io/paused=true
Best-practice checklist. Curated XRDs only — never expose raw Managed Resources to consumers. Pipeline-mode Compositions with tested, versioned Composition Functions, never legacy native patch-and-transform. Provider families, not monolithic providers, to bound CRD/API-server load. Admission-policy enforcement (Kyverno/OPA) on XR fields as guardrails distinct from Composition defaults. Shared, reused supporting infrastructure (subnet groups, VPCs) designed into Compositions from the start, not discovered as a gap during a burst-provisioning incident. Independent HA/DR posture for the hub cluster itself, tracked with its own SLOs.
Troubleshooting checklist. Start with kubectl describe on the XR/Claim for status conditions, then kubectl get managed to find the specific blocking Managed Resource, then provider-pod logs for the raw cloud API error. Aggregate reconcile-failure metrics by Managed Resource kind before assuming unrelated one-off failures. Distinguish "Composition Function pipeline error" (XR never produces the expected Managed Resource) from "Managed Resource never reaches Ready" (cloud API rejected a well-formed request) — different failure classes, different debugging starting points.
Design pattern to remember. Continuous reconciliation beats periodic batch apply for anything that must stay correct between human-triggered runs — and a small number of opinionated, curated platform APIs beats exposing raw provider complexity, whether that complexity is a Terraform module's 40 variables or a cloud provider's raw resource schema.