Deploy a Kubernetes App & AWS Resources using Crossplane on Kubernetes: Part 2

Before Crossplane enters your delivery path, Kubernetes cannot be a black box during incidents. You should already be able to inspect Deployments, Pods, Services, Secrets, CRDs, Events, owner references, finalizers, and controller status conditions with kubectl, and you should know where to look when reconciliation stalls. Crossplane adds another set of controllers; it does not replace the controllers already responsible for scheduling pods, mounting secrets, resolving services, or rolling out workloads. It is not a one-time AWS provisioning script that runs and exits. You declare the desired state, Crossplane providers reconcile continuously, and resource conditions show whether a managed resource is Ready, still converging, waiting on a reference, late-initializing provider defaults, or blocked by an error that needs a human decision. For this walkthrough, use an AWS account and credentials that can create, observe, update, and delete the target resources in the region you choose. Do not stop at “the provider authenticated.” A common failure mode is treating valid credentials as sufficient authorization: Kubernetes accepts the manifest, but the provider keeps retrying because AWS returns AccessDenied, quota, dependency, tagging, region, validation, or immutable-field errors.

This walkthrough builds on Part 1, Deploy AWS Resources using Crossplane on Kubernetes. Before you deploy an application that depends on Crossplane-managed infrastructure, prove that the infrastructure control plane works by itself. Otherwise pod scheduling failures, image pull errors, readiness probes, Helm values, and noisy application logs will distract from the real dependency problem. Start with the checks that feel boring because they prevent expensive debugging later: confirm that the AWS provider is Healthy, that the ProviderConfig name in your manifests matches the object installed in the cluster, that namespace-scoped and cluster-scoped resources are being applied in the right place, and that the configured IAM identity can call the exact AWS APIs required by the resources you are creating. For anything stuck in a non-ready state, inspect conditions and events before tailing application logs. Look for schema validation failures, unresolved references, AWS authorization errors, dependency ordering problems, immutable-field changes, quota limits, finalizers, deletion policies, or unexpected late-initialized values. A missing bucket, a non-ready IAM policy, a failed role attachment, or a blocked managed resource can easily masquerade as a bad application rollout. The first useful question is still simple: did Crossplane create the AWS dependency, and does its status say the workload can safely consume it?

We’ll use the MeteorOps/crossplane-aws-provider-bootstrap repository as the working reference for manifests and configuration. Treat it as an executable example, not a production platform blueprint to copy line for line. Its main value is that it makes an important boundary visible: platform-owned AWS infrastructure on one side, application runtime configuration on the other. Before adapting that boundary for real teams, make explicit decisions about AWS regions, deterministic resource names, mandatory tags, IAM scope, namespace isolation, connection secrets, deletion behavior, drift handling, and promotion from development to staging to production. Also decide which fields developers may set directly and which fields the platform should standardize through compositions, defaults, admission policy, code review, or documentation. These are not cleanup tasks for a later hardening sprint. They determine whether Crossplane becomes a clear internal platform API with supportable guardrails, or a hidden coupling layer made of Kubernetes YAML that nobody fully owns.

Platform teams usually feel pressure from three directions at the same time. Developers want usable environments without waiting in ticket queues. Security wants cloud access to be controlled, reviewable, and auditable. Operations wants deployments, rollbacks, and incidents to follow a path the team has already practiced. A real release is more than a Deployment object: the workload, AWS dependencies, IAM boundaries, configuration, secrets, network assumptions, observability hooks, and runtime wiring all have to line up before the application is actually useful. Creating one bucket, role, queue, database, or workload is rarely the hard part. The hard part is making the whole environment repeatable, reviewable, and recoverable without turning every release into a manual handoff. Crossplane helps when it defines a clear control-plane boundary with explicit ownership, policy, naming, deletion behavior, and visible status. It helps much less when it becomes a directory of cloud YAML that hides unclear ownership, overly broad IAM permissions, inconsistent names, missing lifecycle rules, and runbooks nobody has practiced under pressure.

What We’ll Deploy in Part 2: A Kubernetes Application Environment with Crossplane-Managed AWS Dependencies

In this article, we’ll deploy a small Kubernetes application environment while Crossplane provisions the AWS resources the workload depends on. This is where Crossplane becomes more than “cloud resources written as YAML”: it becomes part of the platform contract. Developers want short-lived environments without tickets, but platform teams still need guardrails for IAM, secrets, naming, cost, drift, deletion behavior, and support ownership. When that contract is vague, delivery fragments across Terraform runs, Helm releases, cloud consoles, Slack threads, and runbooks, and nobody has a reliable answer to the operational question that matters: is this environment ready to run? Crossplane helps by exposing external dependencies through the Kubernetes API, close to the workloads that consume them. That shared surface is powerful, but it has a sharp edge: “accepted by Kubernetes” is not the same as “created correctly in AWS” or “safe for the application to use.” We’ll focus on the boundaries that matter in practice: which AWS objects Crossplane should manage, what stays in application manifests or Helm values, how readiness is surfaced, and how to debug stalls caused by IAM, validation errors, dependency ordering, deletion policies, finalizers, or pods starting before their dependencies are actually available.

This is a step-by-step guide with an example and a Git repository, so by the end of it you should be able to deploy a sample env.

You can run the walkthrough end to end and finish with a working environment, but the commands are not the lesson. The useful habit is tracing ownership and readiness through each layer: AWS, Crossplane, and Kubernetes. When progress stops, resist the urge to blame the Helm chart or the application code first. Read the status conditions, check Events, and then look at the provider logs. Only after the dependency chain is healthy should you move up to the workload and debug pods, services, configuration, or application behavior.

If you are tempted to jump straight to the manifests, read the explanations first. They are the part that will help you adapt the example to a different AWS service, namespace model, or deployment pipeline without cargo-culting YAML.

Hope you enjoy!

‍

What should you expect from this walkthrough?

By the end of it you'll understand:

1. How Crossplane can be used for full environment deployment
2. How to deploy a sample app with AWS resources

What not to expect?

We’ll keep the example intentionally small. The goal is not to build a complete internal developer platform in one pass; it is to make the handoff visible. The application expresses a dependency, Crossplane reconciles the AWS resource, and Kubernetes runs the workload after that dependency is ready enough to use. Once that path is predictable, the production layers are easier to add without hiding the basics: policy, external secret delivery, environment promotion, observability, and CI/CD integration. A narrow example also makes ownership easier to discuss. In a large demo stack, it is too easy to miss which team owns the bucket, the IAM boundary, the chart values, the namespace, or the cleanup behavior.

This is not a full guide to Helm providers or chart lifecycle management in Crossplane. The narrower goal is to make the boundary explicit before Helm releases and Crossplane-managed AWS resources share the same delivery workflow: what creates infrastructure, what configures the workload, and what signal tells the pipeline or operator that the environment is ready.

‍

Why Crossplane for the Full Environment Use-Case?

When you want to deploy a full environment, it usually involves 3 layers:

1. Infrastructure: Resources the application needs to run well
2. Application: The programs built by the company to serve users
3. Data: The data the application uses

But you already know that.

For years, teams treated infrastructure and application delivery as two separate phases. Crossplane challenges that habit.

The traditional sequence was simple: provision the infrastructure first, then deploy the application on top.
Crossplane changes that boundary by letting a Kubernetes-facing resource request the cloud dependency and have a controller reconcile it with the same status-driven model you use for workloads.

Pull-Request Environments are also easier

By creating a namespace with all of the apps and the AWS resources required with Crossplane, the use-case of creating a full environment per Pull-Request as part of the CI becomes much easier.

That's a nice benefit of such setup for companies utilizing the feature-branch or Gitflow approaches.

‍

A Traditional Full Env Example

In many teams, deploying a full environment has meant coordinating several separate steps and hoping each handoff preserves the same assumptions:

1. Provision VPC+EKS+... using Terraform
2. Use Terraform to bootstrap the cluster with a CD tool (e.g., ArgoCD)
3. ArgoCD looks at a repo that deploys all apps from there
4. An application needs a new S3 Bucket, so the developer writes Terraform code for it
5. The application gets removed after a while (but the bucket stays)
6. Someone needs to remember that bucket was owned by that app and remove it from Terraform

‍

A Crossplane Full Env Example

To provision and deploy the environment, the flow stays familiar: start with a Kubernetes cluster, then let Crossplane reconcile the external AWS resources the application depends on.

1. Provision VPC+EKS+... using Terraform
2. Deploy Crossplane's prerequisites to the cluster with Terraform
3. Package app-specific Crossplane resources inside the application Helm chart, so an application release also requests the infrastructure it depends on.
4. Use Crossplane to install the application Helm chart and provision the shared infrastructure required by that class of applications.
5. When an application is removed, its AWS resources are gone with it
6. When an entire environment is terminated, its AWS resources are gone with it

‍

Choosing the Boundary: Crossplane Manages Helm, or Helm Ships Crossplane

Once Helm enters the workflow, the design question becomes concrete:

Do you package Crossplane resources inside the application chart, or do you let Crossplane install the chart after it provisions the supporting infrastructure?

The practical answer is: both, but not at the same point in the lifecycle.

Reasons for Crossplane in Helm:

1. Create or modify app-specific resources when that app is deployed
2. Delete app-specific resources when that app is deleted

Reasons for Helm in Crossplane:

1. Manage dependencies between resources and applications using Crossplane
2. Create shared resources that are not owned by a single application

‍

The Step-by-Step Guide

Deploy the simple application alongside a S3 bucket using a Crossplane Composite Application.

‍

Before proceeding

Before you continue, complete the setup from the first article. The examples here assume the AWS provider is installed, the ProviderConfig points to valid credentials, and the IAM identity behind it can manage the AWS resources declared in this walkthrough. If that baseline is wrong, Kubernetes may still accept every manifest while Crossplane sits in a failed or pending reconciliation loop. Debug that as a control-plane problem first, not as an application bug. Check the IAM policy scope, the ProviderConfig reference, managed resource conditions, Kubernetes Events, and the Crossplane provider logs before chasing pod-level symptoms.

‍

Deploy the Crossplane Kubernetes Provider

Prepare the AWS Credentials for the Application to be able to use AWS

Run the following oneliner to create the Secret containing the AWS credentials in the right format as required by the Application (the application will simply run aws s3 ls to show the bucket):

kubectl create secret generic aws-creds \
--from-literal=aws_access_key_id=$(grep -i aws_access_key_id creds | awk -F' = ' '{print $2}') \
--from-literal=aws_secret_access_key=$(grep -i aws_secret_access_key creds | awk -F' = ' '{print $2}')

Make sure it was created as expected by fetching the secret:

Deploy the Crossplane Kubernetes Provider resources using the k8s-provider-bootstrap.yaml file

kubectl apply -f k8s-provider-conf.yaml

Before you create anything that depends on the provider, verify that it exists and is ready. Kubernetes may accept the custom resources before the provider is healthy, but Crossplane will not reconcile them correctly until the provider is installed, configured, and reporting a ready state.

kubectl get providers provider-kubernetes

When the provider is healthy, the status should look similar to this:

Deploy the Crossplane Kubernetes Provider Configuration using the k8s-provider-conf.yaml file:

kubectl apply -f k8s-provider-conf.yaml

This is done separately as it needs to happen after the Provider resources were created.

This is where we tell the Crossplane Kubernetes Provider in which Kubernetes cluster it should operate when it's creating resources.

‍

Create a deployable unit for an application and its AWS resources with Crossplane

Here we do 3 things with 3 files:

1. The composite-app-xrd file:
   Contains the CompositeResourceDefinition (XRD) for the K8sApplication by using the Composition of a K8s Deployment and S3 Bucket (described below)
2. The composite-app-composition file:
   Contains the Composition definition which creates both the Kubernetes Deployment and the S3 Bucket
3. The composite-app-example file:
   Calls the CompositeResource defined by composite-app-xrd file

‍

Crossplane Resources Files Breakdown & Creation

‍

composite-app-xrd.yaml

~ K8sApplication CompositeResourceDefinition

This defines a composite resource for a Kubernetes application, with bucketName and bucketRegion fields in the spec. Users can claim this resource as K8sApplication.
The K8sApplication CompositeResource (XRD) accepts the bucketName & bucketRegion fields and uses them to create a S3 Bucket, and to create a K8s Deployment of a mock "service" that simply runs aws s3 ls to see the bucket.

~ Deploy the CompositeResourceDefinition (XRD)

kubectl apply -f composite-app-xrd.yaml

‍

composite-app-composition.yaml

Defines a Composition of resources that can be created by a CompositeResource.

This is where we define the Composition that creates a combo of a Kubernetes Deployment with the mock "service" that runs aws s3 ls as well as the S3 bucket - The CompositeResource simply calls this resource.

~ Deploy the Composition

kubectl apply -f composite-app-composition.yaml

‍

composite-app-example.yaml

Deploys the actual K8sApplication CompositeResource, passes the target AWS region and bucket name, and exposes those same values to the Kubernetes Deployment as environment variables so the workload can access the bucket Crossplane created.

As mentioned above, the CompositeResource calls the Composition which creates the resources using the Crossplane providers.

Deploy the app by running the following command:

kubectl apply -f composite-app-example.yaml

‍

Look at your pretty Application

Fetch the K8sApplication resource you've just created by running the below command obsessively until it's marked as Healthy:

kubectl get K8sApplication

‍

When reconciliation finishes, inspect the managed resource status. It should be close to this:

Print the logs of the application and see it fetching the AWS S3 Bucket:

kubectl logs -l app=awscli
# 2024-10-17 16:00:31 my-app-bucket-nqzhx-xzjcq
# 2024-10-17 16:00:50 my-app-bucket-nqzhx-xzjcq

‍

‍

Cleanup

kubectl delete -f composite-app-example.yaml

‍

Recap

To briefly recap what you did here:

1. Prepared Crossplane for deploying a mix of Kubernetes and AWS resources
2. Defined the manifests required to deploy an app built of a Deployment and a S3 Bucket
3. Sharpened your grasp on some Crossplane concepts
4. Discussed some use-cases for which it's useful

If this walkthrough helped, or if you want another platform engineering or cloud-native topic covered in the same practical style, send a note to michael@meteorops.com.

‍

_{Disclaimer: In actual environments or production, it’s essential to fine-tune the permissions in the different manifests. Instead of using access keys and secret keys directly, consider implementing IAM Roles for Service Accounts (IRSA) to manage permissions more securely.}