The safest way to manage Kubernetes secrets with Vault in a startup is to stop storing long-lived secret values in manifests and let each pod fetch short-lived credentials at runtime. In practice, you bind one Kubernetes service account to one Vault role, give it read access to a narrow secret path, and deliver the secret through Vault Agent Injector or a CSI-based volume path.
How the pattern works
Vault handles secret storage, policy, lease, and rotation, while Kubernetes supplies workload identity through service accounts.
The usual flow looks like this:
- A pod starts with a specific Kubernetes service account.
- Vault verifies that service account through the Kubernetes auth method.
- Vault maps the service account and namespace to a Vault role.
- The Vault role attaches one or more narrow policies.
- The pod receives only the secrets allowed by those policies.
- Vault renews or expires the credentials based on their lease settings.
This gives you a clean boundary. A payments API can read payment database credentials, but it cannot read the staging Stripe token, the data warehouse password, or another service鈥檚 signing key.
Key concepts you need before implementing Vault
You need to understand service accounts, Vault roles, policies, leases, and delivery methods before you wire Vault into Kubernetes.
- Kubernetes service account. This is the workload identity. Use one service account per application, not the default service account.
- Vault Kubernetes auth role. This maps a Kubernetes service account and namespace to Vault policies.
- Vault policy. This defines which paths the workload can read, list, create, update, or delete.
- Secret path. This is the location of the secret in Vault, such as
kv/data/apps/prod/payments-api/database. - Lease. This controls how long a dynamic secret is valid. Database users, cloud credentials, and certificates can be issued with leases.
- Delivery method. Vault Agent Injector renders secrets into the pod, while the CSI driver mounts secrets as files.
A practical startup architecture
A startup should start with simple boundaries: one Vault path per application and environment, and one Kubernetes service account per workload.
A workable first layout might look like this:
kv/
data/
apps/
prod/
payments-api/
database
stripe
jwt-signing-key
staging/
payments-api/
database
stripe
jwt-signing-keyFor dynamic secrets, use dedicated engines instead of storing static values in KV where possible. For example, use Vault鈥檚 database secrets engine to issue temporary PostgreSQL users instead of storing one shared database password forever.
Keep the first version boring. You do not need a large role hierarchy, many policy templates, or several delivery methods on day one. Pick one pattern, document it, and make every new service follow it.
Step 1: decide which secrets belong in Vault
Vault should hold application secrets that need access control, auditability, or rotation.
Good candidates include:
- Database usernames and passwords.
- API tokens for third-party services.
- JWT signing keys.
- Webhook signing secrets.
- Private keys used by applications.
- Temporary cloud credentials when your cloud setup supports them.
Do not put every configuration value in Vault. Feature flags, public URLs, log levels, and non-sensitive settings usually belong in ConfigMaps, application config, or your deployment system.
Also avoid using Vault where native identity is better. For example, if a workload can access AWS, GCP, or Azure through workload identity without a static key, prefer that over storing a cloud access key in Vault.
Step 2: enable Kubernetes authentication in Vault
The Kubernetes auth method lets Vault trust a pod鈥檚 service account token without storing a Vault token in Kubernetes.
The exact configuration depends on where Vault runs. If Vault runs outside the cluster, use the Kubernetes API server endpoint, cluster CA certificate, and a token reviewer service account. A typical setup looks like this:
vault auth enable kubernetes
vault write auth/kubernetes/config \
token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
kubernetes_host="$KUBERNETES_HOST" \
kubernetes_ca_cert="$KUBERNETES_CA_CERT"The token reviewer should have only the permissions needed to call the Kubernetes TokenReview API. Do not reuse a broad admin token for this.
Step 3: create a narrow Vault policy
Each application should get a policy that allows access only to its own paths.
For a payments API using KV v2, the policy might look like this:
path "kv/data/apps/prod/payments-api/*" {
capabilities = ["read"]
}
path "kv/metadata/apps/prod/payments-api/*" {
capabilities = ["list"]
}The kv/data path is used for reading KV v2 secret values. The kv/metadata path is used when the application or agent needs metadata or listing access. If the workload does not need to list keys, remove the list capability.
Step 4: bind the policy to a Kubernetes service account
The Vault role should bind one service account in one namespace to the application policy.
vault write auth/kubernetes/role/payments-api \
bound_service_account_names="payments-api" \
bound_service_account_namespaces="prod" \
policies="payments-api" \
ttl="1h"This role means that only pods running as the payments-api service account in the prod namespace can receive the payments-api Vault policy.
Create the matching Kubernetes service account:
apiVersion: v1
kind: ServiceAccount
metadata:
name: payments-api
namespace: prodDo not bind Vault roles to default service accounts. That turns every pod in the namespace into a potential secret reader.
Step 5: deliver secrets with Vault Agent Injector
Vault Agent Injector is a common choice when you want Vault to render secrets into files inside the pod without changing much application code.
With the injector, annotations tell Vault which role to use and which secrets to render:
apiVersion: apps/v1
kind: Deployment
metadata:
name: payments-api
namespace: prod
spec:
replicas: 2
selector:
matchLabels:
app: payments-api
template:
metadata:
labels:
app: payments-api
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "payments-api"
vault.hashicorp.com/agent-inject-secret-database.env: "kv/data/apps/prod/payments-api/database"
vault.hashicorp.com/agent-inject-template-database.env: |
{{- with secret "kv/data/apps/prod/payments-api/database" -}}
DB_USERNAME={{ .Data.data.username }}
DB_PASSWORD={{ .Data.data.password }}
{{- end }}
spec:
serviceAccountName: payments-api
containers:
- name: app
image: example.com/payments-api:1.2.3
command: ["/bin/sh", "-c"]
args:
- . /vault/secrets/database.env && exec ./payments-apiThis keeps the secret out of the Kubernetes manifest and out of the container image. The rendered file lives inside the pod filesystem.
For new applications, prefer reading secrets from files instead of environment variables. Environment variables are easy, but they do not update while the process is running and they often appear in debugging output, crash reports, or process inspection tools.
Step 6: deliver secrets with the CSI driver
The CSI path is useful when you want secrets mounted as files through Kubernetes volumes.
A simplified SecretProviderClass can look like this:
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: vault-payments-api
namespace: prod
spec:
provider: vault
parameters:
vaultAddress: "https://vault.example.com:8200"
roleName: "payments-api"
objects: |
- objectName: "database-password"
secretPath: "kv/data/apps/prod/payments-api/database"
secretKey: "password"The pod then mounts the CSI volume:
apiVersion: apps/v1
kind: Deployment
metadata:
name: payments-api
namespace: prod
spec:
template:
spec:
serviceAccountName: payments-api
containers:
- name: app
image: example.com/payments-api:1.2.3
volumeMounts:
- name: secrets
mountPath: "/mnt/secrets"
readOnly: true
volumes:
- name: secrets
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "vault-payments-api"This pattern works well when the application can read secrets from files such as /mnt/secrets/database-password. If the application only supports environment variables, you may need a wrapper script, an application change, or a different delivery method.
Step 7: use dynamic database credentials where you can
Dynamic database credentials are safer than shared static passwords because Vault can create, renew, and revoke them per workload.
For PostgreSQL, the general idea is to configure Vault with a database connection and a role that creates temporary users:
vault secrets enable database
vault write database/config/postgresql \
plugin_name="postgresql-database-plugin" \
allowed_roles="payments-api" \
connection_url="postgresql://{{username}}:{{password}}@postgres.example.com:5432/postgres?sslmode=require" \
username="$VAULT_DB_ADMIN_USERNAME" \
password="$VAULT_DB_ADMIN_PASSWORD"
vault write database/roles/payments-api \
db_name="postgresql" \
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT payments_app TO \"{{name}}\";" \
default_ttl="1h" \
max_ttl="4h"The application then reads from a dynamic path such as:
database/creds/payments-apiThis gives every pod or agent session its own database credentials. When the lease expires or is revoked, Vault can remove the database user.
Before you use short TTLs in production, check how your application handles connection pools. A pool that keeps old connections forever may fail when credentials expire. Start with a TTL such as 1 to 4 hours, verify renewal, and test what happens when Vault revokes the lease.
Tradeoffs of Vault in Kubernetes
Vault improves secret control, but it adds an operational dependency that you need to run carefully.
- Availability matters. If Vault is unavailable, new pods may fail to start or fail to fetch secrets. Existing pods may keep running until their current credentials expire.
- Rotation needs application support. File updates do not help if the application never reloads them. Some services need a restart, a signal, or explicit reload logic.
- Dynamic secrets are better, but they are more complex. They require database roles, TTL tuning, and revocation testing.
- Agent sidecars consume resources. The overhead is usually acceptable, but it matters in very dense clusters with hundreds of small pods.
- Static KV secrets still need a rotation process. Moving a long-lived API token into Vault does not rotate it automatically.
- Vault itself needs backups and recovery testing. A secure secret store that nobody can restore is a production risk.
Vault Agent Injector versus CSI versus other options
The right tool depends on whether you want secrets rendered by a sidecar, mounted as files, synced into Kubernetes Secrets, or encrypted in Git.
- Kubernetes Secrets. This is the simplest option, but values are only base64 encoded unless your cluster has encryption at rest configured, and rotation is usually manual.
- SOPS or Sealed Secrets. This works well for GitOps workflows because encrypted values can live in Git, but decrypted secrets still become Kubernetes Secrets in the cluster.
- External Secrets Operator. This is useful when applications expect Kubernetes Secrets, but the synced value still lands in Kubernetes and should be protected with etcd encryption and RBAC.
- Vault Agent Injector. This is strong for templating, dynamic secrets, and lease renewal, but it adds init containers or sidecars and webhook behavior to understand.
- Vault CSI provider. This is clean for file mounts and Kubernetes-native volume management, but applications must read from files or adapt to them.
For most startups already running Kubernetes, Vault Agent Injector is often the fastest first step. CSI is a good fit when your platform team wants a consistent volume-based pattern. External Secrets Operator is reasonable when you must support applications that already depend on Kubernetes Secret objects.
Security controls that matter most
The most important controls are narrow identity bindings, narrow policies, audit logs, TLS, and a tested recovery path.
- Use one Kubernetes service account per workload.
- Bind Vault roles to exact service account names and namespaces.
- Avoid broad policies such as
kv/data/apps/*for application workloads. - Use separate paths for production and staging.
- Run Vault over TLS only.
- Restrict network access so only approved namespaces or workloads can reach Vault.
- Turn on Vault audit logging before production use.
- Keep root tokens out of CI, Slack, tickets, and shared password managers.
- Test backup restore before your first incident.
- Document how to rotate each class of secret.
Common mistakes to avoid
Most Vault failures in startups come from overbroad access, unclear ownership, or treating Vault as a place to hide static secrets forever.
- Using the default service account. This makes identity too broad and makes later cleanup painful.
- Giving every service the same Vault policy. This removes the main security benefit of Vault.
- Putting secrets into environment variables by default. This makes refresh harder and can expose values through debugging tools.
- Skipping etcd encryption when syncing to Kubernetes Secrets. If you sync secrets back into Kubernetes, secure Kubernetes storage as well.
- Using short TTLs without testing renewal. Short-lived credentials are useful only when the app and agent can renew or reload safely.
- Forgetting local development. Developers need a safe path for dev credentials that does not copy production Vault tokens onto laptops.
- Running Vault without an owner. Someone must own policies, auth roles, upgrades, backup tests, and incident response.
A startup-friendly rollout plan
The safest rollout is to move one non-critical service first, prove the pattern, and then make it the default for new services.
- Pick one service with a small number of secrets.
- Create a dedicated Kubernetes service account for that service.
- Create one Vault policy scoped to that service and environment.
- Use Vault Agent Injector or CSI to mount secrets as files.
- Deploy to staging and confirm the pod starts without Kubernetes Secret values.
- Rotate the old secret after the service is reading from Vault.
- Test pod restart, Vault downtime, and lease renewal behavior.
- Document the pattern and apply it to the next service.
Do not start by migrating every secret in the company. Start with the pattern you want all future services to copy.
Decision checklist
You are ready to run Vault-backed Kubernetes secrets when you can answer these operational questions clearly.
- Which service account maps to each Vault role?
- Which Vault paths can each workload read?
- Are production and staging secrets separated?
- Are secrets delivered as files, environment variables, or synced Kubernetes Secrets?
- Does the application reload secrets, or does it need a restart?
- What happens if Vault is unavailable during a deploy?
- What is the TTL for each dynamic secret?
- Who owns Vault policy changes?
- Where are Vault audit logs stored?
- When was the last restore test?
How to get this right
Use Vault to remove long-lived secrets from Kubernetes manifests, bind each workload to a narrow Vault role, and deliver secrets at runtime through Agent Injector or CSI. Start with one service, prefer file-based delivery, use dynamic credentials where your application can handle renewal, and keep policies small enough to review in a pull request.
If your startup runs Kubernetes on Azure and wants a senior engineer to set this up cleanly, MeteorOps engineers do this hourly for startups through Azure Kubernetes Service support.
