Secrets Management with HashiCorp Vault in Kubernetes

The Secrets Problem in Kubernetes

Every application needs secrets — database passwords, API keys, TLS certificates, encryption keys. In Kubernetes, the default approach is to store these in Kubernetes Secrets, which are base64-encoded (not encrypted) and accessible to anyone with RBAC access to the namespace.

This is not security. It is security theater.

Common anti-patterns we see in production environments:

Secrets hardcoded in container images — Exposed in every registry and build log
Secrets in environment variables — Visible in kubectl describe pod output
Secrets committed to Git — Even "private" repos get breached
Shared static credentials — No rotation, no audit trail, no expiration

HashiCorp Vault solves these problems by providing centralized secrets management with dynamic credential generation, automatic rotation, detailed audit logging, and fine-grained access control.

Step 1: Deploy Vault on Kubernetes with Helm

The recommended approach is to run Vault in high-availability mode with integrated Raft storage:

# Add the HashiCorp Helm repository
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

# Create a dedicated namespace
kubectl create namespace vault

# Install Vault with HA configuration
helm install vault hashicorp/vault \
  --namespace vault \
  --set server.ha.enabled=true \
  --set server.ha.replicas=3 \
  --set server.ha.raft.enabled=true \
  --set server.dataStorage.size=10Gi \
  --set server.dataStorage.storageClass=gp3 \
  --set server.auditStorage.enabled=true \
  --set server.auditStorage.size=10Gi \
  --set injector.enabled=true \
  --set ui.enabled=true

After deployment, initialize and unseal the first Vault pod:

# Initialize Vault with 5 key shares, 3 required to unseal
kubectl exec -n vault vault-0 -- vault operator init \
  -key-shares=5 \
  -key-threshold=3 \
  -format=json > vault-init.json

# Unseal the first pod (repeat with 3 different keys)
kubectl exec -n vault vault-0 -- vault operator unseal <key-1>
kubectl exec -n vault vault-0 -- vault operator unseal <key-2>
kubectl exec -n vault vault-0 -- vault operator unseal <key-3>

# Join the other pods to the Raft cluster
kubectl exec -n vault vault-1 -- vault operator raft join \
  http://vault-0.vault-internal:8200

kubectl exec -n vault vault-2 -- vault operator raft join \
  http://vault-0.vault-internal:8200

# Unseal the remaining pods
# (In production, use auto-unseal with AWS KMS, GCP KMS, or Azure Key Vault)

For production deployments, always configure auto-unseal to avoid manual intervention after pod restarts:

# vault-config.hcl
seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "alias/vault-unseal-key"
}

Step 2: Configure Kubernetes Authentication

Vault needs to verify that pods requesting secrets are who they claim to be. The Kubernetes auth method uses service account tokens for authentication:

# Login to Vault
export VAULT_ADDR="http://127.0.0.1:8200"
vault login <root-token>

# Enable Kubernetes auth method
vault auth enable kubernetes

# Configure it to communicate with the Kubernetes API
vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc:443"

Create a policy that defines what secrets a service can access:

# payment-service-policy.hcl
path "secret/data/payment-service/*" {
  capabilities = ["read"]
}

path "database/creds/payment-service-role" {
  capabilities = ["read"]
}

path "pki/issue/payment-service" {
  capabilities = ["create", "update"]
}

Apply the policy and bind it to a Kubernetes service account:

# Write the policy
vault policy write payment-service payment-service-policy.hcl

# Create a role that maps Kubernetes SA to Vault policy
vault write auth/kubernetes/role/payment-service \
  bound_service_account_names=payment-service \
  bound_service_account_namespaces=production \
  policies=payment-service \
  ttl=1h

Step 3: Implement Dynamic Database Credentials

Static database passwords are a liability — they never expire, are shared across environments, and are impossible to rotate without downtime. Vault's database secrets engine generates unique, short-lived credentials on demand.

# Enable the database secrets engine
vault secrets enable database

# Configure the PostgreSQL connection
vault write database/config/payments-db \
  plugin_name=postgresql-database-plugin \
  allowed_roles="payment-service-role" \
  connection_url="postgresql://{{username}}:{{password}}@payments-db.production.svc:5432/payments?sslmode=require" \
  username="vault_admin" \
  password="initial-setup-password"

# Rotate the root password so only Vault knows it
vault write -force database/rotate-root/payments-db

# Create a role that generates credentials with a 1-hour TTL
vault write database/roles/payment-service-role \
  db_name=payments-db \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  revocation_statements="REVOKE ALL ON ALL TABLES IN SCHEMA public FROM \"{{name}}\"; DROP ROLE IF EXISTS \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

Now every time a pod starts, it receives a unique database credential that automatically expires. If a credential is compromised, the blast radius is limited to one pod and one hour.

Step 4: Inject Secrets into Pods with Vault Agent

The Vault Agent Injector uses Kubernetes mutating webhooks to automatically inject secrets into pods via annotations. No application code changes required.

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: payment-service
  template:
    metadata:
      labels:
        app: payment-service
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "payment-service"

        # Static secrets
        vault.hashicorp.com/agent-inject-secret-config: "secret/data/payment-service/config"
        vault.hashicorp.com/agent-inject-template-config: |
          {{- with secret "secret/data/payment-service/config" -}}
          export STRIPE_API_KEY="{{ .Data.data.stripe_api_key }}"
          export WEBHOOK_SECRET="{{ .Data.data.webhook_secret }}"
          {{- end }}

        # Dynamic database credentials
        vault.hashicorp.com/agent-inject-secret-db: "database/creds/payment-service-role"
        vault.hashicorp.com/agent-inject-template-db: |
          {{- with secret "database/creds/payment-service-role" -}}
          export DB_USERNAME="{{ .Data.username }}"
          export DB_PASSWORD="{{ .Data.password }}"
          {{- end }}

        # Auto-renew credentials before they expire
        vault.hashicorp.com/agent-cache-enable: "true"
        vault.hashicorp.com/agent-cache-listener-port: "8200"
    spec:
      serviceAccountName: payment-service
      containers:
        - name: payment-service
          image: myorg/payment-service:v1.2.3
          command: ["/bin/sh", "-c"]
          args:
            - source /vault/secrets/config &&
              source /vault/secrets/db &&
              /app/payment-service
          ports:
            - containerPort: 8080
          resources:
            limits:
              cpu: 500m
              memory: 256Mi
            requests:
              cpu: 100m
              memory: 128Mi

The Vault Agent sidecar handles authentication, secret retrieval, and automatic renewal — your application just reads files or environment variables.

Step 5: Alternative Approach with External Secrets Operator

For teams that prefer a Kubernetes-native workflow, the External Secrets Operator (ESO) syncs Vault secrets into Kubernetes Secrets automatically:

# secret-store.yaml
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "http://vault.vault.svc:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "external-secrets"
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets

---
# external-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: payment-service-secrets
  namespace: production
spec:
  refreshInterval: 5m
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: payment-service-secrets
    creationPolicy: Owner
  data:
    - secretKey: stripe-api-key
      remoteRef:
        key: secret/data/payment-service/config
        property: stripe_api_key
    - secretKey: webhook-secret
      remoteRef:
        key: secret/data/payment-service/config
        property: webhook_secret

This approach works well when your applications already read from Kubernetes Secrets and you want to add Vault as the source of truth without changing application code.

Step 6: Enable Audit Logging

In regulated environments, you must demonstrate who accessed what secrets and when. Enable Vault's audit log:

# Enable file-based audit logging
vault audit enable file file_path=/vault/audit/audit.log

# For production, ship logs to your SIEM
# Example: enable syslog backend
vault audit enable syslog tag="vault" facility="AUTH"

Every Vault operation — reads, writes, authentication attempts, policy changes — is recorded with the full request and response (with sensitive values HMAC'd). Forward these logs to your SIEM (Splunk, Elastic, or Datadog) for alerting on suspicious access patterns.

Production Hardening Checklist

Before going to production, verify these items:

Auto-unseal configured with cloud KMS (never store unseal keys on disk)
TLS enabled for all Vault communication (mTLS preferred)
Audit logging enabled and forwarded to SIEM
Root token revoked after initial setup (use identity-based auth)
Namespaces configured to isolate teams (Vault Enterprise)
Disaster recovery replication configured across regions
Backup and restore tested with Raft snapshots
Sentinel policies enforcing organizational rules (Vault Enterprise)
Resource limits set on Vault pods to prevent noisy-neighbor issues
Network policies restricting which pods can reach Vault

# Take a Raft snapshot for backup
vault operator raft snapshot save /tmp/vault-backup.snap

# Verify the snapshot
vault operator raft snapshot inspect /tmp/vault-backup.snap

# Automate daily backups with a CronJob

# vault-backup-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: vault-backup
  namespace: vault
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: vault-backup
          containers:
            - name: backup
              image: hashicorp/vault:1.15
              command:
                - /bin/sh
                - -c
                - |
                  vault operator raft snapshot save /tmp/vault-$(date +%Y%m%d).snap
                  aws s3 cp /tmp/vault-$(date +%Y%m%d).snap \
                    s3://my-vault-backups/$(date +%Y%m%d).snap
              env:
                - name: VAULT_ADDR
                  value: "http://vault-active.vault.svc:8200"
                - name: VAULT_TOKEN
                  valueFrom:
                    secretKeyRef:
                      name: vault-backup-token
                      key: token
          restartPolicy: OnFailure

Conclusion

Secrets management is not optional — it is a fundamental security requirement that only becomes harder to retrofit as your infrastructure grows. HashiCorp Vault, when properly deployed on Kubernetes, provides dynamic secrets, automatic rotation, fine-grained access control, and comprehensive audit logging that static secrets simply cannot match.

At DevOpsVibe, we have deployed and managed Vault across dozens of Kubernetes clusters for organizations ranging from startups to enterprises. Whether you need help with initial deployment, migration from static secrets, or integration with your existing CI/CD pipelines, our team can get you to production-grade secrets management in weeks, not months. Secure your infrastructure today.

filed under

vaultsecrets-management

kubernetessecurityhashicorpdevops