Kubernetes Secrets Management Basics Without Shooting Yourself in the Foot

Secret is a type, not a guarantee

The first time someone told me “it’s fine, it’s in a Secret,” I decoded the base64 in etcd-backed storage and read the database password in cleartext. Base64 is encoding. Anyone with permission to read the Secret object can read the value.

Default cluster encryption of Secret data at rest is not always on. Verify encryptionConfiguration on the API server. Until then, treat etcd access and API read access as equivalent to holding the credential.

Compliance questionnaires often ask how secrets are stored at rest. “We use Kubernetes Secrets” is an incomplete answer without encryption and access control details.

Never commit raw Secret manifests

I do not put plaintext Secret YAML in git. Not even in private repos. Git history keeps them forever. Use a sealed mechanism or generate Secrets in the cluster from an external store at sync time.

For demos, kubectl create secret generic is fine. For GitOps, I use Sealed Secrets or External Secrets Operator pulling from AWS Secrets Manager, GCP Secret Manager, or Vault.

# Create from literals (interactive / CI), not from committed YAML values
kubectl -n payments create secret generic db-credentials \
  --from-literal=username=app \
  --from-literal=password="$(openssl rand -base64 24)" \
  --dry-run=client -o yaml | kubectl apply -f -

# Prefer files mounted as volumes over env for less accidental leakage
kubectl -n payments create secret generic db-credentials \
  --from-file=username=./username.txt \
  --from-file=password=./password.txt

RBAC: who can get secrets

Limit get/list/watch on secrets to the controllers and humans who need them. Developers who only need to restart pods do not need secret read. I have audited clusters where every engineer in a group could kubectl get secret -o yaml across namespaces.

Service accounts should mount only the secrets their workload needs. Do not share one mega-secret across twelve deployments “for convenience.”

• Separate namespaces for trust boundaries when possible
• Avoid cluster-admin for day-to-day humans
• Prefer Role over ClusterRole for secret access
• Audit kubectl auth can-i get secrets -n … for each team role

Mounting: files vs environment variables

Env vars are easy to dump and inherit. I prefer volume mounts at /var/run/secrets/app/… and have the app read files. Many frameworks support both.

If you must use env, inject specific keys, not every key in a blob. Restart pods on rotation—Kubernetes will not always hot-reload env from an updated Secret depending on how you consume it. File mounts from Secret volumes update asynchronously; apps may need to watch or restart.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  template:
    spec:
      containers:
        - name: api
          image: ghcr.io/example/api:1.4.2
          volumeMounts:
            - name: db
              mountPath: /var/run/secrets/db
              readOnly: true
          env:
            - name: DB_PASSWORD_FILE
              value: /var/run/secrets/db/password
      volumes:
        - name: db
          secret:
            secretName: db-credentials
            items:
              - key: password
                path: password

External stores and rotation

External Secrets Operator keeps the cluster Secret in sync with a cloud manager. Rotation happens in one place. I set refresh intervals consciously—too aggressive thrashes; too slow leaves revoked credentials active.

Application support for dual credentials matters. Database users that allow two passwords during rotation avoid downtime. JWT signing with kid headers lets you roll keys without killing sessions instantly.

Annotate Secrets with owning team and rotation date. Stale credentials with no owner are incident fuel.

Common failure modes

Logging the environment at boot. Debug containers with mounted secrets left running. CI jobs that print helm template output including secret values. Backups of etcd without encryption resting in object storage.

Also: copying prod secrets into the dev cluster “to reproduce a bug.” Use scrubbed fixtures. Prod credentials in shared dev clusters are how leaks widen.

Minimum bar I accept for a cluster

Encryption at rest enabled for Secrets, tight RBAC, no plaintext Secrets in git, mounts scoped per workload, and a documented rotation path. That bar is boring and it prevents most of the incidents I get called into.

Sealed Secrets vs External Secrets vs CSI drivers

Sealed Secrets encrypt for a specific cluster controller key so ciphertext can live in git. Simple and good for small teams. Key backup and disaster recovery matter—lose the sealing key and you cannot decrypt.

External Secrets Operator syncs from a cloud manager. Better when security already standardized on Vault or AWS Secrets Manager. CSI secret store drivers mount secrets without always creating a Kubernetes Secret object—useful for stronger isolation, with its own operational learning curve.

I pick Sealed Secrets for small clusters without a central vault. I pick External Secrets when the company already has a secret manager and rotation policies. I avoid inventing a fourth custom operator.

• Sealed Secrets: git-friendly ciphertext, cluster-bound keys
• External Secrets: central source of truth and rotation
• CSI drivers: mount without durable Secret objects when required
• Standardize on one pattern per environment tier

Incident response when a Secret escapes

Assume the value is compromised once it appears in a ticket, screenshot, or CI log. Rotate in the upstream system, update the Kubernetes Secret or external store, roll pods, then invalidate old credentials on the provider side.

Search cluster events, git history, and log systems for the secret prefix patterns. Add detection rules for high-entropy strings in application logs if you do not already have them.

Afterward, fix the path that leaked—boot logging, overly broad RBAC, or a debug container. Rotation without fixing the path is a countdown to the next page.

GitOps workflow that stays honest

Desired state in git should reference secret names and external secret mappings, not values. Pull request reviewers should never need to see live credentials to approve a deploy.

Policy-as-code can deny Kubernetes Secret manifests that contain data: keys in application repos. That guardrail stops the “quick fix” PR that pastes a password into YAML.

Break-glass procedures matter. When the secret manager is down, how do you ship a hotfix? Document a manual path that still audits who accessed what—and retire temporary credentials afterward.

Namespaces, multi-tenancy, and soft walls

Namespaces are not strong security boundaries by themselves, but they help organize RBAC. Tenant A’s secrets should not be readable by tenant B’s deploy role.

On multi-tenant clusters, prefer separate namespaces per team or environment and NetworkPolicies that match. Secrets sitting in default are a smell.

Controllers that sync secrets cluster-wide need careful review. A misconfigured ExternalSecret with the wrong namespace selector is a cross-tenant leak waiting to happen.

• Reference secret names in git, not values
• Policy-deny plaintext Secret data in app repos
• Document break-glass and audit it
• Review cluster-wide secret sync carefully

A short implementation order

If your cluster is early: enable encryption at rest, lock down RBAC for secrets, stop committing plaintext, mount files for the first apps, and pick Sealed Secrets or External Secrets before the third service copies a password into YAML.

If your cluster is mature but messy: inventory who can get secrets, rotate anything that ever lived in git, and migrate the highest-risk credentials to an external manager first—databases, identity providers, payment keys.

Teach the team the base64 myth once. It sticks better than any policy PDF. Then enforce the policy with admission controls so the myth does not have to be relearned in an incident channel.

Key takeaways

• Kubernetes Secrets are base64-encoded objects—protect them with RBAC and etcd encryption at rest.
• Never commit plaintext Secret manifests; use sealing or an external secret store with GitOps.
• Grant secret read access narrowly to humans and service accounts.
• Prefer file mounts over environment variables when practical.
• Plan credential rotation with dual-key or dual-password windows.
• Keep prod secrets out of shared dev clusters and CI logs.