This is part 3 of our HashiCorp Vault production series. In part 1, we explained why we run Vault in a separate cluster. Now comes the next challenge: how do you get secrets from Vault to your applications without storing them in Git?

Series overview:

1. Production Kubernetes Architecture with HashiCorp Vault 
2. Terraform Infrastructure for HashiCorp Vault on EKS 
3. External Secrets Operator: GitOps for Kubernetes Secrets 
4. Dynamic PostgreSQL Credentials with HashiCorp Vault 
5. Vault Agent vs Secrets Operator vs CSI Provider 
6. Securing Vault Access with Internal NLB and VPN

What is External Secrets Operator for Vault Integration?

External Secrets Operator (ESO) is a Kubernetes operator that synchronizes secrets from external systems like HashiCorp Vault into Kubernetes secrets. It solves a fundamental GitOps problem: keeping sensitive data out of Git while maintaining declarative configuration.

In our setup, ESO:

• Reads secrets from HashiCorp Vault
• Creates native Kubernetes secrets
• Keeps them synchronized
• Integrates perfectly with FluxCD for GitOps workflows

The GitOps Secret Management Problem in Kubernetes

In a GitOps workflow, everything should be in Git. But putting secrets in Git, even encrypted, creates risks:

• Accidental exposure in pull requests
• Permanent history of sensitive data
• Complex key management for encryption

ESO solves this by storing only the reference to secrets in Git, not the secrets themselves.

Implementing External Secrets Operator with HashiCorp Vault

Here’s how we configured External Secrets Operator to work with our Vault setup.

Prerequisites

From our architecture:

• HashiCorp Vault running in tooling cluster (see part 1)
• FluxCD managing both clusters
• Vault accessible at vault.tooling.internal:8200

Installing external secrets operator

ESO is deployed via FluxCD in our application cluster:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: external-secrets
  namespace: external-secrets-system
spec:
  interval: 1h
  url: https://charts.external-secrets.io

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: external-secrets
  namespace: external-secrets-system
spec:
  interval: 5m
  chart:
    spec:
      chart: external-secrets
      sourceRef:
        kind: HelmRepository
        name: external-secrets

Configuring ClusterSecretStore for Vault

The ClusterSecretStore tells ESO how to connect to Vault:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "https://vault.tooling.internal:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "external-secrets"
          serviceAccountRef:
            name: "external-secrets"
            namespace: "external-secrets-system"

This configuration:

• Points to our internal Vault instance
• Uses Kubernetes authentication
• References a service account that Vault trusts

Creating external secrets

With the ClusterSecretStore configured, we can create ExternalSecret resources:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
  namespace: production
spec:
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  refreshInterval: 15m
  target:
    name: app-secrets
    creationPolicy: Owner
  data:
    - secretKey: database-password
      remoteRef:
        key: secret/data/production/database
        property: password
    - secretKey: api-key
      remoteRef:
        key: secret/data/production/external-api
        property: key

ESO will:

1. Connect to Vault using the ClusterSecretStore config
2. Fetch the specified secrets
3. Create a Kubernetes secret named app-secrets
4. Refresh every 15 minutes

FluxCD and HashiCorp Vault integration for complete GitOps

The real power comes from combining ESO with FluxCD’s substitution feature. This allows us to use secrets in our GitOps workflow without exposing them.

Secret substitution in FluxCD with external secrets

FluxCD can substitute variables in your manifests using data from secrets:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: app-deployment
  namespace: flux-system
spec:
  interval: 10m
  path: "./apps/production"
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  postBuild:
    substituteFrom:
      - kind: Secret
        name: app-secrets
        optional: false

Using substituted values

In your application manifests, use placeholders:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  database-url: "postgres://app:${database-password}@db:5432/app"
  external-api-endpoint: "https://api.example.com"
  external-api-key: "${api-key}"

FluxCD will replace ${database-password} and ${api-key} with values from the ESO-generated secret.

Production patterns with external secrets operator and HashiCorp Vault

Namespace isolation

We use SecretStore (namespace-scoped) instead of ClusterSecretStore for application-specific secrets:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: team-a-vault
  namespace: team-a
spec:
  provider:
    vault:
      server: "https://vault.tooling.internal:8200"
      path: "secret/team-a"
      # Rest of config similar to ClusterSecretStore

Secret rotation handling

While ESO doesn’t support automatic rotation for dynamic secrets, the refresh interval ensures updates are pulled regularly. For dynamic credentials, we use Vault Agent (covered in part 4).

Monitoring secret synchronization

Check ESO sync status:

kubectl get externalsecrets -A

Look for the READY and LAST SYNC columns to verify secrets are syncing properly.

Troubleshooting common issues

ESO can’t connect to Vault

Verify the ClusterSecretStore status:

kubectl describe clustersecretstore vault-backend

Common issues:

• Service account doesn’t exist
• Vault Kubernetes auth not configured
• Network connectivity between clusters

Secrets not updating

Check the ExternalSecret events:

kubectl describe externalsecret app-secrets -n production

The refresh interval might be too long, or there could be permission issues in Vault.

Production considerations

High Availability

ESO runs as a deployment with multiple replicas. Ensure your configuration includes:

replicaCount: 3
resources:
  requests:
    memory: 64Mi
    cpu: 10m

Security

• ESO service account should have minimal Vault permissions
• Use separate Vault paths per environment/team
• Enable Vault audit logging for ESO access

Performance

With hundreds of ExternalSecrets, consider:

• Increasing ESO replicas
• Tuning refresh intervals
• Using webhook triggers instead of polling

What’s Next?

We’ve covered how to sync static secrets from Vault to Kubernetes using ESO. But what about dynamic database credentials that expire? That’s where Vault’s database secret engine shines – covered in our next post.

The complete configuration shown here is available in our GitHub repository, including:

• Full ESO deployment manifests
• Example ExternalSecrets
• FluxCD integration examples
• Vault configuration for ESO