Vault on Kubernetes security considerations

Vault is a security product responsible for protecting sensitive data and serves as a single source of secrets. The Production Hardening documentation provides recommendations based on the security model assuming that Vault is deployed on VMs instead of containers. When you deploy Vault on Kubernetes, those best practices are not quite transparent. This tutorial highlights where extra precaution is needed when you deploy Vault on Kubernetes in production.

The following topics are addressed in this tutorial:

• Single vs. Multi Tenancy Runtimes
• End-to-End TLS
• Turn Off Core Dumps
• Ensure mlock is Enabled
• Container Supervisor
• Don't Run as Root

Single vs. Multi Tenancy Runtimes

The Vault Production Hardening documentation states that Vault should be the only main process running on a machine to reduce the surface area introduced by other tenants.

Limitations introduced by running Vault on Kubernetes

Consul Helm chart only allows one Consul server cluster per Kubernetes cluster. This makes Consul available to other tenants on the Kubernetes cluster which undermines the security model for Vault when using Consul as its storage backend.

Another security concern is that the Vault container is owned by root, but the Vault executable inside the container is still run as the vault user. This means that the supervisor process is owned by root. Any process that escapes the pod will have root privileges on the Kubernetes worker node. (Also, see the Container Supervisor section.)

End-to-End TLS

Vault should always be used with Transport Layer Security (TLS) in production. If intermediate load balancers or reverse proxies are used to front Vault, they should not terminate TLS to ensure that the traffic is always encrypted in transit.

Limitations introduced by running Vault on Kubernetes

Kubernetes uses TLS throughout the system to connect disparate components. However, Kubernetes does not verify TLS connections by default for certain connections, and portions of the codebase include the use of InsecureSkipVerify which precludes verification of the presented certificate.

Issue Example:

if dialer != nil {    // We have a dialer; use it to open the connection, then    // create a tls client using the connection.    netConn, err := dialer(ctx, "tcp", dialAddr)    if err != nil {        return nil, err    }    if tlsConfig == nil {        // tls.Client requires non-nil config        klog.Warningf("using custom dialer with no TLSClientConfig. Defaulting to InsecureSkipVerify")        // tls.Handshake() requires ServerName or InsecureSkipVerify        tlsConfig = &tls.Config{            InsecureSkipVerify: true,        }    // ...}

Mitigations

• Use TLS certificates signed by a trusted Certificate Authority (CA)
• Use intermediate CA's to protect the root CA
• Verify the authenticity of the server when making client request
• Verify the authenticity of the client when responding to requests
• Require TLS 1.2+

Turn Off Core Dumps

A user or administrator that can force a core dump and has access to the resulting file can potentially access Vault encryption keys. Therefore, ensure that the process core dumps are disabled inside the container.

Limitations introduced by running Vault on Kubernetes

• Root user on the worker node can access procfs (process filesystem) for the containerized process
• In a multi-tenant cluster, containers running with the --privileged flag will be able to access procfs on the host machine
• If this is the same machine that the Vault container is running, a process from the privileged container could exfiltrate the key from a core dump of the Vault process in the container

Mitigations

Ensure RLIMIT_CORE is set to 0 or use the ulimit command with the core flag (ulimit -c 0) inside the container to ensure your container processes can't core dump.

Ensure mlock is Enabled

Enable memory lock (mlock) to prevent memory swap (writing memory to disk).

Limitations introduced by running Vault on Kubernetes

Memory lock (mlock) ensures memory from a process on a Linux system isn't swapped (written) to disk. Extra care is needed to ensure this is properly configured inside your container and enabled through Kubernetes.

Mitigations

• Due to how capabilities are performed in Linux, the parent process that is starting the container which runs the mlock call must have IPC_LOCK capabilities

• To enable IPC_LOCK in the container supervisor process, use a security context:

  securityContext:    runAsNonRoot: true    runAsUser: 1000    capabilities:      add: ["IPC_LOCK"]

• Ensure your container is leveraging a storage driver that supports mlock such as overlayfs2.

Container Supervisor

The supervisor process that starts your container should not run as root.

Limitations introduced by running Vault on Kubernetes

If your container starts as root, the processes that escape that container have root on the node they were started on.

Mitigations

Apply security context for your container and pod to prevent starting your container as root. (A security context is a property defined in the deployment yaml.)

Generic Kubernetes Pod example:

apiVersion: v1kind: Podmetadata:  name: hello-worldspec:  containers:  # specification of the pod’s containers  # ...  securityContext:    runAsNonRoot: true

Don't Run as Root

Vault is designed to run as an unprivileged user, and there is no reason to run Vault with root or administrator privileges which can expose the Vault process memory and allow access to Vault encryption keys.

Limitations introduced by running Vault on Kubernetes

Do not circumvent the entry point for your container. This will also circumvent the user to use PID 1 in the container and use root instead. (Refer to Issue #205.)

Mitigations

Follow the hardening guidelines and ensure your Vault process is not started as root user inside your container.

Help and Reference

• Vault Security Model
• Production hardening documentation