# How to Handle Secrets on the Command Line

Updated on: May 20, 2024

The command line really wasn't designed for secrets. So, keeping secrets secret on the command line requires some extra care and effort. The other day in my homelab I was configuring a TLS client certificate for a Grafana datasource. The intention was to write something I could run on a timer whenever the certificate is renewed. The command needed to:

1. Build some JSON with the renewed certificate and private key injected into it
2. PUT it to Grafana's API to update the datasource configuration.

I thought I was being very clever when I wrote this lil' Bash pipeline:

```
BEARER_TOKEN=MhY3b3i3gFpa9otnLQVznJYoWLxpGJUod3iDJwCKRFUVtuALGJooBJuCUf7w9HJfbu;
jq -n
     --arg ca_cert "$(< $STEPPATH/certs/root_ca.crt)"
     --arg client_cert "$(< $CERT_LOCATION)"
     --arg client_key  "$(< $KEY_LOCATION)"
      -f ./datasource.jq
 | curl -s -X PUT
      -H "Content-Type: application/json"
      -H "Authorization: Bearer $BEARER_TOKEN"
      -d "$(< /dev/stdin)"
      --cacert $STEPPATH/certs/root_ca.crt
      https://grafana:3000/api/datasources/2
```

This uses [`jq`](https://jqlang.github.io/jq/) to populate a JSON template for an API request (`datasource.jq`) with variables passed in with `--arg`, and then PUTs to the API with `curl`. Nice, huh?

It didn't take long to discover that this pipeline leaks secrets _all over the place_:

- The private key credential file is leaked: `"$(< $KEY_LOCATION)"`
- The bearer token env variable is leaked: `-H "Authorization: Bearer $BEARER_TOKEN"`
- And the secret piped data from `STDIN` is leaked: `-d "$(< /dev/stdin)"`

All of these values, including the precious contents of the private key file, can be seen via `ps` when these commands are running. `ps` finds them via `/proc/<pid>/cmdline`, which is globally readable for any process ID.

To make atonement, I'm writing this post. We'll look at three methods for handling secrets on the command line: Using piped data, credential files, and environment variables. We'll look at some of the risks of these approaches, and how to use each of them as safely as possible.

But first, let's look at a sanitized version of the above pipeline:

```
jq -n
   --rawfile ca_cert $STEPPATH/certs/root_ca.crt
   --rawfile client_cert $CERT_LOCATION
   --rawfile client_key $KEY_LOCATION
   -f ./datasource.jq
| curl -s -X PUT
   -H @api_headers
   -d @-
   --cacert $STEPPATH/certs/root_ca.crt
   https://grafana:3000/api/datasources/2
```

Notably:

- The `--rawfile` flag in `jq` _moves the credentials closer to where they are used_ by delegating the responsibility for reading the certificate data from the files to `jq` instead of `bash`.
- Once `jq` pushes the secret JSON into the pipe, `curl` uses the `-d @-` flag to _pull the secret data directly from the pipe_ and use it as the HTTPS request body.
- And `-H @api_headers` will _read the static bearer token from a file_ (`api_headers`).

Now, no secrets will appear in `ps`—only filenames.

## Piped Secrets

As the sanitized example shows, a pipeline is generally an _excellent_ way to pass secrets around, if the program you're using will accept a secret via `STDIN`. Because a pipe only has two ends, right? Imagine yourself whispering a secret into one end of a pipe, and a friend putting their ear up to the other. It's just like that.
Usually. The `$(< /dev/stdin)` leak shown above uses a neat Bash substitution to make an otherwise secure pipe insecure. For example, if you run:

```
$ echo "secret-data" | curl -d "$(</dev/stdin)" https://example.com:3000
```

Then the output of `ps` for `curl` will show:

```
curl -d secret-data https://example.com:3000
```

Other than that, there's not too much to worry about with pipes.

## Credentials Files

What's not to love about a file? It's got an owner. It has permissions and access control. Give each secret a file! Any program that accepts secrets should be able to accept them by passing a filename or by redirecting a file into `STDIN`. You can also use files to pass secrets into Docker containers with mounted volumes.

A few notes about storing and retrieving file secrets:

- You'd better get the permissions right
- Avoid leaking the secret in the command string eg. with `"$(< secret_file.txt)"`
- Be sure your disk is encrypted at rest, eg. with LUKS
- You may want to encrypt the contents of the file — but, then you need to figure out how to handle the encryption key.

## Environment Variables

Using environment variables for secrets is very convenient. And we don't recommend it because it's so easy to leak things:

- Some operating systems still make every process's environment variables world readable. (But, in all the Linuxes I've seen, `/proc/<pid>/environ` is not world-readable.)
- In Docker, anyone with access to the Docker daemon can use `docker inspect` to see all of the environment variables for any running container.
- In systemd, environment variables in unit files are available to users via the dbus interface (see the recent introduction of `LoadCredential=` for an alternative that uses credential files)
- Exported environment variables will get passed to every new process, and then who knows what will happen to them. They might get dumped to `STDOUT` or logged to a debug logfile.
- Local (unexported) shell variables are also easy to leak into `ps` output:

```
$ BEARER_TOKEN=MhY3b3i3gFpa9otnLQVznJYoWLxpGJUod3iDJwCKRFUVtuALGJooBJuCUf7w9HJfbu;
curl -s -X PUT
      -H "Content-Type: application/json"
      -H "Authorization: Bearer $BEARER_TOKEN"
...
```

- Variables can easily end up in shell history. In many shells, adding an extra space before a command will exclude it from shell history.
  - In Bash, the `HISTCONTROL` variable must be set to `ignorespace`.
  - In the Fish shell, run `fish --private` to start a private mode Fish session.
- Secrets can also slip into your editor's search history. For example, in `vim` there's a `$HOME/.viminfo` file that contains search history. Run `vim -i NONE` to disable it for a session.

If a command takes a password _filename_, you can use a password environment variable if you're careful about it. For example, say we have a `$STEP_CA_PASSWORD` environment variable, and we run the following in Bash:

```
$  STEP_CA_PASSWORD=amazingpw; step-ca --password-file <(echo -n "$STEP_CA_PASSWORD") $(step path)/config/ca.json
```

The `/proc/<pid>/cmdline` for this process will contain something like:

```
step-ca --password-file /dev/fd/11 /home/carl/.step/config/ca.json
```

Using this `<()` syntax of [process substitution](https://www.gnu.org/software/bash/manual/html_node/Process-Substitution.html), Bash will create a file using the output of `echo -n "$STEP_CA_PASSWORD"` and supply that file's name to `--password-file`.
Great, right? Except...wouldn't the `/proc/<pid>/cmdline` for the inner `echo` contain:

```
echo -n my-secret-ca-password
```

Thankfully, in Bash, the answer is no. We are saved by the fact that `echo` is a builtin, and a process will never be created. So, if you are able to otherwise secure your environment variables, this approach is safe. The downside is that it _appears_ unsafe, because `$STEP_CA_PASSWORD` is still getting substituted into something that certainly looks like it's a command.

## What About A Secrets Manager?

Secrets managers can be great because they can make it easier to get secrets closer to where they are used.
For example, a Docker container can call out to a secrets manager for its secrets.
But, a secrets manager is an extra dependency.
Often you need to run a secrets manager server and hit an API.
And even with a secrets manager, you may still need Bash to shuttle the secret into your target application.
For this post I'm focused on more lightweight solutions.

Speaking of lightweight solutions,
there is an keyring facility in the Linux kernel.
The Linux keyring offers several scopes for storing keys safely in memory that will never be swapped to disk.
A process or even a single thread can have its own keyring,
or you can have a keyring that is inherited across all processes in a user's session.
To manage the keyrings and keys,
use the [`keyctl`](https://man7.org/linux/man-pages/man1/keyctl.1.html) command or [`keyctl`](https://man7.org/linux/man-pages/man2/keyctl.2.html) system calls.

## Directly in the command

In case it isn't already abundantly clear, this is very unsafe. There is no way for the caller of a command to choose to hide the command line from being world readable. So, any CLI command worth its salt should not accept passwords directly.

You could _maybe_ pass a password directly in a command if you were running inside a secure container with sandboxing around everything. But, why take the risk?

Now, even for this one there's a caveat. Have you ever run `mysql` this way?

```
$ mysql --user carl --password amazingpw db.smallstep.com
```

Or `curl` this way?

```
$ curl -u carl:password https://example.com:3000
```

These commands accept passwords against their own better judgement, for convenience. But, immediately upon startup, they will overwrite `argv` with a blank value, effectively hiding the secret. If you run `ps` during the `curl` command shown here, you'll see:

```
curl -u               https://example.com:3000
```

Now technically, if a system is overloaded enough, it could be possible to grab the secret from `/proc/<pid>/cmdline` before `curl` has a chance to overwrite it. But, this approach is better than nothing at all. And these passwords can easily end up in audit logs or shell history. So, better to avoid this approach entirely.
The alternative for `curl` is a credential file: A [`.netrc` file](https://everything.curl.dev/usingcurl/netrc) can be used to store credentials for servers you need to connect to.
And for `mysql`, you can create option files: a `.my.cnf` or an obfuscated `.mylogin.cnf` will be read on startup and can contain your passwords.