Dotenv-vault: The New Way to Manage .env

Sharing .env values manually through copy-pasting seems to still be the most common approach in the software industry. However, I thought we've always been boasting about our technological prowess, but why are we stuck with this old-school tradition?

In this article, I will be sharing a somewhat opinionated tool, dotenv-vault that modernizes the way we handle .env files!

As a bonus, I'll also show you how it can simplify your docker deployment process.

What's Wrong with Manual Sharing?

While sharing .env through chats doesn't necessarily lead to disaster, but it does come with some unpleasant effects.

PS: I will not consider security here. (not that it's insignificant, but I doubt that multimillion project owner still doing it)

Messy

Here are some scenarios

1. You asked someone for a secret, they sent you through DM's or Telegram.
2. Someone updated a feature to include a new secret, they sent them to a group
3. Your team leader thought the secret might better live in the documentation
   PS: Only some
4. During onboarding, the secrets were sent to you through an email
5. etc.

So where can you find secret A? Telegram? Slack? Direct Message? Group Message? Which Group?

The secrets are scattered everywhere!

Of course, it's not the end of the day, you can always ask another developer for the secrets.

Just that we developers don't like to ask people sometimes.

Error-prone

What if you accidentally deleted a secret from staging/production while editing the file? Or what if you edited the wrong variable? What if you deleted a character while scrolling down?

Maybe it wasn't noticeable right away as .env are cached, but the production might go down during the next deployment!

As long as edits happen in terminals, it's prone to errors, especially when juniors or interns who aren't familiar with bash get access to it. (I believe the readers are not one of these)

PS: From one of my experiences, we update the production/staging .env using ssh, and that's my assumption here.

Again, these two reasons aren't huge enough issues, but they make us less happy at times. Also, it's always a good reason to progress forward, isn't it?

In the worst-case scenario, it may even lead to huge mistakes, such as deleting important env secrets while updating production .env, and breaking the production.

Env Vault

Introducing .env secrets manager, dotenv-vault.

In short, it's like Git, but for .env files.

It's pretty intuitive for most programmers out there, with operations involving only pull, push, and build most of the time.

Benefits

Clearly, when it comes to benefits, it must solve the aforementioned issues. Let's see how dotenv-vault does it

Clean

Even with just a quick peek at the screenshot above, you can already see how the secrets will be kept centralized.

The best part about this is that if you're building with microservices or having multiple projects, then these secrets can be accessed by all of them, without repetition! Say bye to copy pasting values across multiple projects.

Every update will be made directly to the vault, so everyone can access all updated values easily. Beyond that, it also offers different environments to separate the secrets of our environments, so it does cater to most of our needs.

Safety

This is not about security, but corresponds to error prevention.

Remember how I commented on the insecurity of editing file from the terminal? What if you can only edit one variable at one time? While carelessness can still lead to mistakes, the possibility of errors will be minimized now.

Simplicity

How many commands do we need to remember? 2, for most cases.

npx dotenv-vault pull [development/staging/production] # to pull in new changes
npx dotenv-vault push # to push changes from local

Security

I'm no security expert here, but according to the docs, they do look pretty secure! All decrypted values exist only in memory and are flushed after using it.

Others

Unfortunately, the paid version is required to access more features of dotenv-vault. Nonetheless, $5 per month per user seems rather affordable for companies out there.

You probably don't need dotenv-vault for your personal projects anyways, who are you sending your secrets to?

Here are some cool features I find on paid version:

1. Version history, similar to Git, allows you to rollback and examine changes at the individual level
2. Slack notification, who wouldn't want to be notified when someone changes the secrets?
3. User Access Controls, it's probably best to just let the DevOps guys manage these keys. Or perhaps the seniors.

Of course, these are some nice additions, dotenv-vault is totally amazing in its free version anyways.

What about Config Server?

If you come from a Spring Boot background, you might find this surprisingly similar to config server! Nevertheless, config servers aren't the convention for JavaScript & Python projects, so yeah.

There are more differences, but I'd stop here since it's not the purpose of this article. You can read more about it here.

Project Setup

To demo the usage of dotenv-vault, I'll be using a Next.js project. More specifically, I'll be using my next-power-starter template, to get started, simply run

yarn create next-app dotenv-hello-world -e https://github.com/HohShenYien/next-power-starter

Not self-promotion, just simply because it's an empty project on my computer that I can use directly.

Usage

1. Sign Up
   Well, I guess you know how to do that. Here's the link.

2. Create a new vault
   The vault is required to link the project with dotenv-vault server. This step is required for new projects.

   npx dotenv-vault new
   

   

   After pressing y, your browser will launch into a page like this

   

   After clicking next, you will notice that a new file, .env.vault is added to your project!

   

   The value there will point to the remote project on dotenv. (doesn't it sound like Git?)
   This DOTENV_VAULT is not a secret, it's just like the github project link, nothing special about it. Therefore it's okay to be shown.

3. Login from terminal
   The next step will require us to prove that we're a member of the project before we can access the secrets.

   

   npx dotenv-vault login
   

   

   You'll either be logged in already like below, or you'll need to login in dotenv.

   

   Going back to the project, you should see a new file, .env.me. This file is like the auth token that proves that you can access the project. Therefore, if you share the file, then others can also access the secrets without having to login again.

   

   PS: This file is like the top secret, don't share it!

4. Add some secrets
   Next, let's head back to your dashboard. You should see your project being listed there.

   Clicking into the project, you should see something like

   

   Since we have already setup, just click the click here.

   Next, we can add a secret, say NEXT_PUBLIC_APP_NAME and MY_NAME.

   PS: The prefix NEXT_PUBLIC... means that it can be used in the client, while other environmental variables can only be accessed in the server. Read more about it in the docs.

   

   It should look like this in the end.

   

   Note that the environment is development

5. Pull from local
   Now that we've set it up in dotenv-vault, we can now download it to our local project. Let's run

   npx dotenv-vault pull development # depends on the environment you want to pull
   

   and voila, the image is now in your project!

   

6. Push from local
   Maybe you'd prefer to update the .env and publish it on dotenv? Let's do it

   

   I've added a new variable, MY_EMAIL and modified MY_NAME to my full name. Let's push it

   npx dotenv-vault push development # depends on the environment you want to push to
   

   Checking it on the dotenv-vault, you should see the new keys appearing there

   

Checking the environmental variable in project

Let's use the variables from the Next.js project. Open up the page, app/page.tsx

Yes, this project uses the app directory, you can read more about it here.

// src/app/page.tsx
...
export default function Home() {
  // I can do this because this is a server component
  const myName = process.env.MY_NAME;
  ...
              <p className="text-slate-600">
                The template which balances flexibility and simplicity
              </p>
              <p className="font-sm text-slate-600">By {myName}</p>
  ...

Spin up the development server,

yarn dev

You can now find the variable shown up (The By Hoh Shen Yien)

Bonus: Deployment with Docker

Finally, let's look at how we can deploy the project with docker.

Production Environment

Let's first push our .env to production,

1. .env.production
   Let's create a new .env.production file which will be pushed to production

   cp .env .env.production
   

   Before we push, we will also need to edit the first line, which indicates the environment

   # production@v1
   NEXT_PUBLIC_APP_NAME="dotenv-hello-world"
   MY_NAME="Hoh Shen Yien"
   MY_EMAIL="hohshenyien@gmail.com"
   

2. Pushing to dotenv

   npx dotenv-vault push production
   

You should see the secrets appearing on remote dotenv, in the production environment.

Setting up Docker

You can install the Docker here if you haven't already.

Let's add a Dockerfile, here's the starting file from Nextjs

Copy the entire content, and paste it into /Dockerfile.

Finally, also update /next.config.js to use the "standalone" output type

/** @type {import('next').NextConfig} */
const nextConfig = {
  output: 'standalone',
}

module.exports = nextConfig

Setting up Next.js

To deploy on Docker, we'll need to add a new dependency,

yarn add dotenv-vault-core

You'll need this line before you can access the environmental variables.

It will download the .env to the server during execution

require('dotenv-vault-core').config()

Since there's technically no index.js file that will be called on every page (if just page, then it can lie in app/layout.tsx, but they won't be available in API routes) & API.

Hence, I'll create a env.ts file that contains all environmental variables

// src/utils/configs/env.ts
require('dotenv-vault-core').config()

export const MY_NAME = process.env.MY_NAME

And to access these variables, just import them! Simple and straightforward.

Retrieving Production Key

Finally, we will need to pass the production vault key for dotenv to download the .env file. To retrieve the key, you can run

npx dotenv-vault keys production

You'll get the production key in the next line. Copy it, we'll use it to run the docker image.

PS: the key should look like dotenv://key_..../.env.vault?environment=production

Building Docker Image

Finally, just build the docker image as usual

docker build -t docker-next .

and to test it locally on localhost:8080 (Because 3000 is occupied by the running Next.js project earlier)

docker run -e DOTENV_KEY="your_key_here" --rm -it -p 8080:3000 --init docker-next

And that's it. You have a perfectly working Docker image.

You can check out their documentation for more details.

Conclusion

Migrating to dotenv-vault is relatively simple, and its benefits are quite significant.

I used it once and truly enjoyed it, so perhaps you should try it too!

Disclaimer: I am not sure how will dotenv and Docker work in a serverless environment (Vercel), maybe it'll retrieve it for every different server?