It's a Secret! Shhhh.....

Home > Blog > It's a Secret! Shhhh.....

Whether you’re a Developer, IT Infrastructure engineer or a Database administrator, there are times where secrets (or even passwords / keys) become a nightmare to manage in a world where cloud and on-premises applications and workloads meet.

secrets management gif

I remember the days when some of us (yes, I’m guilty of it) relied on excel spreadsheets, or worse post-it notes to quickly write the root password of the server that was just provisioned and might have to whisper that around for our application engineers to install some piece of software on that server (luckily the noise in a datacenter usually drowned my voice out).

Good practice back then was to ensure the spreadsheet was regularly updated. If any changes were made (such as the root password), “when and why” was a column to fill in the excel diary, which was then shared on some “private” storage device. If you lost that post-it notes early on or someone had changed the password without an entry in the long-shared diary, that was the day you had to fill out that TPS report.

You may say “well, I have a password manager and it works perfectly” and to be honest, for personal or small teams use, it still works - securely protects credentials and other various secrets. So, “Check”.

Now think through this for a moment: You have 10 colleagues working on various projects and located in different cities. 10’s or 100’s of applications (or instances, or servers) running 24x7 in your IT infrastructure, multiple secrets that needs to be shared for all colleagues in your company and other sensitive information that needs to be processed by those applications.

You now need to start leveraging cloud infrastructure tomorrow as you need to scale to meet customer demands for the “tomorrow”. So far, this does not look too bad as long as you know and trust all your colleagues and direct reports that they will do the right thing.

Maintaining an online diary instead of your old and legacy excel diary is a huge step up, and as long as you trust your colleagues and know they won’t abuse them in a malicious way (send them on some security training), is still perhaps good enough.

Fast forward to circa 2020 and then comes the era where you now need to trust “the tools”; it needs to be trusted by your colleagues and especially you. When automation is all in harmony, you’re relying on the tools to do the work for you, and as the word suggests: “autonomously”.

That means you need to effectively trust an application or some tool that hosts all of the secrets will be retrieved / used only when necessary and that you know when it’s been accessed or was used the last time by the appropriate application (i.e., only the application that has access should have access). You’ll also want to know if those secrets are stored securely as well (hopefully not in plaintext).

So if you’ve read this far and you’re thinking - well, my excel diary is fairly up-to-date and I trust Maurice who happens to live just across the street with the lock password (and we’re buddies at work), then this is not for you.

secrets management

But if you’ve been wondering, how do you centrally manage and ensure secrets are securely “shared” or injected into you infrastructure (and perhaps now your CI/CD infrastructure), then read on.

I’m a fan of DigitalOcean, for personal use. I still rely on the “big clouds” (A, M and G) for my day-to-day work life, but I do rely on DigitalOcean for my personal work.

I rely on various CI/CD framework to help me test and validate my own code work (and by no means a full-time developer), even though I tend to use “test1”, “test2”, “test3” as my variables in the application (I heard this is really bad practice!). It was a simple decision back circa 2014 when they offered free credits of $20 for the first month, so I took advantage of it and never looked back.

Part of my CI/CD framework relies on containers as an entire “instance” seemed too expensive to not only maintain, but also to keep it up and running / shutting down. As a responsible husband on my personal credit card, it was too much to management outside of personal life.

So, relying on a much simple, dynamic and elastic way to pull the right image with all of the necessary tools preinstalled was a must for me. Kubernetes made a lot of sense for me and ever since I’ve also moved across to a managed kubernetes environment on DigitalOcean.

tokenization to protect sensitive data application

When DOKS initially launched, my number one concern was more on “secrets” as I need to pass many variables / credentials to the pods running in the cluster to automatically pull or fetch my test codes in github. I tend to trust my cloud vendors when it comes to resilient services but trusting my cloud vendor from storing my API keys / passwords didn’t quite sit right for me.

So, I studied up on what a “MutatingAdmissionWebhook” was and found that there was a way to mutate the pod to add a “sidecar” to my pod to securely expose the secrets to my pod by storing them securely (encrypted, OfCourse) in the sidecar container - and that would solve my challenge, Check!

As an engineer, it was up to me to try and test this theory out! But writing Go wasn’t my strength and did I mention that I am a no-good developer?

Turns out Fortanix had already done this for their customers and also securely stores all secrets in an HSM (Hardware Security Module) grade system to truly provide a multi-cloud secrets injection method securely (cause you never know when I might switch over from DigitalOcean to another cloud vendor) - Voila!

secrets management

They already had a sidecar container available and the admission controller container that would service the webhook in kubernetes when a pod is provisioned so that the pods can be mutated whenever the MutatingAdmissionWebjook API is called, all natively within kubernetes.

Best thing is that Fortanix Data Security Manager always stores secrets encrypted (Fortanix’s strength being that even the owner of the appliances can’t even see what’s inside the system given it’s use of Secure Enclaves), coupled with the fact that it was external to DigitalOcean’s ecosystem (or had to paste any plaintext value into the kubernetes native secrets manager), it was the choice for me. Now the rest sits on the access management of the Pods / Containers and the rest is history.

I’m quite happy with the setup I have right now - it works and it’s fairly secure in my opinion. Love to hear from others on what their experiences have been to date.

Checkout how I managed to get it working with various cloud vendors below: