Posted by Mike Racine
Azure Key Vault Basics : How to keep ‘secrets’ secret
Is there such a thing as “too many secrets”? Very much so! In this episode of #KnowOps, Dana shows how you can use Key Vault to not only keep secrets but safely use them in all sorts of different places.
- Too many secrets.
- Dana Epp here. Welcome to the channel that helps aspiring Azure administrators, like you and me, to know ops, and well, master the Microsoft Cloud. I'm glad to have you here. If you haven't yet, smash the subscribe button, so that you can be notified when I release new videos each week.
It's hard to believe it's been almost 30 years since the movie Sneakers came out. It's a great movie showcasing the world of secrets and computers, and it's actually one of my favorite movies. And here we are, decades later, still having to worry about too many secrets. We use secrets in pretty much everything, because they're just so inexpensive and easy to use, and well, abuse. Which is why I wanna show you how to keep your cloud secrets, secret. And that's done through Azure Key Vault.
Key Vault lets you manage secrets, keys, and certificates in a centralized cloud service that can be backed by a tamper resistant hardware security module, or HSM. It removes the need for in-house expertise on how to properly protect secrets and remove the complexity to allow people and different cloud resources to safely use them. Developers no longer have to worry about storing secret information in their applications, like connection strings and encryption keys, and IT staff don't have to worry about credential cruff being left lying around. There's no need to write custom code to protect or retrieve secrets because Key Vault can be hooked into pretty much everything in Azure these days. Best way to demonstrate this would be to jump into the Azure portal and show you how to set up Key Vault and make it work with a few live Azure resources. Let's get right to it.
Okay, to get started, I've already set up a resource group with a bunch of resources, just so they're all prepped up. So I've created a Log Analytics workspace and app service called KV Demo for Key Vault Demo, and I've also created a Logic app, so we'll be able to use these. None of these have been set up or configured with anything, but I just wanted to have those pre-created before we walk through how to build a Key Vault and then set it up to work with some of these services. So to get started, we're just going to add a new Key Vault. And I'll call mine TooManySecrets, why not? And we'll stick this in Canada Central. And I'm gonna review and create. Now, by default, when you first create a Key Vault, whoever is creating a Key Vault will automatically have permission 'cause they're the owner. And that's important to know because one interesting thing about Key Vault is that the typical IAM that you think about which relates to your access through RBAC is only about the management of Key Vault, in in itself doesn't give you the permissions to actually access the values in Key Vault, the secrets and whatnot. So that's important to understand. Because when you go into the Key Vault you'll see the standard IAM section which is what's used for making the role based access control decisions on who can manage the Key Vault. But the actually ability to access the secrets and the keys and the certificates, that's actually done over in Access Policies.
So as you can see here, I'm already in here as a user because I created this Key Vault. If we wanted other users to have access we would have to come back over through here. And we'll use that in a little bit here as we start trying to get applications talking into the Key Vault. To get started though, let's go create our first secret. Let's call it MyAppSecret. And in our case here, let's give it a password. And hit create. And this is the first area of where the secret is. Now, how do you access a secret? Well, if you click back into it after it's been created, you'll see here, current version. If you click into that, you get a secret identifier. That secret identifier is a dedicated URI to access it. In itself it's not a secret. And you can share that and put that into places. But it's good to know where it is and you can copy the clipboard and have access to it. If I ever wanna look at the value, I can just click Show Secret Value, and there it is, my bad password. But, let's go back in and work with it.
Let's say that we wanted to make a change to this. You don't actually go and change the value. What you ultimately do is just create a new version. And what you'll notice is when it's created it has a new identifier. You'll also notice, in this case, both of them are there. So this allows you to have multiple versions, or and they each have its own unique URI, that allows you to, when configuring services, if you haven't had an opportunity to go and rotate the URIs, you can still run the old one and make sure that you can update all the other services before you disable it and make it no longer available. But that's where I just wanted to get started so that we understand that you can have a secret and have multiple versions of it at any one time. When you wanna access it though, you always can go to it by using this false. If I copy this clipboard, you can see here it's going to the vault, toomanysecrets.vault.azure.net/secrets/MyAppSecret/, and then this unique version number. If I ever wanna access just the direct, the latest version, I can always remove the version number, and that would get me to whatever the current version is. So that way when you're in certain applications, if you don't want to always be hard coding the URI, you can always just put the base and that would always go and fetch the latest version.
Okay. So now that we have that, let's get started and see how we can work with it. In our case here, I'm just gonna jump over here and go and open up that Logic app that I created. So I haven't actually done anything with it, so the first time it executes it's gonna want me to use the designer, and so in my case here, I'm just going to create a very simple Logic app here. And the purpose isn't to build a fancy Logic app, it's just to show how Logic apps can talk to Key Vault to get access to secrets.
So in our case here, there's a couple ways that we can do it. The first way is to use connectors that are built right in. So if I was to add a new step, and search for Key Vault, you'll see I have an option here under Key Vault to get a secret. Now, here's an interesting problem. I can try to use this and get it to connect up, but it won't be able to work because this Logic app has not been given permission to access that Key Vault we just created. However, if we go back in to the Key Vault itself, I go into the Access Policy, I can add an access policy here which would be to allow someone, in this case, the Logic app, to have get access, and in this case I also wanna give it list access, which will allow the Logic app to list any secrets that may exist in this Key Vault. And for the principle, I want to be able to go and list the application. The problem is at this particular moment is that that application is never been configured or set up, so we have a slight problem, and that is that we can't see it in Azure Active Directory yet.
However, if we were to take a step back, and go under Identity, we can create a manager identity here by simply turning a system assigned managed identity. With the managed identity turned on inside of any kind of application, Azure Active Directory will manage how that application will have access. This is sometimes called a service principle, but it's a special one, because it's a managed identity that's through there. So by having that now configured, if I go back here and select a principal, and type in here of my Logic app demo, mykvdemo. There it is there, this is the application that I have for that Logic app.
If I now select that and add it, and remember to hit save, we're adding an access policy that is allowing the Key Vault to have access here to be able to access that secret. With that in play now, if I go back into that designer, and use the connector, I can have it get a secret. And the first time it does this, it's gonna ask me what vault do I wanna access? And that's because it's gonna create a connection to the system. In this case, I'm actually gonna sign in with my normal account so that I can just run it here. And in this particular case, it's gonna off me. And now I can access that secret. And let's just use this as a starting point, let's just see how this works. If I now save this, and then run it. This is a manual run, so it's not really gonna have an HTTP trigger, it's just triggering it on its own so that we can see how this actually runs. And what we'll see here under get secret is I can look at the output and we can see there's my new password, No more secrets Marty.
Now that in itself can be a problem because we really don't want to be able to have passwords showing up or secrets showing up inside of our Logic workflow 'cause this actually shows up in the logs. So when you are running this you'll actually be able to see it. So one way that you can correct that when using something like a Logic app in this scenario is that you can actually configure to use secure output. So we go back into the designer and go into the settings. There is an option here to secure output. And now with that in play, if we save that, and run it again. If we go look at that value in the output, you'll see that it won't allow you to see it because of the security configuration.
So that's the first way that through Logic apps you can connect up because we have that all in play. It gives you the flexibility to pull in that information which you could then use in other steps in the Logic app to directly connect up to it. And that's a pretty powerful way to do it. Now, there may be times when you're not using a Logic app, maybe you wanna use this in C# code or you want to be able to do this in PowerShell or in a script of some sort.
Let's give an example of how you could do this using the rest end point directly. So I'm just going to delete this step for a second. And we're gonna add a new one. In this case, we're gonna do a standard HTTP connection. So now what we're gonna do is we're gonna use a get which is the equivalent of connecting up via the web and to pull down that info. So if we go back to those secrets we had and we go get that URI that I was talking about before, and copy that to clipboard, we can enter that URI here. Now, slight change. When we're accessing it in a restful way like this, a better way to do this is not to use the version, so we'll erase that, and we're gonna change it around to say that we wanna access the API version, at least 7.0. And now that we've set that up, it's gonna go straight to that URI that we've made available and it's gonna call a GET. And of course, this is going to fail if we don't have the right authentication in play.
So down over in our case here, there's an option here for authentication types, and we're gonna say we want to use the managed identity. So in this case, we've already bound the managed identity to this Logic app, so it will use that identity when calling into Key Vault. Now, to do that though, there is one little bit that has to change. We have to allow the managed identity to know what audience wants a token from Azure Active Directory. So if we go in and we say use the audience, we can then have a parameter here, which in our case is https://vault.azure.net. And what that's going to do is allow the system to say, I want to go as that managed identity called MyKVDemo as an application, I want a token that will be for the audience or for access to Key Vault. And then because that application has been granted through an access policy, they'll be able to fetch, in our case here, my app secret. So, if I've entered everything correctly here, and I hit save, I can now do a direct web connect to the Key Vault. And if we take a look at the results, you will see there is the value. And in this case, this is a JSON response with a bunch of extra information.
Now, very similar to like how I talked before, there is no secure output when you're directly connecting via an HTTPS connection like this, so this is probably not something you would actually wanna do on a Logic app, but if you were working in any kind of scripting language or anything else where you're not caching the inputs and outputs like you do with a Logic app, you would be able to use this to go an extract that information dynamically as you needed getting it here. And you can do that in lots of different ways. Here we're showing it being an HTTPS request just directly through there.
We could actually go back in the policies and it's possible to set this up, so even your arm templates, if you look at this checkbox here, if you enable this, then arm templates would be able to extract the secrets that you have in Key Vault and use them dynamically as it's deploying a new resources. So these are very useful to have throughout the system. So the nice thing I wanna show you is how we can do this with let's say a web application, very similar, so just like we had our Logic app here, we could very easily go into a normal web app and we'd be able to extract information, let's lay like connection strings or whatnot. In our case here, if we go into the configuration, you have your app settings. Typically, developers are gonna want you to put an app setting in here which would be that connection string to the Cosmos DB or that SQL Server, and those will have secrets that allows people to access that database directly. So you really don't want to be putting those values here. But if you have to, because this is the way that they've set up their application to work, what you can do is move those secrets into Key Vault and then fetch them dynamically using something called Key Vault References. Now, before I can do that, just similar to like the Logic app, I need to give this application permissions to talk to the Key Vault.
So again, we're gonna go back in here, we're gonna go into Identity and create a system managed identity. Turned that on. Hit save. Now that we have that managed identity set up, we wanna go back to the secrets and we're going to add a new access policy. And again, same thing, we wanna give it get and list privileges. And in this case now, this is the web apps demo. And we add that in. Now, to demonstrate how this is gonna work and how it's not gonna work, I'm gonna actually delete this first just to get started, so we don't have that on there. Showing how easy it is to add it, we can just as easily delete access for an application by just hitting delete. Now, let's go back to that demo and I'll show you how you'd be able to add the hook.
We say we want a new application setting. So let's say we wanted to call this setting, MySecret. It could be anything, it could be My DB Connection String or whatnot. What we would do is we'd use the @Microsoft.KeyVault. This is what a reference looks like. And then we have to set up the secret URI. Which, if you recall, back in the secrets. Go into the current version. I'm gonna copy the whole directed value here. And I have a version that's on here, and I'm gonna hit okay. And when I hit save, watch what happens. Now you'll see here that the source says Key vault Reference, but there is a red circle with an X through it. And that's because when it tried to go and validate it failed to access to it. Access to Key Vault was denied.
So again, if we go back in, go into the access policies, add that access policy. Give that web apps managed identity access. And now, what I'm gonna do here, just to cause this to do a proper save is I'm going to remove the version. But I'm gonna keep the slash there, that's very important. Without the slash you will always get a 404. But now, by making that change and hitting okay and making it re-fetch, when I hit save, you'll now see that there is a check mark that's green, which is telling us that the Key Vault reference has worked and now this web application is connected to Key Vault and will extract the latest or current version of a secret related to the one that we configured there. And that becomes quite powerful 'cause now you no longer have to have secrets in your app configs, you can reference them directly in Key Vault, and allows you to maintain those secrets and control access as you need to.
Now, one thing to think about when working with this type of stuff. This isn't the safest way to do it, because if the goal is that you're trying to prevent and protect secrets, understand that application settings and web apps, they end up becoming environment variables when they're deployed, and that means that anybody with permission to use the advance tools like Kudu, has the ability to be able to get that value anyways. So as an example, if you were to go into Develop Tools, Advanced Tools, and launch that, if we go into the Environment and look in the AppSettings, you can see the password, the secret, in clear text. So things like connection strings and access to SQL databases and everything will still be accessible if people have access to these advanced tools.
This is one of the reasons that I highly recommend that developers do not have access to production subscriptions, that they don't have access to the applications to be able to go in and see the production configuration and secrets. When they need to have access to resources, we make that and export that through application insights so that they have access to the data they need without giving them physical control over the environment because Kudu is gonna leak that information which is probably something you don't want. Another way to secure that would be to actually programatically access the Key Vault and use it just in time as a secure string and then destroy it, so it's never actually an environment variable, it's never an application setting, it's loaded in real time.
Of course there are limits with the fetching and everything else, but that's something you and your developmental group need to talk about to see what's the best way to protect your secrets. But, the good news is you can keep it all in Key Vault. So now the last thing I wanna talk about is one of the things that kind of irks me a little bit about Key Vault, and that is the fact that by default it doesn't log anything for you. You wanna be able to track and manage access to these events so you know what's going on. To do that, you actually have to enable diagnostics.
So if you go in to diagnostic settings, there's an option here, and I already have one configured here called KVDiags. And I'm gonna create a new one here 'cause I wanna send this to a new log analytics and that's gonna be KVDemoDiags. And I'm gonna send this to Log Analytics. And I just happen to have had one preconfigured here so you can already see it here. And I wanna set AuditEvent. By doing that, that will send all of the audit events related to people fetching secrets, people querying Key Vault, to allow you to track everything. By default, this isn't turned on, and I highly recommend that A, you set up a Log Analytics workspace, and B, that you set up the Diagnostics to write audit events to that Log Analytics workspace.
That way, you have audit ability and tracking of who and how people are accessing Key Vault. More importantly, when you go and start tracking for things such as 400 errors and 401 errors and 403 errors, you can actually start seeing people that are trying to access your secrets who shouldn't be.
- Too many secrets.
- Okay, so I might not have a little black box with a super decryption chip in it, but I do hope you can see that in a world with too many secrets, you can manage them quite well in the cloud with Azure Key Vault. What do you think? Was this helpful? Let me know by hitting the like button, and if you haven't yet, smash the subscribe button so that you can be notified as I publish new content each week. Until then, thanks for watching. We'll see you in the next episode.
Is there such a thing as “too many secrets”? Very much so! Keep your secrets in Azure secret using Key Vault. #knowops @auditwolf