Posted by Dana Epp
What is Configuration Drift And How Can It Affect Your Azure Cloud Environment?
So your team is ready. You have your security policies in place, and they’re a work of art. Permissions are perfected, every unneeded port is closed off, automated provisioning of patterns is keeping VMs secure whenever they’re spun up, and your admin passwords are under lock and key. Awesome.
Too bad it won’t last.
Configuration drift is one of the most pernicious security problems that impact cloud configuration management. No matter how great your cloud configurations are to begin with, over time they will change and become weakened. Keeping systems secure requires being aware of configuration drift, and having policies in place to mitigate it.
What is configuration drift?
Simply put, configuration drift refers to any situation where changes are made to a software environment over time – particularly in regards to the security environment. Users are added and changed. Software updates necessitate new permissions. New hardware is deployed, and may not have exactly the same configuration tools.
Each change made will chip away at the initially acceptable configuration. Sometimes these changes are benign, but all too often a well-meaning but unwise or poorly-implemented change will open up a hole in the security which wasn’t there before. With each such shift, a once-hardened system becomes a bit weaker, and more vulnerable to intrusion.
However, what really tends to make configuration drift hard to deal with is that it can be extremely difficult to identify, or backtrack to a root cause.
The dangers of ad-hoc modifications
Probably the most common cause of configuration drift is adhoc changes being made by different elements of an organization, without central security oversight or proper communication between teams. It can be all too easy for a single programmer to, for example, open up a new port to enable functionality in a web app, without realizing that port isn’t approved for use within the security policy. A lack of documentation of this change can make it very difficult for a security team to figure out why the port is open, or what it’s doing.
They may even try to close it, only to see tech support flooded with calls about malfunctioning software.
Often, tight deadlines can be a major contributing factor here. If programmers, admins, or other individuals feel pressured to implement changes ASAP, change management processes are often one of the first things to be ignored. Poor lines of communication can compound this issue. In worst case scenarios, systems security becomes a patchwork, with the development team, CloudOps team and security team at odds.
Dealing with configuration drift in your own computing environments.
As mentioned, some level of configuration drift is effectively inevitable. However, it can be contained and controlled. The drift does not have to be negative. Further, it’s entirely possible to implement internal policies that minimize adhoc changes and the issues they can cause. For example:
1 – Maintain a Configuration Management Database (CMDB)
For mid-sized operations, a CMDB can be an effective way to keep track of system security matters. This is a single central database that maintains listings for all configuration items with their security settings and dependency relationships. An up-to-date CMDB makes it easier to understand how changes to one configuration item can affect security elsewhere, and track the downstream impact.
On the downside, CMDBs become increasingly difficult to maintain as the number of IT assets grows.
2 – Have clear lines of communication between your CloudOps team, security team and other teams
Don’t allow your developers and other ‘hands on’ workers to ever be in a situation where they feel like it’s too difficult to consult with the security team, or too time-consuming to fit their schedules. That is a situation which virtually guarantees harmful configuration drift. Look for ways to streamline interactions, without compromising security.
3 – Minimize the number of “snowflake” systems
A snowflake system is one which requires hand configuration, and cannot be configured solely through automated processes. While sometimes unavoidable, such systems can easily be the source of harmful configuration drift since it can be easy to make tweaks which aren’t fully communicated. Have policies in place encouraging proper communication whenever dealing with snowflakes.
4 – Audit, audit, audit
Configuration drift is guaranteed to slip past you without routine auditing of systems. Make security audits a regular feature, at least on a quarterly basis. Even merely auditing a randomized sample of IT assets is far better than nothing.
5 – Utilize automation to regularly audit systems
Where there is a problem in business computing, someone is looking into ways to automate it. Increasingly robust tools are being introduced to market which can monitor all your systems for even minor security policy infractions in near-realtime. This is frequently among the best ways to mitigate configuration drift, and catch those hard-to-find changes before any major security holes are created.
How To Manage Configuration Drift In Your Azure Cloud
One of the biggest long-term security threats to face your Azure cloud environment is configuration drift. As we discussed have discussed here, configuration drift is any situation where a cloud environment begins to drift away from its optimal configuration due to ongoing usage, tweaking, and other changes made by team members in their day-to-day work. Not such a great thing for environments that want to follow healthy cloud configuration management practices.
Over time, this drift can easily open holes in your security which were not there in the first place.
Configuration drift in Azure can be mitigated by good communication and documentation policies, but it really cannot be fully eliminated. That’s why we talk of managing configuration drift. Like a long-term illness, it won’t entirely go away. However, there are still plenty of strategies you can deploy to minimize its effects and ensure you have sufficient warning if new risks unexpectedly emerge due to changing cloud configurations.
Five Workable Strategies For Managing And Mitigating Azure Configuration Drift
1 – Become accustomed to Azure Resource Manager (ARM)
Azure Resource Manager (ARM) is one of the most important assets you have in managing your Azure cloud environment. ARM allows you to easily create deployment templates, which allows you to also setup your approved configuration set as parameters, which is then applied across all your Azure configuration items as needed. It can be applied to a single resource, or to a resource group of related apps and solutions.
The major benefit to ARM templates is their ability to be applied and re-applied consistently as needed, without being particularly susceptible to configuration drift – particularly because you can track and maintain change as code. Infrastructure-as-Code (IaC) is a powerful way for ClouOps personnel to gain the benefits developers have had for years in tracking and monitoring changes through things like source control.
IaC via ARM also allows you to avoid over-reliance on custom adhoc scripting, which is a powerful tool but potentially too powerful and too easy to misuse. In short, ARM is a good choice for ensuring all your cloud resources have the same starting point in terms of it's configuration.
Be sure to have policies in place tightly regulating who has access to your ARM templates, and how any changes to them are checked in and documented. And when deployed, ideally, only a pre-authorized Service Principal has access. You can learn more about this in our Ultimate Guide to Cloud Configuration Management.
2 – Leveraging Desired State Configuration (DSC) automation
One of the most powerful tools that comes with Azure is Desired State Configuration (DSC), an automated system for creating system configurations and propagating them across all relevant cloud resources with PowerShell. This means DSC is also one of your best options for reducing the amount of drift occurring in your configuration when using automation.
Additionally, Azure can utilize DSC across cloud-based VMs, on-premises VMs and physical machines (including both Linux and Windows boxes) and can even be linked to AWS if you also utilize Amazon services. DSC can either be a one-and-done process, or it can continually check for configuration drift and re-apply the original desired settings whenever it detects change.
Of course, the latter option should be used with discretion. If an application or other project requires changes to the configuration settings, DSC will need to be updated accordingly. Managing constantly-evolving cloud resources and DSC automation will require some oversight.
3 – Monitor your Activity Log
Your Azure Activity Log has a wealth of information about changes made to the system in the last 90 days, and it’s a good idea to keep an eye on it. You don’t want to rely entirely on “eyeballing” the log to prevent configuration drift, but if you know what resources shouldn’t have changes, you can easily spot problem areas if you scan the log once a day.
For longer term projects, we suggest storing a secondary copy of your logs for archival purposes. Since the Activity Log only shows the last 90 days by default, archiving it once a month can help pin down changes which flew under the radar.
4 – Improve communication and CloudOps team accessibility
In our experience, one of the biggest sources of configuration drift are team members who are facing time pressure and feel they are unable to fully comply with change management processes while also meeting desired deadlines. Trying to balance speed with security is always a tricky balancing act, but in practical terms, security measures will ultimately end up being ignored if team members feel they’re getting in the way of productivity.
Work with your CloudOps and security teams, and your other team members, to develop procedures for contacting appropriate personnel and having changes approved in a manner that is as streamlined as possible. Consider also building a little leeway into your deadlines, so that nobody feels security needs to be sacrificed for speed.
5 – Deploy third-party automated change detection systems
In the end, it can be difficult to fully manage configuration drift using only Microsoft’s own Azure tools. Currently, there is still a lot of manual work and monitoring needed, even when utilizing tool sets like DSC. Fortunately, third-party applications are picking up that slack, and can fill in the gaps in Microsoft’s own offerings.
Automated change detection systems can watch over all your cloud assets, alert you to changes shortly after they’re made that are drifting from the acceptable baseline, and keep detailed logs of all activity. If needed, they can also be leveraged to revert changes which are particularly disruptive. This can provide a best-of-all-worlds level of oversight of your cloud security, minimizing the need for micromanagement while still providing sufficient information to backtrack any problems.
AuditWolf Can help with managing your configuration drift in Azure
AuditWolf can be your partner in managing Azure configuration drift. Our powerful platform gives you full oversight over your cloud assets, helping you see the big picture of how everything links together, while also providing details on individual security threats.
Topics: Cloud Configuration Management