Security vulnerabilities happen, it is just a fact, not only in technology but in life in general. When we are made aware of those vulnerabilities, we need to "fix" things or mitigate them to the best extent possible. In IT, that is typically synonymous with installing security patches or workarounds. I know that many of our customers have questions about how to best do patch management using the RightScale platform. This post is the first part in helping you accomplish that task and focuses on cases where a vendor patch is available. In my next blog post, I'll talk about best practices for applying workarounds or fixes when there is no vendor patch available.
Within the RightScale platform three primary options can be used to automate the patching of instances:
- Unfreeze Security Repositories and enable automatic updates on systems, hope that the updates don't break anything.
- Manually unfreeze Security Repositories for test systems and update. Perform regression testing, then update & refreeze Security Repositories for production systems and apply updates. Do this regularly (say monthly or weekly).
- Update each ServerTemplate with the latest Security Repository. Regression test each updated ServerTemplate. During a schedule maintenance period, force all servers to be relaunched with updated ServerTemplates.
Of course, there's also always the option to hide underneath a pile of coats and hope it all works out for the best. It goes without saying that while many people de-facto implement this last option, it is not a viable long-term strategy! :)
Let's dive into each of the options a bit more and look at some pros and cons, so you are in a better position to pick the one (or combination) that works best for you.
- Unfreeze with automatic updates: Since many (most?) of the core Linux distributions have functionality to allow selecting of security updates only, you freeze all channels, and then set the security repo to /latest via a RightScript. You then configure the system to install those updates on an interval you desire (daily seems to be a good choice). For example, on Debian based systems, such as Ubuntu, security patches are broken out into a separate repository. For a given release it is possible to only automatically install updates from http://security.ubuntu.com/ubuntu/ instead of http://us.archive.ubuntu.com/ubuntu/, making this very easy to implement. Just unfreeze that repository and updates will apply as they are released. With CentOS you can run "yum update --security" and only install security related patches. Using this method allows rapid access to the latest security updates, with almost no work required to enable this behavior as the unattended upgrade packages do all the work for you.
The downside is that if a broken package is released, say into Ubuntu-security, it could affect production. A side note is that as it relates to security patches, the industry at large has pretty much come to the acceptance that the risk of problems with automatically patching security vulnerabilities outweighs the potential risks with doing it. For example, Debian, Ubuntu and Windows 2003-2008 all ship this way. For those who determine that the risk of automatic patching is too great, there is ...
- Unfreeze in test, test it, update production: This option is to apply the security patches to instances in a test deployment, then after regression testing, deploy them to production using a RightScript to update repositories and perform an update on production servers. This has the advantage of some level of regression testing prior to deploying security patches in production. The downside is that there is a high manpower cost to perform the functional testing on a regular basis. There is also the fact that you should test the specific items that the security fix supposedly touched which involves a bunch of research. This is a non-trivial effort. It would likely require a special test environment dedicated to security testing. From a purely dogmatic standpoint, this is the way it should be done, but the pragmatist in me knows that for many organizations, the additional cost associated with this is not justified by the increase in risk posed by just installing security updates. I'd rather have patched systems, than people not doing it because it was not the absolute best way to go about it.
- Update ServerTemplates and relaunch: This may be the cleanest and seemingly easiest approach. There is relatively little change in current operations, as many of you use this method currently. This also ensures that all packages are tested before being deployed in production. The upside is that systems are cleanly built, and ServerTemplates are updated more often. The downside to this is that your patch level is only as good as your latest ServerTemplate update, and while it works for servers that can be frequently updated (app servers, web servers, etc.), it really doesn't work well for services that are infrequently updated, or difficult to relaunch (databases, load balancers, etc.). Further, it forces you to relaunch servers you wouldn't otherwise relaunch during maintenance windows.
So, you may be asking, "You use RightScale to manage RightScale, so how do you do it?" Well, at RightScale, we have chosen a hybrid approach of #1 and #2. Our default patching policy is "unfreeze with automatic updates." As stated earlier, there is some inherent risk in this stance, but we feel that getting critical fixes in outweighs the incremental risk of taking too long to get the patch deployed. In instances where the risk of any patch (security or not) breaking a system, we use the "Unfreeze in Test, test it, update production" patch policy. Further, we design our platform with mitigating controls to restrict access to systems and services that may not get the latest patches on a daily basis. This policy/stance works for us, and we think it is a reasonable one for others to start with (if you didn't already have a stance).
I would be remiss if I did not point out that there are likely a myriad of other ways that you can perform security patching, but that these are ones you get "out of the box" with the RightScale platform. The specific approach you choose will be driven by your business requirements. Remember that you have options, so use them to develop a process the works for you and your organization. My next blog will be on deploying workarounds and non packaged fixes. Until then, happy security "patching!"