Media Coverages
Today's Keys to Patching: Standardization and Automation
By Matthew Zito
August 31, 2007 -- Database patching is the new challenge for database administrators (DBAs) in the 21st century. In today's world of identity theft, Sarbanes-Oxley and security disclosure requirements, DBAs are expected to be more security aware than ever, and patching is at the forefront of this charge. In the old days, DBAs patched for two reasons - stability issues or vendor recommendations. Security patches were generally considered "baked in" when the database vendor issued a major patch revision, and the result was that they came infrequently at best.
One of the critical challenges today is that DBAs are not security people, and security folk are not DBAs, which forces both groups to try to communicate very complex ideas and concepts in a neutral fashion both sides can understand. For example, databases have very elegant (read: complex) role-based privilege systems. These afford a ton of flexibility to the DBA in creating very specific AAA schemes (accounting, authorization and authentication). Modeling these AAA schemes in a standard security model can be difficult. These problems extend to patching as well, especially because many companies now have a strategy of trying to get security patches to databases as quickly as possible to keep their auditors and security teams happy.
The database vendors aren't helping either. Over the past few years, the drive on the database side has been to add features above all else - part of an increasingly competitive database market. With all of these features, the security side of things got left behind, until a slew of Microsoft and Oracle bugs terrified security teams, and the patch race was on.
Today, database vendors are releasing security patches on a quarterly basis. This most recent quarter, Oracle fixed 51 security bugs, several of which were so severe that a potential hacker didn't really need much database know-how to take advantage of them. The challenge now is to get the patches on the right databases as fast as possible, while still making sure they don't cause issues.
It's a complex problem - arranging to change control windows and scheduling downtime are complicated and time-consuming tasks, plus there's always the risk that a security patch will break something within the database. There are a number of steps in the traditional patching process - education, patch testing, deployment and quality assurance (QA).
Education is researching the patch to find out what it fixes, how to install it, etc. This is typically done by each DBA in turn as they get ready to patch "their" databases. Patch testing consists of performing some basic installations of the patch to look for gross errors, and deployment means actually deploying the patch to a database. Lastly, there's QA, often the longest step overall because each database needs to have QA individually. So how can we shorten and simplify this process?
When there's a pile of databases to patch and not enough time, the answer is "standardization and automation." Standardization is a long-term goal - it will help get all of your databases in a consistent state, from versions to enabled features to installed patches. When the databases are standardized, you can more accurately predict how a patch will behave, especially once it has been successfully deployed on a few similar databases. Standardization also will help save on QA time, remove the need for patch testing, and overall, make the database easier to manage. Automation, on the other hand, will help get patches on databases in a fast, reliable and repeatable fashion - removing the need for all of the DBAs to be trained on the specific process and procedure for each patch.
Standardization, when combined with automation, also helps on an ongoing basis - once the quarterly patch has been deployed, updating your standard build to add the new patch makes it easy to keep brand-new databases stable and secure. The point of patch automation is not only to get the patches on databases faster, but to more accurately deploy the patch over and over again without worrying about the DBA making a mistake.
So how can you automate your patch deployments today? Well, there are companies that make products to help automate things like patching and provisioning. But even without a product to help, step one is setting down ground rules for how patches are applied and tested. Step two is standardizing your database environments - try to get down to a few specific builds, configurations and versions to minimize the complexity around patching. Step three involves having someone script out the patching process. While it may seem silly to write a script to apply a patch that will be deployed once on all the databases, this will help significantly. The last step is automating the QA process. While it's still critical to have application-specific QA, there's a variety of tests that can be done automatically to identify critical failures. Some examples of these include: checking for new invalid objects, creating a new table, dropping a test schema, adding users, etc. Making these QA procedures an automated part of the patch deployment process will help catch errors before they become a big problem.
Database patching is here to stay. There are always going to be security holes, and it's unlikely that security teams and auditors will ease up anytime soon. The key is to make sure that database security patching is an integrated process, not a fire drill or disaster. It's also not acceptable to simply ignore it by pretending that there's no need to put security patches on in your environment. The best strategy is simply good management - standardization, automation and testing. This will keep the environment consistent, the DBAs more productive, and the hackers off bothering someone else.
Matthew Zito is an expert in large-scale infrastructures, with extensive experience in designing and deploying some of the world's largest system, network, and application architectures. Before joining GridApp Systems as Chief Scientist, Zito worked at information storage company EMC and at Register.com, where he pioneered the use of centralized management and automation to run large production application infrastructures.