In a recent survey we conducted, it turned out the DBAs are mostly ignoring security patches. Two thirds of the DBAs have never applied a CPU and only about 10% of them are applying CPUs in a timely fashion. After releasing the survey, we had some interesting responses in online publications and blogs which I would like to address in this post.
Response number 1 – Lies, damned lies, and statistics
The survey is made up, we asked the wrong questions, we did not understand the answers and my favorite – the survey contacted only lazy DBAs between at home between 1PM and 3PM.
Well, let me set the record straight here – the rolling survey was conducted in face to face OUG meetings – 14 different ones across the US, anywhere from 10 people in a room to 45 people. We did not select who was invited or who attended. These were OUG members who chose to attend our sessions. If anything, I would say these were people rather more interested in database security than the average.
We asked two questions -
1) Have you installed the latest Oracle CPU? (actually, since some meetings where right after the CPU we’ve asked about the latest 2 CPUs).
2) Have you ever installed an Oracle CPU?
You can find the actual answers in the survey itself. I should add that the results were quite similar across the different OUG meetings.
I can relate to this disbelief in the results. It sounds amazing that the most important assets of the organization are left un-patched especially after so many publicized incidents. But those in the know already knew this.
Response number 2 – DBAs are just lazy
Well, as a former DBA for many years I definitely do not insinuate that DBAs are lazy. The simple fact is that they just have too much working against them when trying to apply the CPUs:
1. The need to test all applications using the database is a heavy burden
2. Oracle supports only the latest patchsets
3. The lack of application vendor certification of the CPUs
4. The simple fact that it takes a huge amount of work to manually shutdown the database and apply the patch in an organization running hundreds if not thousands of instances
5. For production critical databases you have to consider maintenance windows which might come once a year
6. The lack of understanding by some IT security personnel of the severity of the problem simply does not generate enough pressure in the organization – please see Rich Mogull’s excellent post on this topic.
All in all, I know of companies that analyze and deploy CPUs as soon as three months after release but those companies are very few and usually have budgets in the millions for such things…
Response number 3 – Yeah, we all knew that is the case
I know, I knew it as well being a DBA and all. But I always thought that DBAs at least deploy the patches after testing them. We are talking about severe vulnerabilities. Some of them are remotely exploitable without credentials.
A final remark – as always in security – use common sense:
1) Install only what you use, don’t install features you are not going to use and remove them if installed by default – many vulnerabilities are in rarely used components like Oracle Spatial, etc.
2) Use the least privilege principle – give the minimum permissions required for the task – every permission can be used to attack the database (create view, create procedure, etc.). Many packages can be used for an attack. Lock them down.
3) Check for default and weak passwords – there are many tools out there. Check after every patch as there were cases it restored default accounts.
4) Secure the network – use firewalls, valid node checking, etc.
5) Use secure coding – bind variables, bind variables, bind variables.
There are many white papers out there talking about Oracle security…
Back after a short and much needed hiatus, I came across this piece by security analyst Eric Ogren on Computerworld’s website. He discusses how DBAs have become public enemy number one because of compliance mandates to exercise segregation of duties, and how this has been blown out of proportion to other, greater risks.
A few days pass, and the story about the Fidelity database breach comes to light (incidentally I chose this article from Computerworld as well). A senior DBA sold 2.3 million records, including bank account and credit card details, to a data broker.
So are DBAs “dangerous” or not?
Unfortunately, there is no denying the risk element. If risk is the arithmetical product of the probability of an incident happening and the potential damage that incident could cause, then due to the latter factor DBAs as well as other highly skilled insiders with access privileges pose a significant risk.
This does not mean, however, that there is a high probability of DBAs becoming malicious insiders. Obviously, the vast majority of DBAs pose no threat to their employers or clients, but the old adage of one rotten apple applies nonetheless. While there is a much higher probability that someone who is not a DBA would try and breach the database, the DBA is in a much better position to succeed should he or she really want to do that.
An external hacker would find it very difficult to achieve this kind of scale (millions of records) without insider cooperation. It is difficult to determine what direct damages this will bring to Fidelity and its customers, but the bad publicity is already quite significant: Running a news search on Google for fidelity data breach yielded 529 results at the time of writing.
Clearly, there is a problem here which cannot be ignored, but on the other hand, Eric’s conclusion was absolutely correct – DBAs are a part of the solution, and I would even stress that they are an essential part of the solution. The fact is, DBAs know more about database security than anyone else. They know more about database vulnerabilities, exploits and hacks, and more about how to address them than anyone else. Trying to implement a database security solution by circumventing or ignoring DBAs would be futile.
It is important, for security as much as for regulatory compliance reasons, to monitor and audit DBA activity. In fact, this should be done for all users who access the database. DBAs are first to understand this. If you work in a bank vault, you know there are CCTV cameras on you. You want those cameras on you. DBAs are in a similar situation and they understand this requirement completely.
What DBAs should not accept are two kinds of solutions that one sometimes comes across (sometimes it isn’t the tool but the implementation process):
- Solutions that hinder or interfere with the DBA’s daily tasks – DBAs are primarily concerned with running databases efficiently. Any solution that jeopardizes this primary objective is counter-productive, and doomed to fail anyway because DBAs and other staff will find ways to circumvent it.
- Solutions that ignore DBA input – As I suggested, DBAs are not as opposed to the notion of monitoring their own activities as some people think, so there is no real reason not to involve them. More importantly, I believe it is simply impossible to implement a solid database security solution without DBA cooperation. Any solution that ignores the specific data structures, user profiles, schemas and views simply cannot be doing a good job. Those are all managed by DBAs.
Finally, there is the question of priorities. Obviously my company sells database security monitoring products, so my view is not objective, but I’ll say this: Databases are still the most neglected parts of the enterprise IT infrastructure security-wise, especially when taking the magnitude of the threat into account. The Fidelity incident is just the latest in a long string of examples demonstrating this.