It’s been a while since I’ve blogged. Hit a dry spell, I guess. Will try to post more frequently and about some technical issues as well. Anyway, I’m at the RSA conference in San Francisco for the entire week. It’s been a great conference so far with interesting keynotes and sessions. Also, a lot of evening receptions that basically give you an excuse to drink beer and wine
I visited the PCI reception on Monday evening which was a big success with many interesting conversations. Spoke with many security managers from large organizations about PCI. It turns out that 99% of the people I’ve talked with are either in the midst of a PCI audit or have just undergone one. Interestingly, when asked about database security, most of the security managers I’ve talked with are saying that this is the next thing for them to invest in.
On Tuesday evening, I went to the SC magazine awards gala. My company (Sentrigo) was nominated for “Rookie security company of the year” which is very important to me and shows the security industry’s recognition of the importance of database security. And the best part of the evening was that we actually won!!! It was amazing being called to the stage and later interviewed for the magazine. I felt a bit like at the Oscars… Sorry about the poor image quality…
The only problem with the conference so far is that I actually don’t have enough time to go to all the sessions and keynotes I would like to go to. Too many meetings, I guess…
Next week, I’ll be presenting at Collaborate08 in Denver under the auspices of IOUG - if you’re around come and see me on Monday, or catch me later at our booth (#1826) in the IOUG section.
In the summary of her post she raises an important issue - that most DBAs are reactive rather than proactive when it comes to monitoring their databases. I’ll take this even further… it’s not just DBAs (and I’m not going to get into the whole issue of who owns database activity monitoring…) but companies in general are too reactive when it comes to database security.
Yes, I know that security doesn’t generate revenues, it doesn’t even reduce costs - at least not in any consistent, measurable way. Security is all about reducing risk and the cost associated with that risk. The problem is that by being reactive, companies are addressing yesterday’s risks, not today’s or tomorrow’s risks. There are a several biases in how the IT security budget is allocated, and one of the biggest biases is the visibility bias: The tendency to invest in protecting against visible threats, even if they are small. Spam is a good example. It doesn’t do much harm, but it’s visible every day in everyone’s inbox (I’m not talking about malware, just the “classic” commercial spam which is 99.9% of spam). Companies are investing more today in reducing the marginal spam to the n-th degree, with diminishing returns, than they are in database security. Far more.
Risk, on the other hand, is not just a question of visibility or sheer quantity. It’s also a question of the potential damage of even a single attack, and the probability of such an attack succeeding. The risk posed by inadequate database security is currently greater than the risk posed by spam, given the counter-measures already in place for the latter. Yet many enterprises, by force of habit and inertia, continue to invest in protecting against threats for which they already have good enough solutions, whereas other areas remain barren. Some companies do, of course, shift their attention to new threats every once in a while, but I wonder how many enterprises do an annual (or more frequent) risk assessment that includes a “green field” threat analysis and gap analysis?
I’ll be presenting on Oracle database hacking and security at the UKOUG DBMS Special Interest Group meeting this week. The meeting will take place on Thursday, 20th March 2008 in Baylis House, Slough (UK, obviously). Here’s the link for the agenda and details http://www.ukoug.org/calendar/show_event.jsp?id=3358
Hope to see some of you there - come and say hello…
Just a short announcement this time - Sentrigo is hosting a live webinar/webcast with Pete Finnigan where he’ll share his wisdom on Oracle database security, show some attack vectors and how one can detect and prevent them, as well as other good stuff.
Those of you who’ve ever attended one of Pete’s masterclasses at an OUG or security conference know that they are well worth attending, and those of you who haven’t - you’re now given the chance to attend from the comfort of your own computer…
It takes place on Friday, March 28th. You need to register in advance - here.
Totally unrelated to database security but I’ve read this interesting bit on /. while flying to the US. It got me thinking - how does China prevent people from going to restricted sites like blogger.com? Do Chinese ISPs use some form of IP filtering? Do they parse HTTP and prevent proxies? How about HTTPS? and SSH? I’m thinking that if SSH (or HTTPS for that matter) is allowed, one can easily do port forwarding to surf whatever site he wants as long as he has SSH access to a server outside China. It’s as east as setting local hosts file to point blogger.com to 127.0.0.1 and running ssh -L80:blogger.com:80 user@host.outside.china and then pointing your browser to blogger.com. It can’t be that simple, can it?
In a recent survey we conducted, it turned out the DBAs are mostly ignoring security patches. Two thirds of the DBAs have never applied a CPU and only about 10% of them are applying CPUs in a timely fashion. After releasing the survey, we had some interesting responses in online publications and blogs which I would like to address in this post.
Response number 1 - Lies, damned lies, and statistics
The survey is made up, we asked the wrong questions, we did not understand the answers and my favorite - the survey contacted only lazy DBAs between at home between 1PM and 3PM.
Well, let me set the record straight here - the rolling survey was conducted in face to face OUG meetings - 14 different ones across the US, anywhere from 10 people in a room to 45 people. We did not select who was invited or who attended. These were OUG members who chose to attend our sessions. If anything, I would say these were people rather more interested in database security than the average.
We asked two questions -
1) Have you installed the latest Oracle CPU? (actually, since some meetings where right after the CPU we’ve asked about the latest 2 CPUs).
2) Have you ever installed an Oracle CPU?
You can find the actual answers in the survey itself. I should add that the results were quite similar across the different OUG meetings.
I can relate to this disbelief in the results. It sounds amazing that the most important assets of the organization are left un-patched especially after so many publicized incidents. But those in the know already knewthis.
Response number 2 - DBAs are just lazy
Well, as a former DBA for many years I definitely do not insinuate that DBAs are lazy. The simple fact is that they just have too much working against them when trying to apply the CPUs:
1. The need to test all applications using the database is a heavy burden
2. Oracle supports only the latest patchsets
3. The lack of application vendor certification of the CPUs
4. The simple fact that it takes a huge amount of work to manually shutdown the database and apply the patch in an organization running hundreds if not thousands of instances
5. For production critical databases you have to consider maintenance windows which might come once a year
6. The lack of understanding by some IT security personnel of the severity of the problem simply does not generate enough pressure in the organization - please see Rich Mogull’s excellent post on this topic.
All in all, I know of companies that analyze and deploy CPUs as soon as three months after release but those companies are very few and usually have budgets in the millions for such things…
Response number 3 - Yeah, we all knew that is the case
I know, I knew it as well being a DBA and all. But I always thought that DBAs at least deploy the patches after testing them. We are talking about severe vulnerabilities. Some of them are remotely exploitable without credentials.
A final remark - as always in security - use common sense:
1) Install only what you use, don’t install features you are not going to use and remove them if installed by default - many vulnerabilities are in rarely used components like Oracle Spatial, etc.
2) Use the least privilege principle - give the minimum permissions required for the task - every permission can be used to attack the database (create view, create procedure, etc.). Many packages can be used for an attack. Lock them down.
3) Check for default and weak passwords - there are many tools out there. Check after every patch as there were cases it restored default accounts.
4) Secure the network - use firewalls, valid node checking, etc.
5) Use secure coding - bind variables, bind variables, bind variables.
There are many white papers out there talking about Oracle security…
Mike Rothman (of Security Incite) has a new series of podcasts over on eBizQ (where my VP marketing was interviewed a while back on the same topic). In the latest podcast, the 2nd in the series, Mike interviews Rich Mogull on the topic of database security.
If you didn’t “get it” until now, or if you have a hard time explaining database security to your colleagues/boss - just send them the link.
At one point in the interview, Rich mentions the availability of some free tools, so this is the time to remind everyone that my company, Sentrigo, provides a free, downloadable version of Hedgehog, our database activity monitoring solution.
Oracle OpenWorld came and went. I had some interesting sessions which I’ll summarize shortly, some less interesting sessions, lots of beer and a great concert by Billy Joel and Lenny Kravitz. I arrived in SF on Friday night from Philadelphia (after being selected again at the airport for “random” inspection). I had several interesting meetings with customers in Philadelphia so I was very much exhausted and went straight to sleep. Saturday, I met with friends and registered to the conference.
Sunday started with a nice security session from Oracle’s Chad Hughes and two other guys. The interesting part in the session was a sneak peak into Oracle’s internal secure coding standards. It looks like Oracle is running Fortify on their code for code analysis. I believe they are also running it on their PL/SQL packages to catch un-validated user input passed to dynamic statements. In fact, I heard that one of the reasons for DBMS_ASSERT.NOOP was to remove false positive alerts coming from Fortify. Some other interesting stuff was related to Oracle’s “Secure by default” initiative. Things like auditing turned on by default, smarter password profiles that will lock users, etc. are interesting indeed but on the other hand, I got the distinct feeling that Oracle is talking the talk but not walking the walk, so to speak. If you check the attack surface in Oracle 11g, you can easily see that the number of public packages has increased tremendously and you have APEX installed by default?! The rest of the talk was dedicated to ISO-17799 (later renumbered as ISO-27002). All about control, asset management, etc.
Later that night we had Mr. Larry Ellison starring in the Sunday Night Live show. It was very interesting hearing about Oracle’s first days. I heard some other keynotes from Larry Ellison but none was so nostalgic and so informal. Ah, the early days of a young company - living in the office, surviving on pizza and coke. Reminds me of myself this past year.
Sunday we also had the nostalgic party which was fun…
Monday was filled with many announcements and coverage of new Oracle 11g features. Among them we heard about Oracle VM, which caught VMWare by surprise. Another highlighted though badly named feature was RAT (Real Application Testing) - a truly interesting feature. As I always tell my customers, you must test it before deploying in production This feature makes testing so much easier.
The best session of the day was from Tom Kyte of asktom in the no slide zone. It was pure entertainment. Tom hosted a contest between DBA 1.0 with scripts and command line against DBA 2.0 with the new Oracle tools like EM, ADDM, AWR and some more TLAs in “real live” scenarios. As you might expect, DBA 2.0 won the contest while showing the effectiveness and ease of use of the tools. Although, because of the slow WIFI, DBA 2.0 actually almost lost in the first scenario. It was hilarious. I must admit that although traditionally I’m more of a command line type of guy, the presentation was very convincing.
Monday night was OTN night and of course lots of beer and other liquids.
On Tuesday, I had more security sessions but I managed to squeeze in another Tom Kyte session where he counted his 11 best features of 11g. Again, very nicely delivered.
Another interesting session was the “Oracle CPU best practices” session. CPU stands for Critical Patch Update and Oracle has a predictable process to deliver them to customers. I really feel Oracle’s pain here. The process has to be predictable and ordered but this means that vulnerabilities like this one are published without a patch being available for 3 months. Also, from my experience, many customers find the CPUs too hard to follow and either skip them entirely (and rely on patch sets) or install every other one. Here are 10 interesting random facts about the CPU:
1. They are mostly tested on common platforms. Tests are hardly performed on non-common ones.
2. Pre-release information is available on last Thursday before CPU
3. The CPU is released on Tuesday
4. There are 3 types of patches - security, security dependent and customer conflicts in patches. From 10.2.0.3, the conflicts patches are removed.
5. Sometimes, patches are released without information because they are not available on all platforms
6. October 07 contained 82 combinations (5 supported versions ported to 12 platforms).
7. Testing is done in a 6 week cycle
8. 75% of bugs are found by Oracle internally (1% - open source, 10% - customers, 15% - researchers)
9. Oracle prioritizes by severity - source of discovery, availability of exploit code, CVSS score, etc.
10. Next dates - 01/15, 04/15, 07/15, 10/15
The funniest question from the audience was “the rate of vulnerabilities is not declining. When is Oracle going to fix all problems?”. And the truthful answer was “never”
On Tuesday night, Oracle Israel invited all the Israeli guys to Beni Hanna (a Japanese restaurant) where again, we ate and drank mojitos and sake.
The best part of Wednesday was the Billy Joel concert. Oracle organized the entire event superbly and besides entertainment we had plenty of food and drinks.
Of course, the day after, we saw a lot of tired Oracle attendees.
That’s it… Another year of OOW came and went… As always, it was a great experience and besides beer provided many interesting insights into Oracle’s current features and future plans.
Well, finally I’m writing the third part of the blog. The thing that pushed me to finish this was a talk I had with Tim Hall of Oracle-base fame after his Unconference presentation in Oracle OpenWorld. Tim told me that his Java developers are claiming that adding user context information in an already existing application (Swing) is a non trivial task. You know, I’ve been hearing this from a lot of our customers and while I agree it is not trivial, I will try to outline a method of doing so without changing application code. In this day and age when there are advanced tools such as AspectJ and Spring framework, adding cross-cutting concerns to an application should not be an insurmountable task.
So, without further ad0, I will detail an AspectJ aspect that will wrap around an Oracle connection and add user context information to every statement. This aspect can be used with existing programs and also adapted and extended to catch login information in a Swing based application. I will build of the previous examples in providing the necessary infrastructure of domain and DAO classes.
The rumors about my death have been greatly exaggerated, to paraphrase Mark Twain. I guess I’m a burst-blogger, at least for as long I’m also the CTO of a growing start-up.
The credit card companies started to make good on their threats and levy hefty fines like this one issued against TJX and its banks. This makes the pain of non-compliance very real, and I think we are going to see more of it as the credit card companies demonstrate that they mean business. This is one of the benefits of having an industry-regulated standard as opposed to laws and regulations - the incentives to enforce are business incentives, so they work…
A-propos, another recent development around PCI, which I think has not been receiving the attention that it should, is the passing of the first state law to augment PCI DSS the standard. Minnesota, the state that passed this law, is home to some of America’s largest retailers, such as Target and Best Buy, so on its own this law may have far reaching impact. Moreover, similar to California Senate Bill 1386 that deals with privacy breach notification and spawned copycat laws in some 38 other states, I expect the Minnesota law to be the harbinger of additional state laws (Texas, Massachusetts and Illinois are contemplating it), although in California it was shot down by the governator.
It may seem redundant to enact laws where an industry standard is already working well, but I understand the lawmakers’ perspective. You can’t just leave everything to market forces. Yes, right now it seems PCI is on the right track to provide protection for consumers. But this may not necessarily be the case in the future. Call it short term overkill, long-term insurance.
In the meantime, the retailers are trying to play “pass the hot potato” with the credit card issuers. While I agree that less data storage is less potential for data theft, there are accounting issues and simple business streamlining issues that need to be addressed. Guess what? The retailers’ gambit is not going to work. PCI DSS is not reversible, it’s only going forward. Credit card companies provide a valuable service to both consumers and retailers, and in this game, they have the power. Don’t like the requirements VISA is imposing? You have a choice - either comply, or don’t accept VISA anymore (and good luck with that…!), or outsource CC processing entirely.
The reality is that PCI is going to become part of the cost of doing business. It’s several years too late, but better late than never.
