Counting Vulnerabilities
Posted by Gunter Ollmann on May 29, 2007 at 4:41 PM EDT.

It would seem to me that, on a daily basis, I get asked way too often “how many vulnerabilities are there in popular software?”

If you have read the 2006 Trend Statistics report – you will have observed that X-Force tracked, analyzed and researched 7,247 public vulnerability disclosures last year.  If you’re also following the X-Force Threat Insight Monthly report, you’ll see that there’s already been over 2,500 new vulnerability disclosures in 2007.  But what do we actually mean with “public vulnerability disclosures”?

“Public vulnerability disclosures” encompass all software vulnerabilities that have had details of their existence discussed and disclosed on the Internet and can be referenced (typically through a URL).  For example, the April 2007 vulnerability in Universal Plug and Play referred to in http://www.microsoft.com/technet/security/Bulletin/MS07-019.mspx .  0-day vulnerabilities and other under-ground vulnerabilities are also included a public disclosures because, once they are uncovered, an advisory is issued informing customers of their existence – hence, they too become disclosed.

Does that mean that there were only 7,247 vulnerabilities in 2006?  The unfortunate answer is a resounding No - not by a long shot.

So, what vulnerabilities manage to escape these kinds of statistics?

1. Vulnerabilities that have been disclosed to the vendor and are currently undergoing remediation, but will not be publicly disclosed until a suitable fix can be made available to their customers.  This delay in public disclosure is typically part of a responsible disclosure strategy.  For example, take a look at X-Force’s responsible disclosure guidelines.
2. Vulnerabilities that were discovered internally by the vendor and silently patched.  For example, the vendor may conduct a biannual security code review and find several new flaws.  They develop fixes for these vulnerabilities and do not need to “credit” their discovery to an external researcher.
3. Vulnerabilities that have been “purchased” by organizations and have been released under NDA to that organizations customers or used stealthily in malware or other weaponization contracts.  Typically these types of vulnerabilities remain out of the public arena until either the organization extracts all the value they can, or the vulnerability is uncovered by an another researcher (either independently, or through observation of the vulnerabilities use in an attack) and is then responsibly disclosed.
4. Vulnerabilities that have been discovered under contract (e.g. penetration testing of a new application deployment) and consequently belong to the organization that’s paying the bill (it’s then up to that organization to decide whether they wish to work with the software vendor to fix it).
5. Vulnerabilities discovered by dedicated researchers and deemed “too lame” and consequently never disclosed to the vendor for fixing.  Depending upon the experience of the researcher and their focus upon peer-recognition, anything less than remote-root against a default Microsoft service may fall into this “lame” bucket.
6. As much as I hate to say it; vulnerabilities that affect software specific to a non-English speaking region which cannot be understood or verified by the analysts reviewing vulnerability disclosures.

From my own experience, I would say that the 4th category – vulnerabilities discovered under contract – is probably the biggest catchall for undisclosed vulnerabilities (by way of volume).  For example, when assessing applications (whether they be web-based applications, competitive reviews of compiled business applications, custom deployment of mainstream applications, or even in-house developed software) I would guesstimate an average consulting penetration-tester/researcher would uncover about 5-10 new vulnerabilities per day.  If you were to include a typical web-application (non-financial – because financial web applications are ‘usually’ more secure than other categories), then the same consultant could uncover 40+ new vulnerabilities in a single day – mainly due to lots of pages with submission forms suffering from the same types/classes of programming flaws (e.g. cross-site scripting, SQL injection, etc.).

The bigger the application, the more vulnerabilities will be discovered.  For example, in some of the larger engagements I’ve worked on in the past, a four man team working for five days have uncovered over 600 vulnerabilities in a single commercial application. 

How many new vulnerabilities are discovered annually?

So, I guess the question is how many new vulnerabilities are discovered annually each year – including those that are not publicly disclosed?

Any answer is going to be a bit of a guess – but I’m willing to have a controversial stab at it.

Lets focus on six categories from above, and see what number of non-public vulnerabilities we come up with…

1. Vulnerabilities being fixed – If I look at all those security vendor sites that indicate “upcoming” vulnerability disclosures, I can see about 20 noteworthy vulnerabilities at any point in time.  If I examine past experiences in how long it takes to get vulnerabilities fixed by vendors and publicly disclosed – I’d say that, at the end of a year, between 2-5 percent of any particular years vulnerability total is probably still being being worked on by the vendors.  So, conservatively, we’re looking at about 165 additional vulnerabilities per annum.
2. Internally discovered vulnerabilities – This ones pretty tricky.  Any vendor worth their salt has a decent QA team responsible for uncovering bugs.  But when does a bug become a security vulnerability?  Probably when someone working for the vendor puts on a security hat and says “that’s a security issue” and demands that it is fixed sooner rather than later.  For major software vendors, I’d guess that internally discovered (and silently fixed) security vulnerabilities probably account for 80-90%.   Therefore, lets say public disclosures only make up one in five of the vulnerabilities actually fixed by the major vendors.  Referring to the 2006 statistics for the top 10 vulnerable vendors, they publicly disclosed 964 vulnerabilities (14% of the total for the year).  Therefore, we’re probably looking at a further 4,800 vulnerabilities per annum.
3. Purchased vulnerabilities – Not too many legitimate companies undertake this kind of business.  For a vulnerability to be of value to their customers, it’ll have to be “sexy” – and there aren’t really too many of those – let alone ones that can stay out of the public eye for that long.  So, at a guess, I’d limit this to maybe a couple of hundred per annum.
4. Contract discovery – This is that BIG category I was talking about before - so let’s throw some numbers at it.  I guess that there must be around 5,000 dedicated security consultants and researchers around the world that regularly contract for penetration testing and vulnerability discovery, and let’s assume that they are gamefully employed for 150 working days per annum (the rest of the time it’s a mix of travel, report writing, conferences, pre-sales, holidays, weekends, etc.).  If we assume the lower number of 5 vulnerabilities per working day, we get to an amazing 3,750,000 vulnerabilities per annum (yes, this includes cross-site scripting, SQL injection, buffer overflows, authentication bypasses, etc.).  Now, since it’s quite likely that a lot of these applications are covered multiple times by multiple consultants throughout the year (for example, a custom SAP Enterprise deployment), let’s say that 90% of these vulnerabilities are also repeat finds each year – and that the average life of the products is three years.  Therefore we arrive at around 125,000 newly discovered vulnerabilities per annum.
5. "Too Lame" - The number of dedicated vulnerability researchers and bug hunters that would "throw out" a vulnerability as being lame is pretty small.  New and junior researchers are constantly looking for vulnerabilities in an effort to establish their credentials - consequently they are unlikely to not follow a vulnerability through to public disclosure.  Senior and experienced researchers are more likely to not pursue Medium and Low impact vulnerabilities - however, they often pass these vulnerabilities on to junior members of their research teams for disclosure.  Let's assume that there are about 200 senior [dedicated] bug hunters (5+ years commercial experience) out there and choose not to pursue 2-20 vulnerability discoveries per year all the way through to disclosure because they are deemed "lame".  This would result in about 1,000 additional vulnerabilities per annum. 
   Since commercial researchers often focus on the same business sofwtare for analysis, the same "lame" discovery may be made by multiple researchers, so lets drop this number in half to 500.
6. “Foreign” vulnerabilities – Again, not a particularly easy one to grasp… but I’m going to guess (and I do mean guess) that the English-centric research groups are probably missing about 20% of global public disclosures – which would equate to around 1,450 last year.  Luckily, I think that just about every “major” vulnerability will eventually be disclosed in English (regardless of the language it was originally disclosed in), and the vulnerabilities being missed in any global statistics are more likely to be Low and Medium impact in nature.

Summing it all up, then we’re probably looking at around 132,115 non-public vulnerabilities for last year – making a grand total of 139,362 new vulnerability discoveries (give or take quite a few).

A Grand Total

To be sure, 139,262 new vulnerabilities in a single year is a colossal number – but is it wrong?  Too many people underestimate the number of vulnerabilities in the software they use at home and in the enterprise office.  Public vulnerability disclosures provide only a small window in to the total number of vulnerabilities uncovered on an annual basis.

Since no analysis is complete with a suitable graph – using the numbers from above we observe that the contracted discovery of new vulnerabilities dwarf all other non-public vulnerability rates and even public disclosures.

 
What does this mean to businesses trying to protect themselves?  The key points to take home are:

• If you’re basing your protection strategy upon keeping up solely with public vulnerability disclosures, you’re missing almost 95% of the vulnerabilities actually out there (this year).
• Just because a vendors software patch isn’t marked as a “Critical Security Update” doesn’t mean that vulnerabilities aren’t being fixed with it.  Silently patching an internally discovered vulnerability appears to be increasingly common.
• If your defense systems are designed to protect against specific vulnerabilities (i.e. signature-based) – it probably means that it was designed to protect a subset of publicly disclosed vulnerabilities.  Preemptive protection engines are needed for the remaining 97% of annual vulnerabilities.
• Security vendors that focus solely upon the vulnerabilities they discover (or purchase) themselves are missing the point.  Even if they uniquely uncover 100 new vulnerabilities in a year, that still equates to less than 0.1% of the annual total. For a vendor to provide the best protection, they have to analyze and research every new vulnerability and attack methodology.
• If in doubt, call in the professionals.  At the end of the day, the vulnerabilities you most need to be concerned about are the ones that exist and will affect your own environment.  Professional security consultants can point out all the publicly disclosed vulnerabilities that affect your organization – as well as those undisclosed vulnerabilities that are unique to your organization.