contributed articles What does vulnerability and patch data say? BY GUIDO SCHRYEN Is Open Source Security a Myth? DURING THE PAST few decades we became accustomed to acquiring software by procuring licenses for a proprietary, or binary-only, immaterial object. We regard software as a product we have to pay for, just as we would pay for material objects. However, in more recent years, this widely cultivated habit has begun to be accompanied by a software model characterized by software that comes with a compilable source code. This type of software is referred to by the term “open source software” (OSS). While there is consensus that opening up source code to the public increases the number of reviewers, the impact of open source on software security remains controversial. Propo nents of OSS stress the strength of the resulting review process19 and argue in the sense of Raymond20 that, “Given enough eyeballs, all bugs are shallow.” while some opponents follow the argu ment of Levy,11 who remarks “Sure, the source code is available. But is anyone reading it?” Interestingly, Ozment and Schechter18 report that in the Open- BSD source, foundational vulnerabili ties have a median lifetime of at least 2.6 years, which seems to refute Ray mond’s argument. While the security discussion is rife with beliefs and guesses, only a few quantitative models and some empirical studies appear in the litera- ture.1,9,10,13,17,21,24,25 Many of these em- pirical studies investigate one package or a few software packages only, and to my best knowledge, no prior study has been conducted to comprehensively study differences between open source and closed source security. The reason why comprehensive empirical studies have been neglected is probably due to the need for laborious collection and analysis of reliable data and the associ- ated manual work. This article presents a comprehen sive empirical investigation of pub lished vulnerabilities and patches of 17 widely deployed open source and closed source software packages. The empirical analysis uses comprehen sive vulnerability data contained in key insights ■ The security discussion of open source and closed source software is rife with beliefs and guesses. Data-driven insights based on an empirical analysis, as examined here, provide new insight into such security issues. ■ The analysis illustrates there is no empirical evidence that the particular type of software development is the primary driver of security. Rather, the particular policies of vendors determine the patching behavior. ■ It is most important to provide economic incentives for software producers to make software less vulnerable and to provide patches. doi:10.1145/1941487.1941516 130 communications of the acm | May 2011 | vol. 54 | no. 5 --- the NIST National Vulnerability Da tabase14 and a newly compiled data set of vulnerability patches. Based on these comprehensive data sets, this study is capable of providing empirical evidence that open source and closed source software development do not significantly differ in terms of vulner ability disclosure and vendors’ patch ing behavior, a phenomenon that has been widely assumed, but hardly in vestigated. Open and Closed Source Software Generally, the availability of source code to the public is a precondition for software being denoted as “open source software.” Beyond this require ment, the Open Source Initiative (OSI) has defined a set of compliance criteria for open source software.15 The (open source) definition (OSD) includes per- mission to modify the code and to re- distribute it. However, it does not gov- ern the software development process in terms of who is eligible to generate and to modify software. In this regard, two options are distinguished by Ray- mond:20 When what is called “bazaar style” is in place, any volunteer can provide source code submissions. In a more closed environment, software is crafted by individual wizards, and the development process is characterized by a relatively strong control of design and implementation. This style is re- ferred to as “cathedral style.” The OSI approved several licenses, including the Apache License, the BSD license, and the GNU General Public Li- cense (GPL), which is maintained by the Free Software Foundation (FSF). The FSF provides a definition of “‘free soft- ware’ [as] a matter of liberty, not price.”8 In contrast to the OSD definition, the FSF definition explicitly focuses on the option of releasing the improvements to the public. Software is usually regard ed as “closed” if the source code is not available to the public. The categorization of software and its development process as “open source software (development)” or “free software (development)” in contrast to “closed source software (development)” mirrors the perspective of developers and specifies the type of development. Complementarily, one could also adopt the software users’ point of view by distinguishing between software that is charged for and software that is free of charge. The resulting classification scheme is shown in Table 1. The Life Cycle of Vulnerabilities and Patches When software is executed in a way which is different from that which the original software designers intended, this misbehavior is rooted in software bugs. Anderson3 assumes the ratio between software bugs and software lines of code (SLOC) to be about 1:35. When bugs can be directly used by at- Table 1. Classification of software. Developers’ perspective Users’ perspective Open Source (license) Closed Source Free of charge Linux, Apache web server Adobe Acrobat Reader Subject to charge MySQL (dual licensing1) Microsoft Windows operating systems 1 “[D]istributors that…do not wish to distribute the source code for the commercially licensed software under version 2 of the GNU General Public License (the “GPL”) must enter into a commercial license agreement with Sun.“ (http://www.mysql.com/about/legal/licensing/oem/) 132 communications of the acm | May 2011 | vol. 54 | no. 5 --- tackers to gain access to a system or network, they are termed “vulnerabili ties” by the U.S. MITRE Corporation.12 Although there are other definitions of “vulnerabilities,”16,23 the adoption of the MITRE definition is useful in a pragmatic sense for three reasons: ˲ Most empirical studies implicitly use this definition by analyzing “Com- mon Vulnerability and Exposures (CVE)” entries, which are provided by MITRE. CVE identifiers are not only widely used by researchers, they are also used by information security product/ service vendors. Thereby, the CVE defi- nition has become a de facto standard. ˲ The process of accepting a po- tential software bug as a CVE vulner- ability is well documented, and the assessment is conducted by security experts.12 ˲ The U.S. National Institute of Stan dards and Technology (NIST) adopts the MITRE understanding of vulner abilities in its National Vulnerability Database (NVD), which is probably the largest database of security-critical software bugs and which provides comprehensive CVE vulnerability data feeds for automated processing. Vulnerabilities and their dynamic behavior can be described through the “vulnerability life cycle,” which is shown in Figure 1 as a UML statechart diagram. The diagram provides a pro- cess-oriented perspective on a single vulnerability and its patch (for the consideration of exploits see the study of Frei9), integrates states that have been introduced by Arbaugh et al.,5 and uses a cycle to account for the fact that patching vulnerabilities can even create new vulnerabilities.5 Injection: The life cycle starts with the injection of a vulnerability into the software. In principle, a vulner- ability can find its way into software through the intentional behavior of software developers, who strive to sell or exploit vulnerabilities, or to harm their employer, or unintentional be- havior, which can be rooted in careless programming or in using “insecure” development tools. After testing, the software is finally released, the public starts searching for vulnerabilities, and the software vendor potentially continues searching. Discovery and publication: The dis- covery of vulnerabilities can be based on coincidental detection or on the active search of persons with intrinsic or with extrinsic motivation. The emer- gence of vulnerability markets,22 in- cluding bug auctions, bug challenges, and vulnerability brokers,7 provides economic incentives (at least for “white hats”) to search for and to disclose in- formation on vulnerabilities. When a vulnerability is discovered, the question of whether it should be published or not occurs. If a “black hat” detects the vulnerability, his or her decision depends on whether s/he aims at making the vulnerability avail- able to as many other “black hats” as possible and to gain reputation, or to a closed group of potential attackers. If the vulnerability is detected by a “white hat,” including the software vendor, it is still not clear whether the vulner- ability should be published or not, as vulnerability information is useful for both the “good guys,” who can provide patches, and the “bad guys,” who prob- ably would not have gained knowledge of the vulnerability otherwise. Some researchers have addressed this ques- tion: Rescorla21 argues against dis- closure, as he finds the probability of vulnerability rediscovery to be vanish- ingly small. However, investigating the operating system OpenBSD, Ozment17 finds that vulnerabilities are correlated regarding their rediscovery, and argues in favor of disclosure. Patching: Once a vulnerability is published, at first glance it seems ob- vious that the vendor should provide a patch as soon as possible. But it can be economically reasonable for the vendor not to provide a patch when it is the customers who suffer the most cost of failure and when competitors behave likewise. If the vulnerability is not published (and detected by “white hats” other than the vendor), again, the question arises of whether the vendor should provide a patch or not. While the aforementioned economic argument still holds, the decision not to provide a patch may be rooted in the assumptions that a nonpublished vul- nerability is hardly exposed to attacks; any vulnerability disclosure reduces the vendor’s reputation; and the patch reveals the vulnerability to attackers, who then try to compile exploits and to use them to attack unpatched systems. When a vulnerability patch is avail- able, the search for newly injected vulnerabilities starts since it is known may 2011 | vol. 54 | no. 5 | communications of the acm 133 --- that patches can contain new vulnera bilities.6 As the injection refers to a new vulnerability, Figure 1 shows a dashed line. An overview of the classification of vulnerabilities is provided in Figure 2. The previous discussion of the life cycle stresses that the empirical securi ty of software goes beyond technologi cal phenomena and also depends on economic conditions. In the particu lar context of open source and closed source software, Anderson2 shows that although, under ideal conditions, open and closed systems are equally secure, this symmetry can be broken owing to economic phenomena. Investigated Software Packages In order to draw a picture of empiri cal open source and closed source software security, it seems alluring to consider as many software packages and vulnerability data as possible. But this (quantitative) approach suffers from at least two limitations. First, for many software packages only (too) few vulnerability data is available, as the packages are rarely deployed and prob- ably hardly attractive for attackers. Sec- ond, a comparison of open source and closed source software remains strong- ly biased, unless the software packages under consideration are comparable in terms of functionality. However, for many open source and closed source software packages, no functional coun- terparts are available. Due to these issues, I decided to follow a qualitative approach, and to manually select widely deployed soft- ware packages for the empirical analy sis. Assuming that most software is usually attacked through the Internet, I adopt the client-server perspective to frame the selection of software pack- ages (see Figure 3). On the client side, the most widely deployed operating systems (OS) are Microsoft OS, MAC OS X and Linux derivatives (http://market- share.hitslink.com). Among the Mi- crosoft OS, Windows 2000, Windows XP, and Windows Vista are the leading ones in terms of market share, but I excluded the latter due to its short his- tory (release date: January 30, 2007). Regarding Linux, I selected Red Hat Linux and Debian Linux, which are widely deployed Linux distributions. In addition to operating systems, I an alyze web browsers, email clients, and office software. Regarding web brows- ers, Internet Explorer and Firefox are the most widely used programs (http:// marketshare.hitslink.com), regard ing email clients and office software, I found no reliable statistics. I selected Outlook Express and Thunderbird, which are comparable in terms of functionality in contrast to Outlook, which integrates much more function- ality, and MS Office (Word, Excel, and Powerpoint) and OpenOffice. On the server side, I analyze Web servers and (relational) database man- agement systems (DBMS), which are widely used application types. Internet Information Services and Apache are the most frequently used Web servers (http://news.netcraft.com/archives/ web_server_survey.html). Oracle and DB2 are two of the mostly used closed source DBMS (http://www.gartner.com/ it/page.jsp?id=507466), while for open source DBMS no reliable data could bve found. Having explored database-relat- ed Web sites, I decided to use mySQL and PostgreSQL, which are widely de- ployed. The specific versions of the soft- ware packages are given in Table 2. Vulnerabilities Having decided to adopt the vulner ability definition of the MITRE CVE group (discussion earlier), the ques- tion remains of how to gather CVE data as the CVE group “…only contains the standard identifier number with status indicator, a brief description, and ref- erences to related vulnerability reports and advisories” (http://cve.mitre.org/ Table 2. Vulnerability data. Application Type Product Devel. Type Release date #vuln MTBVD [days] Development of vulnerability disclosure over time Browser Internet Explorer 7 Closed 2006-10-18 74 13.29 Linear Firefox 2 Open (BS) 2006-10-24 167 5.16 Linear Email client MS Outlook Express 6 Closed 2001-10-25 23 120.73 Linear Thunderbird 1 Open (CS) 2004-12-07 110 13.79 Not linear Web server IIS 5 Closed 2000-02-17 83 40.90 Not linear Apache2 Open (CS) 2000-03-10 80 40.63 Linear Office MS Office 2003 Closed 2003-11-17 99 19.22 Not linear OpenOffice2 Open (CS) 2005-10-20 19 63.16 Linear Operating system Windows 2000 Closed 2000-02-17 385 9.35 Linear Windows XP Closed 2001-10-25 297 8.97 Linear Mac OS X 10.x Closed2 2005-04-29 300 4.64 Linear Red Hat Enterprise Linux 4 Open (CS) 2005-02-14 2641 5.48 Not linear Debian 3.1 Open (BS) 2005-06-06 2071 6.45 Linear Database Management System mySQL 5 Open (BS) 2005-10-24 33 46.00 Linear PostgreSQL 8 Open (CS) 2005-01-19 25 58.96 Linear Oracle 10g Closed 2004-01-15 63 29.72 Not linear DB2 v8 Closed 2004-03-26 13 136.38 Linear BS: Bazaar style CS: Cathedral style 1 The NVD lists linux kernel vulnerabilities separately from vulnerabilities of specific Linux distributions. Red Hat Enterprise Linux (RHEL) 4 uses kernel 2.6.9, (http://www.linuxcompatible.org/Red_Hat_En- terprise_Linux_4_Nahant_Beta_2_Public_Availability_s36797.html), Debian 3.1 uses kernels 2.4.27 or 2.6.8 (http://www.debian.org/News/2005/20050606). I consider only those kernel vulnerabilities that were published after the release date of RHEL 4 and Debian 3.1, respectively. 2 Some open source components are included. 134 communications of the acm | May 2011 | vol. 54 | no. 5 --- about/faqs.html). I decided to use the NIST National Vulnerability Database (NVD) (http://nvd.nist.gov/)14, which provides full database functionality for the complete MITRE CVE diction ary. Information on MITRE CVE vul- nerabilities and the NIST NVD is pro- vided on the Web sites of MITRE12 and NIST,14 respectively. Vulnerability data. I discuss the quality of used data and the implica tions of how the MITRE CVE diction ary and the NIST NVD are built for the analysis of vulnerabilities. As the data sources of the CVE group are manifold and include trustworthy organizations, such as US-CERT and SecurityFocus, the CVE input can be assumed to be comprehensive, although it cannot be guaranteed that all disclosed vulner- abilities are considered. The analysis of potential vulnerabilities by the MI TRE CVE group assures that each CVE candidate has been inspected by secu- rity professionals. In cases where soft- ware vendors dispute vulnerabilities, I chose to use MITRE data in favor of an unbiased assessment. Overall, the CVE dictionary is a valuable resource for vulnerability analysis in terms of both quantity and quality. As the NVD acquisition procedure considers all CVE vulnerabilities in a timely manner and provides them in xml data feeds, the NVD is an appropriate database for the analysis of vulnerabilities in gen eral. However, while the NVD provides a comprehensive database of (CVE) vulnerabilities, the properties of some vulnerability attributes added by NVD analysts need some more attention: ˲ Original release date (ORD): Owing to two potential time gaps, the ORD as- signed to a CVE identifier by the NVD does not necessarily mirror the actual date of disclosure: Time between the actual disclosure of a vulnerability and its consideration in the “Assigned” phase of the MITRE CVE workflow. This gap is zero when the vulnerabil- ity has not been disclosed to the pub lic. Moreover, time between the “As- signed” date and the NVD publication date. As no information on the overall time gaps is available, the computation of patch times and exploit times would contain errors of unknown size. How ever, I assume the effect of the errors on the particular types of development of vulnerability disclosure over time (for example, linear or S-shape) to be less important, as this development is not affected by the full time gaps, but only by differences between the time gaps (standard deviation of time gaps). In addition, unusually large time gaps would be detectable in the graphics shown in Figure 4. ˲ Common Weakness Enumeration (CWE): The NVD analysis team assigns a type (for example, buffer overflow) to a vulnerability, based on a subset of the MITRE CWE structure. However, by Dec. 31, 2008, only about one fourth of all NVD CVE entries (9,748 out of 34,091) contained a CWE name so that an analysis of vulnerability types is not reasonable. ˲ Common Platform Enumeration (CPE): The NVD applies the structured naming scheme CPE, provided by MI TRE, to assign names of vulnerable product versions to CVE identifiers. ˲ Common Vulnerability Scoring System (CVSS): Vulnerabilities are scored by the NVD analysis team re garding their severity. CVSS 2.0 pro- vides for three scores, with each score value being between 0 and 10 (highest severity). The base score is an aggre- gation of six base score metrics. This score is mandatory, and specified by vulnerability bulletin analysts and soft- ware vendors. The NVD team works closely with the CVSS working group, MITRE, public vulnerability sites, ven dors and security researchers to come to a consensus on scoring some of the more commonly occurring vulnerabili ties. Other score types are the temporal score and the environmental score. For our analysis, only the base score is ap- plicable. It should be noted that the NVD scoring system changed over time: CVSS 2.0 scores for the CVE vulnerabil- ities published prior to 9/11/2005 were converted by the NVD team from prior CVSS metric data. The investigation of the NVD conversion script provided by the NVD Program Manager (C. John son) reveals that for all CVSS 2 charac teristics corresponding CVSS 1 charac teristics are available, and a “natural” conversion was conducted. To sum up, older scores that have been converted into CVSS 2 are comparable with “new” CVSS 2 scores. Based on the afore- mentioned analysis, I regard CVSS 2.0 scores to be useful for further analysis. The following analysis of NVD vul nerabilities is based on NVD xml data feeds as available on Jan. 31, 2009. All feeds were imported into MS Office Excel 2007 and processed using fil- ters and MS Query. In order to assure that vulnerabilities listed in the NVD data feeds have not been accidentally misattributed regarding the affected software version, I double-checked the affected software versions of each vul- nerability on the Web sites of vendors, MITRE, and SecurityFocus. In very few cases of inconsistencies, I excluded the particular vulnerability from any fur ther analysis. Mean time between vulnerability disclosures. Table 2 lists for each soft- ware package the number of published vulnerabilities and the “mean time between vulnerability disclosures” (MTBVD) defined—analogously to the software engineering term “mean time between failures”—as the average number of days since software release divided by the number of published vulnerabilities. With regard to deter- mining the MTBVD, I consider only those vulnerabilities that have been published after the release date. Vul- nerabilities that have been published earlier than the release date and that also affect the version under consid- eration are due to the development process of earlier versions. A simple comparison of the MTBVD of software packages does presumably not provide reliable results regarding the level of security because of two reasons: the se- verity of vulnerabilities should be con- sidered, for example by using vulner- ability weights, and the vulnerability detection and publication are probably correlated with market and software factors. For example, an important market factor is the attractiveness of the software for “vulnerability search- ers,” an important software factor is software size, as given by “software lines of code” (SLOC). However, reli able and precise figures are available for few software packages only, so that figures must not be used for a fair com- parison of security levels. On the other hand, MTBVD data is not useless as it provides an impression of how often vulnerabilities are published and how insecure especially operating systems are, regardless of the particular devel opment style. Overall, data suggests the vulnerability publication rate is may 2011 | vol. 54 | no. 5 | communications of the acm 135 --- probably determined by other factors than the software development style. Development of vulnerability disclo sure over time. While the previous dis- cussion provides a static picture of the history of vulnerabilities, I now address the development of vulnerabilities over time. In an earlier empirical investiga tion, Alhazmi et al.1 assume the devel- opment of vulnerability discovery can be described by an “S” shape. However, the analysis of vulnerability data shows that for 12 of 17 considered software packages a linear correlation between time and the number of published vul- nerabilities is found (see Table 2 and Figure 4). This supports Rescorla’s ar gument that the rate of vulnerability finding is constant over long periods of time. The reason why five packages show a different behavior is not clear. Interestingly, three closed source pack- ages (MS Office, Oracle, and DB2) and one open source package (Apache) show no vulnerability disclosure during the first 316, 202, 460, and 367 days, re spectively, while for all other packages vulnerabilities were published quickly after their release. The available vulner- ability data explains these large time gaps only partially: ˲ In contrast to the other three pack- ages, Apache 2 was a substantial re- write of much of the Apache 1.x code. As knowledge of vulnerabilities of Apache 1.x was of very limited use for finding vulnerabilities in Apache 2, the discovery of the first vulnerabilities could be expected to take a while, but it seems questionable as to whether this explains a one-year gap. Data on pub- lished vulnerabilities does not tell us whether the quality of code prevented vulnerabilities from being detected quickly or whether discovered vulnera bilities were published later or are even still unpublished. ˲ In the case of DB2, the software showed only 13 vulnerabilities, so that a large time gap is less surprising. ˲ Regarding Oracle, the NVD pub lished the first (ten) vulnerabilities altogether on Aug. 4, 2004. It seems reasonable to assume that these pub- lished vulnerabilities had been de tected much earlier and that their pub- lication was delayed either by Oracle, MITRE, or NIST NVD. ˲ The case of Office 2003 showed none of the aforementioned reasons. Vulnerability data does not reveal why the first vulnerabilities were published so late. Despite the aforementioned minor issues in analyzing vulnerability data, overall there is no observable differ ence between open source and closed source software with regard to the (type of) development of vulnerabilities over time, and there is also no observable difference between open source soft- ware developed in bazaar style and open source software developed in ca- thedral style. Severity of published vulnerabili ties. Having considered the number of 136 communications of the acm | May 2011 | vol. 54 | no. 5 --- vulnerabilities, I now analyze whether open source and closed source soft- ware differ in the severity of published vulnerabilities. This perspective is im- portant as well, because a single highly severe vulnerability that enables at- tackers to get root access to a system is probably more crucial than 10 low severe vulnerabilities that grant only reading access to unauthorized users. I analyze the severity of vulnerabilities for each software package in terms of the median and the proportion of highly severe vulnerabilities. Mean, standard deviation and, for each ap- plication type, the median of medians is also given (see Table 3). The analysis provides the following results: ˲ The medians of medians reveal that the vulnerabilities of office prod- ucts are much more severe (8.45) than those of Web servers (5.0), while the values of the other application types are close to each other. However, a sta- tistical analysis of the medians is not reasonable here due to the low number of values.” ˲ When we determine the medians of medians of open source software (5.7) and closed source software (6.8) and also the corresponding medians of the proportions of highly severe vulnerabilities (30.28% and 45.95%, re spectively), the first impression is that open source software is more secure in terms of the severity level. However, applying statistical analysis (Mann Whitney U-test) on the medians,a no statistically significant differences can be found: the two-tailed test provides a high number for p (p=0.11). Applying the same test to the proportion figures, the test, again, does not indicate that the samples are significantly different at the 0.05 level (p=0.06).b a Although the severity of a vulnerability is given by a number, this number is at ordinal scale level only (due to the characteristics of the CVSS evaluation process). Consequently, vari ances of the samples cannot be determined. b In contrast to severities of vulnerabilities, pro portions of highly severe vulnerabilities are at cardinal level. As the equality of the sample variances cannot be rejected at the 0.05 level (Fsample = 1.62 < 3.5 =F(7,8), the assumption that samples are normally distributed cannot be re jected at the 0.05 level (Kolmogorov-Smirnow test), we assume that variances are approxi mately equal so that the Mann-Whitney U-test can be applied. ware communities in terms of creating, detecting, and publishing vulnerabili ties, the investigation of the provision of patches aims at identifying how vendors behave in actively addressing and finally removing vulnerability is Vendors’ Patching Behavior Patch data. While the analysis of vul nerabilities and their publication re fers to the first three phases of the software vulnerability life cycle, and thereby mirrors the behavior of soft Table 3. Severity of published vulnerabilities. Application Type Product Severity (range=[0;10]) mean median std. dev. Proportion of highly severe vulnerabilities ([7;10]) Median of medians Browser Internet Explorer 7* 6.65 6.80 2.07 45.95% 6.6 Firefox 2** 6.38 6.40 2.11 36.53% Email client MS Outlook Express 6* 6.18 5.10 1.76 39.13% 5.95 Thunderbird 1** 6.53 6.80 2.23 47.27% Web server IIS 5* 6.00 5.00 1.55 36.14% 5.00 Apache2** 5.36 5.00 1.50 18.75% Office MS Office 2003* 8.11 9.30 1.91 67.72% 8.45 OpenOffice2** 7.61 7.60 1.79 63.16% Operating system Windows 2000* 6.58 7.20 2.10 57.92% 6.8 Windows XP* 6.67 7.20 2.16 58.92% Mac OS X 10.x* 6.18 6.80 2.13 41.33% Red Hat Enterprise Linux 4** 4.72 4.90 2.20 23.11% Debian 3.1** 4.75 4.90 2.21 23.19% Database Management System mySQL 5** 5.05 4.90 2.02 12.12% 6.15 PostgreSQL 8** 6.17 6.80 1.89 36.00% Oracle 10g* 5.96 5.50 2.05 33.33% DB2 v8* 6.22 7.2 2.75 53.85% * closed source software median of medians = 6.8 ** open source softwareµ median of medians = 5.7 may 2011 | vol. 54 | no. 5 | communications of the acm 137 --- sues. In order to detect differences in the patching behavior of open source and closed source vendors, I analyze how many of the vulnerabilities re mained unpatched and whether any correlation between the patch status and the severity of vulnerabilities ex- ists. Although vendor sites provide patch dates, I do not analyze the time gap between vulnerability disclosure and vendors’ provision of patches, as the vulnerability publication dates contained in the NVD do not necessar- ily give the actual publication date (dis- cussion earlier). In contrast to vulnera bility publication data, reliable data on patches can be (manually) collected by directly looking up vendors’ sites and vendor-neutral Web sites. More specifi- cally, I used the following data sources to obtain reliable patch data: NVD, MI TRE site, US-CERT Vulnerability Notes Database, SecurityFocus, Microsoft Se- curity Bulletins, OpenOffice.org, The Open Source Vulnerability Database, The X-Force database (IBM), Mozilla Foundation Security Advisories, Red Hat Network, Apache Security Reports, Apple Mailing Lists, IBM FixPaks, VU PEN Security, mySQL Forge, and Oracle Security Alerts and Patch Updates. The newly compiled data pool con tains patch data on the aforemen- tioned browsers, email clients, Web servers, office products, operating systems, and database management systems. In cases where vulnerabilities have not been patched by those ven- dors regarded as “responsible” by NIST (NVD), I manually checked the MITRE description of these vulnerabilities. On the basis of this inspection I decided whether it’s the responsibility of a par- ticular vendor to provide a patch or not. In the few cases where I regarded the assigned vendor as not being responsi ble, I did not count the vulnerability as unpatched (by that particular vendor). (Un)patched Vulnerabilities Table 4 shows aggregated patch data for each software package. Vulnerabili ties for which I could not find any patch information by Feb. 28, 2009 are classi fied as unpatched. As I used NVD xml data feeds as available on Jan. 31, 2009, the analysis considers a time gap of four weeks in order to account for delays in vendors’ patching behavior, such as those due to “patch release days.” It is remarkable to see that 17.6% (30.4%) of the published open (closed) source software vulnerabilities (in terms of the median) are still un- patched. However, standard devia- tions differ enormously (6.9% and 24.7%, respectively). Apparently, the proportion of still unpatched vulner- abilities largely depends on the spe- cific vendor. I discuss this behavior in detail below. Interestingly, the case of Microsoft also shows that even the same vendor can display different patching behav- ior, depending on the particular ap- plication type: while only 4% of MS Office 2003 vulnerabilities remain un- patched, one out of three vulnerabili- ties of both operating systems remain unpatched, half of the vulnerabilities of IIS are still open, and even two out of three vulnerabilities of the Internet clients remain unpatched. The case of operating systems shows that the proportion of unpatched vulnerabili- ties of software cannot be explained by simply considering the number of vul- nerabilities, it rather depends on the vendors’ patching priorities. Severity of (un)Patched Vulnerabilities It is interesting to compare the severity median of unpatched vulnerabilities with the median of patched vulner- abilities, in order to detect vendors’ patching priorities, and to detect dif- ferences between open source and closed source software. The data in Ta- ble 4 reveals that, for all six Microsoft products, there is a strong bias toward patching severe vulnerabilities. This result indicates that Microsoft decides to leave less severe vulnerabilities un- patched, probably because the eco- nomic efforts would not be compen- sated by the (minor) gain in software security. However, on the other hand, the result also shows that Microsoft is interested in patching the most se- vere vulnerabilities, which reveals that software security is regarded to be a serious market issue. Apple (MAC OS X) shows a similar behavior in their op- erating system in terms of the severi- ties of patched and unpatched vulner- abilities, but, in contrast to Microsoft, Apple seems to be interested in patch- ing most of the vulnerabilities. We find this strong interest in patching vulner- abilities also in the cases of Oracle and IBM (DB2), but the severity medians of Table 4. Patched and unpatched vulnerabilities. Application Type Product Vulnerabilities (un)patched #vuln. #vuln. unpatched Prop. of unpatched vuln. Median of severities unpatched patched overall Browser Internet Explorer 7 74 49 66.22% 5.0 9.3 6.8 Firefox 2 167 34 20.36% 5.0 6.8 6.4 Email client MS Outlook Express 6 23 15 65.22% 5.0 7.3 5.1 Thunderbird 1 110 6 5.45% 3.45 6.95 6.8 Web server IIS 5 83 40 48.19% 5.0 7.2 5.0 Apache2 80 21 26.25% 4.7 5.0 5.0 Office MS Office 2003 99 4 4.04% 5.05 9.3 9.3 OpenOffice2 19 4 21.05% 5.25 9.3 7.6 Operating system Windows 2000 385 117 30.39% 5.1 7.2 7.2 Windows XP 297 91 30.64% 5.0 7.5 7.2 Mac OS X 10.x 300 20 6.67% 5.0 6.8 6.8 Red Hat Enterprise Linux 4 264 39 14.77% 4.9 4.9 4.9 Debian 3.1 207 30 14.49% 4.9 4.9 4.9 Database Management System mySQL 5 33 8 24.24% 4.6 4.9 4.9 PostgreSQL 8 25 3 12.00% 9.0 6.3 6.8 Oracle 10g 63 8 12.70% 7.35 5.5 5.5 138 communications of the acm | May 2011 | vol. 54 | no. 5 --- unpatched vulnerabilities are higher than those of the patched ones. To sum up, three out of four closed source software vendors leave only a few vul- nerabilities unpatched, while the other vendor focuses on patching more se- vere vulnerabilities. Regarding the medians of patched and unpatched vulnerabilities of open source vendors, I do not find any pat- tern. In addition, the patching behav- ior of open source vendors shows that the proportion of unpatched vulnera- bilities varies between 12% and 26.25%, and can differ considerably. On the oth- er hand, none of the eight open source software packages shows an outlier, in contrast to closed source software. As a result of the analysis of the patching behavior of software vendors, it turns out the behavior is not mainly determined by the particular software development style, but by the policy of the particular vendor. Threats to Validity Although the presented empirical study uses comprehensive data on vulnera- bilities and patches of widely deployed software packages, and manual work was carried out in order to check and to improve data quality, some threats to the validity of results remain. First, the analysis investigates only those vulner- abilities that have been published as CVE vulnerabilities, that is, it excludes vulnerabilities that have not been dis- closed at all or have not been included in the MITRE CVE dictionary. Second, neither the “Assigned date” provided by MITRE nor the “Original release date” included in the NVD necessarily mirror the actual date of vulnerability disclo- sure to the public. I am not aware of any data sources that provide reliable and complete information on actual disclo- sure dates. Third, the CVE-to-vendor/ product mapping (CPE) in the NVD is incomplete and not always clear with regard to which vendor is responsible for releasing a patch. The mapping was manually inspected by myself. In some cases, conflicts were resolved by adopt- ing my point of view. Fourth, MTBVD values are limited in their validity, as missing market and software factors of packages (for example, market share, SLOC) would need to be considered. Discussion The analysis and comparison of open source and closed source software packages reveals that the type of soft- ware development is not the primary driver of software security in terms of the development of vulnerability dis- closure over time, the severity of pub- lished vulnerabilities, and unpatched vulnerabilities and their severity. Con- sequently, we should explore other factors rather than asking whether open source or closed source software leads to higher levels of security. One approach would be to investigate the technical roots of vulnerabilities, for example by adopting the “Common Weakness Enumeration” scheme pro- vided by MITRE. This approach might help to explain differences between various software types with regard to the severity of vulnerabilities. Unfortu- nately, the NVD provides CWE values for only a few vulnerabilities, so that we need a more comprehensive data set. A second approach would be the com- parison of MTBVD values by identifying and considering software and market factors. We could then apply regres- sion models, and control for these fac- tors. As the sound application of such models also requires the availability of the actual dates of vulnerability disclo- sures, we would need more precise data than is currently available. To sum up, we still face several data problems that impede the identification and explana- tion of differences in software security. In contrast to vulnerability data, patch data is even more difficult to ob- tain. In order to facilitate further analy- sis and to avoid the tedious collection of patch information, comprehensive data pools would be useful. The analy- sis of vendors’ patching behavior shows a diffuse picture that misses clear pat- terns. We find patching behaviors that do not only differ between vendors but also between different products of the may 2011 | vol. 54 | no. 5 | communications of the acm 139 --- same vendor (Microsoft). However, high figures of unpatched vulnerabili ties show that exogenous incentives for software vendors to avoid vulnerabili ties and/or to provide patches still need to be amplified, although a bias toward patching most severe vulnerabilities occurs. Economic countermeasures may provide such incentives.3,4 Conclusion This work has presented the first com- prehensive empirical study on the security of open source and closed source security. It compared 17 well- known and widely deployed software packages regarding published vulner- abilities and software vendors’ patch- ing behavior. The empirical results have shown that open source and closed source software do not signifi- cantly differ in terms of the severity of vulnerabilities, the type of develop- ment of vulnerability disclosure over time, and vendors’ patching behavior. Although open source software devel opment seems to prevent “extremely bad” patching behavior, overall there is no empirical evidence that the par ticular type of software development is the primary driver of security. Rather, the policy of the particular vendor de- termines the patching behavior. Con- sequently, in order to make software less vulnerable, it is most important to provide strong economic incen- tives for software producers to provide patches (at least for published vulner- abilities) or, even better, to avoid vul- nerabilities from the outset. Acknowledgement The author would like to thank Chris topher Johnson, Program Manager of the NIST National Vulnerability Data- base, for valuable discussions and the provision of background information, and Eliot Rich for his valuable support in writing this article. I appreciate the critical remarks of three anonymous re viewers. All errors are the sole responsi- bility of the author. References 1. Alhazmi, O., Malaiya, Y., Ray, I. Measuring, analyzing and predicting security vulnerabilities in software systems. Computers & Security 26, 3 (2007) 219-228. 2. Anderson, R. Open and closed systems are equivalent (that is, in an ideal world). Perspectives on Free and Open Source Software. J. Feller, B. Fitzgerald, S.A. Hissam, and K.R. Lakhani (eds). MIT Press, Cambridge, MA, 2005, 127–142. 3. Anderson, R. Why information security is hard—An economic perspective. In Proceedings of the 17th Computer Security Applications Conference, (New Orleans, LA, Dec. 10-14, 2001), 358-365. 4. Anderson, R. and Moore, T. Information security economics—and beyond. Information Security Summit 2008; http://www.cl.cam.ac.uk/~rja14/ Papers/econ_czech.pdf. 5. Arbaugh, W.A., Fithen, W.L. and McHugh, J. Windows of vulnerability: A case study analysis. IEEE Computer 33, 12 (2000), 52–59. 6. Beattie, S., Arnold, S., Cowan, C., Wagle, P., Wright, C. and Shostack, A. Timing the application of security patches for optimal uptime. In Proceedings of 16th Systems Administration Conference, (Berkeley, CA, 2002), USENIX Association, 233–242. 7. Böhme, R. Vulnerability markets. What is the economic value of a zero-day exploit? In Proceedings of 22nd Chaos Communication Congress, (Berlin, Germany, Dec. 27-30, 2005). 8. Free Software Foundation (FSF). The free software definition; http://www.fsf.org/licensing/essays/free-sw. html, 2007. 9. Frei, S., May, M., Fiedler, U. and Plattner, B. Large-scale vulnerability analysis. In Proceedings of the ACM SIGCOMM 2006 Workshop, (Nov. 11, 2006, Pisa, Italy). 10. Gopalakrishna, R. and Spafford, E. H. A trend analysis of vulnerabilities. Technical Report 2005-05, CERIAS, Purdue University, May 2005. 11. Levy, E. Wide open source; http://www.securityfocus. com/news/19, 2000. 12. MITRE. Common vulnerabilities and exposures; http:// cve.mitre.org, 2009. 13. Neuhaus, S., Zimmermann, T., Holler, C., Zeller, A. Predicting vulnerable software components. In Proceedings of the 14th ACM Conference on Computer and Communications Security (CCS 2007), (Alexandria, VA, Oct. 2007), 529–540. 14. NIST. National Vulnerability Database; http://nvd.nist. gov, 2009. 15. Open Source Initiative (OSI). The Open Source Definition; http://www.opensource.org/docs/osd, 2006. 16. Ozment, A. Improving vulnerability discovery models: Problems with definitions and assumptions. In Proceedings of the 3rd Workshop on Quality of Protection, (Alexandria, VA, Oc. 29, 2007). 17. Ozment, A. The likelihood of vulnerability rediscovery and the social utility of vulnerability hunting. In Proceedings of the 4th Workshop on the Economics of Information Security, (Harvard University, June 2–3, 2005, Cambridge, MA), 1–21. 18. Ozment, A. and Schechter, S.E. Milk or wine: Does software security improve with age? In Proceedings of the 15th Conference on USENIX Security Symposium, (Vancouver, B.C., July 31-Aug. 4, 2006). 19. Payne, C. On the security of open source software. Information Systems Journal 12, 1 (2002), 61–78. 20. Raymond, E.S. The Cathedral and the Bazaar: Musings on Linux and Open Source by an Accidental Revolutionary. O’Reilly, Beijing, China, 2001. 21. Rescorla, E. Is finding security holes a good idea? In Proceedings of the 3rd Annual Workshop on Economics and Information Security, (University of Minnesota, May 13-14, 2004). 22. Radianti, J., Rich, E. and Gonzalez, J.J. Vulnerability black markets: Empirical evidence and scenario simulation. In Proceedings of the 42nd Hawaii International Conference on System Sciences, (Big Island, Hawaii, Jan. 5-8, 2009). 23. US-CERT. Vulnerability notes database field descriptions (2009); http://www.kb.cert.org/vuls/ html/fieldhelp/ 24. Woo, S.-W., Alhazmi, O.H. and Malaiya, Y. K. An analysis of the vulnerability discovery process in Web browsers. In Proceedings of the 10th International Conference on Software Engineering and Applications, (Dallas, TX, Nov. 13-15, 2006). 25. Woo, S.-W. , Alhazmi, O. H. and Malaiya, Y. K. Assessing vulnerabilities in Apache and IIS HTTP servers. In Proceedings of the 2nd International Symposium on Dependable, Autonomic and Secure Computing, (Indianapolis, IN, Sept. 29-Oct. 1, 2006), 103-110. It is still challenging to retrieve consistent and comprehensive vulnerability data and patch data. In order to facilitate further analysis and to avoid tedious data collection, comprehensive data pools would be useful. Guido Schryen (guido.schryen@wiwi.uni-regensburg.de) is a professor of information systems at the University of Regensburg, Germany. © 2011 ACM 0001-0782/11/05 $10.00 140 communications of the acm | May 2011 | vol. 54 | no. 5