I’ve spent the past 3.5 years working on a team where my primary responsibilities involved “application security”. Now that this era has come to an end, I’d like to share some of the initiatives and define their successes and shortcomings. This is part 4 of 5 so please be sure to read parts 1, 2, and 3.
Metrics and defining success
I feel that it would be easier to give you a view of the ultimate goal behind the application security metrics rather than explain how it evolved over time. The ultimate goal was for us to be able to:
Track vulnerability metrics and relevant meta data across all security assurance services.
Derive relevant data from vulnerability metrics data to aid in assigning the appropriate resources to address vulnerability trends.
Demonstrate the effectiveness, reach, and customer feedback of each security assurance service.
Provide a snap shot view of overall risk to the enterprise based on known application vulnerabilities and demonstrate trends over time.
The rational behind these goals was so that we could demonstrate the effectiveness of our services (and therefore our work), provide reliable vulnerability management data, and give us the data points needed to effectively make use of the resources we’d been allocated. I should note that we never really had an accurate picture of just how many applications were out there, so we had difficulties determining service coverage, so instead we decided to nail a stake in time as the starting point for our date based metrics.
First, we had to build a system that would collect and report on the vulnerability data points we’d identified. I’ve blogged about the system here and here where you can read the details, but to summarize, we wanted to classify the vulnerability, define risk, provide detailed description and any proof of concept data, define the product that is vulnerable, define the development team that created the vulnerable code, define contact information, and track resolution status. With this system, not only did we have each vulnerability we’d identified documented and meta-tagged, but we also had the data that allowed us to determine which development groups had the worst vulnerability rates (most vulnerabilities per product) and should be targeted for training. We also used the vulnerability data and meta-tags to provide executive level views to answer the “how many?” “what type?” “who’s doing it?” and “how bad?” questions from executives.
Findings from the penetration testing, threat modeling, and security consulting services fit very nicely into the vulnerability management system. A simple customer feedback form could have been used to rate the effectiveness of the service delivery and the derived metrics could have been used to refine our processes, we just never got that far in the project. Of course, there were several other activities which didn’t get accounted for such as training and outreach and communications who’s demand grew organically throughout the year and who’s reach in the company could have been measured over time to (hopefully) demonstrate a downward trend in vulnerability metrics for the trained development teams.
I think that the metrics plan was strong and would provide us the data we needed, I just wish we’d started implementing the full plan sooner.
In order to effectively implement my plan for tracking enterprise vulnerability metrics and their associated data points, it was clear that a custom application for data entry and report viewing would be required. As such, I wrote a PHP/MySQL web application with the following features and functionality.
Data Model – In order to structure the collected data in the database in a manner which facilitates data extraction from reports, I had to define the reporting needs early in the development process. By doing so, I was able to create a relational data model which makes reporting fast, accurate, and reliable.
Data Input – I’ve provided a series of HTML forms which provide the user with all the data points from my previous article. These forms have been designed with a heavy focus on ease of use and rapid data entry. To provide these features, the input for the required data points have been grouped logically and implement a feature for quick input where the user can select from a list of recent entries for the given data points (eg: recent contacts, recent CVE numbers, etc).
Reporting – I’m a big fan of keeping reporting as open as possible: it makes life much easier down the road. As such, I created a reporting framework which is capable of outputting a list of vulnerabilities based on a query for any data point stored in the database. For example, if you want to know which reports are associated with a given manager, business unit, vulnerability class, or keyword just select your preferences on the search page, and view the results.
Statistics – A data collection tool is worthless unless you have a way to quickly determine the characteristics of the data stored in the database. As such, I created a statistics page which provides the viewer with a snapshot of the vulnerability metrics based on similar characteristics. The following details the statistics and their intended purpose:
Vulnerabilities Created and Resolved
Simple count of vulnerabilities created and resolved (separate counts) in the system for the given time period.
Vulnerabilities Created and Resolved per Category
Simple count of vulnerabilities created and resolved (separate counts) in the system for the given time period per vulnerability category.
Vulnerabilities Created and Resolved per System Type
Simple count of vulnerabilities created and resolved (separate counts) in the system for the given time period per system type.
Vulnerabilities Created and Resolved per Source
Simple count of vulnerabilities created and resolved (separate counts) in the system for the given time period per vulnerability source.
Vulnerabilities Created and Resolved per Impact Rating
Simple count of vulnerabilities created and resolved (separate counts) in the system for the given time period per impact rating category and value.
Vulnerabilities Created and Resolved per Risk, Severity, and Probability
Simple count of vulnerabilities created and resolved (seperate counts) in the system for the given time period per risk, severity, and probability rating.
Vulnerabilities Created and Resolved per Domain
Simple count of vulnerabilities created and resolved (seperate counts) in the system for the given time period per domain name.
Top Report Owners
Simple count of vulnerabilites created and resolved with average time to resolve in the given time period per report owner (the security engineer assigned to the vulnerability).
Top Business Units
Simple count of vulnerabilites created and resolved in the given time period per business unit listed in the contact field.
Top Managers
Simple count of vulnerabilities created and resolved in the given time period manager listed in the contact field.
The tool also contains several usability features such as mass actioning of vulnerabilities (eg: resolve multiple reports at once), a data authorization model defined by report owner (owner can edit and view, others can only view), and a “my” page which displays reports relevant to the user.
In the future I hope to seperate the parts of the tool which are specific to my environment and open source the project so that others can gain visibility into their vulnerability posture.
Some time back I posted about designing and implementing the processes and tools which would enable me to track vulnerabilities affecting the enterprise throughout their security life cycle (identification to mitigation validation). In addition to providing a central repository for all enterprise vulnerabilities, I also wanted to make sure that we could use the metrics derived from this data to support investments in other areas of IT Security. My target was to have everything ready to start tracking vulnerabilities on the first day of business in 2009.
The data points I decided to track were primarily aimed at providing visibility into how vulnerabilities were identified, who was responsible for them, establishing an enterprise risk profile, and categorizing vulnerabilities according to risk factors. The following is a more detailed list of the metrics I’m tracking and their intended purpose.
The Data Points
Report Source – Tracking where the vulnerability reports are originating from. As options, I decided on direct report, findings from automated scanning, and all relevant existing security processes (security assurance, incident response investigation, etc) as sources.
System Type – Tracking the type of system affected by the vulnerability. As options, web applications, desktop applications, hosts/networks/operating systems, and web platforms seem to encompass all major system types.
Vulnerability Class – Being able to abstract individual vulnerabilities into something that is digestible by a non-technical audience is critical in this field. The National Vulnerability Database (NVD) has adopted a cross section of the Common Weakness Enumeration (CWE) hierarchy that is broad enough for the diversity of vulnerabilities most enterprises would encounter. For more information, checkout the CWE page on the NVD website, http://nvd.nist.gov/cwe.cfm.
Impact Ratings - Giving us a smart way to answer the “how bad is it?” question. The NVD has developed a framework (Common Vulnerability Scoring System – CVSS) where the characteristics of the vulnerability are supplied and the high, medium, low ratings for risk, severity, and impact are derived mathematically based on the supplied characteristics. The mathematical formula for the computation is available in the CVSS documentation, but I had a different idea for computing the h/m/l ratings. First, here are a few of the characteristics, their ratings, and descriptions:
Access Vector (AV)
Local - vulnerability can only be exploited from the local system. LAN - vulnerability can only be exploited from the local area network. Network - vulnerability can be exploited from the internet.
Authentication (AU)
User - User level authentication is required to exploit the vulnerability. Admin/Root – Administrator/root level authentication is required to exploit the vulnerability. None - No authentication is required to exploit the vulnerability.
Remediation Level (R)
Official Fix – The vulnerability has been fully remidiated and tested according to guidance given by security staff. Temporary Fix – The vulnerability has been mitigated temporarily while a permanent solution is devised. Workaround – A workaround exists to circumvent the vulnerability. Unavailable – No remidiation is available for the vulnerability.
Not Defined – Remediation for the vulnerability is unknown.
Additional impact ratings include Access Complexity (AC), Confidentiality Impact (C), Integrity Impact (I), Availability Impact (A), Exploitability (E), and Report Confidence (RC). Each of the values associated with the impact ratings is assigned a whole integer from 1 – 3. For impact ratings where there are more then 3 options, I found that in each case, 2 of the options were close enough to be assigned the same value. For determining which value I would assign to each impact rating, I assumed the greater the value of the impact rating, the more serious the vulnerability for the given impact. For example, for Authentication, User is assigned a value of 2, Admin/Root a value of 1, and None a value of 3. I then weighted each impact rating based on the importance attributed to the impact. For example Access Vector (weight of 2) is much more of a factor when determning risk than Report Confidence (weight of 1). To further clarify the impact of a vulnerability, I wanted to produce ratings for risk, severity and probability which are determined with the following equations:
Risk: (2*(av) + 2*(ac) + 2*(au) + c + i + a + 2*(e) + 4*(r) + rc)/15. Severity: (c + i + a + 2*(e) + rc)/6. Probability: (2*(av) + 2*(ac) + 2*(au) + 4*(r))/10.
Once the risk, severity and probability have been determined, I use the following ranges to assign high, medium, and low:
High: >= 2.00. Medium: >= 1.00 and <= 1.99. Low: >= 0 and <= .99.
Contact Information – Tracking who is reporting the issues, and who is responsible for fixing them. For each contact entered into the system, I wanted to capture their name, contact information, business unit (only for enterprise contacts), and their manager (again, only for enterprise contacts). The purpose of the contact information data points is to be able to determine which group within the enterprise we can target for education regarding specific topics in security. For example if we find that product team X has an abnormally high number of input validation vulnerabilities, we could justify investing in some XSS and SQLi training and know that it will be beneficial.
URL – I wanted to be able to present the data by fully qualified domain name (FQDN) to easily present the data to site owners.
Other - In addition to the data points mentioned above, I’ve also provided inputs for tracking of CVE numbers and internal ticket tracking system numbers. While this isn’t particularly relevant to vulnerability tracking, it will give my management a tie in to existing processes.
Provided that the data entered into the vulnerability tracking system is reliable and up to date, answering the “What’s wrong?“, “How widespread?“, “Who’s responsible“, “What’s the exposure?“, “What if we don’t fix it?“, “Where should we invest resources?“, and “How’s the security?” questions should be no problem from here on out.
In business, you typically find that a critical question associated with investing in security is “How much will it cost if we don’t do X?”. It’s no secret that it’s still very difficult to justify a web application security program to those who approve the dollars. Sure, for breaches involving financial loss or financial information theft, justification for prevention can leverage the cost of the financial loss. But consider an XSS vulnerability in a webmail program or a CSRF vulnerability in the social network du jour. How do you determine the cost of an attacker SPAMMING your users or performing unauthorized changes to personal data in order to justify your program? You need to scare executives enough to care…any they only speak numbers.
Numbers can be obtained using existing sources of knowledge including findings from existing security mitigation, entries from web server access logs, and Web Application Firewall (WAF) deployments. There are several methods of extracting and consuming this data such as Abnormally Detection Systems (ADS), vendor provided data viewers, or custom scripts. Each organization will have different needs and available budget, so careful consideration must be taken when considering the various options for collecting and analyzing the data.
Metrics from Existing Security Mitigation
I am making the assumption that there is already some form of Quality Assurance (QA) for security taking place on the web applications your company produces. Even if your QA program is a single person with a knack for running some tools, the findings from these efforts can be tracked, labeled, and categorized. Over time, these data points can be leveraged to justify additional investments in staff, training, tools, or outreach efforts. The Common Vulnerabilities and Exposures (CVE) and National Vulnerability Database (NVD) have developed a common way of defining, tracking, and measuring vulnerabilities. The following are a few examples of the kinds of metrics which can be obtained from CVE/NVD style vulnerability tracking:
Vulnerability risk scoring
Providing a consistent rating for risk for vulnerabilities which is based on a repeatable scoring of business and technical vectors will provide a metric from which prioritization for mitigation can be derived. Additionally, integration of this score into vulnerability reporting will provide an easy way to to quickly communicate the importance of any given issue. CVE/NVD scoring.
Vulnerability classification distribution
The CVE/NVD methodology defines categories in which to classify vulnerabilities called the Common Weakness Enumeration (CWE). These categories include authentication issues, XSS, path traversal, and design errors amongst others. The metrics obtained from this data will provide justification for security development for the design and development teams.
Vulnerability totals per project or development team
Being able to track vulnerability totals across projects or development teams will provide additional leverage for targeted training for development staff. Imagine being able to walk into the Director’s office and justifying the cost of bringing in an expert to discuss secure authentication and authorization in web systems through hard numbers demonstrating the needs of the business.
Metrics from Web Server Logs
Web server logs contain a wealth of information relating to the sort of attacks taking place across your company’s web properties. The key to making use of this data is consistent monitoring and tracking of the chosen data points. Some key data elements worth tracking include:
Source IP address
The source IP address of the HTTP request can be matched against an DNSBL database or a private IP blacklist in order to track use of the application by known threat agents.
HTTP methods RFC-2616 defines a finite number of HTTP methods allowed in the HTTP protocol. Any deviation from the defined standards could adversely impact misconfigured webserver and should be considered an offensive action.
Requested URL combined with HTTP response status codes
Being able to track the number of requests against URLs will help in determining the presence of rogue files on webservers and help identify scanning for common hidden directories. Using the HTTP response codes will help identify instances where the requested resource returned status codes other then the 200 and 300 series.
Query string parameters
Query string parameters are the most prevalent location for input based attacks. An easy way of identifying offensive name/value pairs is to use the regular expressions in the relevant mod_security rules (see mod_security rule file modsecurity_crs_40_generic_attacks.conf).
Client browser strings
The client browser string identifies the type of browser and it’s capabilities to the web server. This information can be instrumental in identifying automated attacks against your company’s web properties. There is a lot of knowledge which can be gained from tracking bad browser strings, but this certainly goes beyond the scope of this post.
Metrics from WAF
Similar to obtaining metrics from web server logs, metrics obtained from Web Application Firewalls (WAFs) can be used to identify attacks taking place against your company’s web applications. I personally recommend using mod_security as it’s free, flexible, and powerful. Initial deployments of mod_security should be in log only mode until it has been determined that the enabled rules are not prohibiting legitimate users from using the application and that the rules are enforcing the intended restrictions.
Of course, I understand that getting this information together might present somewhat of a challenge, but the data points obtained from this metricizing excercise will provide irrefutable hard data on existing security controls as well as areas for improvement. I’m currently going through this excercise, so over the next couple months, I’ll hopefully be able to provide some numbers to demonstrate the effectivness of my efforts.
I’ve spent the past 3.5 years working on a team where my primary responsibilities involved “application security”. Now that this era has come to an end, I’d like to share some of the initiatives and define their successes and shortcomings. This is part 4 of 5 so please be sure to read parts 1, 2, and 3.
DandyID
Friendfeed
claimID
Disqus
Facebook
Twitter
myOpenID
Google Profiles
Linkedin