I’ve spent the past 3.5 years working on a team where my primary responsibilities involved “application security”. Now that this era has come to an end, I’d like to share some of the initiatives and define their successes and shortcomings. This is part 4 of 5 so please be sure to read parts 1, 2, and 3.
Metrics and defining success
I feel that it would be easier to give you a view of the ultimate goal behind the application security metrics rather than explain how it evolved over time. The ultimate goal was for us to be able to:
Track vulnerability metrics and relevant meta data across all security assurance services.
Derive relevant data from vulnerability metrics data to aid in assigning the appropriate resources to address vulnerability trends.
Demonstrate the effectiveness, reach, and customer feedback of each security assurance service.
Provide a snap shot view of overall risk to the enterprise based on known application vulnerabilities and demonstrate trends over time.
The rational behind these goals was so that we could demonstrate the effectiveness of our services (and therefore our work), provide reliable vulnerability management data, and give us the data points needed to effectively make use of the resources we’d been allocated. I should note that we never really had an accurate picture of just how many applications were out there, so we had difficulties determining service coverage, so instead we decided to nail a stake in time as the starting point for our date based metrics.
First, we had to build a system that would collect and report on the vulnerability data points we’d identified. I’ve blogged about the system here and here where you can read the details, but to summarize, we wanted to classify the vulnerability, define risk, provide detailed description and any proof of concept data, define the product that is vulnerable, define the development team that created the vulnerable code, define contact information, and track resolution status. With this system, not only did we have each vulnerability we’d identified documented and meta-tagged, but we also had the data that allowed us to determine which development groups had the worst vulnerability rates (most vulnerabilities per product) and should be targeted for training. We also used the vulnerability data and meta-tags to provide executive level views to answer the “how many?” “what type?” “who’s doing it?” and “how bad?” questions from executives.
Findings from the penetration testing, threat modeling, and security consulting services fit very nicely into the vulnerability management system. A simple customer feedback form could have been used to rate the effectiveness of the service delivery and the derived metrics could have been used to refine our processes, we just never got that far in the project. Of course, there were several other activities which didn’t get accounted for such as training and outreach and communications who’s demand grew organically throughout the year and who’s reach in the company could have been measured over time to (hopefully) demonstrate a downward trend in vulnerability metrics for the trained development teams.
I think that the metrics plan was strong and would provide us the data we needed, I just wish we’d started implementing the full plan sooner.
In business, you typically find that a critical question associated with investing in security is “How much will it cost if we don’t do X?”. It’s no secret that it’s still very difficult to justify a web application security program to those who approve the dollars. Sure, for breaches involving financial loss or financial information theft, justification for prevention can leverage the cost of the financial loss. But consider an XSS vulnerability in a webmail program or a CSRF vulnerability in the social network du jour. How do you determine the cost of an attacker SPAMMING your users or performing unauthorized changes to personal data in order to justify your program? You need to scare executives enough to care…any they only speak numbers.
Numbers can be obtained using existing sources of knowledge including findings from existing security mitigation, entries from web server access logs, and Web Application Firewall (WAF) deployments. There are several methods of extracting and consuming this data such as Abnormally Detection Systems (ADS), vendor provided data viewers, or custom scripts. Each organization will have different needs and available budget, so careful consideration must be taken when considering the various options for collecting and analyzing the data.
Metrics from Existing Security Mitigation
I am making the assumption that there is already some form of Quality Assurance (QA) for security taking place on the web applications your company produces. Even if your QA program is a single person with a knack for running some tools, the findings from these efforts can be tracked, labeled, and categorized. Over time, these data points can be leveraged to justify additional investments in staff, training, tools, or outreach efforts. The Common Vulnerabilities and Exposures (CVE) and National Vulnerability Database (NVD) have developed a common way of defining, tracking, and measuring vulnerabilities. The following are a few examples of the kinds of metrics which can be obtained from CVE/NVD style vulnerability tracking:
Vulnerability risk scoring
Providing a consistent rating for risk for vulnerabilities which is based on a repeatable scoring of business and technical vectors will provide a metric from which prioritization for mitigation can be derived. Additionally, integration of this score into vulnerability reporting will provide an easy way to to quickly communicate the importance of any given issue. CVE/NVD scoring.
Vulnerability classification distribution
The CVE/NVD methodology defines categories in which to classify vulnerabilities called the Common Weakness Enumeration (CWE). These categories include authentication issues, XSS, path traversal, and design errors amongst others. The metrics obtained from this data will provide justification for security development for the design and development teams.
Vulnerability totals per project or development team
Being able to track vulnerability totals across projects or development teams will provide additional leverage for targeted training for development staff. Imagine being able to walk into the Director’s office and justifying the cost of bringing in an expert to discuss secure authentication and authorization in web systems through hard numbers demonstrating the needs of the business.
Metrics from Web Server Logs
Web server logs contain a wealth of information relating to the sort of attacks taking place across your company’s web properties. The key to making use of this data is consistent monitoring and tracking of the chosen data points. Some key data elements worth tracking include:
Source IP address
The source IP address of the HTTP request can be matched against an DNSBL database or a private IP blacklist in order to track use of the application by known threat agents.
HTTP methods RFC-2616 defines a finite number of HTTP methods allowed in the HTTP protocol. Any deviation from the defined standards could adversely impact misconfigured webserver and should be considered an offensive action.
Requested URL combined with HTTP response status codes
Being able to track the number of requests against URLs will help in determining the presence of rogue files on webservers and help identify scanning for common hidden directories. Using the HTTP response codes will help identify instances where the requested resource returned status codes other then the 200 and 300 series.
Query string parameters
Query string parameters are the most prevalent location for input based attacks. An easy way of identifying offensive name/value pairs is to use the regular expressions in the relevant mod_security rules (see mod_security rule file modsecurity_crs_40_generic_attacks.conf).
Client browser strings
The client browser string identifies the type of browser and it’s capabilities to the web server. This information can be instrumental in identifying automated attacks against your company’s web properties. There is a lot of knowledge which can be gained from tracking bad browser strings, but this certainly goes beyond the scope of this post.
Metrics from WAF
Similar to obtaining metrics from web server logs, metrics obtained from Web Application Firewalls (WAFs) can be used to identify attacks taking place against your company’s web applications. I personally recommend using mod_security as it’s free, flexible, and powerful. Initial deployments of mod_security should be in log only mode until it has been determined that the enabled rules are not prohibiting legitimate users from using the application and that the rules are enforcing the intended restrictions.
Of course, I understand that getting this information together might present somewhat of a challenge, but the data points obtained from this metricizing excercise will provide irrefutable hard data on existing security controls as well as areas for improvement. I’m currently going through this excercise, so over the next couple months, I’ll hopefully be able to provide some numbers to demonstrate the effectivness of my efforts.
I’ve spent the past 3.5 years working on a team where my primary responsibilities involved “application security”. Now that this era has come to an end, I’d like to share some of the initiatives and define their successes and shortcomings. This is part 4 of 5 so please be sure to read parts 1, 2, and 3.
DandyID
Friendfeed
claimID
Disqus
Facebook
Twitter
myOpenID
Google Profiles
Linkedin