Blog :: by Wade Woolwine

Thoughts on an AppSec Program (Pt. 4) – Metrics and defining success

I’ve spent the past 3.5 years working on a team where my primary responsibilities involved “application security”. Now that this era has come to an end, I’d like to share some of the initiatives and define their successes and shortcomings. This is part 4 of 5 so please be sure to read parts 1,  2, and 3.

Metrics and defining success
I feel that it would be easier to give you a view of the ultimate goal behind the application security metrics rather than explain how it evolved over time. The ultimate goal was for us to be able to:

• Track vulnerability metrics and relevant meta data across all security assurance services.
• Derive relevant data from vulnerability metrics data to aid in assigning the appropriate resources to address vulnerability trends.
• Demonstrate the effectiveness, reach, and customer feedback of each security assurance service.
• Provide a snap shot view of overall risk to the enterprise based on known application vulnerabilities and demonstrate trends over time.

The rational behind these goals was so that we could demonstrate the effectiveness of our services (and therefore our work), provide reliable vulnerability management data, and give us the data points needed to effectively make use of the resources we’d been allocated. I should note that we never really had an accurate picture of just how many applications were out there, so we had difficulties determining service coverage, so instead we decided to nail a stake in time as the starting point for our date based metrics.

First, we had to build a system that would collect and report on the vulnerability data points we’d identified. I’ve blogged about the system here and here where you can read the details, but to summarize, we wanted to classify the vulnerability, define risk, provide detailed description and any proof of concept data, define the product that is vulnerable, define the development team that created the vulnerable code, define contact information, and track resolution status. With this system, not only did we have each vulnerability we’d identified documented and meta-tagged, but we also had the data that allowed us to determine which development groups had the worst vulnerability rates (most vulnerabilities per product) and should be targeted for training. We also used the vulnerability data and meta-tags to provide executive level views to answer the “how many?” “what type?” “who’s doing it?” and “how bad?” questions from executives.

Findings from the penetration testing, threat modeling, and security consulting services fit very nicely into the vulnerability management system. A simple customer feedback form could have been used to rate the effectiveness of the service delivery and the derived metrics could have been used to refine our processes, we just never got that far in the project. Of course, there were several other activities which didn’t get accounted for such as training and outreach and communications who’s demand grew organically throughout the year and who’s reach in the company could have been measured over time to (hopefully) demonstrate a downward trend in vulnerability metrics for the trained development teams.

I think that the metrics plan was strong and would provide us the data we needed, I just wish we’d started implementing the full plan sooner.

Tags: CVE, CWE, Enterprise Security, Metrics, NVD