I’ve spent the past 3.5 years working on a team where my primary responsibilities involved “application security”. Now that this era has come to an end, I’d like to share some of the initiatives and define their successes and shortcomings. This is part 5 of 5 so please be sure to read parts 1, 2, 3, and 4.
Training, outreach, and networking
We decided to leverage existing communications avenues (mailing lists, newsletters, status reports, etc) as well as setup a Wordpress blog. We used these tools to publish information about security news, link to 3rd party security documentation, security guidance, solicit feedback and most importantly, identify individuals throughout the organization that had interest or skills in secure software development. Our goal with the outreach program was not only to make information resources available, but also to ensure that our services were transparent and accessible. Like us, most security groups have to overcome the stigma of being a crazy bunch of paranoid hackers who cost the company money and cause deadlines to slip. As such, the outreach program coupled with our threat modeling and security consulting services were delivered with clarity, transparency, and comprehensiveness.
We also delivered numerous training courses aimed at educating developers and architects in defensive programming, software vulnerabilities, and threat modeling. These courses were typically delivered to smaller audiences and accompanied with hands on activities. We found that these courses were not only being well received, but also that attendees would contact us to request additional training tailored to a specific topic that would be relevant to their products. I feel that we had the distinct benefit of having team members who were very adept at delivering training and realize that not all organizations have this sort of resources. Given a high enough priority in company goals, training can easily be purchased, and employees who are members of professional groups can leverage relationships with professionals outside the organization who would be willing to deliver the training.
For more information on homegrown security teams, checkout my post.
Plain and simple: this book is a must read if you’re either thinking about deploying ModSecurity to protect your web applications or already have a basic ModSecurity deployment and want to learn how to customize it for your environment. Because I’m already well versed with ModSecurity, I decided read this book cover to cover without the distraction of the keyboard so that I could truly focus on how well complex topics like web application firewalls (WAF) and web application security could be covered in 250 pages. After about 5 hours of reading, I had reached the appendices and spent the next 2 days working back through the rulesets I use to protect my own web properties. Not only do I feel that I now have a more secure set of rules, I have also documented the performance impacts of ModSecurity on the Apache processes and reduced the overall response time of my applications by about 6ms.
The first 2 chapters of the book deal with installing, configuring, and rule writing basics for ModSecurity 2.5 with Apache 2.x. Magnus Mischel walks the reader through installing and configuring the module in Apache and making sure the installation is working properly. The chapter assumes best case scenarios for defaults in the operating system while in my experience, ModSecurity installs rarely go that easily. That being said, the preface of the book specifies that a good understanding of system administration principles is suggested to get the most out of the book and distribution specific installation guides for ModSecurity are widely available online. Mischel then goes on to introduce the ModSecurity rule writing syntax which he very astutely presents in two sections: rule grammar and structure followed by practical examples of how to write rules to mitigate vulnerabilities. Even after spending 1.5 years running numerous production instances of ModSecurity with custom rules, Mischel’s in depth coverage of the topic still had me hunting around for my highlighter to mark sections of the chapter containing syntax features that I didn’t know existed.
The book also does a great job of covering ModSecurity performance and logging. Mischel uses httpperf to benchmark server response time, memory usage, and processor load and presents the results for Apache with/without ModSecurity and with/without the default ruleset. Readers will find that this data will be invaluable if they ever need to sell the security benefit vs. system performance degradation to system administrators and executives. The chapter on logging and auditing offers an in depth look at the features available in ModSecurity offering insight on real world log and audit needs balanced with the processing overhead associated with excessive data collection. Mischel also provides a detailed implementation guide for deploying the ModSecurity Console with mlogc giving administrators and analysts a dashboard for monitoring alerts in a environment using multiple instances of ModSecurity.
One of the chapters I was most looking forward to reading was “Blocking Common Attacks” where Mischel presents a number of common web application vulnerabilities, offers an overview of the causes and impacts of said vulnerabilities and presents a virtual patch using a series of ModSecurity rules. To be perfectly honest, I was left a little disappointed with lack of depth in some of the content. While I understand that in depth coverage of web application vulnerabilities is probably beyond the scope of the book, the additional knowledge required to fully understand the causes and exploits of the issues that are presented in this book is likely beyond the average skill set of a system administrator. I was also left a little disappointed with the section on cross site request forgeries (CSRF) which never offered any virtual patching examples to mitigate the issue. Negatives aside, Mischel does a great job at presenting ModSecurity rules for mitigating 14 types of web application vulnerabilities such as cross site scripting (XSS), sql injection (SQLi), and shell code execution.
Mischel spends an entire chapter discussing configuring ModSecurity using a positive security model tailored to a YABB (Yet Another Bulletin Board) deployment. For a veteran ModSecurity user, this was by far the best chapter in the book, even though Mischel suggest you use Ethereal (Wireshark) as your local web proxy (you should be using Burp Suite by Portswigger). I can only hope that readers who are rolling out new ModSecurity installations or who are looking to improve existing deployments read this chapter and realize the value of this approach and choose to do the same to protect their applications.
By including the full ModSecurity directive and variable reference and a detailed guide on Regular Expressions, Magnus Mischel has created a complete guide to ModSecurity that is not only a great introduction to the WAF technology, but also a great desk reference for the experienced administrator.
Title: ModSecurity 2.5
Author: Magnus Mischel
Publisher: PACKT Publishing
Front page taglines:
* Securing your Apache installation and web applications
* Prevent web application hacking with this easy-to-use guide
I’ve spent the past 3.5 years working on a team where my primary responsibilities involved “application security”. Now that this era has come to an end, I’d like to share some of the initiatives and define their successes and shortcomings. This is part 3 of 5 so please be sure to read parts 1 and 2.
Threat modeling, architecture reviews, and security consulting
Our application security program always offered a design review service. Initially this service wasn’t very well defined and simply involved a security engineer performing a review of available documentation and interviews with the development team. The outcome of this engagement was a series of recommendation and requirements for the project to implement during the development phase. As you can imagine, this approach was very subjective and lacked documentation, but formalization of the offering was put on the back burner due to the successes with the penetration testing offering.
A little over a year and a half after the application security program was defined, we started focusing on assimilating some of the industry proven approaches to threat modeling and security architecture best practices into our existing design review service. We leveraged the opportunities presented by the local OWASP chapter and were able to obtain the training in formal threat modeling approaches that we were easily able to adapt and offer as a fully documented and repeatable process. Unfortunately, my time with the team was up before we could really determine the effectiveness of this service offering. My hope was that we could use this service to not only identify deficiencies in secure design early in the lifecycle, but also to setup a plan for providing training, code review / SCA services, and penetration testing at the appropriate times. This would effectively setup a consulting relationship between the individual security engineers and the project team where a single person / team could be leveraged for all aspects of application security.
I’ve spent the past 3.5 years working on a team where my primary responsibilities involved “application security”. Now that this era has come to an end, I’d like to share some of the initiatives and define their successes and shortcomings. This is part 2 of 5 so please be sure to read part 1.
Penetration Testing
Like most application security programs, ours was started by 2 of the team members who had some understanding of the vulnerability space in applications (mostly web based) and knew how to test for them. In a company with a seemingly unlimited number of custom web applications the target space was wide and varied which provided a vast playground for exploration and learning. Over time, the value of our hacking away at these web properties coupled with a few juicy findings quickly became apparent to the leadership and we were asked to work on expanding this capability to the rest of the team while maintaining the high level of thoroughness, accuracy, and professionalism our internal customers had grown accustomed to.
We were faced with the unique challenge of defining and documenting the vulnerability space, provide documented examples of how to identify and exploit the vulnerabilities, provide in person training to explore the complexities of application architectures and the interconnectedness of various components, and provide the development background necessary to work with developers to resolve the issues. Fortunately for us, the Open Web Application Security Project (OWASP) had already published some early versions of the OWASP Testing Guide and the OWASP Development Guide. These two documents provided the written documentation necessary to begin familiarizing the team with the general concepts and jargon associated with web application security. With the help of the OWASP WebGoat tool, we were able to provide a platform on which the engineers to practice what they were learning in the documents. We also conducted some additional internal training which took and in depth look at the vulnerabilities which were most common across our specific platforms and explored the best ways to address these issues in code. After a few years of regular training and refinement of the methodology has yielded a very efficient, accurate, and repeatable security testing service. If I had to go back and do it again, I really wouldn’t change anything. The resources obtained through the OWASP organization served us well and the existing internal knowledge was sufficient to bring the rest of the team up to speed.
I also wanted to address the issue of automated scanning. We did indeed purchase a solution which we worked tirelessly to implement and customize for our needs only to find that our manual approach with a couple open source tools was performing much better than the automated solution. This was partially due to the application we were testing, we had some particularly unique extremes in size of the application to be tested, and the automated tools simply didn’t complete scans in an amount of time that we found acceptable. If I had to do it again, I would explore some more varied options for automated testing that could be better fit for the environment. I realize the importance of an automated solution to validate applications which do have strong security requirements, but that do require some level of security assurance.
I’ve spent the past 3.5 years working on a team where my primary responsibilities involved “application security”. Now that this era has come to an end, I’d like to share some of the initiatives and define their successes and shortcomings. Since this promises to be a pretty long post, I’m going to break it up into more digestible bits containing an initiative or two.
Dedicated application security team
When I was first hired on to the team, it was responsible for both security assurance (application security) and incident response. One of the first milestones which lead to the success of the program was the separation of the team into 2 groups with distinctly different areas of responsibility: one team did security assurance, the other did incident response. I credit this move as a success because when the responsibilities were shared, engineers were required to share their focus between the proactive mindset of security assurance, and the reactive mindset of incident response. Inevitably, certain engineers were more skilled in one or the other of the disciplines and the quality of service across the different offerings would be degraded at times. While the teams now had a different and separate structures and managers, the importance of information sharing wasn’t lost as the teams were already so used to collaborating as a single team.
If your company is serious about application security and the workload justifies it, I would suggest that a dedicated application security team be created to provide the expertise needed to address the unique challenges of a complete application security problem space. The charter and team responsibilities must be carefully selected, if the mission is too narrow certain critical of the supporting components in application architectures might be out of scope, while a mission that is too broad will reduce the amount of time engineers will be able to dedicate to appsec will be limited. If I had to do it over again, I would certainly narrow the scope of the team to only include application security and associated disciplines (training, threat modeling, secure design and architecture consulting, security assessments, code reviews and analysis – SCA).
Even though there have already been some great posts (Rafal Los, Gunter Ollmann, RSnake, John Steven…and again John Steven) I felt like I wanted to offer my commentary and hopefully convince some of you to attend the next OWASP event close to you. Quick disclaimer: I helped Doug, Rex, Mark, Kate and the rest of the volunteers at the conference, so I might be a little bias.
First – if you’re going to host a conference in DC, there’s really no better venue than the DC Convention Center. This is really a top notch venue in the center of DC that is built for conferences. It’s metro accessible and the conference services (food, beverages, A/V, wireless, etc) are top notch. I’m not saying the venue makes or breaks the con, but it helps.
The speakers, technical content of the presentations, and variety of topics exceeded that of much more expensive conferences I’ve attended. Joe Jarzombek kicked things off with the keynote, David Byrne and Charles Henderson can’t filter the stupid, Jon Rose and Tom Leavey brought the drinking game with a chance of 0-day, Jeff Williams tackled the insider threat, Kevin Johnson and Tom Eston think our friends want to eat our brains, Josh Abraham brought synergy to our pen-testing tools, RSnake gave security experts happy hour chatter for the next year with the 10 least-likely and most dangerous people on the web, John Steven condensed 6 hrs of Threat Modeling training into a 45 minute talk (good thing we had him scheduled at the end of the day), and Chris Weber dazzled with unicode. Not to be outdone, the OWASP projects were equally represented with Pravir Chandra on OpenSAMM, Jeff Williams on ESAPI followed by Arshan Dabirsiaghi on the ESAPI WAF, Sebastien Deleersnyder and Fabio Cerullo brought the OWASP tools together to deploy web applications, Matt Tesauro did his thing with the Live CD, Dr. Boaz Gelbord touched on security spending, and of course, who could forget Dave Wichers at the OWASP Top 10 2010 RC1! The conference also features 2 panels, the Federal CISO panel with Ray Letteer (USMC), Timothy Ruland (US Census), Richard Smith (TSA), and lead by Matt Fisher. The SDLC Panel features Michael Craigue (Dell), Dan Cornell (Denim), Dennis Hurst (HP), Joey Peloquin (FishNet), David Rook (Realex), Keith Turpin (Boeing), and lead by Pravir Chandra. The conference also featured a CTF running the new OWASP CTF project and hosted by Martin Knobloch.
For those of us who have been in the security industry for a few years, these conferences are a great chance to catch up with old friends and make new acquaintances. It was great to see familiar faces like Tom Brennan, @grecs, Ken Van Wyk, Matt Fisher, Dinis Cruz, Jon Rose, John Steven, Lee Anne Hart, Gracie Daniel, Jon McCarty, Jeremy Long, Rob Fuller, Jack Mannino, Rex Booth, Mark Bristow, Doug Wilson and others. At the same time, it was great to make new contacts like Josh Feinblum, Pravir Chandra, Robert Hansen, Matt Tesauro, Arshan Dabirsiaghi, Rafal Los, Jeff Williams, and hell, even the great Dan Kaminski made an appearance!
Just like any good conference, the awards and closing remarks held on the last day were full of thanks, toys, flying vendor squishy balls, foam rockets (courtesy of Tom Brennan), cheers, and clapping. It was truely a great way to wrap up a top notch con!
Finally, and although it’s been done many times already, I want to take a second to recognize all those OWASPers and DCers that came together to make this event what it was. I’m really copying this list verbatim from the last page of the conference booklet:
Rex Booth, Mark Bristow, Doug Wilson, and Kate Hartmann who provided the leadership without which this conference wouldn’t have come together.
The OWASP Board – Jeff Williams, Dinis Cruz, Dave Wichers, Tom Brennan, and Sebastien Deleersnyder who gave us “carte blanche” and trusted us to get this conference done.
The lead volunteers Barry Austin, Angel Contreras, Josh Feinblum, Lee Anne Hart, Martin Knobloch, Jeremy Long, Jon Rose, David Sachdev, Mike Smith, and myself.
The red shirt people of which there are way too many to name…THANK YOU!
And all those who spoke at or attended the conference!
So if you get the chance to attend a future OWASP event, or if you haven’t checked out your local chapter, hopefully this blog post and the others I mentioned in the first paragraph will shed the spotlight on the OWASP organization and how WE work to improve application security worldwide.
In this time of shrinking budgets, reduced staff, and other various financial constraints, security departments world wide are looking for ways to justify the expense of a well rounded application security program. Jeremiah Grossman (WhiteHat Security / OWASP) and Jeff Williams (Aspect Security / OWASP) have collaborated on a fantastic article that is sure to get security engineers, developers, and managers alike lining up at their executives’ door armed with some great arguments in support of creating or expanding their application security programs.
While I fully support and endorse the content of the article, I can already see the responses from some executives:
“Wow, this is really great, and I fully buy into it, but as it stands, there is REALLY no money extra money in the budget.”
What now? Hopefully, there are employees in your company who are responsible for managing servers, the domain, the networks/firewalls/IDS/IPS, and development efforts; but most importantly, you, who is concerned about the state of application security. Why not use some of the best resources from those groups to form an application security working group? I’ve found that no matter what the company, there are ALWAYS people that are interested in being able to devote some time to learning about how those sneaky hackers are able to break into so many systems. Furthermore, what (good) technologist wouldn’t be interested in expanding their skills?
I’m sure that you’ll encounter some resistance from employees and management alike, after all, time is valuable, but remember, Jeremiah and Jeff have already given you all the ammo you need; use it wisely.
Now that you’ve identified folks who are willing to donate some of their valuable time to this effort, your first step will be educating them about the various activities and processes involved with a complete application security program. Here’s a quick reminder:
Design and architecture reviews and threat models where you work to identify flaws in the application design and architecture as well as imposing security requirements.
Secure coding standards where you work to ensure that developers implement features such as input validation, authentication, authorization, and avoid creating flaws in the business logic as they move through implementation.
Code reviews where through either automated or manual means, developers review their code with a focus on identifying any security issues.
Testing and validation where you ensure that all requirements have been implemented according to design specs.
Deployment and maintenance where you ensure that logs are being monitored, system patches are being kept up to date, and 3rd party libraries used in the application are kept up to date.
The chances are good that some of the folks that have chosen to join the security working group already have some valuable input to provide. For example, the sysadmins might have expertise in Apache, host configuration management, or IPTables. The network folks might have been exposed to web application firewall technology. The developers might have some great security specific libraries that they’ve used on past projects. All of this knowledge is valuable and already at your finger tips. For other topics, search around your social network, if you’re on Twitter, checkout @securitytwits, if you’re a member of OWASP, attend the meetings, talk to folks, try to get an idea of their expertise. Many of the seasoned security professionals who attend these meeting would be happy to come in and present at one of your groups meeting.
So, get out there, talk to people both within and outside of your company, not only will you expand your social network and improve your companies application security program, but you’ll also be giving new opportunities to those who join the security working group.
In business, you typically find that a critical question associated with investing in security is “How much will it cost if we don’t do X?”. It’s no secret that it’s still very difficult to justify a web application security program to those who approve the dollars. Sure, for breaches involving financial loss or financial information theft, justification for prevention can leverage the cost of the financial loss. But consider an XSS vulnerability in a webmail program or a CSRF vulnerability in the social network du jour. How do you determine the cost of an attacker SPAMMING your users or performing unauthorized changes to personal data in order to justify your program? You need to scare executives enough to care…any they only speak numbers.
Numbers can be obtained using existing sources of knowledge including findings from existing security mitigation, entries from web server access logs, and Web Application Firewall (WAF) deployments. There are several methods of extracting and consuming this data such as Abnormally Detection Systems (ADS), vendor provided data viewers, or custom scripts. Each organization will have different needs and available budget, so careful consideration must be taken when considering the various options for collecting and analyzing the data.
Metrics from Existing Security Mitigation
I am making the assumption that there is already some form of Quality Assurance (QA) for security taking place on the web applications your company produces. Even if your QA program is a single person with a knack for running some tools, the findings from these efforts can be tracked, labeled, and categorized. Over time, these data points can be leveraged to justify additional investments in staff, training, tools, or outreach efforts. The Common Vulnerabilities and Exposures (CVE) and National Vulnerability Database (NVD) have developed a common way of defining, tracking, and measuring vulnerabilities. The following are a few examples of the kinds of metrics which can be obtained from CVE/NVD style vulnerability tracking:
Vulnerability risk scoring
Providing a consistent rating for risk for vulnerabilities which is based on a repeatable scoring of business and technical vectors will provide a metric from which prioritization for mitigation can be derived. Additionally, integration of this score into vulnerability reporting will provide an easy way to to quickly communicate the importance of any given issue. CVE/NVD scoring.
Vulnerability classification distribution
The CVE/NVD methodology defines categories in which to classify vulnerabilities called the Common Weakness Enumeration (CWE). These categories include authentication issues, XSS, path traversal, and design errors amongst others. The metrics obtained from this data will provide justification for security development for the design and development teams.
Vulnerability totals per project or development team
Being able to track vulnerability totals across projects or development teams will provide additional leverage for targeted training for development staff. Imagine being able to walk into the Director’s office and justifying the cost of bringing in an expert to discuss secure authentication and authorization in web systems through hard numbers demonstrating the needs of the business.
Metrics from Web Server Logs
Web server logs contain a wealth of information relating to the sort of attacks taking place across your company’s web properties. The key to making use of this data is consistent monitoring and tracking of the chosen data points. Some key data elements worth tracking include:
Source IP address
The source IP address of the HTTP request can be matched against an DNSBL database or a private IP blacklist in order to track use of the application by known threat agents.
HTTP methods RFC-2616 defines a finite number of HTTP methods allowed in the HTTP protocol. Any deviation from the defined standards could adversely impact misconfigured webserver and should be considered an offensive action.
Requested URL combined with HTTP response status codes
Being able to track the number of requests against URLs will help in determining the presence of rogue files on webservers and help identify scanning for common hidden directories. Using the HTTP response codes will help identify instances where the requested resource returned status codes other then the 200 and 300 series.
Query string parameters
Query string parameters are the most prevalent location for input based attacks. An easy way of identifying offensive name/value pairs is to use the regular expressions in the relevant mod_security rules (see mod_security rule file modsecurity_crs_40_generic_attacks.conf).
Client browser strings
The client browser string identifies the type of browser and it’s capabilities to the web server. This information can be instrumental in identifying automated attacks against your company’s web properties. There is a lot of knowledge which can be gained from tracking bad browser strings, but this certainly goes beyond the scope of this post.
Metrics from WAF
Similar to obtaining metrics from web server logs, metrics obtained from Web Application Firewalls (WAFs) can be used to identify attacks taking place against your company’s web applications. I personally recommend using mod_security as it’s free, flexible, and powerful. Initial deployments of mod_security should be in log only mode until it has been determined that the enabled rules are not prohibiting legitimate users from using the application and that the rules are enforcing the intended restrictions.
Of course, I understand that getting this information together might present somewhat of a challenge, but the data points obtained from this metricizing excercise will provide irrefutable hard data on existing security controls as well as areas for improvement. I’m currently going through this excercise, so over the next couple months, I’ll hopefully be able to provide some numbers to demonstrate the effectivness of my efforts.
I’ve spent the past 3.5 years working on a team where my primary responsibilities involved “application security”. Now that this era has come to an end, I’d like to share some of the initiatives and define their successes and shortcomings. This is part 5 of 5 so please be sure to read parts 1, 2, 3, and 4.
Even though there have already been some great posts (
In business, you typically find that a critical question associated with investing in security is “How much will it cost if we don’t do X?”. It’s no secret that it’s still very difficult to justify a web application security program to those who approve the dollars. Sure, for breaches involving financial loss or financial information theft, justification for prevention can leverage the cost of the financial loss. But consider an 
DandyID
Friendfeed
claimID
Disqus
Facebook
Twitter
myOpenID
Google Profiles
Linkedin