Thoughts on an AppSec program – The Team

Tuesday, December 29th, 2009

I’ve spent the past 3.5 years working on a team where my primary responsibilities involved “application security”. Now that this era has come to an end, I’d like to share some of the initiatives and define their successes and shortcomings. Since this promises to be a pretty long post, I’m going to break it up into more digestible bits containing an initiative or two.

Dedicated application security team
When I was first hired on to the team, it was responsible for both security assurance (application security) and incident response. One of the first milestones which lead to the success of the program was the separation of the team into 2 groups with distinctly different areas of responsibility: one team did security assurance, the other did incident response. I credit this move as a success because when the responsibilities were shared, engineers were required to share their focus between the proactive mindset of security assurance, and the reactive mindset of incident response. Inevitably, certain engineers were more skilled in one or the other of the disciplines and the quality of service across the different offerings would be degraded at times. While the teams now had a different and separate structures and managers, the importance of information sharing wasn’t lost as the teams were already so used to collaborating as a single team.

If your company is serious about application security and the workload justifies it, I would suggest that a dedicated application security team be created to provide the expertise needed to address the unique challenges of a complete application security problem space. The charter and team responsibilities must be carefully selected, if the mission is too narrow certain critical of the supporting components in application architectures might be out of scope, while a mission that is too broad will reduce the amount of time engineers will be able to dedicate to appsec will be limited. If I had to do it over again, I would certainly narrow the scope of the team to only include application security and associated disciplines (training, threat modeling, secure design and architecture consulting, security assessments, code reviews and analysis – SCA).

Building the Security Team

Monday, February 23rd, 2009

hackers_aheadMubix recently had a blog post where he detailed an approach for building a team to defend the infrastructure in a hacking contest. This got me thinking about how I might recommend going about building an enterprise security team.

Know that you are the right person for the job – Building a team of security professionals requires a strong understanding of many different areas of specialization in the computer field. You must not only have the people and business skills typically required in any other leadership role, but you must also have a deep understanding of technologies (both security and other) and how they inter operate to ensure reasonable protections against disasters. I’m not saying that you must be able to configure the PIX firewall, but you should understand it’s purpose, differences in firewall technologies, their challenges and how they play into the overall security architecture.

Know your mission – the first item of business is to determine what the security team is going to provide in terms of services for the enterprise. What are the problems that you were asked to solve by creating a security team? Some examples might include PCI/SOX/HIPAA compliance. Perhaps a savvy CEO has asked you to build enterprise security guidance in the form of policies, standards and baselines. Maybe you’ve inherited the network firewall ACL management, penetration testing, enterprise authentication management and product security design services to better serve the company’s need for IT Security.

Know your infrastructure – now that you’ve determined all aspects of the enterprise that are going to be under your control, it’s time to audit them and determine whether any security controls are already in place. Remember, you’re going to need to build the team to support and enhance these controls, this exercise will help you determine a required skill set. This is also the point at which you’re going to audit the parts of the enterprise you’re being tasked to protect. It is important to involve all stakeholders so that a single set of requirements can be derived for each of the security services. For example, if one of your tasks is to evaluate and standardize security controls over all production hosts, it’s important to determine the operational needs of the system administrators who manage these hosts and the developers who write the applications running on these hosts.

Team leads – Here’s where the size of the enterprise you’ve been tasked with protecting comes in. If you’re a small company, your “leadership team” might be 3 really talented individuals with whom you plan on dividing the responsibilities. On the other end of the spectrum, your leadership team might be 5 managers with teams of engineers to accomplish the work in a large enterprise environment. The people in your leadership team (whether the enterprise is big or small) must be colleagues that you trust will not only be able to help you execute your vision, but will also be able to provide valuable input to the plan. The leadership team must all share the same general understanding of enterprise security architecture components while maintaining a strong subject matter expertise in the specific technology areas which they are directly responsible for. For example, I would expect that your network team leader could not only lead the engineers on the team, but also be able to step up to the keyboard and fill an engineer’s shoes.

The engineering teams – Provided you’ve been given enough budget and are looking to secure a large enterprise, you’ll be looking to fill at least a couple vacancies on each team. The interviews should be conducted such that the candidate is not only demonstrating knowledge in the areas directly associated with the position they’re applying for but also in the other areas of responsibility for the security team. In addition to the usual traits you look for in a good candidate (good resume, demonstrated technical knowledge, good writing/communication skills, overall fun person), you are also looking for candidates who are not only experts in their fields, but possess a broad knowledge of technology in general. Don’t be afraid to conduct several interviews with several different people and keep the interviews challenging by presenting a real life security concern and have the candidate and interviewers solve it as a team.

Solve the problem on paper – Now that you have your awesome security team in place, it’s time to get them all together and lay out the problems you’ve been tasked with solving. This is when your vision for solving the problems gets mulled over and revised by your trusted colleagues. This exercise should not only add specific technology solutions to the plan, but will also allow everyone the opportunity to apply their knowledge and be invested in the final solutions.

Measure it – In any enterprise, progress must be measured to ensure some form of Return on Investment (ROI). Costs associated with technology purchases, staff salaries, and other activities will have to be be justified by demonstrating that the cost provides a reasonable assurance of protection against threats. Additionally, in the security world, a lot of useful planning information can be derived from tracking security metrics. For example, if the security assurance team is noticing a high number of input based vulnerabilities during their application assessments, perhaps some investment in secure development training for the dev teams is in order. Knowing which metrics to track as the plan is being built will facilitate the collection and tracking of the numbers.

I do realize this is pretty high level and lacking in specifics, but I wanted to be sure I highlighted the importance of shared technical security knowledge as a key factor in the success of the security plan and subsequent implementation. Furthermore, I beleieve that the same planning and executing techniques can be applied from the executive goals to the engineers’ individual goals and when given the chance to build those goals together, the likelyhood of a good plan being build and implemented increases greatly.