From: ComputerWeekly.com

How can businesses measure the effectiveness of their IT security teams to ensure they are getting value?

Make sure security information is available at the right level, writes Vladimir Jirasek, director of communications, CSA UK & Ireland and project lead CAMM.

The question of measuring the value of security in an organisation has not been fully answered since the creation of information security discipline. And this fact is, in my opinion, one of the reasons security teams find it difficult to convince business to invest in security, except perhaps immediately after an incident.

The management of any organisation is typically good at managing based on information (metrics, KPIs, scorecards, traffic lights and others) available to them. However, the information needs to be at an appropriate level. Consider a CEO. Is s/he really interested in a number of vulnerabilities in all IT systems? Or would he be more interested in knowing how much exposure (in monetary terms) these vulnerabilities present?

There should be three types of security risk metrics in an organisation (top to bottom): a) Monetary-based risk exposure for an organisation, b) policy compliance scorecard, and c) detailed technology and procedural metrics. This systems needs to be connected from top down and bottom up as outputs from the bottom feed into the upper level metrics.

Let’s have a look at these in bit more detail. At the top level, I believe there is a need to create a standardised metric similar to Basel II ‘Value at risk’ (VaR), adapted for information security. This would inform CEOs of monetary exposure for all company’s assets due to missing and inadequate security controls. Such a metric could be used to build a business case for security investments: “Is the investment going to lower the VaR by more then it will cost?”.

At second level, the compliance to security policy needs to be measured. If there are 20 high level policy statements, use the scorecard metric to show company compliance with each one.

Finally, at the lowest level detailed procedural and technology metrics are needed. That is where metrics such as “a number of critical vulnerabilities” or “number of level 5 application errors” are appropriate. These are typically used in day to day operations.

In summary, metrics are needed for strategic, tactical and operational decisions. We, information security community, need to work on these together.

Use local, basic security as well as up-to-date procured technologies, says John Walker, London Chapter ISACA Security Advisory Group and director of communications common assurance maturity model.

Today’s technological business environments can represent a labyrinth of interconnected systems, infrastructures, applications, and services which can be as complex as it gets – and with the awareness, that such complexity can only grow, then we should expect some brain swell to be on the operational support menu.

When we come to the security expectations which underpin such complex environments, these can be equally challenging insofar as the modern day security technologies and applications must accommodate leading-edge and legacy systems at the same time. Such operational needs must also accommodate a resilient level of security to satisfy those standards we have all come to love and embrace, such as DPA, PCI-DSS, and SOX. To accomplish this in the majority of cases high investments have been made in complex Security Commercial of the Shelf (COTS) systems, and applications to drive the security profile, and to maximise the overall level of protection – job done, right!

One problem that seems to be a regular occurrence on the consultative radar is what may only be described as over-reliance on procured technologies, which are expected to deliver a kill-bill silver bullet, resolving all security issues known to man. The second observation on the radar represents the dismissal of Best and Accepted Security Practices at the lower end of the security spectrum – please allow me to expand.

Whilst current-day, leading-edge security technologies are of course a must in most operational environments, it is nevertheless equally important to accommodate the security mission with the basics of security in the guise of standards, secure builds, accompanied with pragmatic, workable policies, which underpin the overall delivery of security. As such it is also highly recommend that familiarity is maintained with the good old command-line interfaces to facilitate operational agility, to facilitate timely extraction of information; or to apply some on the fly, authorised change. It may also be that, if a back-to-basics approach is maintained by elements of the security team, it just may be possible to leverage some owned security technologies to accommodate a business security requirement at zero cost. For example, I have written a very simple forensics extraction batch file, which upon execution will request and acquire around 90 security queries, writing them into a file (report) which can be generated between 2 to 60 seconds flat (dependent on connectivity, and environment).

It may be that my approach is not a best fit for all purposes. However, over the years, when faced with difficult demands of audit reports, or operational expectations to deliver a solution with zero budgets, I have found that maintained familiarity with the back-to-basics approach can pay dividends, and at least for a short time, and can place the security professionals in a good light.

Decide what KPIs will show value for your particular organisation, urges Dani Briscoe, research services manager at The Corporate IT Forum.

Measuring the value of a team that doesn’t directly link to the bottom line is very intangible. Some members have likened this to asking a finance department to show value for their services – in pure profit terms, it’s hard to justify.

Whilst this is still a big challenge for The Forum’s members, it is occupying teams currently with plans to put benchmarks and measures in place. With all investment still under a high level of scrutiny, having the ability to show value is exceedingly desirable.

A recent Q&A exchange on the topic highlighted clearly that there is no “one size fits all” with security teams. There are variable team sizes, scope and responsibilities across industries and even across individual organisations within a group. With this inconsistent structure it’s difficult to establish a baseline against which to compare a team’s performance and effectiveness. In addition to measuring the value of a team, whose contribution is quantitatively intangible, measuring a success – where only failures are easy to count – is hard.

Members are currently involved in determining and implementing KPIs that show value and progress. Just over 60% did have some measure in place and this varied from showing alignment to legislation and regulations to meeting and achieving SLAs agreed with the business. Value is demonstrated through the reduction in security issues or failed changes; alignment to external compliance and audit. One head of IT from the food distribution sector commented “We measure success in terms of compliance to the external audit and PCIDSS requirements. We will also demonstrate through the reduction in security issues and failed changes, etc.”

One commonly used tool is the annual satisfaction survey of internal customers, including various key stakeholders, across the business. This measures the quality of specific information security services delivered such as good best practice sharing via policies and standards and exploring how to benchmark the organisation’s ability to react to security issues.

Corporate IT Forum members are able to participate in both formal and informal group benchmarks on an annual basis. One informal measure covers eCrime with which members can monitor their own performance against frameworks, standards and policies in place through to breaches, risks recognised and mitigated against a known group of peer organisations. A more formal benchmark is available through the continuous performance improvement (CPI) model.

Set achievable targets to derive effective KPIs, advises Adrian Davis, ISF principal research analyst.

First, let’s remember what effectiveness means: it means doing the right things and setting the right targets to achieve an overall goal. ISF research on the Role of Information Security in the Enterprise has emphasised that the business and information security must work together to set goals and manage expectations. Without such teamwork, effectiveness becomes irrelevant.

The goal should be framed by what can be controlled and what cannot. Goals such as ‘no attacks’ or ‘100% security’ are unrealistic – but a goal of ‘maintaining and enhancing our current security posture’ is. The goal can then be broken down into a small number of key performance indicators or strategic milestones to be tracked, e.g. benchmarking results and budget performance.

ISF research on metrics and measurement has uncovered a significant number of lower level, tactical and operational measures that can be tracked and how management tools can be used to aggregate, and report these measures to various audiences. Management tools promote a structured approach to capturing and trending performance. A balanced scorecard, for example, examines financial, process, learning and customer perspectives. Under the financial perspective, ROI for projects, hours booked to projects or consultancy, cost of breach/incident resolution can be measured; the process perspective can include average time to resolve incidents, productivity increases, audit findings and BCP/DR measures. The learning perspective can include measures such as effect of awareness programmes, increases in information security qualifications, business-level certifications, whilst the customer perspective can cover the perception of information security and its value: customer satisfaction, helpdesk queries resolved – and average time taken.

Every business has a different model and measures; the ISF view is that information security professionals need to work with the business to jointly set the desired goals. A joint understanding will enable a valuable contribution to the organisation.

IT security teams must focus on threat management and execute threat containment, says Alessandro Moretti, volunteer member from Switzerland of the (ISC)2 Board of Directors; and a senior risk and security executive in financial services.

Measuring the effectiveness of a team, person or process always relies on relative objectivity. Typically, a consumer of the production process tends to have a different expectation vis a vis the producer of the process. This applies to IT security teams too, and therefore measurement should be applied to gauge personnel and team performance.

Measurement of effectiveness of IT security teams is often correlated to performance. For example, a criterion of measurement could be how many more exception tickets the security team closes compared to the previous year. If the team closed 10 per cent more exception tickets than the previous year, then management may argue that the IT security team is effective.

However, it is all too easy to focus on KRIs, SLAs, KPIs and so forth, without really understanding what IT security teams should be doing effectively. Security of organisations is best managed effectively through proactive threat management. People, teams, processes, technology must all be focused on it. To illustrate threat management forms part of the core operations of anti-virus solution providers – they continuously adapt their anti-virus engines to mitigate potential threat of new viruses. Similarly, even for IT security teams, being passive and reactionary is perhaps the easy option. Measuring the number of security incidents dealt with is not the best measure of performance.

Alongside being focused on threat management, IT security teams must have the agility and ability to execute threat containment. For instance, to derive maximum value from an IT security team, an organisation could set a target that 80 per cent of resources must be focused on threat management, and the balance on threat containment and security incident management after the event has taken place. If the organisation finds that in reality the reverse is true, then clearly the security strategy is not working, which may be due to a number of reasons such as lack of automated processes or necessary skill set. Either way, the result would be a measure of effectiveness and the value it adds to the organisation.

Use metrics to satisfy business impact, organisational control and measurability, suggests John Pescatore, vice-president and distinguished analyst at Gartner.

The most effective security metrics will satisfy three closely related criteria:

• Business impact – Apply security metrics to areas that have a significant impact on important business processes.

• Organisation control – Apply metrics to areas where you can exercise control or choice.

• Measurability – Be able to obtain the data that is required for a given metric, as well as measure phenomena on which you can take action (for example, vulnerabilities and the time required to reduce them).

The most important goal of a security system is to help the organisation meet its business performance goals. This means useful security metrics have to provide indication of not just effectiveness of the security program in meeting business requirements but also efficiency – not quite return on investment, but a measurement of how much is being spent on achieving the security level demonstrated. The percentage of the IT budget (or overall corporate revenue/turnover) dedicated to information security is always an important metric to track, as is the ratio of employees to security staff.

Technical security metrics should easily translate in terms of direct impact on the enterprise in areas such as the downtime of critical systems, regulatory compliance and customer trust and should be used to drive meaningful increases in efficiency in dealing with mature threats or improved security against emerging threats. Measuring the number of malicious-code attacks or attackers, for example, is not productive because these are factors beyond your control. However, the number of vulnerabilities at any given time, and the time needed to reduce them, or the number of resources impacted by each security incident, are under the control of the IT security organisation and are meaningful metrics.

To be really effective, security metrics need to be compared against baselines which are hard to find. Industry verticals that have industry consortia or other cooperative organisations should work to shared and compare common security metrics.

Understand the cost of your organisation’s security and the value of the assets it protects, says Mike Westmacott, chair of BCS Young Professionals Information Security Group and security consultant at Information Risk Management plc

The primary objective when understanding value of security teams and systems is to determine the value of the assets that they are protecting. It is quite possible, and not at all unknown, for organisations to spend a greater amount on protecting an asset than the actual asset is worth – essentially a negative return on investment. The first step is to work out where value lies in the organisation.

Information assets are notoriously difficult to attach a pound value to, but there are some simple ways to get a fair valuation. Some types of information can be valued by the time it took to create them: the source code of a software vendor can be valued by the man days it took to produce, hence complete loss of that source code would require a complete replacement costing the around the same amount of time it took to originally create. Such intellectual property may also have a value associated with their disclosure. Imagine the same software vendor is hacked and loses its source code to a competitor – suddenly their competitive advantage has been reduced and a loss in revenue will become apparent. Exact costs here are extremely difficult to determine, but on a rather crude level one could imagine that a competitor that is as equally capable has been formed and that equates to a reduction by half of the number of clients that are won.

Other information assets have a different intrinsic value – such as information that has not been created by an organisation, but instead is collated as part of the normal business process. Of most significance in the current climate is personal information recorded about individuals, and in particular payment card data. The loss of such data may not impact the operations of an organisation (at least not immediately) but will cause severe damage to reputation with secondary and tertiary losses through recompense, fines, and an increase in payment card merchant costs levied by the card providers. Organisations should look at breach reports and identify companies that are similar in size and trade in order to value this information. Here the point at which value is measured is the point at which the cost to recover from such a breach should be greater than the cost to protect and avoid that breach in the first place.

Once these values are understood then both existing and desired security controls can be reviewed and effectiveness measured. For the purposes of this article I will group together both technical and human controls – the equipment and teams that form the IT security groups. At this point careful measurement of many aspects of the organisation are needed, and statements of operation should be created to determine what these measurements mean. Such a statement may be ‘the number of attacks detected against our web servers has increased’. The metric here could be a count of the number of web application attacks against an eCommerce system, and the response might be to launch an investigation. The key point is that attacks were detected – therefore validating the cost of the relevant security control.

This ability to detect changes is where the value of the controls lie – anything which occurs that is a change from how normal operations should run is important. Ensuring that the these attacks and breaches are detected should be the secondary objective for measuring effectiveness: If controls are in place, but the reports they provide are not interpreted correctly then they are worthless. Similarly if the controls are not configured or installed correctly then their effectiveness is massively reduced. One solution to this issue is to deploy secondary controls backed up by dedicated third party security teams. Such controls can be brought in at relatively low cost as the expense is related to analysis time and is not allocated to capital expenditure. The result of an investigation of this time would be to provide insight into the technical effectiveness of other controls and to report on issues such as malware infections, acceptable use policy violations, attack attempts, PCI-DSS compliance breaches and other incidents. The outcome would be the ability to tune existing controls better, and to make the decision about what controls may be required in the future.

Without knowing the value of information assets within your organisation it is impossible to gauge the effectiveness of security countermeasures and determine the return on investment or to budget for future requirements. Monitoring and successfully interpreting changes in the results provided by implemented security controls is the key to understanding whether they are operating correctly. If they operate correctly, and if the cost to purchase and operate them is less (significantly) than the cost of the asset they protect then they are to some measurable degree effective.

Make the business case for your organisation’s security, says Des Ward, president of the Cloud Security Alliance – UK & Ireland Chapter.

If we are to be seen as an enabler to the business, rather than a cost centre, we need to understand what constitutes value from a business perspective. It’s long been established thinking that business value occurs in one of three conditions, where the organisation can:

• do new things it couldn’t before

• stop doing things that was losing it money (eg duplication of services)

• do things more efficiently

Note that most compliance frameworks that are currently in existence will fit these conditions, with the exception of ISO/IEC-27001:2005. We have become great as an industry at pushing ‘good enough’ proscriptive security to protect against fines, but this compliance overload has meant that we’re merely painting the equivalent of the Forth Rail Bridge; with each new revision to an outdated concept of securing the network rather than working with the business to provide value to them through managing risks to their information assets.

How can we show value when we practice security rather than application of controls to manage risks? We need to understand the information assets that have a tangible value and/or impact from a Confidentiality, Integrity and Availability perspective; there is no possibility of understanding the impact and/or value, however without speaking to the business functions.

This communication has to occur from the viewpoint that we have to be here to provide value in the following ways:

• understand what the business wants to do, and provide controls to manage the risks from doing it

• understand the location of the datasets within the organisation, to reduce the impact from excessive information storage

• re-use existing technical compliance controls to provide operational benefits (eg can you use scanning tools to validate the CMDB?)

Business value can be shown, but it comes from interaction and communication; we need to identify information assets that business units trust and value, which they will then wish to keep safe.

Take a strategic view of the business’s security needs and key elements, advises Gerry O’Neill, director at Inforisca and vice-president of the Cloud Security Alliance, UK & Ireland.

On first reading this question, the tendency might be to simply launch into a standard response around the topic of security metrics – a topic with which as professionals we have been grappling for well over a decade now, and I’m not sure if we’ve reached consensus on what to measure, or how to measure it.

Instead, we should step back and take a more strategic look at the needs of the organisation, its objectives as a business or provider of services, and the elements which the security function needs to provide optimally to support those objectives. Those key elements could be summarised as:

• Be clear about the mission – What is it that the organisation is trying to achieve? And so, what should the Security function be providing as support for those business outcomes? What will help are some clear objectives around risk and security.

• Organisation – Are we optimally organised to achieve the above? What is the best organisational positioning for the security function, are the function’s representatives present on all the relevant decision-making committees? Is there effective stakeholder liaison (internal and external), and do team members have the skills and capabilities to be effective influencing decisions in all these groups?

• Skills and competences – There are many definitions of skills in this field defined by professional bodies, some with a tendency to over-focus on the technical dimension. Those most relevant to the question of effectiveness would be: intelligence, insight, understanding of business context, strategic to tactical, appropriate technical, process and influencing skills. Validation of these, and processes for development and support, are also key.

• Agreeing indicators and measurements – It is important to agree risk thresholds or appetites and targets for reduction in risk impacts. The challenges are: What can we define? – which of these are quantifiable? – are we speaking the same language? Key assets will be sources of operational and situational awareness, combined with efficiency in capturing, assessing and responding to these.

• Feedback – reporting results to Executive management and other stakeholders – providing assurance, proof that we are doing our job, and doing it well, clearly benefiting the business. And also doing this in a meaningful format/language. Dashboard charts, governance assurance and KPI reports, all have their place, but often timely verbal updates can be more effective.

• Benchmarking – one final perspective that senior management may wish to see is that of a comparative assessment, how are we performing in this dimension compared to other similar organisations? But that’s potentially a whole other topic …

In addressing the broad overall framework above, we should hopefully be able to develop an approach to ensure that we are achieving a balanced and effective service to our organisations.