Measuring Your INFOSEC

Metrics are reflective of a mature information security program. When you have policies and processes in place and you are actually measuring their effectiveness then you are able to make objective decisions about what you can do to improve the effectiveness of the program. If a senior member of the organization asks how secure are we, would you have an answer?  What if the question were am I more secure today than I was yesterday?  How do I know if I'm spending money on securely effectively?  What's my security posture relative to my peers?  These are all questions that cannot be answered within policies and procedures alone. In instances where an information security program must take corrective action, it will often be met with resistance from detractors. In order to make a substantive argument in your favor, you need objective numbers that speak in a way that is understood throughout the organization.

Effective Metrics Tell A Story 

From that story the organization should be able to derive a lesson that born from conclusions that were found in analysis of the data. Good metrics demand a story. The story reveals a lesson. learning from the conclusions drawn by analysis of the data. Like any good story, you have to know your audience and select your theme to connect with their frame of reference." Metrics should be able to fit into one of three categories that support different levels of organization decision making:

• Strategic:  speak to overarching corporate objectives
• Managerial: speak to initiatives that are in place to meet the corporate objectives 
• Operational: low level details that speak to the day-to-day activities of the corporation

The characteristics of a good metric in information security are:

• Predictive: provides the ability to forecast security outcomes with a high correlation between the metric and the outcome. This type of metric allows the organization to look reliably into the future based on historical trends. These metrics can be used to stall, limit, or prevent incidents before the occur.  
• Relevant: focused on the specific needs of the organization and provide actionable data that steers direction of information security.  The options for metrics are boundless so the selected metrics must mean something that will enhance knowledge.  
• Actionable: provides the organization metrics that the organization has the ability to influence.  An effective metric is not one that the organization is powerless to do anything about to exercise some control.  
• Accurate: metrics may or may not need pinpoint precision to be effective.  This characteristic is dependent on the requirements of it by its recipients.  However, the level of accuracy needs to be reflected in its repeatability.  
• Independent: reflect the metric's ability to be resistant to manipulation by outside forces.  Shows the trustworthiness and integrity of the metric.
• Tangible: metrics should be quantitative.  Ratings such as high, medium, and low should not be used to convey any information in security metrics.  The metrics should fall into either a state or rate.  You can measure the current position of something or the direction/rate of change of something.  
• Repeatable: metrics should be easily gathered and updated via automated means as much as possible.  This allows the collection to have the same impact and accuracy throughout the history of the metric's collection.
• Cost: the cost of the metric does not necessarily mean that the metric needs to be cheap.  But it's value does need to outweigh the cost.  Most metrics are cheap to collect but in instances where a survey may need to be conducted and analyzed there may be a heavier cost to bear.  

Sample Metrics

Finding meaningful metrics for your organization are really dependent on your unique organizational needs and objectives.  However there are a number of metrics out there that are good for a base example.  When selecting metrics define the metric name, metric description, metric purpose/objective, required data sources, frequency of measurement, units of measure, and benchmark or goal. Below are some sample metrics to consider.

Time taken to Remediate Security Events

The amount of time between the detection of a security incident/event  and when it has been marked as closed/remediated.  This metric shows how efficient your security operation center may be operating.  This could be considered a management and operational metric.

Mean Time to Detection

The amount of time between the actual security incident and when it was detected/reported.  This metric shows how effective your detection and reporting mechanisms are.  This would be considered a management and operational metric.

Number of high/medium/low risks unresolved

The number of risks identified in the organization that remain unaddressed.  This metric measures the overall effectiveness of the risk management program.  This metric covers strategic, management, and operational.

Percentage of assets in risk management program

A useful metric as you are rolling out your program the tracks its progression.  This would be considered a strategic and management metric.

Patch Latency

Tracks the amount of time that has passed between the availability of a patch and it's implementation. The larger the amount of time that has elapsed the more susceptible the environment may be to attack. Patch latency measures the effectiveness of a patch management program. This would fall into the management and operational category.

Percentage of Change Authorizations

Measures the effectiveness of the change management program. A metric that insures that personnel are following the change management process that has been defined.  This could be an expensive metric to track as it may be difficult to discover qualifying changes throughout the asset inventory.  This would fall into the management and operational category.

Percentage  of Employees Who Have Completed Security Training

Measures the coverage of your security training program.  This would fall into management and operational metrics.

Percentage of Systems That Have Been Validated for Compliance

When an organization is undergoing compliance efforts (PCI/SOX/HIPAA) this is a metric that tracks its progression.  This metric could have financial implications as well as operational.  This would fall into strategic, management, and operational metrics.

Implementation of a measurement program can be a daunting task. In order to make sure that you are measuring pertinent information and that it continues to be worthwhile, you should take a systems approach to implementation.  State the requirements, have organizational backing, design and develop the measurement system and the metrics themselves, test the collection and dissemination mechanisms, execute the system, and monitor its effectiveness. A measurement program will arm a security manager with meaningful data that facilitates decision making that's not based in FUD (fear, uncertainty, doubt).

Buiding an Enterprise Risk Management Program

In this post, I outlined some high level steps to begin an Enterprise Information Security Program.  Once you have established policies and implemented the basic steps of an Information Security program, the next phase is to develop its capabilities and further align the effort with business objectives. The sign of a mature Information Security program is the integration of Risk Management. Where policies are the essential foundation of the program, risk management is the rudder that steers the ship. Policies, procedures, guidelines, and baselines provide the building blocks of the security program's direction where implementation of Risk Management identifies the impact to the organization if security controls fail.  

First let's define risk. It is common in the information security community to interchange the words risk and threat. These are in fact two different things.  An example is that you buy a lock for your door to prevent the threat of someone just walking into your home. While the risk of someone walking into your home is much lower in remote areas verse urban areas due to differences in population density. So people are often more likely to leave their doors unlocked in remote areas because while the threats are the same, the likelihood of occurrence is much lower. A more formal definition of risk is the "probable frequency and probable magnitude of future loss of confidentiality, integrity, availability, or accountability."

So while building your security program you've identified all of your information resources and established patch and vulnerability management policies. Upon your initial scans you've discovered 230 vulnerabilities and 125 missing patches. With your limited resources how do you decide what to patch first? What do you do if a critical vulnerability is discovered on a system that cannot be patched for business reasons? How do you decide if an expensive security control should be implemented? These are the kinds of questions a developed risk management program can answer by providing measurable business context.  

As with building a security program in general, the number one advice I can offer is to start small. Out of the gate do not try to building complex qualitative risk models that will bogged down in a quagmire of mathematical models that your organization most likely does not have the necessary historical data to build in the first place. Do not try to apply a risk category to every single component of your information system environment on day one.  Refrain from attempting to identify every single risk scenario before you've identified all of the critical risks. Do not attempt to achieve a "perfect" environment early on. Quick demonstrable wins and early successes will lead to a Risk Management program that has real, positive impact.  

One of the initial tasks in building an Information security program was to build an inventory of information assets. This is a key step in establishing a Risk Management Program as well. Risk Management requires an extra degree of detail in order to establish an accurate risk profile. Risk management requires knowledge that includes regulatory compliance details, housing of any personally identifiable information, credit card processing, etc. The process of discovering all of these types of details is known as risk profiling. This information can be gathered via interviewing or having the resource owner complete a questionnaire. The resource owners provide the details but the risk manager is assigned the task of assigning the overall sensitivity of the resource.  

The answers are fed into a predefined sensitivity scale that will assign low-medium-high sensitivity ratings. The risk sensitivity scales provide context so that risk evaluators are able to readily understand what makes one resource more sensitive than another. An official scale provides a universal metric across the enterprise that everyone understands is the standard.  

Your questionnaires have been completed and you have defined what it means to have a sensitivity of low, medium, and high.  Now you can begin creating a security profile of your information system resources.  Here is where to apply the approach of making these initial stages as easy as possible.  Start by profiling large chunks of the environment and then gradually increasing focus. If your environment consists of multiple datacenters, provide a profile by datacenter. If you outsource aspects of your information system, profile that environment vs what is locally managed. Provide a profile of a DMZ compared to internal environment. Whatever makes sense for your environment, create chunks of resources to provide a sensitivity rating to and gradually narrow your focus.  

Now that you've got your security profiles and have defined your most critical/sensitive assets, you can begin to leverage your vulnerability management processes that were put in place when building your security program. The vulnerability management  process will assign a weight to each vulnerability that is based on each assets sensitivity. Optimally your policy will outline things like the amount of time to remediate vulnerabilities based on sensitivity, processes to accept risks in instances they cannot be fixed, and processes to validate vulnerability fixes.  

An effective risk management program permeates throughout the organization. Risk liaisons are placed in each department to evaluate if their initiatives have adverse impacts to the overall risk of the organization. Risks are mapped to business objectives. Information security is represented at the enterprise risk committee in reporting to senior management risks to be prioritized along with other organizational risk. Having the  risk program at the executive level enables the program to be aligned to business objectives providing focus in support of the mission. 

Building An Enterprise Information Security Program

In the modern day, it is difficult to turn on the news, pick up a paper, or mill about everyday life without being exposed to the details of the latest hack that has disclosed personal information, tarnished a reputation, or pilfered currency. The threat landscape has lead to organizations looking inward to assess their current state and realize that they may not have the means or processes to deal with a hacking incident. What do we do if we’re being hacked? What if we’re already hacked? How would we know? What would we do? Seeking the answers to these types of questions typically lead to the establishment of a security program that puts in place processes and procedures that guide organizations to make the right decisions in the face of active attackers.

But where to begin? It can be a long, arduous road from knowing you need answers to actually having answers. In order to be successful in the establishment of an enterprise security program, it is necessary to define clear goals. In order for goals to be effective they must have a clear definition and be assigned deadlines. Deadlines must be practical yet challenging so that the success of the program is obtainable and measurable. The goals of a security program should be unique to your organization and reflect challenges specific to you. The outlined goals should lend themselves to the creation of a mission statement that functions as the guiding light of the security program.

After the goals have been articulated, the next step is to obtain official buy in from executive management. Often the buy in is presented in the form of a charter or something similar. It is essential that formal recognition of the security program is granted so that they are effectively empowered to meet their goals. This empowerment optimally places the security program in a branch of an organizational tree that is independent of the rest.

So you’ve set your goals, the executive management has sung your praises…..now what? It is untenable to attempt to make broad, sweeping changes to the organization effectively. The best approach is to follow the philosophy of crawl, walk, run. As with most things, it is best to try to set yourself up for success with quick wins. Identify and attack some of the low-hanging fruit within the company. An effective security program cannot begin without a thorough understanding of the systems, software, and connections. A good place to begin is defining the system boundary by gathering a detailed inventory of systems, understanding their function, and gathering a high level understanding of the most sensitive systems. This process can be completed with a mix of technologies that scan your systems, and interviewing your constituents to understand what is important to them.

The interview process will serve to provide two functions. Not only do you gather more details about the environment, but it also provides an opportunity to enhance the relationship between the security team and the other organizational groups. Early on it is imperative to evangelize the initiative. The message to the organization should relay the mission and goals. The personnel should also understand that security is not something that is going to be happening to them. It is meant to enhance awareness and increase effectiveness (note: effectiveness may need to be redefined to mean something other than as fast as possible). While conducting evangelization you may find that some of your most staunch supporters and enablers reside within the user base. Leveraging them may prove to be your most valuable resource.

Now that you’ve identified systems and interconnections, gathered an understanding of organizational processes, and weighed the critical assets, policies and procedures can begin to be formulated. This is another area to seek quick wins. Seek to find policies that may already in place and just haven’t been formalized yet. You don’t want to introduce any drastic changes to the community culture right out of the gate. This sort of action will not endear you to the user base. A good place to look would be the Acceptable Use Policy or Account Management. Once you have achieved some easy wins, seek to define policies and procedures that are more mature and are integrated throughout the enterprise. Policies and procedures should be woven throughout the software development lifecycle, operations, desktop support, and physical security. This integration ensures that security is embedded throughout the organization’s culture.

Now you’ve entered the walk portion of the program. Enter automation. It is important to establish strong policy prior to automation because they will shape the requirements for the solutions you put in place. Automation generally encompasses solutions like patch management, vulnerability management, and configuration management. Entire textbooks have been written on each topic but I will touch on a couple of key points for each.

When you were crawling you gathered a comprehensive list of systems. Now that you’re ready to walk you’re able to gather the current patch level of each individual system throughout the enterprise. You should know what’s up to date and what’s lagging. You should be in a place where you should strive to make sure that everything is patched. You are able to prioritize what should be patched immediately and what could wait until a later date. While the goal is to make sure everything is patched all the time, the cold, stark reality is that this is not always possible. Systems that cannot be patched should be documented and tracked.

For vulnerability management the goal should be to find all of the vulnerabilities in the enterprise. This is obtainable but it requires different automated scan types. Authenticated, network, external and passive scanning are all tools that can be deployed to detect vulnerabilities throughout the enterprise. Once a vulnerability is detected the process to remediate as defined in your vulnerability management policy initiates. The vulnerability should be assigned a score based on its criticality. This score dictates the immediacy of remediation or acceptance of the risk.

An effective configuration management program is a sure sign of a mature security program. The road to establishing a streamlined configuration management process can be daunting as you are inundated with terms such as CMDB (configuration management database), configuration items, and artifacts. As with the security program in general, a proper configuration management program begins with starting small and scaling out with success. It can begin with small wins such as establishing change control for a firewall or creating a gold workstation image. Configuration management begins in the walk phase but extends well into the run phase of the security program. 

Once you’ve mastered walking, you are ready to run with the establishment of correlation engines, threat hunting, incident detection and response, and digital forensics. By utilizing continuous monitoring, you will be able to dynamically detect when bad things are happening in your environment, and launch processes that quarantine the attack and allow for further investigation limiting impact as much as possible. This phase has close ties with lawyers, insurance and law enforcement so that they are able to quickly act in the event of a breach. A security program that is running is able to measure how fast and efficiently it’s running and provide quantitative data points that gives feedback on how it can improve.

Whether you’re a team of one or a part of an organization that has substantial resources that can back a full fledged security program with specialized teams, the process of beginning an effective program begins with starting with small, visible wins that garner willing and enthusiastic participation throughout the organization.  Identify low-hanging fruit to be plucked to gain momentum and ride successes until you've met your goals for the program and progressed into a mature enterprise.