Measuring Efficacy

There’s a better way to define security effectiveness than through acronyms, technobabble and hyperbole. At OpenDNS Security Labs we use three categories to define security effectiveness: coverage, accuracy, and performance.  

These are the three measures that should be calculated, analyzed, and scored when evaluating security effectiveness. Today, security effectiveness is most commonly measured by a technology’s ability to prevent known (and in some cases unknown) threats. Prevention is usually defined as stopping malicious code from running on or infecting a device. This is not an effective or scalable measure moving forward because where cybercrime is far from single-dimensional, this method of evaluation is.

• 

  Coverage

  There are two fundamental elements missing from traditional evaluations of security effectiveness in the context of coverage. The first is prevention of data leaks. Traditionally, prevention is about stopping threats from getting in. Umbrella Security Labs considers prevention in the larger context of both stopping threats from getting in and preventing data from getting out. The second is location. Solutions that claim 99% effectiveness fail to consider that members of today’s nomadic workforce lose secure coverage the moment they step outside the office.  That needs to be accounted for in measurement of coverage.

• 

  Accuracy

  Accuracy in security today is largely measured by false negatives, or threats that should have been blocked that weren’t. In order to give a complete, rounded picture of accuracy, this measure should also consider false positives and their potential impact on user experience or worker productivity. Moreover, as threats become increasingly complex and nefarious the ability to accurately contain them is becoming more important.

• 

  Performance

  An evaluation of performance is curiously missing from most security effectiveness, equations, mostly because it is complex and difficult to measure. But it’s not impossible. By measuring speed, latency and scale a security company can get an honest picture of performance.

• 

  How to Compare

  1. Statistics offered by vendors Vendor-sponsored third party test
  2. An independent third party test that does not correctly cover the key areas of coverage, accuracy, and performance.
  3. Additionally many use antiquated methods to give numbers that are not near reality.
  4. Self-test the solution on your company’s production network or a network that is a reasonable simulation.
  5. Talk to reference customers who are using the proposed solution about their experience in efficacy of the solution.