NSS Labs Announces Breach Detection Systems Test Results

AUSTIN, TX--(Marketwired - Aug 2, 2016) - NSS Labs, Inc., the world's leading cyber security product research, testing, and advisory company, today released the results from its Breach Detection Systems (BDS) group test. Seven of the leading BDS vendors were examined for security effectiveness, performance, and total cost of ownership: Check Point, Cisco, FireEye, Fortinet, Lastline, Palo Alto Networks and Trend Micro -- with a total of nine products.

BDS products claim to provide detection of potentially malicious files and traffic that bypass traditional security products. Therefore, attacks that leading antivirus products detected were discarded so that only unknown and highly evasive attacks were tested. 

Test highlights include:

• Products have improved significantly since prior tests; overall Security Effectiveness ranged from 86.5% to 100.0%.
• While many attacks were detected immediately, testing uncovered some attacks that took close to 24 hours to be detected; other attacks were never detected by some products.
• Average time to detect was 15 minutes 12 seconds across all products; the fastest average time to detect was 4 minutes 6 seconds while the slowest was 38 minutes 36 seconds.
• False positive rates ranged from 0% to 2.63% of traffic.
• Offline infections -- where a user is infected while working remotely and then attaches to the corporate network and spreads the infection -- proved to be problematic for several products where detection rates ranged from 69.2% to 84.6%.
• Total Cost of Ownership (TCO) per Protected Megabit per Second ranged from US$19 to US$147.
• 6 products achieved a Recommended rating; 1 product received a Caution rating, and 2 products received a Neutral rating.

The BDS market is growing quickly. In 2015, the BDS market expanded at a rate of 52 percent to US $1.086 Billion. The BDS market is projected to have a compound annual growth rate (CAGR) of 25 percent through 2020, reaching a market size of over US $3.315 Billion.

The latest NSS Breach Detection Systems test report includes:

• Over 5 billion discrete data elements.
• Thousands of victim machines.
• Collection and analysis of Terabytes of logs.
• Hundreds of discrete samples used in current campaigns.
• Exploits, malware, and evasion testing was performed using regularly abused compromise mediums such as web, email and offline infections -- leveraging multiple common document types.
• Over 100 unique evasion mechanics were tested against each product.

"Breach Detection Systems are supposed to find attacks that other security products miss," said Thomas Skybakmoen, Distinguished Research Director at NSS Labs. "Detection rate, time to detect, and false positive rates are critical metrics for enterprises to consider when purchasing a breach detection system."

The following Vendors and Products Tested during this test:

• Check Point Software Technologies, Ltd. 13500 Next Generation Threat Prevention and SandBlast™ (NGTX) Appliance R77.20
• Cisco Firepower 8120 with NGIPS v6.0 and Advanced Malware Protection v5.3.2016071117
• FireEye Network Security NX 10450 and EX 8420 v7.7.90
• Fortinet FortiGate 500D v5.4.1 with FortiSandbox Cloud Service
• Fortinet FortiSandbox-3000D v2.1.3 with FortiClient v5.4.1.0840
• Lastline Enterprise v7.10
• Palo Alto Networks Next-Generation Security Platform 5020 PAN-OS 7.0.3 - Inline Mode
• Palo Alto Networks Next-Generation Security Platform 5020 PAN-OS 7.0.3 - Tap Mode
• Trend Micro Deep Discovery Inspector Model 4000 v3.8SP2 (a.k.a. TippingPoint Advanced Threat Protection) with OfficeScan 11.0.5102 Service Pack 1

View the NSS Labs Breach Detection Systems (BDS) Security Value Map™ graphic here. 

About NSS Labs, Inc.

NSS Labs, Inc. is the global leader in security product testing and advisory services, providing businesses with the objective information, analysis, and tools they need to successfully manage cyber security risk. NSS offers advanced solutions including the Cyber Advanced Warning System^™ (CAWS) -- a cloud service that continuously captures live attacks being used by threat actors, and tests security products against those attacks in real time. CEOs, CIOs, CISOs, and information security professionals from many of the world's largest and most demanding enterprises rely on insight from NSS. For more information, visit www.nsslabs.com.