Evaluating Security Product Vendors
In light of recent news about “Vendors Behaving Badly” we want to talk about how a security professional should evaluate vendors and their products.
Tanium exposed hospital’s IT while using its network in sales demos: https://arstechnica.com/security/2017/04/security-vendor-uses-hospitals-network-for-unauthorized-sales-demos/
Lawyers, malware, and money: The antivirus market’s nasty fight over Cylance: https://arstechnica.com/information-technology/2017/04/the-mystery-of-the-malware-that-wasnt/
- There are so many different sources of information about vendors and their products. You owe it to yourself to evaluate not just the vendor but also each source of information.
- Analyst Firms: Gartner/Forrester/etc
- Always remember they take a very generic view using a notional enterprise as the standard.
- Current customer interviews are important but, remember, those customer contacts likely came from the vendor.
- The perception of “Pay for Play” is there no matter how much the firms want to squelch that.
- These tests presume a lot so make sure you understand what the conditions of the test were.
- The “Pay for Play” perception exists here too….
- The results of the testing aren’t specific but can help show outliers in a group
- 3rd Party Testing: NSS Labs, etc.
- Obviously your best and most relevant source of information. :-)
- If you have developed a reliable network of peers you can reach out and ask folks. But, remember, buy them a beer for their troubles…
- Always remember perspective is everything. Some people just don’t like Company_Z and will always hate their products.
- Information Sources
- Start with 3rd party data and demos. This will determine if your requirements (you did write out your requirements, right?) are met by the product
- Do not allow the vendor to drive the definition of “success” in a PoC
- Try to break it. I mean REALLY try to break it.
- Remember during the PoC is going to be the best support and interaction you will ever get. If that sucks you might want to move along.
- Test *all* of your use cases. (you do have documented use cases, right?)
- Do a PoC (Proof of Concept).
- Product Evaluation Rules
- Service providers such as penetration testers and MSSPs
- Edge Cases