The Southern Fried Security Podcast
Join Andy Willingham, Martin Fisher,Steve Ragan, Yvette Johnson, and Joseph Sokoly as they discuss information security, news, and interview interesting people. Get in the discussion at

Evaluating Security Product Vendors


In light of recent news about “Vendors Behaving Badly” we want to talk about how a security professional should evaluate vendors and their products.


Recent News:

Tanium exposed hospital’s IT while using its network in sales demos:

Lawyers, malware, and money: The antivirus market’s nasty fight over Cylance:


  1. There are so many different sources of information about vendors and their products.  You owe it to yourself to evaluate not just the vendor but also each source of information.
    1. Analyst Firms:  Gartner/Forrester/etc
      1. Always remember they take a very generic view using a notional enterprise as the standard.
      2. Current customer interviews are important but, remember, those customer contacts likely came from the vendor.
      3. The perception of “Pay for Play” is there no matter how much the firms want to squelch that.
      1. These tests presume a lot so make sure you understand what the conditions of the test were.
      2. The “Pay for Play” perception exists here too….
      3. The results of the testing aren’t specific but can help show outliers in a group
    2. 3rd Party Testing:  NSS Labs, etc.
      1. Obviously your best and most relevant source of information.  :-)
    3. Podcasts
      1. If you have developed a reliable network of peers you can reach out and ask folks.  But, remember, buy them a beer for their troubles…
      2. Always remember perspective is everything.  Some people just don’t like Company_Z and will always hate their products.
    4. Networking
  2. Information Sources
    1. Start with 3rd party data and demos.  This will determine if your requirements (you did write out your requirements, right?) are met by the product
      1. Do not allow the vendor to drive the definition of “success” in a PoC
      2. Try to break it.  I mean REALLY try to break it.
      3. Remember during the PoC is going to be the best support and interaction you will ever get.  If that sucks you might want to move along.
      4. Test *all* of your use cases. (you do have documented use cases, right?)
    2. Do a PoC (Proof of Concept).
  3. Product Evaluation Rules
    1. Service providers such as penetration testers and MSSPs
  4. Edge Cases
Direct download: SFS_Podcast_Ep_-_194.mp3
Category:podcasts -- posted at: 9:53pm EDT

Tonight's episode is all about those learning moments. 

CISOs and security orgs find new and interesting way to screw up all the time.  Leaving that Any-Any rule in place on the new firewall…  Disabling the CEOs account by accident…  Not realizing that Shadow IT had just installed a new egress point…

Here are our stories.  The name have been changed to protect the culpable.

Direct download: SFS_Podcast_Ep_-_193.mp3
Category:podcasts -- posted at: 10:02am EDT