15d ago

0 Views

0 Downloads

2.87 MB

57 Pages

Transcription

How to Measure Anything inCybersecurity RiskPresented by:Douglas HubbardHubbard Decision Research

My Co-Author and IRichard SeiersenCurrently the General Manager of Cybersecurity and Privacy at GE HealthCare. Data driven executive with 20 years experience spanning subjectmatters in Cyber Security, Quantitative Risk Management, PredictiveAnalytics, Big Data and Data Science, Enterprise Integrations andGovernance Risk and Compliance (GRC). Led large enterprise teams,provided leadership in multinational organizations and tier one venturecapital backed start-ups.Douglas HubbardMr. Hubbard is the inventor of the powerful Applied Information Economics(AIE) method. He is the author of the #1 bestseller in Amazon’s math forbusiness category for his book titled How to Measure Anything: Findingthe Value of Intangibles in Business (Wiley, 2007; 3rd edition 2014). Hisother two books are titled The Failure of Risk Management: Why It’sBroken and How to Fix It (Wiley, 2009) and Pulse: The New Science ofHarnessing Internet Buzz to Track Threats and Opportunities (Wiley, 2011).2

The Biggest Cybersecurity RiskQuestion: What is Your Single Biggest Risk in Cybersecurity?Answer: How You Measure Cybersecurity Risk3

Current Solution Here are some risks plotted on a“typical heat map”. Suppose mitigation costs were:o Risk 1: 725K – Higho Risk 2: 95K – Lowo Risk 3: 2.5M – Criticalo Risk 5: 375K – Moderate What mitigations should befunded and what is the priorityamong these?4

Current SolutionsMost standards and certification tests promote risk analysis as a type of ordinal scoringmethodThe “Risk Rating Methodology” on OWASP.org states: “Once the tester has identified a potential risk and wants to figure out how serious it is, thefirst step is to estimate the "likelihood". At the highest level, this is a rough measure ofhow likely this particular vulnerability is to be uncovered and exploited by an attacker. It isnot necessary to be over-precise in this estimate. Generally, identifying whether thelikelihood is low, medium, or high is sufficient .”5

Can Analysis or Expertise be a“Placebo”?“The first principle is that you must not fool yourself, and youare the easiest person to fool.” — Richard P. Feynman Collecting more than a few data points on horses makes experts worse at estimatingoutcomes. (Tsai, Klayman, Hastie) Interaction with others only improves estimates up to a point, then they get worse. (Heath,Gonzalez) Collecting more data about investments makes people worse at investing. Collecting moredata about students makes counselors worse at predicting student performance.(Andreassen) An experiment with a structured decision analysis method shows confidence increasedwhether decisions are improved or degraded. (Williams, Dennis, Stam, Aronson)In short, we should assume increased confidence from analysis is a “placebo.” Realbenefits have to be measured.6

What the Research Says There is mounting evidence against (and none for) the effectiveness of“risk scores” and “risk matrices.” Fundamental misconceptions about statistical inference may keep somefrom adopting quantitative methods. Experts using even naïve statistical models outperform human expertswho do not.Note: Every improvement we are about to has already been adopted inseveral cybersecurity environments.7

Summarizing Research on OrdinalScales Bickel et al. “The Risk of Using Risk Matrices”, Society ofPetroleum Engineers, 2014 They performed an extensive literature review to-date as well as astatistical analysis of RM used in Petroleum Engineering Risk(which are nearly identical to RM’s in Cyber) – including computinga “Lie Factor” of the degree of distortion of data.“How can it be argued that a method that distorts the information underlying anengineering decision in nonuniform and uncontrolled ways is an industry bestpractice? The burden of proof is squarely on the shoulders of those who wouldrecommend the use of such methods to prove that these obvious inconsistencies donot impair decision making, much less improve it, as is often claimed.”8

What if We Could Actually MeasureRisk in Cybersecurity?What if we could measure risk morelike an actuary – “The probability oflosing more than 10 million due tosecurity incidents in 2016 is 16%”What if we could prioritize securityinvestments based on a “Return onMitigation”?DB AccessPhysical AccessData in TransitNetwork Access ControlFile AccessWeb VulnerabilitiesSystem ConfigurationExpectedLoss/Yr 24.7M 2.5M 2.3M 2.3M 969K 409K 113KCost ofControl 800K 300K 600K 400K 600K 800K 500KControlReturn teMitigateMonitorTrackTrackThis means there is about a 40% chance oflosing more than 10M in a year and abouta 10% chance of losing more than 200M.9

Why Not Better Methods? Cybersecurity is too complex or lacks sufficient data for quantitativeanalysis yet can be analyzed with unaided expert intuition or soft scales. Probabilities can’t be used explicitly because . yet we can imply probabilities with ambiguous labels.Remember, softer methods never alleviate a lack of data, complexity, rapidlychanging environments or unpredictable human actors they can only obscure it.10

A Major Fallacy RegardingComparing Methods Don’t make the classic “Beat theBear” fallacy.Exsupero Ursus If you doubt the effectiveness of quantitative methods, remember, all youhave to do is outperform the alternative: unaided expertise or soft scoring methods.11

Your Intuition About SampleInformation is Wrong Cybersecurity experts are not immune to widely held misconceptions about probabilitiesand statistics – especially if they vaguely remember some college stats. These misconceptions lead many experts to believe they lack data for assessinguncertainties or they need some ideal amount before anything can be inferred.“Our thesis is that people have strongintuitions about randomsampling these intuitions are wrong infundamental respects.[and] are sharedby naive subjects and by trainedscientists”Amos Tversky and Daniel Kahneman,Psychological Bulletin, 197112

You Need Less Data Than You Think A beta distribution computes the probability of a frequency being below a given amount (e.g. chancethat rate of occurrence is 2/100) In Excel it can be written as “ Betadist(frequency,alpha,beta)” A uniform prior can be made with alpha 1 and beta 1. This can be used as a starting point formaximum uncertainty. “Hits” and “Misses” can be simply added to the priors ( Betadist(frequency,hits 1,misses 1))2 events in 7 companies over 5 years (35 company –years) of data: 2 hits, 33 misses00.050.10.150.20.250.30.350.40.450.513

Survey Results: The “StatsConcepts” Quiz We conducted a survey of 171 Cybersecurity professionals One Finding: Strong opinions against “quant” are associated with poor statsunderstanding.“It’s not what youdon’t know thatwill hurt you, it’swhat you knowthat ain’t so.”MarkTwain14

Historical Models – Still BetterThan ExpertsWhen experts assess probabilities, many events “. . .are perceived as so unique that past history does notseem relevant to the evaluation of their likelihood.” Tversky, Kahneman, Cognitive Psychology (1973)Yet, Historical models routinely outperform experts in a variety of fields (even considering “Black Swans”)Paul Meehl assessed 150 studies comparingexperts to statistical models in many fields (sports,prognosis of liver disease, etc.).“There is no controversy in social sciencewhich shows such a large body of qualitativelydiverse studies coming out so uniformly in thesame direction as this one.”Philip Tetlock tracked a total of over 82,000forecasts from 284 political experts in a 20 yearstudy covering elections, policy effects, wars, theeconomy and more.“It is impossible to find any domain in whichhumans clearly outperformed crudeextrapolation algorithms, less stillsophisticated statistical ones.”15

Monte Carlo: How to ModelUncertainty in DecisionsOutageFrequencyand DurationFrequencyandmagnitudeof breaches 20 25 30 35 404%5% 6% 7% 8%Cost PerOutage hourLegalLiabilities( MM)10% 15% 20% 15% 30%Losses 30 40 50 60 70? Simple decomposition greatly reduces estimation errorfor estimating the most uncertain variables (MacGregor,Armstrong, 1994) As Kahneman, Tversky and others have shown, wehave a hard time doing probability math in our heads In the oil industry there is a correlation between the useof quantitative risk analysis methods and financialperformance – and the improvement started after usingthe quantitative methods. (F. Macmillan, 2000) Data at NASA from over 100 space missions showedthat Monte Carlo simulations beat other methods forestimating cost, schedule and risks (I published this inThe Failure of Risk Management and OR/MS Today). 1M 2M 3M 4M 5M16

A Simple “One-For-One Substitution”Each “Dot” on a riskmatrix can be betterrepresented as a rowon a table like thisThe output can thenbe represented as aLoss ExceedanceCurve.17

Loss Exceedance Curves: Beforeand After How do we show the risk exposure after applying available mitigations?Chance of Loss or GreaterRisk ToleranceStochasticDominanceInherent RiskResidual RiskGiven Loss or Greater (Millions)18

Overconfidence “Overconfident professionals sincerelybelieve they have expertise, act as expertsand look like experts. You will have tostruggle to remind yourself that they may bein the grip of an illusion.” Daniel Kahneman, Psychologist, EconomicsNobel Decades of studies show that most managers are statistically “overconfident” when assessing theirown uncertainty. Studies also show that measuring your own uncertainty about a quantity is a general skill that can betaught with a measurable improvement Training can “calibrate” people so that of all the times they say they are 90% confident, they will beright 90% of the time.19

Inconsistency vs. Discrimination Discrimination is how much your estimates vary whengiven different information. Inconsistency is the amount of your discrimination thatis due to random differences in estimates - this maybe in addition to differences in interpreting verbalscales, so let’s assume we are using explicitprobabilities. Experts are routinely influenced by irrelevant, externalfactors - anchoring, for example, is the tendency for anestimator to be influenced by recent exposure to ananother unrelated number (Kahneman).20

Inconsistency Measurement Results Total: Over 30,000 individual estimates ofprobabilities These estimates included over 2,000 duplicatescenarios pairs.Judgment 2 We have gathered estimates of probabilities ofvarious security events from:o 48 experts from 4 different industries.o Each expert was given descriptive data forover 100 systems.o For each system each expert estimatedprobabilities of six or more different types ofsecurity events.Comparison of 1st to 2nd Estimates of Cyberrisk judgements by same SME1.00.90.80.70.60.50.40.30.20.10.00.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0Judgment 121% of variation in expert responses areexplained by inconsistency.(79% are explained by the actualinformation they were given)21

Modeling Group Estimates of ITSecurity Event Likelihood Examples of Models vs. Group Averages: Probabilities of different security events happening in the next12 months for various systems prior to applying particular controls.Internal Unauthorized AccessResulting In Productivity LossConfidentiality BreachResulting In 0.500.1Model Estimate0.20.30.40.5Model Estimate The models created produce results which closely match the group’s average. A large portion of the model error is due to judge inconsistency. This nearly eliminates the inconsistency error.22

Effects of Removing Inconsistency Alone A method of improving expertestimates of various quantitieswas developed in the 1950’s byEgon Brunswik. He called it the “Lens Method” It has been applied to severaltypes of problems, includingexpert systems, withconsistently beneficial results.23

Measurement Challenge:Reputation Damage One of the perceived most difficultmeasurements in cybersecurity is damage toreputation.eBay Trick: There is no such thing as a “secret”damage to reputation! How about comparing stock prices afterincidents? (That’s all public!)Home Depot So what is the REAL damage?o Legal liabilities,o Customer outreacho “Penance” projects (security overkill) The upshot, damage to reputation actuallyhas available information and easilyobservable measured costs incurred to avoidthe bigger damages!Target201120122013201424

Supporting Decisions If risks and mitigation strategies were quantified in a meaningful way, decisions could besupported. In order to compute an ROI on mitigation decisions, we need to quantify likelihood, monetaryimpact, cost, and effectivenessLikelihood / YrImpact / YrMitigationEffectivenessRisk 137% 2M to 40M95% 725K725%MitigateRisk 211% 50K to 400K100% 95K-80%TrackRisk 334% 5M to 80M90% 2.5M329%MonitorRisk 429% 500K to 20M98% 375K437%MitigateRiskMitigation Cost/ YrMitigation ROIAction The optimal solution would be to mitigate Risks1 & 4 first. If you have the resources, then mitigate Risk 3. Risk 2 is not worth fixing.25

Call to Action for Cybersecurity Organizations should stop using risk scores and risk matrixes andstandards organizations should stop promoting them Adopt simple probabilistic methods now: They demonstrate a measurableimprovement over unaided intuition and they have already been used. Sothere is no reason not to adopt them. Build on simple methods when you are ready – always based on whatshows a measurable improvement.26

Supplementary Material27

Parameters Cybersecurity Models Experts are given values on avariety of parameters as a basis fortheir estimates. For each scenario they may beasked to estimate a probability of abreach, outage, legal liability, etc. Some companies estimated risks ofincidence for particular systems,others estimated threats oradditional detail for types of losses,but there were some commonthemes (see table).28

“Opinion Toward QuantitativeMethods” (18 Questions)18 questions on opinions of the use of quantitative methods in cybersecurity were asked.Here are some examples:(Responses: Agree, Disagree, No Opinion/Don’t Know)Information security is too complex to model with probabilistic methods.Management and users won't understand the quantitative methods’ output.An expert using quantitative probabilistic methods will do better risk assessments then an expertusing intuition alone.RESULTS: 80% of respondents had more “pro” than “anti” quantitative responses. Only 22%were consistently “pro” on quantitative and “anti” on softer scoring methods.29

The Stats Concepts Quiz (10 Questions)EXAMPLE: Assume that you have a portfolio of systems for which you haveobserved no securit

matters in Cyber Security, Quantitative Risk Management, Predictive Analytics, Big Data and Data Science, Enterprise Integrations and Governance Risk and Compliance (GRC). Led large enterprise teams, provided leadership in multinational organizations and tier one venture capital backed start-ups. Douglas Hubbard