Quantitative Risk Assessment PDF

15d ago
0 Views
0 Downloads
2.42 MB
36 Pages
Transcription

Quantitative Risk AssessmentKendrick GlennMichael JansenOscar GutierrezOH3/ Assessments, Cost Estimates & SchedulesOffice (ACES)August 27th, 2015

JSC Strategic Plan Strategy 3.1 Lead through innovative technical and businessmanagement practices Success Factors: Aggressively pursue innovative technical and business approaches that driveaffordability, sustainability, and accountability Develop a customer-focused approach, streamlining policies, processes, andrequirements such as agreements, pricing, and intellectual property to meetinternal/external stakeholder needs Promote the development of business acumen and situational awareness Develop and implement an investment plan that provides critical capabilities whilereducing infrastructure costs and meeting green technology goals Emphasize life-cycle affordability and risk-informed decision processes in Program /Project management2

QRA Defined The Quantitative Risk assessment (QRA) is an objective riskassessment tool used to project threat impacts The QRA provides an estimate of the magnitude of consequencesfor each identified budget threat The estimated costs to the program are summarized into a totalprobabilistic budget threat estimate An estimate is derived using a range of values rather than a single value An estimate can be a range of possible costs from a range of possible values;meaning the cost will fall within the estimated range QRA systematically determines the likelihood of threats occurringand evaluates the cost (cents/ ) of the occurrence QRA sets out to define, measure, predict, and provide a confidencelevel of likelihood and occurrence of threat impacts3

Risk Defined Any risk consists of 3 questions: What can happen? Captured by risk identification and descriptionHow likely is it to happen? Represented by probability of occurrenceWhat is the impact if it did happen? Described by cost impacts and uncertainty around the impacts ISSP Risk Scorecard Definition:A future event with a negativeconsequence that has some probability of occurring. An item whoseresolution is unlikely without focused management attention. Wikipedia Definition: Risk assessment is a step in a risk managementprocedure. Risk assessment is the determination of quantitative or qualitativevalue of risk related to a concrete situation and a recognized threat (alsocalled hazard). Quantitative risk assessment requires calculations of twocomponents of risk (R):, the magnitude of the potential loss (L), and theprobability (p) that the loss will occur. Thein thisWill ofI beableButto eat?“ ThereRiskare risksand instance:costs to a programaction.they are far lessthan the long-range risks and costs of comfortable inaction. ”John F. Kennedy4

Risk Measuring Explicit recognition of possible outcomes Highlight key factors (major drivers) Decision analysis involves selecting among alternatives A risk analysis of any particular decision tries to establish the range ofoutcomes for each decision that could occur were that decision taken The overall aim of risk analysis is to make better decisions; there is alink to optimization – we are aiming to select best decision5

Risk Acceptance Similarities between road trip and 100M mission Mitigation measures“Only those who dare to fail greatly can ever achieve greatly.”Robert F. Kennedy6

Stop and GoFromDeterministicExcel ModelIdentify InputExposureQuantifyInputExposureProbabilistic [email protected] tiesCorrelated?Is Uncertainty at apoint in time or overtimeWhich Alternative isBest? Mitigate,Terminate,Proceed?7

Hunger Pains Let us reflect on the saying: “A Bird in the hand is worth two inthe bush” The expected value of a decision does not incorporate anyinformation about our attitudes to possible outcomes We might prefer the bird in the hand if we are really hungry (even ifwe think that there is a 99% chance of catching each bird, we mightbe too hungry to take the risk) and this is the only bush in theworld. We might prefer the birds in the bush if: We are not particularly hungry and we believe the probability of catching eachbird is 50% or more We are not hungry and we enjoy trying to catch birds whatever the probabilityis! Practical Example: Select a project with lower average value than one with a higher average valueif the latter project has some possible outcomes which are very poor Certainty v. Expectation8

Satisfied Customer Bird In Hand:uncertainty.100% Satisfies Risk. Extremely averse to Two Birds in Bush: Ammunition – Option to buy more (buy down, eliminate) Marksmanship – Practice (buy down) Environment – No control over (Accept as is)“Only those who dare to fail greatly can ever achieve greatly.”Robert F. Kennedy9

Monte Carlo History Thomas Bayes and French Mathematician Pierre-Simon Laplace Stanislaw Ulam and John von Neumann“View of Monte Carlo (and Monaco) from the east” Hampus Cullin“Take risks: if you win, you will be happy; if you lose, you will be wise.”Author Unknown10

The Monte Way Evolve “point estimate” spreadsheets into modeling tools thatprocess combinations of variables and thereby facilitate morerobust analysis Recognize risk and uncertainty, and understand variability throughsimulation To capture the effect of changes to the inputs, especially incontexts where traditional sensitivity analysis is weak: To capture relationships between variables Valuation of contract clauses and contingenciesCasino de Monte-Carlo Fair price for a game involving uncertain rewards11

Sampling No. of Iterations: Rule of Thumb Rapid convergence to reasonable accuracy with small number of iterations Inverse square root law: doubling the accuracy requires four times as manyiterations Convergence monitoring possibilities Display updates Manual and automatic convergence monitoring Sampling Type Monte Carlo simple random number sampling Latin Hypercube intervals of equal probability Randomness is a way to try to achieve results without creating abiased sample12

13

FunnelBallPegsChannels14

15

16

17

18

19

20

11Cumulative number of balls1 35 8 12 17 25 36 49 63 78 92 105116124129133136138140141Channel number211Cumulative number of balls1 35 8 12 17 25 36 49 63 78 92 105116124129133136138140141ORCumulative number of channels1 23 456789 10 1 1 13 14 15 16 17 18 19 20 211 2Ball range, converted to 0-100 basis (%)0Random resultguaranteed to be thisvalue’s y-axiscounterpart (channel 1)50100Even chance of randomRandom resultresult being or this guaranteed to be thisvalue’svalue’s y-axisy-axis counterpart counterpart (channel 21)(channel 11)21

22

LowLowMostLikelyHighMostLikelyHigh23

# of cases(iterations)Probability relations(math model)Outcome MagnitudeCalculationsRange of possible outcomes& most likely outcome, asdetermined, e.g., by a CEtool such as PRICE or SEERif trying to predict a costoutcome, v. channel(triangular distribution)PercentileOverall (repeatable)outcome(S-curve)Individual (random) outcomes9/14/2015 5:37 PM24

ISS QRA ProcessData SourcesModel Specific InputsCost [email protected] (MonteCarlo Simulation)ISS RiskManagementApplicationAssign ProbabilityDistributionCost Risk byLevel 1 50%Level 2 50/50Level 3 50%L1 (.17,.27,.47)L2 (.0, .05, .15)L3 (0, .03, .1)Assign CostUncertainty DistributionPull Reports viaExcel exportRun AutomationMacrosModel OutputsRisk TriangleAuto ConvergenceQRA Estimate (Probability Distribution) X(Uncertainty Distribution) X (Threat )FY15,FY16,FY17,FY18,FY19,FY20,FY21Uncertainty Distribution(Triangle)Design (.33, 1.00, 2.71)Process (.06, 1.0, 2.65)PM (.51, .95, 3.30)LowMostLikelyHi25

QRA purpose ISS Program viewpoint At the macro level, the QRA allows the ISS Program Management to forecast andmanage both near term and far term program budget reserve requirements andallocations QRA forecasts have been integral to the development of recommendations to theProgram Manager relative to program cost control and informed risk managementand effective reserves management approaches The QRA enables Program Management to measure the expected impact theprogram threats will have on program reserves Program also uses QRA to aid proactive planning in order to respond to potentialAgency funding changes ISS Program Planning & Control (PP&C) Viewpoint QRA tool is currently used in several Assessments team reports The QRA projection of reserves impacts is also an integral part of theProgram Risk Advisory Board (PRAB) quarterly activities The QRA data also supports the ISS Resources team cost containment analyses26

QRA Output What ISS reports use QRA? To support the PRAB as well as EWS and IMPR, ISS Program Threats reports fromIntegrated Risk Management Application (IRMA) are used What method does ACES use to do the QRA? ACES uses a combination of Palisades [email protected] Risk software package and modelingcapabilities of Microsoft Excel @Risk uses a certain number of simulations to combine all uncertainties identified andpossible value and likelihood of occurrences in the model to determine a possible costimpact Microsoft Excel is used for developing simulation models to allow risk analysis capabilitiesthrough probability distribution using functions that accept varied distribution types for cellvalue27

Levels and Uncertainties As input for the QRA model, ACES uses a combination of probability and Kfactor distributions, applying triangular distributions Threats are categorized into “Probability Levels” based on likelihood of their occurrenceand are expressed as a triangular distribution1. Level 1 will most likely occur (0.15, 0.27, .47)2. Level 2 are not likely to occur (0, 0.05, 0.15)3. Level 3 are not likely to occur (0, 0.03, 0.1) Threats are also categorized into “K factors” based on the type of task and the likely cost ofoccurrence expressed as a triangular distribution1. Design & Development (.33, 1.00, 2.71) – development, design analysis, testing2. Process (0.06, 1.00, 2.65) – operation and maintenance of existing systems3. Management (0.51, .95, 3.3) – rate increases, contract negotiations, major mods* QRA estimate (Probability Distribution) X (K factor Distribution) X (Threat )28

To-Go Factor Since factor is not part of QRA’s probabilistic role, expected behavior is for to-goactuals to vary slightly about a descending line connecting 100% FY start and 0% atEOY Most years were as expected (only 2 outliers for early/late starts; 2 for step-functionrealization of threats)Low variability goodcorrelation (high R2)Impacts toReserves to-go100%High variability poorcorrelation (low R2)0%FYstartFYend – xFYend29

Fall off Fact that reserve impact history seems to follow a well-correlated linear pathsupports randomness assumption central to use of Monte Carlo techniques inISS QRA Fact that EOY total RIT is reached by the end of August in every FY studiedsupports use of to-go factor Recommend implementation of modified to-go formulation as tabulated aboveand illustrated below30

QRA Annual ShiftsPrior FY04-FY08 QRA (%)Improved FY15 QRA 47206120515302403131

ANYQUESTIONS?32

Back Up33

K factorsUncertainty Factors: The Uncertainty Factors are defined as: Design &Development, Process, ManagementHistorical spacecraft design, development and operations cost data werecollected from 1964 to 1993 in 347 programs and used to determine theuncertainty factors:Shuttle: 19 FlightsSolar System Exploration: 51Manned (I.e. Apollo, Skylab and Spacelab): 29Planetary Landers: 41Normal (a catch-all for all others and the experiment packages onboard):20734

K Factor definitions Design & Development: Any threat associated with a WBS whose primary purpose is the performance ofsystems engineeringtasks which produce an original design or redesign anexisting system. These tasks would include requirements development (derivation,allocation, integration, etc.), design analysis (thermal, stress,FMEA, etc.),testing (developmental, qualification, integration, etc.) and other systems engineeringtask normally associated with the design of the new equipment. Process: Any threat associated with a WBS whose primary purpose is the accomplishment ofa systems operations process. Processes that would fit into this category are thoseassociated with the operation and maintenance of existing systems. Theseprocesses range from hardware and software ground processing, to equipmentmaintenance record keeping. The key characteristic of these activities is the workassociated with these WBS’s as a function of system operations loading. Management: Any threat associated with the performance of program management activities.Program management activities deal primarily with the administrative aspects ofthe program. Activities that would belong to this category would include contractnegotiation outcomes (rates) and major modification approvals35

References Case Study: Application of Quantitative Risk Analysis (QRA) to RiskManagement in the International Space Station (ISS) Program: MichaelJansen & Richard Fox Decision Making and Quantitative Risk Analysis using the Decision Tool Suite:Palisade Software Seminars ISS QRA Validation Summary NASA Space Flight Program and Project Management Handbook (NPR7120.5E) Monte Carlo Simulations: Eric Druker Booz Allen Hamilton36

Wikipedia Definition: Risk assessment is a step in a risk management procedure. Risk assessment is the determination of quantitative or qualitative value of risk related to a concrete situation and a recognized threat (also called hazard). Quantitative risk assessment requires calculations of two components of risk (R):, the magnitude of the ...