An Evaluation Framework and Decision-Making Model for Defense Acquisition Projects Real Options and Multi-Criteria Decision Analysis (MCDA) by Seungbae Park A dissertation submitted to the Graduate Faculty of Auburn University in partial ful llment of the requirements for the Degree of Doctor of Philosophy Auburn, Alabama August 2, 2014 Keywords: Real options, Compound Option, Volatility Estimation, Monte Carlo Simulation, Bayesian Revision Process, Multi-Criteria Decision Analysis (MCDA), Value-Focused Thinking (VFT), Analytical Hierarchy Process (AHP) Copyright 2014 by Seungbae Park Approved by Chan S. Park, Chair, Professor of Industrial and Systems Engineering John L. Evans, Professor of Industrial and Systems Engineering Ming Liao, Professor of Mathematics and Statistics Jung Chul Park, Associate Professor of Finance Abstract In this dissertation, strategic decision{making models based on real options and multi- criteria decision analysis (MCDA) are proposed to assess the value of risky defense acquisition projects. A large{scale acquisition project requiring a signi cant nancial investment must be evaluated to verify its economic feasibility and policy e ectiveness before it is introduced. In the past, such evaluations were typically made in practice using quantitative cost-bene ts (B/C) ratio analysis to evaluate economic feasibility and analytical hierarchy process (AHP) to synthesize the results of the economic and qualitative policy analysis. However, these traditional valuation models fail to capture the full values created by a new project because they do not correctly capture the nature of the process of developing a new acquisition project. Therefore, we present real options and value{focused thinking (VFT) models to improve the current preliminary feasibility assessment process in planning defense acquisition projects. First, the rational project volatility estimation method for real options is explored. We develop a procedure to estimate a volatility based on various information unique to the defense eld and evolving market information using the Monte Carlo simulation method. Then we evaluate the economic value of the Korea Helicopter R&D Program (KHP), a multi{ phased investment decision problem, using a compound real options model and empirical data to validate its practicality. The results verify that a considerable amount of real option value exists for KHP, while conventional estimation methods fail to capture the potential positive value. Second, we examine the most appropriate way to update the volatility estimates based on the arrival of new information during the acquisition process. A Bayesian revision process ii is presented to analyze given data according to project progression and provide a post audit information used for updating the volatility parameter of the real options model. Finally, we explore a way to integrate the economic and policy feasibility analysis results. Two types of multi{criteria decision analysis (MCDA) methods, analytical hierarchy process (AHP) and value-focused thinking (VFT), are compared and applied to choose the optimum alternative of KHP. The results show that decision{makers can make better decisions through completion of the VFT process. Rather than simply comparing available alternatives, VFT helps the decision{makers truly understand what they want and value in their decision and re ects all these factors in the investment valuation. iii Acknowledgments First, I would like to thank my committee chair Dr. Chan S. Park for taking me on as a student and for sharing his wisdom during various stages of the research, as well as thank him for believing in my ability to undertake this work. His counsel during this research was invaluable and I certainly would not have been able to accomplish it without him. I would like to thank my committee members, Dr. John L. Evans, Dr. Ming Liao, and Dr. Jung Chul Park, for their great guidance and support throughout the course of this dissertation e ort. My appreciation goes out to my parents and siblings who have always been there for me with prayers. Finally and most importantly, I would like to thank my lovely wife Yoonjae and my son Jaeyeong(Martin) for encouraging me to complete this work with unwavering support and love. iv Table of Contents Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 Preliminary Feasibility Assessment of a Defense Acquisition Program . . . . 1 1.2 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.3 Research Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.4 Organization of the Dissertation . . . . . . . . . . . . . . . . . . . . . . . . . 5 2 Background and Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1 The Defense Acquisition Environment . . . . . . . . . . . . . . . . . . . . . . 7 2.2 Real Options Applicability in the Defense Acquisition Field . . . . . . . . . . 10 2.3 Previous Research on the Valuation of Defense Acquisition Projects Using Real Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.4 Project Volatility Estimation Methods in Real Options . . . . . . . . . . . . 12 2.4.1 Logarithmic Rate of Return Approach . . . . . . . . . . . . . . . . . 12 2.4.2 Least Squares Regression Method (LSRM) . . . . . . . . . . . . . . . 14 2.4.3 Monte Carlo Simulation with Time Frame (MCST) . . . . . . . . . . 15 2.5 Bayesian Revision Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.6 Compound Real Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.6.1 Geske Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.6.2 Herath & Park Approach (HPA) . . . . . . . . . . . . . . . . . . . . 20 2.7 Multi-Criteria Decision Analysis (MCDA) . . . . . . . . . . . . . . . . . . . 22 v 3 An Economic Evaluation Framework for Defense Acquisition Programs : Com- pound Real Options Applied to the Korea Helicopter R&D Program (KHP) . . 24 3.1 Korea Helicopter R&D Program(KHP) Outline . . . . . . . . . . . . . . . . 24 3.2 Traditional Approach for KHP Assessment . . . . . . . . . . . . . . . . . . . 26 3.2.1 Cash Flows Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . 26 3.2.2 Risk-Adjusted Discount Rate (k) and Risk-Free Rate (rf) . . . . . . . 27 3.2.3 Net Present Value (NPV) of KHP by Discounted Cash Flow (DCF) Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 3.3 Handling Risk and Uncertainty of the Defense Project . . . . . . . . . . . . . 29 3.3.1 Technical Uncertainty of the Defense Project . . . . . . . . . . . . . . 30 3.3.2 Modeling Technical Uncertainty with Expected Mission Ful llment Rate (EMFR) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 3.3.3 Economic Uncertainty of the Defense Project . . . . . . . . . . . . . 38 3.3.4 Modeling Economic Uncertainty with a Consolidated Approach . . . 39 3.4 Estimating Project Volatility in Defense Acquisition Planning . . . . . . . . 41 3.5 Compound Real Option Model for KHP . . . . . . . . . . . . . . . . . . . . 45 3.5.1 Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 3.6 Conclusion Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 4 Bayesian Real Options Approach to Evaluate Dynamic Defense Acquisition Pro- grams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 4.1 Sample Information and Bayesian Inference . . . . . . . . . . . . . . . . . . 53 4.2 The Beta Conjugate with Binomial Sampling . . . . . . . . . . . . . . . . . . 55 4.2.1 Prior Distribution of EMFR . . . . . . . . . . . . . . . . . . . . . . . 58 4.2.2 Sampling and Posterior Distribution of EMFR . . . . . . . . . . . . . 60 4.3 Log{normal Conjugate and Sampling . . . . . . . . . . . . . . . . . . . . . . 63 4.3.1 Sampling and Posterior Distribution of the Project Value . . . . . . . 64 4.4 Bayesian Real Option Model for KHP . . . . . . . . . . . . . . . . . . . . . . 67 vi 4.5 Conclusion Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 5 Analytical Hierarchy Process(AHP) Model for Defense Acquisition Project . . . 70 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 5.2 Analytical Hierarchy Process (AHP) to evaluate for the KHP . . . . . . . . . 72 5.2.1 Conceptualizing and Selecting Alternatives . . . . . . . . . . . . . . . 73 5.2.2 Structuring Hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 5.2.3 Weighting of Assessment Criteria . . . . . . . . . . . . . . . . . . . . 74 5.2.4 Scoring and Synthesizing . . . . . . . . . . . . . . . . . . . . . . . . . 77 5.2.5 The AHP Approach for the KHP Decision Problem . . . . . . . . . . 78 5.2.6 Conclusion Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 6 Value-Focused Thinking (VFT) Model for Defense Acquisiton Project . . . . . . 93 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 6.2 Value Focused Thinking(VFT) to evaluate for the KHP . . . . . . . . . . . . 94 6.2.1 Identifying the Problem . . . . . . . . . . . . . . . . . . . . . . . . . 94 6.2.2 Creating Value Hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . 97 6.2.3 Development of Evaluation Measures . . . . . . . . . . . . . . . . . . 98 6.2.4 Creating Value Function . . . . . . . . . . . . . . . . . . . . . . . . . 99 6.2.5 Determining Weights of Value Hierarchy . . . . . . . . . . . . . . . . 100 6.2.6 Alternatives Generating and Scoring . . . . . . . . . . . . . . . . . . 101 6.2.7 The VFT Approach to KHP Decision Problem . . . . . . . . . . . . . 102 6.2.8 Conclusion Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 vii List of Figures 1.1 Basic assessment framework for preliminary feasibility . . . . . . . . . . . . . . 1 2.1 F-35 Lightning II Program[4]. . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.2 Using Monte Carlo Methods to estimate of the Project Volatility. . . . . . . . . 13 2.3 Di erence between the Project Volatility at T > 1 and C&A Volatility. . . . . 15 2.4 Bayesian Revision Process. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.5 Call value payo s associated with the real calls[16]. . . . . . . . . . . . . . . . . 21 3.1 Korea Helicopter R&D Program(KHP)Process. . . . . . . . . . . . . . . . . . . 25 3.2 Investment Costs and Incremental Cash Flows. . . . . . . . . . . . . . . . . . . 27 3.3 Considering Various Uncertainties for estimating the Project Volatility. . . . . . 29 3.4 DOD Acquisition Process. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 3.5 GAO Assessment of RAH{66 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.6 Variability of EIC at Each Phase by EMFR. . . . . . . . . . . . . . . . . . . . 36 3.7 World Rotary{Wing Aircraft Market Forecast . . . . . . . . . . . . . . . . . . . 41 3.8 Mathematical relationships between and T[13]. . . . . . . . . . . . . . . . . . 44 3.9 The Project Present Value Distributions at Each Phase. . . . . . . . . . . . . . 45 viii 3.10 Evaluation Process of HPA for KHP. . . . . . . . . . . . . . . . . . . . . . . . . 47 3.11 Gross Project Value and Real Call Value. . . . . . . . . . . . . . . . . . . . . . 49 3.12 Sensitivity Analysis for the KHP Compound Real Options Model. . . . . . . . . 51 4.1 Bayesian Inference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 4.2 Beta Conjugate Distribution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 4.3 Prior Beta Distribution for the Initial EMFR at Each Phase. . . . . . . . . . . . 59 4.4 Posterior Beta Distribution for the EMFR at Each Phase. . . . . . . . . . . . . 62 4.5 Bayesian Postaudit Distribution of the Project Value at Each Phase. . . . . . . 66 4.6 Bayesian Postaudit Gross Project Value and Real Call Value. . . . . . . . . . . 68 5.1 Steps of the Analytical Hierarchy Process (AHP). . . . . . . . . . . . . . . . . . 73 5.2 Standard Hierarchy of Analytical Hierarchy Process (AHP). . . . . . . . . . . . 74 5.3 Hierarchy of KHP for AHP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 5.4 Pairwise-Comparison of alternatives with respect to the criterion of \economic feasibility". . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 5.5 The Overall Results of the KHP Analysis. . . . . . . . . . . . . . . . . . . . . . 83 5.6 Sensitivity of the \Technical Feasibility" Weight. . . . . . . . . . . . . . . . . . 84 5.7 Sensitivity of the \Economic Feasibility" Weight. . . . . . . . . . . . . . . . . . 85 5.8 Sensitivity of the \Policy Feasibility" Weight. . . . . . . . . . . . . . . . . . . . 85 6.1 \Value-Focused Thinking" (VFT) Process. . . . . . . . . . . . . . . . . . . . . . 95 ix 6.2 Decision Context. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 6.3 Value Hierarchy of KHP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 6.4 Examples of Increasing and Decreasing SDVFs. . . . . . . . . . . . . . . . . . . 105 6.5 Sensitivity of the \Maximize E ects on Technology" Weight. . . . . . . . . . . . 109 6.6 Sensitivity of the \Maximize E ects on National Economy" Weight. . . . . . . . 109 6.7 Sensitivity of the \Maximize E ects on Defense Policy" Weight. . . . . . . . . . 110 x List of Tables 2.1 The Military Strength of the Powers Surrounding the Korean Peninsula[3]. . . . 8 2.2 Technology Cycle Times[5]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.3 Comparing Military and Commercial Aircraft Programs? Cycle Times[6]. . . . . 10 3.1 KHP Total Life Cycle Cost and Economic Impact(Revenue). . . . . . . . . . . . 26 3.2 KHP Cash Flows and Expected Net Present Values ($million). . . . . . . . . . . 28 3.3 Required TRL at Each Stage of Defense Acquisition Process. . . . . . . . . . . . 31 3.4 Technology Readiness Level(TRL) de nitions. . . . . . . . . . . . . . . . . . . . 32 3.5 Technology Readiness Level(TRL) Assessment Information. . . . . . . . . . . . 32 3.6 Three{point estimates of EMFR ("i;i = 1;2;3;4). . . . . . . . . . . . . . . . . 37 3.7 Two{point estimates of the Forecasting Market Share. . . . . . . . . . . . . . . 41 3.8 Initial Estimated Volatility of Each Phase. . . . . . . . . . . . . . . . . . . . . . 46 3.9 Parameters for the Binomial Lattices. . . . . . . . . . . . . . . . . . . . . . . . 46 4.1 Initial Three{point estimates of EMFR ("i;i = 1;2;3;4). . . . . . . . . . . . . 59 4.2 Sampling Information of EMFR ("i;i = 1;2;3;4). . . . . . . . . . . . . . . . . 60 4.3 Posterior Beta Distribution of EMFR ("i;i = 1;2;3;4). . . . . . . . . . . . . . 61 4.4 Prior Log{Normal Distribution of PVn(n = 1;2;3;4). . . . . . . . . . . . . . . . 65 4.5 Parameter Values for the Monte Carlo Simulation. . . . . . . . . . . . . . . . . 65 4.6 Sample Information and Posterior Distribution of Each Phase. . . . . . . . . . . 66 4.7 Bayesian Postaudit Parameters for the Binomial Lattices. . . . . . . . . . . . . 67 5.1 Classi cation of MCDA Techniques [74]. . . . . . . . . . . . . . . . . . . . . . . 72 xi 5.2 Pairwise-Comparison Scales of AHP. . . . . . . . . . . . . . . . . . . . . . . . . 74 5.3 RI(Random Index). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 5.4 Pairwise-Comparison between Main Considerations (2nd{Level). . . . . . . . . . 80 5.5 Pairwise-Comparison of Alternatives with respect to the Criterion of \Economic Feasibility" based on the Strategic NPW. . . . . . . . . . . . . . . . . . . . . . 82 5.6 Pairwise-Comparison of sub-criteria (3rd Level) w.r.t the \Technical Feasibility". 87 5.7 Pairwise-Comparison of sub-criteria (3rd Level) w.r.t the \Policy Feasibility". . 87 5.8 Pairwise-Comparison of sub-criteria (4th Level) w.r.t the \Adequacy of Technol- ogy Development Planning". . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 5.9 Pairwise-Comparison of sub-criteria (4th Level) w.r.t the \Success Possibility of Technology Development". . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 5.10 Pairwise-Comparison of sub-criteria (4th Level) w.r.t the \Overlapping with Ex- isting Technology". . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 5.11 Pairwise-Comparison of sub-criteria (4th Level) w.r.t the \Necessity and Consis- tency of Defense-Policy". . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 5.12 Pairwise-Comparison of sub-criteria (4th Level) w.r.t the \Risk factors of Project Implementation". . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 5.13 Pairwise-Comparison of Alternatives w.r.t the \Adequacy of Planning Process and Schedule". . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 5.14 Pairwise-Comparison of Alternatives w.r.t the \Adequacy of Development Goal and Contents". . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 5.15 Pairwise-Comparison of Alternatives w.r.t the \Adequacy of Support System". . 89 5.16 Pairwise-Comparison of Alternatives w.r.t the \Technology Development Trend". 89 5.17 Pairwise-Comparison of Alternatives w.r.t the \Technology Interoperability and Gap". . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 5.18 Pairwise-Comparison of Alternatives w.r.t the \Overall Overlapping Review". . 90 5.19 Pairwise-Comparison of Alternatives w.r.t the \Individual Overlapping Review". 90 5.20 Pairwise-Comparison of Alternatives w.r.t the \Necessity for strengthening na- tional defense ". . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 xii 5.21 Pairwise-Comparison of Alternatives w.r.t the \Consistency and Connectivity with Defense Policy". . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 5.22 Pairwise-Comparison of Alternatives w.r.t the \Restriction of Legal and Infring- ing Patent Rights". . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 5.23 Pairwise-Comparison of Alternatives w.r.t the \Require Operational Capabilities Changing". . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 5.24 Pairwise-Comparison of Alternatives w.r.t the \Steadiness of Related Private Enterprises". . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 5.25 Pairwise-Comparison of Alternatives w.r.t the \Financial Resource Procurement". 92 6.1 Comparison of Fundamental and Means Objective Structures. . . . . . . . . . . 98 6.2 The Evaluation Measures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 6.3 Strategic NPW of Each Alternative. . . . . . . . . . . . . . . . . . . . . . . . . 105 6.4 Analysis of Alternatives. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 6.5 Di erences between AHP and VFT. . . . . . . . . . . . . . . . . . . . . . . . . 111 xiii Chapter 1 Introduction 1.1 Preliminary Feasibility Assessment of a Defense Acquisition Program To successfully pursue a large{scale defense acquisition program which requires a signif- icant nancial commitment by the government, it is necessary to assess potential nancial risk along with the technical attributes of the defense project. Such an assessment also al- lows optimal budget appropriation and e cient nancial operation. Typically, in Korea, a program feasibility analysis must be carried out for any acquisition project costing more than KRW 50 billion (US $47 million). Figure 1.1 outlines steps that must be taken in evaluating a large-scale defense acquisition project by the Korean government. Program Overview / Research Analysis ? Program background, demand, purpose, expected impact ? Program strategy/issues, etc. Economic Analysis ? Demand estimates ? Cost/benefit analysis ? Sensitivity analysis, etc. Policy Analysis ? Effects on domestic industry ? Ripple effects of technology ? Foreign effects, etc. Integration: Multi-Criteria Analysis (MCDA, AHP, Etc.) ? Program feasibility ? Policy recommendations Figure 1.1: Basic assessment framework for preliminary feasibility 1 As shown in Figure 1.1, a preliminary feasibility assessment for a proposed program consists of two main analyses: a policy analysis and an economic analysis. A policy anal- ysis examines factors including military strategic feasibility, tactical e ects, defense policy linkages, and e ects on domestic industry. Some of the results of the policy analysis may be converted to currency units and incorporated into the economic analysis. Non-quanti able policy factors are presented as a ratio scale or utility to evaluate program. An economic analysis explores economic and nancial feasibility by calculating the cost-bene t ratio, net present value, and internal rate of return. If necessary, a sensitivity analysis is undertaken to examine the impacts of uncertain variables such as demand, unit cost, and discount rate of the program to compensate for measurement errors.[1] The nal part of the preliminary feasibility research uses the whole results of the eco- nomic and policy analyses to determine whether the program should be pursued. To reach an accurate determination, it is necessary to integrate the results of quantitative and qualitative analyses that di er in terms of scale and to consider the assessment consistency, program distinctiveness, and opinions of various stakeholders. For this purpose, a multi-criteria deci- sion analysis is often used to identify the best alternative that accomplishes these multiple objectives. However, the process of identifying such a program is quite challenging, so there is great demand for a decision-support tool that is theoretically sound but practically ac- ceptable to the practitioners. 1.2 Problem Statement Many factors such as policies, strategies, and economic feasibility must be considered in defense acquisition projects. However, the process of developing and acquiring state-of- the-art weapon systems requires extremely large investment costs and a long period of time. Moreover, once a government has committed to a decision, it is very di cult to change the course of action without enormous costs of money and time. This lack of exibility is one of the common problems cited for traditional economic analysis. Recently, the real options 2 model is the most commonly used technique for the valuation of a strategic investment project under signi cant uncertainty. This technique alleviates the limitations of the traditional methodology and actively manages the uncertain investment environment, giving strategic exibility to postpone, extend, reduce, or abandon the project and re ecting all these factors in the investment valuation. However, the real options model has seldom been used in the defense acquisition eld. Although a few attempts have been made to use the real options model for defense acquisitions, most of these studies have focused only on a real option model and program value assessment and have not included a speci c and systemic estimation of volatility, which is even more di cult to quantify in defense acquisition projects than in other investment strategies. Rational estimation of volatility is essential to accurately assess real option value. Therefore, a study is needed to rst estimate the volatility of the project analysis phase, and then to explore a method for updating volatility that changes over time. Additionally, there is the problem of combining the results of economic analysis with the qualitative factors that are di cult to quantify in currency units, such as risks, time limits, and the distinctiveness of national security. While economic analysis can be done with a real options method, qualitative factors require multi-criteria analysis. One of the primary concerns in national defense projects is to integrate all these factors (sometimes competing with each other) and to come up with a decision model that has a practical application in the defense eld. Although there have been many studies to assess large-scale investment programs or defense acquisition programs, not much e ort has been made to combine the results of economic analysis and qualitative factors analysis. Therefore, a method is needed to rationally consider both methods, real options and multi-criteria decision analysis. 3 1.3 Research Objectives The main objective of this study is to develop a strategic decision-making model that assesses the value of a defense acquisition program. This objective is accomplished through the following steps: 1. Develop an appropriate real-option decision model for a defense acquisition project. 2. Develop a procedure for estimating a volatility parameter for the real options model, based on evolving macro market information and various information unique to the defense eld. 3. Validate the decision support model and provide detailed economic inferences by com- paring the outcomes with the traditional decision model through the Korea Helicopter R&D Project (KHP), one of the largest defense acquisition projects in Korea. 4. Develop a procedure for updating a initial estimated volatility using the Bayesian revision process, 5. Develop a scheme to integrate the value assessment of economic merits and other qualitative factors by considering various multi-criteria decision models. First, we explore the rational initial volatility estimation method for real options. In estimating volatility of the initial analysis time, it is preferable to use methods that can replicate the given circumstances and determine the volatility easily rather than methods that require complicated and unrealistic assumptions. We represent the Monte Carlo simulation with time frame as a suitable method to estimate phase{speci c volatility measures. Second, the economic value of the Korea Helicopter R&D Program (KHP), a multi{ phased investment decision problem, is evaluated by a compound real option model and its practicality is validated by empirical data. Even when a project is selected and executed, it occasionally fails because of unexpected situations along the way. Thus, to solve this problem, 4 it is important to create a proper valuation and exible decision-making methodology. The compound real option model is the most appropriate approach for these objectives and will be utilized for evaluating the strategic net present worth (SNPW) of KHP. Third, we present an enhanced real option estimation method, which can update volatil- ity and option value by utilizing such new information through the Bayesian revision method. Analysis of the project?s uncertainty is a process of overcoming limitations caused by un- available data by gaining new information and using it. During this process, the Bayesian revision method can be used to revise the initial volatility using actual market data. Finally, to evaluate the feasibility of defense acquisition programs, various factors need to be considered and comparatively analyzed. These factors include the purpose of the project, required operational capability, military strategy, the distinctiveness of national security, and the political commitment to the program?s success, as well as the results of economic analysis. To solve this problem, multi-criteria decision analysis (MCDA) model is combined with a real option method to integrate the results of economic and policy analysis for nal determination. Among various MCDA methods, we examine two methods, analytical hierarchy process (AHP) and value-focused thinking (VFT), for a preliminary feasibility assessment for KHP. 1.4 Organization of the Dissertation This study will examine how 1) the real option and 2) the multi-criteria analysis model a ect the decision-making process by exploring expenditures and elements mentioned in an assessment of the Korean Helicopter Program (KHP) which was carried out by South Korea?s National Assembly Budget O ce[2]. To accomplish these goals, the remainder of this dissertation is organized as follows. Chapter 2 reviews the defense acquisition environment and real options applicability and investigates previous studies and some theories related to the real options and multi- criteria decision analysis models. Chapter 3 presents the Monte Carlo simulation (MCS) to estimate the initial volatility and evaluates the economic value of the Korea Helicopter 5 R&D Program (KHP) using a compound real option model and empirical data to validate the practicality. Chapter 4 discusses the Bayesian revision process to analyze given data according to project progression and to provide post audit information used for updating the volatility parameter of real option model. Chapter 5 explores a way to integrate the economic and policy feasibility analysis results. Analytical hierarchy process (AHP), one of the multi-criteria decision analysis (MCDA) methods, is applied to identify the optimum alternative for KHP. Chapter 6 presents a value-focused thinking (VFT) approach, which is similar to AHP but accesses issues through di erent ways, to develop a suitable decision model for the preliminary feasibility assessment of KHP. Then, AHP and VFT are compared to suggest a more re ned analysis to assess the defense acquisition decision problems. Chap- ter 7 consists of a brief conclusion along with some suggestions for future research. 6 Chapter 2 Background and Literature Review There are three main objectives in this dissertation. The rst objective is to develop an appropriate real options decision model for a defense acquisition program. The second objective is to develop a procedure for estimating a volatility parameter for the real options model and to devise a method to update the volatility based on the new information. The third objective is to develop a scheme to integrate the value assessment of economic merits and other qualitative factors. Therefore, this chapter reviews the current defense acquisition environment and real options applicability for evaluating defense programs and then reviews previous studies related to the research objectives. Finally, it discusses limitations of previous assessment methods and ways to improve on them. 2.1 The Defense Acquisition Environment The occurrence of full-scale wars has become less likely in the post-Cold War era, but traditional con ict factors such as territorial disputes, competition for natural resources, religious and ethnic con icts, and separatist and irredentist movements still remain. As shown in Table 2.1, the global security environment has been complicated by the threats of weapon system development in convergence with state-of-the-art scienti c technology and the prolonged global economic crisis[3]. Accordingly, countries around the world are allocating huge budgets and focusing on acquiring new weapon systems to secure their rights and interests and to prepare for an uncertain security environment as well as potential future battle elds. In particular, the Republic of Korea is the world?s only divided nation and is geopolitically located at the center of East Asia where the interests of major powers around the world are in complicated tangles. The Korean government is under constant pressure to 7 come up with e cient defense acquisition projects as a key factor to assure national security and prosperity. Table 2.1: The Military Strength of the Powers Surrounding the Korean Peninsula[3]. S.Korea N.Korea China Japan Russia United State Number of Troops (thousand) 650 1,200 2,285 250 960 1,570 Defense Budget ($Billion) 25.7 0.8 76.4 54.4 41.9 693.6 Key Weapon System 3 Aegis-class Ships 10 Submarines 460 Fighters and Bombers 420 Combatants 70 Submarines 820 Fighters and Bombers 1 Aircraft Carrier 71 Submarines 1,751 Fighters and Bombers 6 Aegis-class Ships 18 Submarines 348 Fighters and Bombers 1 Aircraft Carrier 65 Submarines 1,765 Fighters and Bombers 11 Aircraft Carriers 71 Submarines 3,191 Fighters and Bombers Force Upgrades Next-generation fighters, improve capabilities to carry out long range precision strikes, acquire battlefield surveillance Develop nuclear and ballistic missiles, as well as chemical and biological weapons to secure a strategic offensive capability New strategic missiles, stealth fighter, nuclear attack submarines (Jin-class), strengthening of space capabilities Strengthening of the MD (missile defense) system, introduction of next generation fighters, enhancement Aegis ship capabilities Nuclear capability enhancement, development of stealth fighters and missiles, strengthening of space capabilities Force upgrade of the Pacific Command, realignment of U.S. Forces in the ROK and Japan, Pursuit of strategic flexibility However, while the costs of acquiring and managing state-of-the-art weapon systems have increased astronomically, the nances to support them are strictly limited owing to the global economic crisis and the pressure to reduce national defense budgets. Accordingly, an accurate and scienti c valuation of new defense acquisition projects is more necessary than ever in making investment decisions. The U.S. predicts that the expected costs of promoting 96 major defense acquisition projects will be $1.6 trillion as of 2011. Despite having the world?s greatest military force and largest economy, the U.S. is facing the issue of managing the shrinking national defense budget while costs are rapidly increasing every year[4]. Moreover, the development of innovative scienti c technologies has led to reducing the service life of weapon systems already acquired[Table 2.2][5]. Therefore, it is necessary to consider the risks and uncertainty associated with technology obsolescence along with economic feasibility in evaluating modern defense acquisition projects. For example, the U.S. F-35 Lightning II (Joint Strike Fighter) acquisition project in Figure 2.1 [4] can be considered a representative case in which a project faced extensive 8 Table 2.2: Technology Cycle Times[5]. System Primary Structural Materials / Platforms 15 ? 30 years Mechanical Systems / Weapons 10 ? 15 years C4I Infrastructure 5 ? 8 years Components Sensors 3 ? 5 years Communications 1 ? 3 years IT Hardware 0.5 ? 2 years IT Software 0.5 ? 1 years damage in the process of project promotion owing to insu cient economic feasibility analysis in the national defense eld and an in exible response to uncertainty factors.[4] Concept System Development Production 11/1996 Program Start 10/2001 Development Start 06/2007 Production Decision TBD Initial Capability USMC,USAF,USN 2035 Last Procurement Program Performance(fiscal year 2012 dollars in millions) As of 10/2001 Latest 12/2010 Percent change R&D cost $38,977 $58,388 49.8 Procurement cost $172,921 $267,596 54.7 Total Program cost $213,708 $326,535 52.8 Unit cost $75 $133 78.2 Total quantities 2,866 2,457 -14.3 Acquisition cycle time(months) 116 TBD NA Figure 2.1: F-35 Lightning II Program[4]. The U.S. Department of Defense (USDOD) decided to purchase 60 F-35s in 2007 without a test ight and adequate economic evaluation. However, various design problems in the ight system and software, delayed the schedule for development and production. Later, the costs for acquisition and maintenance rose astronomically, and the success of the project became uncertain; but although the USDOD could not con dently proceed with the project, it could not abandon it either. A. T. Kearney claimed that defense acquisition projects have more delayed acquisition periods than private investment projects, resulting in various issues such as increased costs and lack of exibility in delayed procurements[6]. 9 Table 2.3: Comparing Military and Commercial Aircraft Programs? Cycle Times[6]. Acquisition cycle time(months) Target Actual Military Aircraft Global Hawk 55 125 IE-2D Advanced Hawkeys 95 136 Gray Eagle UAV, etc 50 TBD Average 88 114 Commercial Aircraft Boeing 777 60 60 Airbus A-380 44 49 Boeing 787 65 83 Average 56 64 Therefore, the evaluation methods that have traditionally been used for the economic evaluation of defense acquisition projects are irreversible and costly. 2.2 Real Options Applicability in the Defense Acquisition Field A nancial option is a contract that gives its holder the right but not the obligation to take action at a predetermined price within a speci ed time period in the nancial market. Similarly, a defense planner with an opportunity to invest in acquisition plans can be viewed as having a right, but not necessary an obligation, to invest at the time of decision. The real options approach will allow a defense planner to value the option by assessing the potential risk of each acquisition weighed against the expense of the required investment. Normally, once a typical decision making model is undertaken, it is irreversible, so that decision{makers risk a great deal of sunk costs if the plan does not pan out the way it was intended. However, the real option decision framework is intended to limit the downside risk at a reasonable cost. Thus, to cope with the risks and the uncertainty of investment in defense acquisition projects, real options analysis can be an e ective planning and evaluation tool. 10 2.3 Previous Research on the Valuation of Defense Acquisition Projects Using Real Options Glaros [8] proposed the applicability of real options to alleviate the rigidity of defense acquisition projects in promoting the Force Transformation project of the USDOD and to enhance the business value and exibility of decision making under future uncertainties. Komoroski et al.[9] used a Knowledge Value Added/Real Options (KVA + RO) method to conduct an economic analysis of IT (information technology) required for navy ship main- tenance. They veri ed the e ects of possible cost savings scenarios by using a KVA method, evaluated the value of each scenario by using the real option method, and presented a decision-making model. However, when a new technology was introduced, they focused only on evaluating the cost savings e ects by using KVA and utilized the real option method as an alternative decision-making value assessment tool, such as a decision{tree. In other words, this study lacked an analysis of the volatility of the rst process of real option and a method of updating such estimated volatility. Olagbemiro et al.[10] explored the value of the United States Defense Software Acqui- sition Program based on a real option analysis. What is notable in their study is a detailed analysis of acquisition demand, which is considered the most important estimation for the software acquisition program. In other words, estimation of demand volatility changes be- cause of con icts between cost and schedule. They estimated the very rst volatility by utilizing historical data (an objective approach) and expert opinions obtained using the Del- phi method (a subjective approach). Then they re ned the very rst volatility by using Dempster-Shafer Theory (DST). The DST represents uncertain factors as belief functions instead of probability. Belief functions have a value ranging from 0 to 1, called the \M" value, while an evidence function is acquired through a combination of belief function. Fi- nally, volatility is estimated by converting an evidence function value to a compound growth rate of the nancial market. This study proved the e ectiveness of DST by not only using the value assessment method but also conducting a detailed analysis of volatility. However, 11 there are limitations to the DST method. First, its application is limited to programs that utilize probability theory to describe uncertainty. For example, the uncertainty of an invest- ment project is represented as the volatility of the project?s rate of return; such volatility is expressed as a standard deviation of the return probability distribution. Another major shortcoming of the DST method is the estimation of the belief function value (M value), which is arbitrary and dependent on the subjective judgment of an evaluator. The DST method lacks a well-established decision theory to solve complicate defense acquisition prob- lems. Park [11] developed a real option model to determine when to purchase a weapon sys- tems. Jang [12] introduced a compound real option model to assess the value of defense R&D programs. However, their study are limited because they do not address the important issue of how the project volatility should be estimated. In summary, no comprehensive tools are available to help address the \project volatility" estimation of a real option, which is even more di cult to quantify in defense acquisition projects than in other investment projects. Therefore, this study is intended to provide a rational volatility estimation method to rst evaluate the volatility of the project analysis phase and then to update the volatility to respond to change in a timely manner. 2.4 Project Volatility Estimation Methods in Real Options 2.4.1 Logarithmic Rate of Return Approach Copeland and Antikarov (C&A) [27] considered a project itself as the underlying asset of real options, used a simulation to estimate the logarithmic rate of return of the future cash ow and its volatility incurred in the project, and presented ways to use this estimate as the volatility of the project. Here, the denominator, PW0, is xed as the constant expected value, and only the numerator, PW1, is calculated by simulation. This method is used because the important concept is the discount value PW1 of the FCF calculated by simulation, which is the future 12 Uncertainty 1 Uncertainty 2 Uncertainty n ? ? ? Monte Carlo Simulation Year 1 2 3 ?T Present Value Model Volatility Estimation Output Probability of PV 0 PV 1 0 ln [] [] Project Volatility PW z EPW Var z? ?? = ?? ?? = = Calculation Inputs 1 0 () :discount rate [] T tk kt ttt rkt PV FCF e r PW PV FCF NPW E PW =+ ?? = ? = + = ? Figure 2.2: Using Monte Carlo Methods to estimate of the Project Volatility. value a year later, and not the expected value of the present value calculated by the tradi- tional DCF method. This concept is also used to simulate stock price in the nancial market when the present stock price is a variable that is already known[29]. On the other hand, unlike C&A, Herath and Park (H&P) [16] considered the denomi- nator PW0 as a random variable, calculating the volatility of the logarithmic rate of return after estimating it by simulation. The underlying variables to calculate the FCF are consid- ered independent variables. Later, Cobb and Charnes(C&C) [30] extended the H&P method and presented the volatility output result of the logarithmic rate of return, taking into con- sideration the correlated underlying variables. The C&A method considered PW0 as an already-known constant similar to stock prices in the nancial market, while H&P and C&C considered PW0 in real options as a random variable that occurred from future cash ow and estimated it by simulation. The de nition of PW0 is the only di erence between the C&A methods and the other two methods. Otherwise, the basic volatility estimation method is the same. However, all these methods based on the C&A?s logarithmic rate of return concept do not consider time frame or option life. The C&A volatility estimation method considers only one unit period (generally one year) as option life. 13 2.4.2 Least Squares Regression Method (LSRM) Smith [32] stated that the C&A method overestimated project volatility as ~PW1 includes uncertainty that may occur in the future. He argued that to estimate ~PW1, it is necessary to consider only the volatility of FCF1 which is information available thus far, and use the average value (expected value) of the simulation result for FCFn(n = 2;3;:::) that may occur in the future. Based on Smith?s argument, Brand~ao et al.[34] modi ed the C&A method as shown in Equation 2.1, and proposed a volatility estimation method of the logarithmic rate of return. ~ = ln ~PW 1 PW0 ! = ln ~FCF 1 +E[ ~PV 1 j ~FCF1] PW0 ! = ln 0 BB BB @ ~FCF1 + nX t=2 h E[ ~FCFt]e r(t 1) j ~FCF1] i nX t=1 E[ ~FCFt]e rt 1 CC CC A (2.1) However, numerous calculation processes are needed to evaluate Equation 2.1. Godinho [35] referred to this evaluation procedure as Two-Phase Simulation (TPS), and used Least Square Regression Method (LSRM) to solve the inconvenience of the TPS procedure. LSRM is proposed by Longsta and Schwartz (2001) to evaluate an American option in the nancial market using the Monte Carlo Simulation(MCS) method, and can be considered as another method for obtaining conditional expected value. However, Godinho?s LSRM also uses the logarithmic rate of return concept of C&A and thus does not consider time frame or option life. Also, complicated statistical analysis procedures must be followed to prove its accuracy and suitability. 14 2.4.3 Monte Carlo Simulation with Time Frame (MCST) The C&A method estimates the volatility of the project?s logarithmic rate of return at T + 1, one unit period (generally one year) after the point at which the project begins, with the basic premise of not considering the option. However, when option life is considered be- fore beginning the project, the characteristic of the GBM model with the consistent volatility of the unit period makes the volatility of C&A equivalent to the volatility?s logarithmic rate of return between T 1 and T, and therefore the C&A method is only considered as 1 in terms of option life. Thus, if the volatility of C&A is considered equivalent to that of a project with an option life greater than 1, the volatility tends to be overestimated. This over- estimation occurs because the volatility of the project?s logarithmic rate of return increases by p T along with the period. Thus, the volatility( ) of the initial analysis time(T = 0) must have a smaller value than the volatility( pT) of the project?s starting time. 0 T T+1 T+n ? ? ? Option Life = T T-1 ? ? ? C&A Volatility = Project Volatility at T=0 ? T? ? ( ) ( ) 1 1 1 ln ln ln ln [] [] TT TT VV Var Var Var V Var V EV EV ? + + ? ? ??? ?= = = = ? ??? ? ??? T? Project Launching Point Project Analysis Point Increase Increase T???= Therefore, C&A Volatility at time T = C&A Option Life Figure 2.3: Di erence between the Project Volatility at T > 1 and C&A Volatility. To improve the C&A method considering only T = 1, Park [13] presented a volatility estimation method considering T > 1. This method draws the project value distribution VT ( T; 2T) at the project launching time(T) by the simulation method and calculates the project volatility( ) using a mathematical interaction formula of the GBM model [31]. 15 = vu uu uu t ln 2 4 T T !2 + 1 3 5 T (2.2) Here, if option life T is 1, it is equivalent to the volatility of the C&A method as explained above; this is because the volatility of the C&A logarithmic rate of return only considers that of the numerator ~V1 estimated by simulation as in Equation 2.3. C&A Method ( 0) ) Var " ln ~V 1 E[V0] !# = Var h ln ~ V1 i = vu utln 21 21 + 1 ! (2.3) The C&A method has the shortcoming of not considering option life; however, the MCST method by Park [13] supplements this shortcoming by estimating volatility while considering the time frame of option life. If a certain rm is considering a large investment project, it will conduct an initial evaluation of its value and uncertainty many years before and attempt to make strategic decisions(delay, expansion-contraction, and abandonment, etc.) through additional data collection and project re-evaluation until the project begins. This ability to collect and reassess new data is the fundamental advantage of using real options as a tool for project valuation and decision{making. When project volatility is initially evaluated, a rational volatility estimation method must consider the option life before the project starts. Therefore, this research will use the MCST method to estimate the project volatility for a real options model. 2.5 Bayesian Revision Process The project volatility estimation methods examined thus far assume that project value follows the characteristics of the GBM model just as stock price does in the nancial market. In other words, they are based on the assumption that volatility of a unit period (generally one year) of a project is consistent and does not change until project completion. However, 16 in reality, volatility varies according to time and the surrounding environment, and thus it is not rational to assume that the volatility estimated in the initial project evaluation will not change until project completion. To address this problem, Haahtela [37] and Brand~ao et al.[38] each introduced di erent volatility estimation methods that use the aforementioned LSRM to estimate volatility that changes in each unit period, obtaining the conditional expected value of the random variable or each periodical PVt through regression analysis. However, the statistical analysis procedure is more di cult and complex than obtaining the constant volatility of the GBM model. Also, the di erent volatility for each project unit period requires a practicality review as well as many statistical follow-up studies to prove the model?s suitability and e ectiveness. In estimating volatility of the initial analysis time, it is necessary to use methods that are suitable to the given circumstances and easy to estimate rather than methods that require complicated and unrealistic assumptions. A technique to calculate the estimates that considered everything in the initial analysis time is neither e cient nor accurate enough. The changing volatility models using LSRM are methods to calculate new estimates based on the initially revised information without additional data collection. However, the Bayesian analysis can calculate the highly reliable and newest volatility according to project progression as it revises volatility with actual data acquired from the market. Project uncer- tainty analysis is a process of acquiring and using various data to overcome initial analysis results that are limited by a lack of available data, and the Bayesian Revision Process is the most tting approach for this analysis. The Bayesian Revision Process uses the following steps: 1. infers the prior distribution considering the unknown parameters as random variables 2. uses information that has owed in additionally or is acquired through data sampling along with prior distribution, and 17 Prior Distribution What we knew before Data Sampling What the data tell us Posterior Distribution What we know now Figure 2.4: Bayesian Revision Process. 3. draws the posterior distribution. If this model is applied to the volatility estimation process of real options, it is possible to 1. calculate the volatility of the initial analysis time 2. collect additional information on random variables during the period t , and 3. calculate a new volatility of the relevant time through the Bayesian analysis. Miller and Park [42] estimated a value of acquired information by using the Bayesian revision process for the normal distribution, and presented a procedure for applying this value to the assessment of a real option. They predicted the posterior distribution by using the conjugate normal prior distribution, and evaluated the value of a European option by estimating an expected value (mean) of the posterior distribution under the assumption that the variance of a variable is known. Herath and Herath [43] demonstrated that a real option approach with Bayesian post auditing o ers a systematic valuation and risk management framework for evaluating infor- mation security spending by rms. In their model, the conjugate gamma distribution is used to update the technological parameters that are inputs in the real options model. In this research, the Bayesian revision process for a beta and log{normal distribution are developed to estimate a posterior distribution for updating the project volatility. The common estimating procedure for uncertainty factors is to make three{point estimates(Beta{ PERT) and simulation results of the project?s cash ow must be a form of the log{normal 18 distribution to use the MCST method. Therefore, the use of the Bayesian revision process for these distributions becomes appropriate during a process of updating volatility of real options. We will discuss this matter in detail in chapter 4. 2.6 Compound Real Options Large-scale capital-intensive investment projects such as R&D and the construction of energy generating units are conducted in phases according to a series of expenditures, and each phase may be considered an option for the value of the next phase. In other words, the investment in the current phase may lead to a new investment opportunity, which is referred to as a compound option. Therefore, a compound option is an option that can provide e ective evaluation results and promotion strategies for most defense acquisition projects that require phased investment costs and decision-making mechanisms as Figure 2.1. The general methods of evaluating options are the Black-Scholes and the binomial lattice model. However, these two models cannot be used directly to evaluate compound options. Instead, extended models, such as the Geske model [15] or the Herath & Park Approach (HPA) [16], must be used. 2.6.1 Geske Model Geske proposed an evaluation method of compound options, which consists of the initial investment and the follow-up investment, under the assumption that 1) the distribution of asset value follows the log{normal distribution and 2) volatility is xed in the option life. If the option exercise time of initial investment and follow-up investment is T1; T2, the expected discounted cash ow is S1; S2, the current price of investment costs is K1; K2, and the volatility of the underlying assets is , the option value(C) of the Geske model is[17] C = S2N2(a1;b1; ) K2e rf 2N2(a2;b2; ) I1e rf 1N1(a2) (2.4) 19 where, a1 = ln(S2=S 2) + (rf + 2=2) 1 p 1 a2 = a1 p 1 b1 = ln(S2=K2) + (rf + 2=2) 2 p 2 b2 = b1 p 2 = s 1 2; 1 = T1 t; 2 = T2 t However, under the assumption that phased volatility is always xed, the Geske model is a call on call option model { follow up investment option for initial investment option { and thus not suitable for multi-phased investment projects with two or more phases which have di erent volatility for each phase. Cassimon et al.[18] referred to the Geske model as two-fold option, and presented the extended model of a six-fold option to overcome the disadvantage of this model, applying it to new drug R&D by a pharmaceutical company. However, this model also has the shortcomings of assuming xed volatility for each phase. Later, Cassimon et al.[19] proposed the n{fold option model applying phase-speci c volatility, and evaluated software R&D using this model. However, these models based on the Geske model are extended models of the Black-Scholes model. The calculation procedure becomes more complicated as the phases of the project become longer and its use may be limited when evaluating investment projects in the form of American options with a long term for the option expiration in each phase. 2.6.2 Herath & Park Approach (HPA) Herath and Park [16] presented a compound options evaluation method of multi{phased investment projects that have speci c volatility for each phase, using the extended binomial lattice model to overcome the disadvantage of the Geske model, which can only be applied to two{phased investment projects with xed volatility for each phase. The calculation procedure shown in Figure 2.5 [16] uses a two-phased(2-fold) model as an example, but it can be extended to a multi-phased(n{fold) model when repeating the 20 3 33 max[ , 0]C VI +? +? = ? 3 33 max[ , 0]C VI ++ ++ = ? 3 33 max[ , 0]C VI ?? ?? = ? 3 33 max[ , 0]C VI +? +? = ? 3 33 max[ , 0]C VI ++ ++ = ? 3 33 max[ , 0]C VI ?? ?? = ? 3 33 max[ , 0]C VI +? +? = ? 3 33 max[ , 0]C VI ++ ++ = ? 3 33 max[ , 0]C VI ?? ?? = ? 3 33 max[ , 0]C VI +? +? = ? 3 33 max[ , 0]C VI ++ ++ = ? 3 33 max[ , 0]C VI ?? ?? = ? 3 C + 3 C ? 3 C ? 3 C + 3 C ? 3 C + 3 C ? 3 C 3 C 3 C 3 C 2 322 max[ , 0]C CVI ++ = +? 2 322 max[ , 0]C CVI ?? = +? 2 322 max[ , 0]C CVI ++ = +? 2 322 max[ , 0]C CVI ?? = +? 2 C 2 C 1 C 1 211 max[ , 0]C CVI ++ = +? 1 211 max[ , 0]C CVI ?? = +? Call Value [Product Introduction] Call Value [First Expansion] Call Value [Second Expansion] 3 C + Real Call ( 1 C ) Real Call ( 2 C ) Real Call ( 3 C ) 1 1 T ue ? ? = 2 2 T ue ? ? = 3 3 T ue ? ? = 1 1 T de ??? = 2 2 T de ??? = 3 3 T de ??? = 1 1 11 (1 ) f rd p ud +? = ? 2 2 22 (1 ) f rd p ud +? = ? 3 3 33 (1 ) f rd p ud +? = ? 11 1 1 1 (1 ) 1 f pC p C C r ++ +? + +? = + 22 2 2 2 (1 ) 1 f pC p C C r ++ +? + +? = + 33 3 3 3 (1 ) 1 f pC p C C r ++ +? + +? = + 11 1 1 1 (1 ) 1 f pC p C C r +? ?? ? +? = + 22 2 2 2 (1 ) 1 f pC p C C r +? ?? ? +? = + 33 3 3 3 (1 ) 1 f pC p C C r +? ?? ? +? = + 11 1 1 1 (1 ) 1 f pC p C C r +? = + 22 2 2 2 (1 ) 1 f pC p C C r +? +? = + 33 3 3 3 (1 ) 1 f pC p C C r +? +? = + Figure 2.5: Call value payo s associated with the real calls[16]. 21 same process using the binomial lattice model. It is the most tting analytic tool of real options for multi-phased investment projects and can overcome the limitations of methods based on the Geske model. Therefore, this study will use HPA to develop a model for the valuation of defense acquisition projects and decision making. 2.7 Multi-Criteria Decision Analysis (MCDA) As mentioned in Chapter 1, the nal part of the preliminary feasibility assessment uses the whole results of the economic and policy analyses to determine whether the program should be pursued. For this purpose, MCDA is often used to identify the best alternative that accomplishes the multiple objectives. There are numerous MCDA techniques that can be used to identify a single most pre- ferred alternative, to rank alternatives, to list a limited number of alternatives for subsequent detailed evaluation, or to distinguish acceptable from unacceptable possibilities[23]. These techniques have been very widely used to evaluate multiple con icting criteria to arrive at better solutions. Hajkowicz and Higgins [24] applied MCDA methods including weighted summation, range of value, PROMTHEE II, Evamix and compromise programming, to water management decision problems. They showed that di erent MDCA methods were in strong agreement with high correlations amongst rankings. Maldonado et al. [25] applied six MCDA methods for selection of a orestation sites: Analytical Hierarchy Process (AHP), ELECTRE III, PROMETHEE II, Compromise Programming (CP), Stochastic Multi{Criteria Accept- ability Aanalysis (SMAA-2) and Iterative Ideal Point Thresholding (IIPT). In this study, all methods designated the same four land units as the most suitable alternatives, while all methods except IIPT designated the same land unit as the least suitable. Among these numerous MCDA methods, Analytical Hierarchy Process (AHP) is the most popular technique and has been utilized in various areas for making the best decision with the available alternatives. A critical feature of AHP is that it creates a hierarchy which consists of criteria and sub-criteria as assessment elements and measures the level of 22 relative importance of these criteria through pairwise-comparison. It helps decision{makers in organizing their values and preferences to make e ective decisions and transfer them into quantitative ratios to weight criteria.[68]. In Korea, the AHP method must be used for preliminary feasibility assessments in any defense acquisition programs to integrate economic and policy analysis results for determining whether to implement or do{nothing. However, relatively little e ort has been devoted to comparing the strengths or weak- nesses of those methods and determining which method is most suitable for current decision situations. This research aims to establish a decision model for assessing the feasibility of an acquisition project before it is introduced. Given this decision situation, decision-makers should question which MCDA method should be utilized to integrate the results of economic and policy analysis instead of just utilizing the most popular method without comparative analysis. Therefore, we will compare AHP, the representative method of \Alternative{Focused Thinking", with the \Value-Focused Thinking" method, which is similar to AHP but accesses an issue through di erent ways to develop a more suitable MCDA model for preliminary feasibility assessment. We will discuss this matter in detail in chapter 5 and 6. 23 Chapter 3 An Economic Evaluation Framework for Defense Acquisition Programs : Compound Real Options Applied to the Korea Helicopter R&D Program (KHP) In this chapter, the compound real options model is applied to a real defense program, KHP. First, we overview KHP to explain the characteristics of the program and evaluate its net present value (NPV) using the traditional discounted cash ow(DCF) approach. Second, we address critical uncertainties a ecting the project value and set stochastic characteristics of these uncertainties for project evaluation. Third, we estimate the probability distribution of the project?s present value and determine the project volatility with the Monte Carlo sim- ulation, which consolidates all elements of defense project uncertainty. Finally, we evaluate KHP using the compound real options model. 3.1 Korea Helicopter R&D Program(KHP) Outline The armed forces of the Republic of Korea (ROK) have purchased all their helicopters from overseas, and over 70% of them are no longer produced, causing many problems such as excessive costs and time required for maintenance. Accordingly, the ROK Armed Forces intend to replace the deteriorating airmobile helicopters through KHP. There are two ways to acquire weapons systems: overseas introduction (production by technical transfer or direct purchase) and research and development (R&D). Overseas introduction enables the weapons to be stably acquired within a relatively short period of time, whereas R&D requires high initial costs and a long period of time, as well as various risks and uncertainties in pro- moting the projects. However, in the long-term view, securing technologies through R&D 24 will create many good e ects such reducing management and maintenance costs of the rele- vant weapons, improving operation ratios, accumulating domestic technology, and increasing related business pro ts. The main objectives of the KHP are as follows: Localization and development of the major dual{use components for military and civil helicopters Construction of military helicopters with the localized components R&D of new and core advanced technologies for securing the independent capability to develop helicopters To achieve these objectives, the Ministry of National Defense, the Ministry of Knowledge and Economy, and various academic and research institutes, as well as about 20 domestic and overseas companies, will participate in KHP. With the goals of mass{producing helicopters through R&D and securing technology for key components, the ROK Armed Forces aim to promote a project in which the follow{up investment is executed according to the success or failure of plans in each phase, as shown in Figure 3.1. NPV1 Expand Success Expand Success Failure NPV2 Expand Success Failure Discontinuation Success Failure Start R&D Discontinuation Failure Discontinuation [Phase 1] Design [Phase 2] System & Prototype Production [Phase 3] Testing Phase 4 Mass Production & Deployment T=1 T+3 T+5 T+7 T+27 T=0 [Phase 0] Program Analysis Figure 3.1: Korea Helicopter R&D Program(KHP)Process. 25 3.2 Traditional Approach for KHP Assessment 3.2.1 Cash Flows Estimation Before we use the real options model to capture exibility in decision making, the cash ows from the project need to be de ned and carefully reviewed to estimate the net present value(NPV) of the project without exibility. NPV is the foundation for real options analysis. [27] The cash ow of KHP is estimated by analyzing total life-cycle cost presented in the assessment data(input) and the economic impact of R&D(output), and dividing by the year. Firstly, the total life-cycle cost in the assessment data is estimated to be $9.1 billion, in- cluding R&D, mass production, and maintenance for 20 years. The economic impact was calculated by the sum of 1) e ect on production inducement, 2) productivity of added value, and 3) technology ripple e ect in the aviation industry. The e ect on production induce- ment is an index that represents the direct and indirect level of production inducement in other industries required to ful ll a unit of nal demand in a certain industry. Productiv- ity of added value was calculated by multiplying the added value ratio by gross national production according to the promotion of KHP. The economic impact is de ned as bene ts generated from conducting the project. The economic impact of the entire helicopter project is estimated to be $13.9 billion. Table 3.1: KHP Total Life Cycle Cost and Economic Impact(Revenue). dollars in billions Total Life-Cycle Cost Economic Impacts (Revenue) Phase 1, 2, 3 (R&D) 1.3113 2.6163 Phase 4 (Mass Production) 4.1565 7.9897 Phase 4 (Deployment & Maintenance) 3.6351 3.2684 Sum 9.1029 13.8744 Total life{cycle cost and economic impact are assumed to be the total of annual esti- mates, and the costs in Phases 1, 2, and 3 are distributed di erentially according to year 26 based on the defense intermediate budget plans for KHP. Costs in Phase 4 (mass produc- tion/deployment & maintenance) and the ripple e ects of all phases are equally distributed for each year, resulting in the following estimate of cash ow: 0 T=1 T=3 T=5 T=7 Phase 1 Expansion Phase 2 Expansion Phase 3 Expansion Phase 4 Expansion E x p ec t ed C a s h F l o w s : $ m il l io n 34.1 86.2 104.3 776.0 571.5 716.6 254.9 390.9 ? ? ? ? ? ? T=27 T=26 T=8 T=6 T=4 T=2 34.1 86.2 104.3 843.7 1371.9 1397.4 Phase 0 Phase 1 Phase 2 Phase 3 Phase 4 T+1 T=0 T+3 T+5 T+7 T+27 791.4 T=9 T=10 1 1 1 1 1 1 1 1 1 Figure 3.2: Investment Costs and Incremental Cash Flows. 3.2.2 Risk-Adjusted Discount Rate (k) and Risk-Free Rate (rf) There are various ways to obtain the risk-adjusted discount rate (k), such as CAPM(capital asset pricing model), WACC(weighted average cost of capital), MAPT(multifactor asset pricing model), comparability analysis, and a rm-or project-speci c hurdle rate [29]. The long-term government bond earnings rate is generally applied to the risk-free rate (rf). However, the National Assembly Budget O ce?s assessment data do not include the discount rate and risk-free rate, and calculating these values is a di erent eld of study. Thus, this study will use a risk{adjusted discount rate(k) of 5.5%, the real social discount rate for national public investment projects presented by KDI [1], and a risk{free rate(rf) of 3.21%, the ve-year government bond interest rate. 27 3.2.3 Net Presen tV alue (NPV) of KHP by Discoun ted Cash Flo w (DCF) Approac h Table 3.2 sho ws the NPV results through the traditional DCF metho d based on the National Assem bly Budget O ce?s assessmen tdata. Eac h phase resulted in gains, as the ind ustrial and tec hnological ripp le e ects exceeded inv estmen tcosts. Therefore, when the costs and economic impacts (b ene ts) estimated by the National Assem bly Budget O ce are divided by year and discoun ted by the tradit ional DCF metho d, the Exp ected Net Presen tV alue(ENPV) of KHP is an inv estmen tv alue of $7.10 billion. Table 3.2: KHP Cash Flo ws and Exp ected Net Presen tV alues ($mill ion). Y ear 0 1 2 3 4 5 6 7 8 9 ? ? ? 26 27 P has e 1 A nnual C as h F l ow 34. 1 34. 1 34. 1 34. 1 34. 1 35. 0 35. 0 35. 0 35. 0 35. 0 I nv es t m ent ( 571. 5) EN PV1 ( 92. 6) P has e 2 A nnual C as h F l ow 51. 4 52. 4 52. 1 52. 1 51. 4 52. 7 51. 7 51. 7 I nv es t m ent ( 716. 6) EN PV2 ( 27. 0) P has e 3 A nnual C as h F l ow 18. 1 19. 1 18. 8 19. 1 26. 4 26. 8 I nv es t m ent ( 254. 9) EN PV3 ( 6. 9) P has e 4 A nnual C as h F l ow 670. 9 684. 6 1, 258. 8 1, 284. 0 I nv es t m ent ( 390. 9) EN PV4 7, 231. 2 EN PV R &D 7, 104. 8 P has e1 473. 8 449. 1 P has e2 684. 9 615. 3 phas e3 246. 0 221. 0 phas e4 10, 910. 1 9, 802. 2 0. 5% 1. 5% 2. 5% 3. 5% 4. 5% 5. 5% 6. 5% 7. 5% 8. 5% 9. 5% v1 847. 0 732. 0 642. 0 566. 0 503. 0 449. 1 403. 0 364. 0 330. 0 301. 0 v2 1, 162. 5 1, 012. 0 886. 0 780. 0 691. 0 615. 3 550. 0 495. 0 447. 0 405. 0 v3 408. 0 357. 0 314. 0 278. 0 247. 0 221. 0 198. 0 178. 0 161. 0 146. 0 v4 10, 801. 0 10, 590. 0 10, 384. 0 10, 184. 0 9, 990. 0 9, 802. 2 9, 619. 0 9, 440. 0 9, 267. 0 9, 099. 0 I1 I2 I3 I4 28 3.3 Handling Risk and Uncertainty of the Defense Project As mentioned in Chapter 2, it is possible to calculate KHP volatility( ) from the project value distribution, VT ( T; 2T), using Equation 2.2. This project volatility consolidates the uncertainties of multiple input variables such as bene t and costs. Thus, various types of uncertainty a ect the design and operation of the defense project. Uncertainty 1 Uncertainty 2 Uncertainty n ? ? ? Monte Carlo Simulation Year 1 2 3 ?T Present Value Model Project Volatility Estimation Output Probability of PV 0 PV 2 , 2 () ln 1 TT T T PV T ?? ? ? ? ?? ?? +?? ?? ?? ?? = ? Inputs Figure 3.3: Considering Various Uncertainties for estimating the Project Volatility. Uncertainties are things that are not known, or known only imprecisely. Estimates of these separate uncertainties are taken either from historical data or from the subjective estimates of management. Uncertainties lead to risks(potential negative outcomes) or oppor- tunities(potential positive outcome).[44] Defense planners must develop strategies to hedge against uncertainty in multi{stage R&D projects where managers can consider continuing, improving, or abandoning development at each decision point. This need to reduce uncer- tainty is the fundamental reason for using the real options technique. Increased uncertainty improves upside potential and option values while limiting downside losses. Huchzermeier and Loch [45] evaluate changes in option values of an R&D project in the presence of ve types of operational uncertainty: 1)market payo variability, 2)budget variability, 3)perfor- mance variability, 4)market requirement variability, and 5)schedule variability. They con- clude that the value of increased managerial exibility through the use of real options in- creases with increased variability in market payo s and budgets.[46] Therefore, uncertainties 29 should be addressed in defense project valuation and decision{making by limiting risks while increasing opportunities. The real options approach can include two broad categories of uncertainties:{technical uncertainties and economic uncertainties[27][31]. Economic uncertainty is a function of fac- tors exogenous to the project, such as general market conditions. In other words, eco- nomic uncertainty is correlated with the general movement of the economy. It increases over time and is identi ed through objective market information. On the other hand, technical(project{related) uncertainty is a function of factors endogenous to the project, such as the success of di erent phases of R&D. Thus, technical uncertainty is not correlated with the general movement of the economy.[55] Technical uncertainty is greatest at the start of the project, but diminishes as the defense planner invests and learns more. It is identi ed through subjective judgment. [43] 3.3.1 Technical Uncertainty of the Defense Project In addition to the various cost, schedule, and programmatic uncertainties, defense R&D projects have to contend with a high degree of technical uncertainty. This uncertainty is due to broadly de ned threshold performance levels, insu cient technological maturity to produce the desired capability, or changing required operational capability(ROC) during the course of the acquisition. Defense acquisition planners often mitigate the technical uncertainty through a combination of formal milestone decision points strategies as shown in Figure 3.4.[46] Concept Refinement Technology Development System Dev & Demonstration Production & Deployment Operations & Support Decision Milestones A B C Concept Decision Design Readiness Assessment Full Rate Production Review Figure 3.4: DOD Acquisition Process. 30 This multi{stage decision{making strategy can be a useful tool for mitigating technical uncertainty. However, because R&D costs are a ected by the degree of technical uncertainty, defense acquisition managers must accurately measure the degree of technical uncertainty to successfully employ this strategy and acquire the desired capability on time and within budget. Insu cient technological maturity with a high level of technical uncertainty requires more money to produce the desired capability and satisfy ROC. Technological maturity is measured by the adjustment parameter and is captured by a defense acquisition expert?s assessment of how secure the required technology is at a given point in time. A common metric that can be employed in the defense eld to assess the ma- turity of evolving technologies is the Technology Readiness Level(TRL). TRL is determined by Technology Readiness Assessment, a process that assesses the maturity of and the risk associated with critical technologies to be used in defense acquisition projects. This process provides a basis for milestone decisions about whether the technologies of the project have acceptable levels of maturity in a relevant defense environment. Thus, TRL can serve as a helpful knowledge{based standard and shorthand for evaluating technology uncertainty, but it must be supplemented with expert professional judgment[47]. Table 3.3 and Table 3.4 show de nitions of TRL and required TRL at each phase of the acquisition process. Each phase of TRL can be assessed by various information and standards as shown in Table 3.5. Table 3.3: Required TRL at Each Stage of Defense Acquisition Process. Concept Refinement Technology Development System Dev & Demonstration Production & Deployment Operations & Support TRL 1~4 TRL 5~6 TRL 7 TRL 8 TRL 9 Decision Milestones A B C If a project does not attain the required TRL at each phase, it incurs an increased risk of technical problems, resulting in potential cost and schedule growth. Most of the defense 31 Table 3.4: Technology Readiness Level(TRL) de nitions. TRL Definition 1 Basic principles observed and reported 2 Technology concept and application formulated 3 Analytical and experimental critical function and characteristic proof-of-concept 4 Component and breadboard validation in laboratory environment 5 Component and breadboard validation in relevant environment 6 System / subsystem model or prototype demonstration in a relevant environment 7 System prototype demonstration in an operational environment 8 Actual system completed and qualified through test and demonstration 9 Actual system proven through successful mission operations Table 3.5: Technology Readiness Level(TRL) Assessment Information. TRL Supporting Information 1 Published research that identifies the principles that underlie this technology 2 Publications or other references that out-line the application being considered and that provide analysis to support the concept 3 Results of laboratory tests performed to measure parameters of interest and comparison to analytical predictions for critical subsystems 4 System concepts that have been considered and results from testing laboratory-scale breadboard(s). Provide an estimate of how breadboard hardware and test results differ from the expected system goals 5 Results from testing laboratory breadboard system are integrated with other supporting elements in a simulated operational environment 6 Results from laboratory testing of a prototype system that is near the desired con-figuration in terms of performance, weight, and volume 7 Results from testing a prototype system in an operational environment 8 Results of testing the system in its final configuration under the expected range of environmental conditions in which it will be expected to operate. Assessment of whether it will meet its operational requirements 9 Reporting of Operational Test and Evaluation (OT&E) Results 32 programs assessed by GAO proceeded with less knowledge(TRL) at critical junctures than suggested by best practices, although several came close to meeting best practice standards. The inaccuracy of early investment cost estimates for developing major USDOD systems is well documented, and cost overruns have been a common problem[48]. For example, in the case of the Comanche reconnaissance attack helicopter(RAH-66), USDOD did not attain the required TRL at the system development stage but still launched the production stage. As a result, costs were increased and the schedule was delayed as shown in Figure 3.5[49]. Concept System Development Production 6/88 Program Start 4/00 Development Start 12/06 Low-Rate Decision 5019 Last Procurement Program Performance(fiscal year 2004 dollars in millions) As of 10/2001 Latest 12/2010 Percent change R&D cost $8,905.1 $12,573 41.2 Unit cost $33 $53 61.9 Total quantities 1,213 650 -46.4 Acquisition cycle time(months) 223 256 14.8 11/09 Full-Rate Decision . Figure 3.5: GAO Assessment of RAH{66 33 3.3.2 Modeling Technical Uncertainty with Expected Mission Ful llment Rate (EMFR) We use Sohn and Kim?s [50] expected mission ful llment rate(EMFR) approach to estimate the degree of attaining TRL at each stage of R&D process. The EMFR approach is based on the cost of ownership(COO) approach, which was developed to address the economic and productive performance of a project by estimating the total life{cycle cost of a speci c processing step. This approach analyzes all costs associated with the acquisition, use and maintenance of a good or service. It considers not only price but also product quality, failure costs, administrative costs, and maintenance, among other factors. The basic COO algorithm is described below[51]: COO(Cu) = Cf +Cv +CyTPT Y c U (3.1) where; Cu = cost per good unit Cf = xed cost; Cv = variable cost; Cy = cost of yield loss TPT = throughput; Yc = composite yield; U = utilization The denominator of Equation 3.1 is an estimate of the number of good units produced during the life of the project. Throughput rate(TPT) is based on measurement and handling times, including factors such as sample preparation, loading and unloading, reporting, and other overhead operations. Composite yield(Yc) is de ned as the ratio of good units compared to the total number of units produced. Utilization(U) is the ratio of actual usage compared to total available time. Utilization includes repair and maintenance time, both scheduled and unscheduled. This equation shows the impact of non-productive time on cost and normalizes ideal throughput to a realistic estimate[52]. 34 Sohn and Kim?s [50] expected mission ful llment rate(EMFR) approach is similar to the COO model and is described as follows: COO(Cu) = CDP +CVAP +CYTAU EMFR (3.2) where; Cu = cost per acquisition unit CDP = cost of development and production CVAP = variable cost by acquisition plan CY = cost of yield loss TAU = total number of acquisition units Sohn and Kim estimated COOs of four alternatives (four types of helicopter) having di erent performance levels. EMFR has values ranging from 0 to 1. The alternative with the highest possible performance has an EMFR value of one point. If another alternative has 75% of performance level as compared with the best alternative, it has an EMFR value of 0.75. Most common defense acquisition decisions rely heavily on the life cycle costs. But the EMFR approach considers the performance level of ROC as well as life cycle costs for estimating acquisition costs including defense project characteristics. In this research, EMFR is de ned as the percentage(%) of achievement of the required TRL at each stage of the acquisition process. EMFR is measured by defense acquisition experts? assessment of how secure the required TRL is at the current stage. It is assumed that the initial estimated investment cost is the minimum value for the investment cost; thus if EMFR is 1 (i.e., if required TRL is expected to be 100% achieved at the current stage), there is no additional cost and no technical uncertainty at the current stage. A lower value of EMFR means that the current stage of project development has less knowledge and a lower TRL than expected by best practices and goals, in which case the investment costs 35 will be increased, as shown in Figure 3.6. This relationship is described as follows: EICi = IICiEMFR(" i) ; i = 1;2;:::;n (number of stage) (3.3) where; EIC = expected investment costs IIC = initial estimated investment costs Figure 3.6: Variability of EIC at Each Phase by EMFR. To construct probability models, the common estimating procedure for EMFR is to make three point estimates (PERT): a minimum (pessimistic) value, a maximum (optimistic) value, and a most likely value. These three estimates are used as the upper bound, the lower bound, and the mode of the corresponding EMFR probability distribution of "n at the end of stage n. Esto("n) = U (upper bound) = optimistic estimate Estp("n) = L (lower bound) = pessimistic estimate Estm("n) = Mo (mode) = most likely estimate 36 This distribution is a useful tool for modeling expert data and is designed to generate a distribution that more closely resembles realistic probability distribution. When used in a Monte Carlo simulation, the PERT distribution can be used to identify risks in project and cost models based on the likelihood of meeting targets and goals. The PERT distribution constructs a smooth curve which places progressively more emphasis on values around (near) the most likely value, in favor of values around the edges. In practice, this distribution of values means that we \trust" the estimate for the most likely value, and even if it is not exactly accurate (as estimates seldom are), we expect that the resulting value will be close to that estimate[53]. Three-point estimates of EMFR at each phase are given in Table 3.6. These values are a little higher than typical defense R&D because Eurocopter, as the primary partner of KHP, will provide technical assistance and supply the transmission and autopilot subassemblies for the helicopters. Eurocopter is a wholly{owned subsidiary of EADS, a global leader in aerospace, defense, and related services. Eurocopter has a stake of 30% in the development phase and 20% in the production phase of KHP. This partnership will reduce the development risk and achieve timely deployment through actively induceing foreign capital (joint ventures, consortiums, etc.) and encouraging joint development of domestic technology and advanced technology[54]. Thus, both EMFR and the success possibility of KHP are higher than the normal defense R&D. Table 3.6: Three{point estimates of EMFR ("i;i = 1;2;3;4). Phase 1 2 3 4 Optimistic 1 1 1 1 Most Likely 0.85 0.87 0.92 0.95 Pessimistic 0.60 0.7 0.80 0.90 37 3.3.3 Economic Uncertainty of the Defense Project For project valuation, economic (or market) uncertainty is a function of factors exoge- nous to the project, such as general market conditions. In other words, economic uncertainty is correlated with the general movement of the economy such as market demand and the price of goods[55]. Typically, economic or market uncertainties are not considered in traditional defense R&D projects because the ultimate goal of defense R&D is to satisfy military requirements and the military is considered the only major consumer. However, KHP is di erent from conventional acquisition programs of the Ministry of Defence. KHP is intended not only to economically and timely develop helicopters that will satisfy the performance needs of the ROK military, but also to reduce maintenance costs and acquire technology related to heli- copter and core components to promote the domestic aviation industry. KHP will provide momentum for jump{starting domestic aviation industries, as well as accelerate develop- ment of future civilian helicopters and overseas expansion if it can acquire the capability to develop system integration and core components. These goals are common characteristics of the most modern defense acquisition projects requiring huge capital investments and ad- vanced technology. The United States, the United Kingdom, Germany, Russia, and most of the developed countries are planning and implementing acquisition projects to achieve a variety of purposes beyond simply procuring military weapon system, including technology development, increased employment, national revenue growth, etc.. Therefore, in order to estimate the cash ow, we must consider the uncertainty of KHP?s bene t to the economy through the above purposes and characteristics. If KHP develops the capability of system integration, core components, and overseas expansion, KHP bene ts will be a ected by the domestic and world demand for rotary wing aircraft and the ROK?s share of the aircraft market. 38 3.3.4 Modeling Economic Uncertainty with a Consolidated Approach As shown in Figure 3.3, the consolidated approach allows the uncertainties of multiple input variables such as bene t functions and various costs, all of which drive volatility, to be considered as a single source of market uncertainty. The estimates of uncertainties pertaining to speci c input variables can be obtained from objective or subjective estimates[43]. We use the costs and bene t information estimated by South Korea?s National Assembly Budget O ce[2]. This estimate consolidates all possible costs, such as xed costs and variable costs, and bene t uncertainties, such as bene t of production inducement, productivity of added value, and technology ripple e ect. However, it does not consider incremental costs for operational maintenance and the demand and growth rate of the aviation industry market. Thus, Thus, we re{estimate the total bene ts created by KHP by considering incremental costs and the possibility of a share in the overseas helicopter industry market. First, one of the simplest assumptions is that the uncertainty follows Geometric Brow- nian Motion (GBM) where its value next period, Vt+ t, is equal to its value current period, Vt, multiplied by a continuous growth factor at rate r for an interval t. Vt+ t = Vt er t (3.4) The growth rate, r, is a normally distributed random variable with constant expected growth ( r) and constant standard deviation. At the end of one period (a year), r lies with 95% con dence within the following interval: r2[ r 2 ; r + 2 ] (3.5) For a period of time T = n t, the interval is r2[ r T 2 pT; r T + 2 pT] (3.6) 39 Then, the random variable has values with 95% con dence within the following interval:[27] VT 2[V0e r T 2 pT ; V0e r T+2 pT ] (3.7) If the experts or reliable historical data provide the higher or the lower values of V, the volatility of the rate of growth will be derived: = ln (Vupper T V0 ) nX i=1 ri 2pT or nX i=1 ri ln (Vlower T V0 ) 2pT (3.8) The current expected annual operational maintenance costs are 12.1($M) at T + 8 and annual growth rate is 3.5%, due to increasing costs of raw materials, labor, and oil price, etc.. At 20 years later (T + 27), if the expected lower level costs are 23.5($M), the volatility of cost is C = 20 0:035 ln 23:5 12:1 2p20 = 3:62% (3.9) Thus, the annual growth rate of operational maintenance costs is rC N(0:035;0:03622) and we generate costs for each year with the Ct = Ct 1erC. Next, Figure 3.7 shows forecast data for the world rotary{wing aircraft market after KHP is launched[56]. Complete products and MRO(Maintenance Repair and Overhaul) includes civilian, military, and unmanned helicopters classi ed into light, medium, and heavy classes. Table 3.7 shows the forecasted results of possible market share in the world rotary wing aircraft industry after successful R&D[57]. Market share is di cult to estimate precisely, thus, we use two{point estimates (uniform distribution) for sampling in the Monte Carlo simulation. Using the uniform distribution is useful in situations in which we have a minimum and maximum estimate available, but no other information. If we can get a most likely 40 F+ year b i lli o n $ Figure 3.7: World Rotary{Wing Aircraft Market Forecast estimate in addition to the minimum and maximum, we can use the additional information to create a more realistic probability model.[53] Table 3.7: Two{point estimates of the Forecasting Market Share. Class F+7 ~ F+17 F+18 ~ F+27 Pessimistic Optimistic Pessimistic Optimistic Large (25,000lbs~) 5% 7% 7% 10% Medium (16,000lbs~25,000lbs) 10% 15% 15% 30% Light (12,000lbs~16,000lbs) 3% 7% 7% 10% 3.4 Estimating Project Volatility in Defense Acquisition Planning As mentioned in chapter 2, the Monte Carlo Simulation with time frame method(MCST) is used for estimating KHP volatility. While a multitude of uncertainties may exist in de- fense projects, these uncertainties must be quanti ed and reduced because it is exceedingly 41 di cult to develop analytical models with more than two sources of uncertainty. To solve this problem, MCST, the consolidated approach, can be used to combine the multiple un- certainties that drive the value of a defense project via Monte Carlo simulation into a single uncertainty: project volatility.[27][43] MCST considers the time frame of option life in esti- mating project volatility, thus overcoming the shortcoming of the C&A method, which does not consider option life. When defense planners promote a large investment project, they conduct an initial evaluation of its value and uncertainty years before, and attempt to make strategic decisions using real options as a tool for project valuation and decision making. Thus, a rational volatility estimation method considers the option life before the project starts. If we assume that an project value(V) follows the Geometric Brownian Motion(GBM), the stochastic process model of V is equivalent to: dV = rVdt+ Vdz (3.10) r : Growth Rate(Mean); : Project Volatility; dz : Wiener Process If Equation 3.10 is converted to the discrete time model using Euler?s Discretization method for the simulation of the project value, it is V = Vt+ t Vt = rVt t+ Vt"p t; " N(0;1) (3.11) V of Equation 3.11 is the changed portion of V during the short period of time, indi- cating that expected return changes as much as rVtdt but the uctuation of expected return Vt"p t is uncertain and thus depends on probability distribution (" N(0;1)). However, as errors may occur in simulation of the stochastic process model using Equation 3.11, it uses the stochastic process model of lnV in which the project value takes a natural loga- rithm. This model re ects that the project value follows the log{normal distribution from 42 0 to the positive in nity(+1), and the logarithmic rate of return with project taking natu- ral logarithm follows the normal distribution from the negative in nity(-1) to the positive in nity(+1). These are same concepts that a stock price follows the log-normal distribution from 0 to the positive in nity(+1) and a rate of return of stock follows the normal distri- bution from the negative in nity({1) to the positive in nity(+1) in the nancial market. Thus, the stochastic process is drawn by Ito?s Lemma: d[lnVT] = r 12 2 dt+ dz (3.12) lnVT N lnV0 + r 12 2 t; 2t (3.13) From Equation 3.13, the expected value and variance of VT are given as: E(VT) = T = elnV0+(r 12 2)T+ 2 2 T = V0erT (3.14) Var(VT) = 2T = e2(lnV0+(r 12 2)T)+ 2T e 2T 1 = V 20 e2rT(e 2T 1) (3.15) If Equation 3.15 is rewritten with (project volatility): = vu uu uu t ln 2 4 T T !2 + 1 3 5 T (3.16) Figure 3.8 represents these mathematical relations. In order to estimate the KHP volatility parameter, we need to obtain the present value distribution of KHP using the Monte Carlo simulation. KHP will be conducted in four phases, in each of which a series of expenditures and current investments will lead to the next investment opportunity. In other words, each phase is considered an option for the value of the next phase. Thus, the gross project value of each phase is determined by the discounted value of future cash ows(excluding the investment costs) and speci c volatility 43 0 0 V T V T ? ? ? T+1 T+2 T+n ? ? ? 2 , () T TT V ??? Option Life Use the Monte Carlo simulation to obtain the distribution by aggregating the periodic random cash flows T V 0 V T V Option Life : T ? ? ? t ue ? ? = t de ??? = 2 , () T TT V ??? 2 ln 1 T T T ? ? ? ?? ?? ??+ ?? ?? ?? = If is small compared to 1, is Log-Normally distributed t? T V Figure 3.8: Mathematical relationships between and T[13]. 44 for the compound option model. Therefore, using 10,000 simulation trials for each phase, we estimate the project value as V1;:::V4, respectively as shown in Figure 3.9. Histogram of x PV f(PV) 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 0.0 0.2 0.4 0.6 0.8 (a) Phase 1: V1 LN(449:1;70:92) Histogram of x PV f(PV) 1.0 1.5 2.0 2.5 3.0 3.5 4.0 0.0 0.2 0.4 0.6 0.8 1.0 1.2 (b) Phase 2: V2 LN(615:4;115:12) Histogram of x PV f(PV) 0.8 1.0 1.2 1.4 1.6 1.8 2.0 0.0 0.5 1.0 1.5 2.0 (c) Phase 3: V3 LN(221:0;51:22) Histogram of x PV f(PV) 2468 0.00.10.20.30.40.5 (d) Phase 4: V4 LN(9;802:1;2;904:32) Figure 3.9: The Project Present Value Distributions at Each Phase. Next, we can calculate phase{speci c volatility parameters using Equation 3.16 as shown in Table 3.8. Notice that the present value distribution of the project must be log{normal distribution to use the Equation 3.16 as shown in Figure 3.8. 3.5 Compound Real Option Model for KHP Using the extended binomial lattice compound real options model as shown in Figure 3.10, we evaluate KHP, which is a multi-phased investment project having speci c volatility for each phase. In the binomial lattice framework, the GBM process for a project value is 45 Table 3.8: Initial Estimated Volatility of Each Phase. Phase 1 Phase 2 Phase 3 Phase 4 Option Life 2 years 2years 2years 2years Volatility 11.10% 13.11% 16.10% 20.51% modeled by a multiplicative binomial process in discrete time. To calculate the option value, major parameters for each investment phase are de ned in Table 3.9. The project value is the Table 3.9: Parameters for the Binomial Lattices. [Phase 1] Design [Phase 2] System & Prototype Production [Phase 3] Testing [Phase 4] Mass Production & Deployment Project Values 1 449.1V = 2 615.4V = 3 221.0V = 4 9,802.1V = Volatility 1 11.10%? = 2 13.11%? = 3 16.10%? = 4 20.51%? = Investment Cost 1 571.5I = 2 716.6I = 3 254.9I = 4 390.9I = Upside Potential 1 1.117u = 2 1.140u = 3 1.175u = 4 1.228u = Downside Potential 1 0.895d = 2 0.877d = 3 0.851d = 4 0.815d = Risk-free probability of upside change 1 0.6166q = 2 0.5893q = 3 0.5591q = 4 0.5266q = present value of the cash ows discounted by the risk{adjusted rate (k = 5:5%) at the time of investment. The exercise price for the compound real call option is the investment costs of each phase and the sum of each exercise price is the total investment costs of KHP. Risk{ free interest rate(rf) is basically used as the interest rate of three{year maturity government bonds with the percentage stated per annum(3:21%). Since there are multiple assets, the risk{free(neutral) probabilities qi have to be calculated separately for each of the real call options. The equation for risk{free probability for each call using the standard binomial model is qi = ((1 +rf) di)=(ui di), where ui = exp( ip t) and di = exp( ip t). 46 P h a s e 0 C on c ep t i on P h a s e 1 De s i gn P h a s e 2 Sy s te m & Pr o t o ty p e Pr o d u c ti o n P h a s e 3 T e st P h a s e 4 Ma s s Pr o d u c t i o n & D e pl o y m e n t 4 44 m a x[ , 0] ud ud C VI = ? 4 44 m a x[ , 0] uu uu C VI = ? 4 44 m a x[ , 0] dd dd C VI = ? 4 d C 4 C 1 4 4 4 44 4 4 m a x [ , { (1 ) } (1 ) ] e x e r c i s e ke e p ope n u uu ud f C uV I q C q C r ? = ? +? + ?? 4u C 1 4 4 4 44 4 4 m a x [ , { (1 ) } (1 ) ] e x e r c i s e ke e p ope n d ud dd f C dV I q C q C r ? = ? +? + ?? 4 1 q ? 4 q 4 1 q ? 4 q 4 1 q ? 4 q 1 4 44 4 4 3 m a x [ 0 , { (1 ) } (1 ) ] do not i nv e s t i nv e st ud f C qC q C r I ? = +? + ? ?? 3 3 43 m a x[ , 0] ud ud C V CI = + ? 3 3 43 m a x[ , 0] uu uu C V CI = + ? 3 3 43 m a x[ , 0] dd dd C V CI = + ? 3 d C 3 C 1 3 3 3 33 3 3 m a x [ , { (1 ) } (1 ) ] u uu ud f C uV I q C q C r ? = ? +? + 3u C 1 3 3 3 33 3 3 m a x [ , { (1 ) } (1 ) ] d ud dd f C dV I q C q C r ? = ? +? + 3 1 q ? 3 q 3 1 q ? 3 q 3 1 q ? 3 q 1 3 33 3 3 2 m a x [ 0 , { (1 ) } (1 ) ] ud f C qC q C r I ? = +? + ? 2 2 32 m a x[ , 0] ud ud C V CI = +? 2 2 32 m a x[ , 0] uu uu C V CI = +? 2 2 32 m a x[ , 0] dd dd C V CI = +? 2 d C 2 C 1 2 2 2 22 2 2 m a x [ , { (1 ) } (1 ) ] u uu ud f C uV I q C q C r ? = ? +? + 2u C 1 2 2 2 22 2 2 m a x [ , { (1 ) } (1 ) ] d ud dd f C dV I q C q C r ? = ? +? + 2 1 q ? 2 q 2 1 q ? 2 q 2 1 q ? 2 q 1 2 22 2 2 1 m a x [ 0 , { (1 ) } (1 ) ] ud f C qC q C r I ? = +? + ? 1 1 21 m a x[ , 0] uu C V CI = + ? 1 1 21 m a x[ , 0] dd C V CI = + ? 1 C 1 1 q ? 1 q 1 1 11 1 1 0 m a x [ 0 , { (1 ) } (1 ) ] ud f C qC q C r I ? = +? + ? T 1 T + 2 T + 3 T + 4 T + 5 T + 6 T + 7 T + 1 T + 3 T + 4 T + 5 T + 3 T + 2 T + 6 T + 7 T + 5 T + ? ? ? Figure 3.10: Ev aluation Pro cess of HP A for KHP . 47 KHP is a large{scale capital{intensive R&D project conducted in phases according to a series of expenditures as Figure 3.1. Each phase is considered an option for the value of the next phase. Investment in the current phase may lead to an investment opportunity in the next phase, an opportunity which is referred to as a compound option. In the binomial lattice model for the compound options, an upward movement is shown to the right and a downward movement is shown to the right but one step down. For the real call C1, the initial gross project value V1 is 449.1. Therefore the two values at T = 1 would be Vu1 = 501:8 and Vd1 = 401:9. Similarly, for the two period real call C2, we obtain the three values at T = 3: Vuu2 = 799:9, Vud2 = 615:4, Vdd2 = 473:5. We repeat this procedure for the two period real calls C3 and C4 to obtain the gross project values. Next, to obtain the compound option value, we start with the inner most real call option C4 and then work backwards to value the nested real options C3, C2, and C1. The real call C4 would be valued by multiplying the terminal call values by the risk{free probabilities and then discounting by the risk{free interest rate(rf) at each node in the two period lattice. The terminal call values are Cuu4 = max[14;772:9 390:9;0] = 14;382:0 Cud4 = max[9;802:1 390:9;0] = 9;411:2 Cdd4 = max[6;503:9 390:9;0] = 6;113:0 then by risk{neutral discounting we obtain the values of Cu4 and Cd4 Cu4 = max[14;382:0(0:5266) + 9;411:2(0:4734)1 + 0:321 ; (12;033:5 390:9)] = 11;654:8 Cd4 = max[9;411:2(0:5266) + 6;113:0(0:4734)1 + 0:321 ; (7;984:5 390:9)] = 7;605:7 48 Finally, we nd the value of the real call C4 by the repeating the risk{neutral discounting again. C4 = max[11;654:8(0:5266) + 7;605:7(0:4734)1 + 0:321 ; 0] = 9;435:1 To obtain the real call C3, we need to consider both the real call C4 and the incremental cash ow. Therefore, the terminal payo s are Cuu3 = max[9;435:1 + 305:0 254:9; 0] = 9;485:2 Cud3 = max[9;435:1 + 221:0 254:9; 0] = 9;401:2 Cdd3 = max[9;435:1 + 160:2 254:9; 0] = 9;340:4 We repeat this procedure to calculate the real call option values as shown in Figure 3.11. V1 449.1 V2 615.4 V3 221.0 Lattices for the Gross Project Values V4 9,802.1 0 1 2 3 4 5 6 7 ??? ??? I1 571.5 449.1 501.8 I2 716.6 401.9 I3 254.9 615.4 701.6 799.9 I4 390.9 539.8 615.4 ?T 1 473.5 r 3.21% 221.0 259.6 305.0 u1 1.117 188.1 221.0 d1 0.895 160.2 u2 1.140 9,802.1 12,033.5 14,772.9 d2 0.877 7,984.5 9,802.1 u3 1.175 6,503.9 d3 0.851 u4 1.228 Real Call Option Value d4 0.815 0 1 2 3 4 5 6 7 ??? ??? Volatility 1 11.10% 7,879.6 8,170.8 Volatility 2 13.11% 8,070.9 Volatility 3 16.10% 8,240.5 8,571.5 8,922.4 Volatility 4 20.51% 8,409.6 8,737.9 q(1) 0.6166 8,595.9 1-q(1) 0.3834 8,839.1 9,154.3 9,485.2 q(2) 0.5893 9,082.9 9,401.2 1-q(2) 0.4107 9,340.4 q(3) 0.5591 9,435.1 11,654.8 14,382.0 1-q(3) 0.4409 7,605.7 9,411.2 q(4) 0.5266 6,113.0 1-q(4) 0.4734 Phase 1 Design Phase 2 System & Prototype Production Phase 3 Test Phase 4 Mass Production & Deployment Phase 0 Conception Figure 3.11: Gross Project Value and Real Call Value. 49 There are four expanded sequential options inherent in KHP R&D. For valuation, it is necessary to rst calculate the value of the option in Phase 4 and re ect it in the option of Phase 3, and develop the same process for each of the initial phases to obtain the value of the nal compound options as shown in Figure 3.11. This calculation results in a compound option value of $7.88 billion. Therefore, when using real options, the project value increases by $7.88B(option premium), while the DCF method gives an evaluation result of $7.10B. Moreover, this method leads to calculating the strategic present worth of $16.03B of a new project[13]. Real Option Premium($7.88) + Conventional NPW($7.10) = Strategic NPW($14.98) 3.5.1 Sensitivity Analysis The compound option value can be varied according to the speci c characteristics of each defense R&D project. In other words, the option value depends on various parameters related to the option model. Therefore, it is important to understand what the impact is on the compound option value with respect to changes in some of its major value{drivers through sensitivity analysis. Figure 3.12 shows the sensitivity analysis for the KHP compound options model. For this model, the estimates for the furthest downstream option to implement Phase 4 impact the SNPW the most. This occurs because the upstream option values are a ected by downstream opportunities and the project value of Phase 4 are larger than the project value of any other phase. Increased underlying asset?s value increases the real call option value. But the large investment costs a ect the SNPW adversely. Increased strike price decreases call option value. The impact of the each phase?s volatility is relatively insigni cant due to the hedging nature of the multiple{phased implementation. The increased number of phases allows more information to be gathered for downstream decision{making[60]. The longer unit period ( T) increases the SNPW because the longer time to expiration provides more time for the value of the underlying asset to move. Additionally, the present value of investment 50 costs decreases as the life of the option increases and this makes the increasing the value of the call. Our compound options model uses a risk{adjusted discount rate(k) of 5.5%, the real social discount rate for national public investment projects presented by KDI [1], and a risk{free rate(rf) of 3.21%, the ve-year government bond interest rate. These parameters play an essential part in the option model. Because a decreasing risk{adjusted discount rate increases a project?s present value and leads to earlier initiation of a project. Also a decreasing risk{free rate increases call option values and leads to invest the project. ?50 ?40 ?30 ?20 ?10 0 10 20 30 40 500.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 2.4x 104 Percent(%) Change SNPW (mil $) V1V2 V3V4 (a) Project value of each phase ?50 ?40 ?30 ?20 ?10 0 10 20 30 40 501.42 1.44 1.46 1.48 1.5 1.52 1.54 1.56 1.58x 104 Percent(%) Change SNPW (mil $) Investment Cost 1Investment Cost 2 Investment Cost 3Investment Cost 4 (b) Investment costs of each phase ?50 ?40 ?30 ?20 ?10 0 10 20 30 40 501.48 1.485 1.49 1.495 1.5 1.505 1.51 1.515 1.52 1.525 1.53x 104 Percent(%) Change SNPW (mil $) ?T?1 ?2?3 ?4 (c) Volatility of each phase and T 0 1 2 3 4 5 6 7 8 9 101.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2x 104 Interest rate(%) SNPW (mil $) risk?adjustrisk?free (d) Risk-adjust(k) and Risk-free(rf) rate Figure 3.12: Sensitivity Analysis for the KHP Compound Real Options Model. 51 3.6 Conclusion Remarks A defense R&D acquisition project is a task characterized by its highly irreversible investment, long project life cycle, and high degree of uncertainty. Thus, it is di cult to derive an accurate estimate of the project value through traditional economic evaluation methods such as DCF, B/C ratio, etc. To solve this problem, in this chapter, we developed a compound options model as an economic evaluation framework for a defense acquisition project and applied it to KHP, an actual defense acquisition project. Typically, a defense R&D project experiences various types of uncertainties throughout its life cycle. These uncertainties involve either technical uncertainties related to the proper- ties of the project, or market uncertainties related to the future market conditions. Thus, we modeled the technical uncertainty through the EMFR approach to estimate the investment costs of each phase. Also we considered uncertainties of operational maintenance costs and world rotary{wing aircraft market share. Even if a project is selected and executed, it occasionally fails due to unexpected situa- tions along the way. The binomial lattice compound real options model provides a strategic and exible decision{making method to mitigate various uncertainties. Also, this model considers phase{speci c volatility. The standard compound option approach assumes a con- stant volatility over the lifetime of the project. However, it is more realistic to assume that the volatility varies over the project procedure. We represent MCST as a suitable method to estimate phase{speci c volatility measures. The evaluation results of the option model verify that there exists a considerable amount of real option value in KHP, while conventional estimation methods fail to capture the potential positive value. But to improve the reliability of the valuation models, it is necessary to estimate a correct distribution of the cash ows and consider the correlation of various uncertainty factors. 52 Chapter 4 Bayesian Real Options Approach to Evaluate Dynamic Defense Acquisition Programs In this chapter, a Bayesian revision process is applied to the learning real options frame- work for strategic defense acquisition decision{making. In the learning real options frame- work, defense managers actively work to improve their decision{making by resolving uncer- tainties of the defense project during the option life. After quantifying acquired information by merging statistical decision theory, the project volatility, which is estimated by consoli- dating various uncertainty factors, is updated by the Bayesian revision process. In chapter 3, the estimating procedure for EMFR is to make three point estimates(Beta-PERT) and the project value distribution of each phase is log{normally distributed. Therefore, a beta conjugate and log{normal conjugate Bayesian analyses are applied to estimate the posterior distributions of these random factors. 4.1 Sample Information and Bayesian Inference The Bayesian revision process is a method of analyzing given data and reaching a rational conclusion by using the Bayesian inference technique. In other words, Bayesian inference is a calculation of the posterior distribution by specifying a probability model that includes some prior knowledge about the unknown parameters and updating knowledge about the unknown parameters by conditioning this probability model on observed sample data.[41] Prior Distribution + Sampling Information (Likelihood) ? Posterior Distribution Figure 4.1: Bayesian Inference. 53 In Bayesian inference, the unknown quantity of interest is modeled as a random variable or as a nite set of random variables. Bayesian theory quanti es uncertainty by determining the prior distribution. Bayesian inference treats unknown parameters as random variables with known prior distributions. This prior distribution is determined before any data is observed by sampling process. Then, the conditional posterior distribution is determined by Bayes? rule. To perform a Bayesian inference, a likelihood function and the prior distribution are needed. Unlike a likelihood function that is usually determined through a statistical model, in this case, the prior distribution has to be chosen. Bayesian inference is based on a joint posterior distribution of parameters or on the marginal distribution of data. As these distributions are dependent on prior distribution, selecting the prior distribution is very important. There are various forms of prior distributions such as conjugate prior distribution, non- conjugated prior distribution, non-informative priors, and Je rey priors. Among them, the one used most often is the conjugate prior distribution. Generally, from the point of view of Bayesian analysis, the use of the conjugate prior has the advantage of compatibility to a form of posterior distribution and parameter estimation and of mathematical calculation. Especially when the distribution is of a form of known posterior distributions, such as normal distribution, gamma distribution, or beta distribution, prior and posterior distributions have a identical form with di erent parameters. Thus, estimation of the posterior distribution becomes easier when new data is added. On the other hand, a disadvantage of the conjugated prior distribution is that the appli- cation of various prior data is restricted since the distribution form remains constant depend- ing on parameters. The posterior distribution may not have the form of known distributions if a non-conjugated prior distribution is used instead of a conjugated prior distribution due to the restrictive conditions of the parameters. In this case, di erent conjugate prior density functions need to be combined, or more statistical analysis procedures may be required. 54 However, a parameter in Bayesian inference used to update the volatility of a real option represents information about the project cash ow. The posterior distribution can be deter- mined using the conjugated prior distribution, because a form of a probability distribution of such information related to the project?s cash ow is a form of the known distributions stated earlier. 4.2 The Beta Conjugate with Binomial Sampling In chapter 3, we de ne the estimate of Expected Mission Ful lment Rate(EMFR) as a key technical uncertainty factor. To construct probability models, the estimating proce- dure for EMFR is to make three{point estimates(PERT): a minimum (pessimistic) value, a maximum (optimistic) value, and a most likely value. These three estimates are used as the upper bound(U), the lower bound(L), and the mode(Mo) of the corresponding EMFR probability distribution of "n at the end of stage n. If we have three point estimates, this information can be used to construct a proba- bility distribution that favors the most likely value. The simplest distribution taking these estimates into account is the triangular distribution. The distribution resembles a triangle with the most likely value at the point of the triangle. The distribution is not required to be symmetrical about the mean. The shape of the triangle may be skewed to the left(minimum) or right(maximum) values. In this way, the triangular distribution can model a variety of di erent circumstances. However, the triangular distribution may place too much emphasis on the most likely value and be limited in its ability to model real world estimates[53]. The PERT distribution also uses the most likely value, but it is designed to gener- ate a distribution that more closely resembles realistic probability distribution. Unlike the triangular distribution, the PERT distribution constructs a smooth curve which places pro- gressively more emphasis on values around the most likely value. This PERT distribution can be represented by the beta probability distribution. The beta distribution, Beta( ; ), can express most of the random distributions by varying the parameter values. Therefore, 55 we use the beta distribution derived by three{point estimates to create a model for updating information of EMFR using the beta conjugate Bayesian analysis. If a beta distribution is used as the prior belief, then a Bernoulli or binomial distri- bution is needed as new data (sampling information) to estimate a form of posterior beta distribution. Beta Prior Distribution + Bernoulli / Binomial Sampling Information (Likelihood) ? Beta Posterior Distribution Figure 4.2: Beta Conjugate Distribution. The Bernoulli probability can be assigned to the speci c set of observations S = fx1;:::;xng. If each trial is treated as independent events, this probability is described below: P(Sj ;n) = nY i=1 P(xij ) (4.1) where, P(xij ) = 8 < : if the outcome is \Success", i.e.,xi = 1 1 if the outcome is \Failure", i.e., xi = 0 If \k" is the number of success times, then P(Sj ;n) = k(1 )n k (4.2) where, X S P(Sj ;n) = 1 (4.3) The binomial distribution is related to the number of successes observed(denoted \k") rather than the speci c set of outcomes S in the Bernoulli distribution. The number of observation n is a xed characteristic of the data, so in most models only is treated as a parameter to be learned. The binomial likelihood can be derived from the Bernoulli 56 likelihood as follows:[65] P(kj ;n) = X k;n2S P(Sj ;n) (4.4) = X k;n2S k(1 )n k = n k ! k(1 )n k = n!k!(n k)! k(1 )n k (4.5) As mentioned before, if we have a Bernoulli or binomial likelihood as sampling informa- tion, the beta distribution can be used as the prior. ( ) Beta( ; ) / 1(1 ) 1 (4.6) If S = fx1;:::;xng is the sample data set with = limn !1 1n nX i=1 Xi, and X j binomial(n; ) with Sj iid Bernoulli( ). Then, the posterior distribution of is decribed as follows: ( jx) = f(xj ) ( )f(x) = n k k(1 )n k ( ) f(x) / k+ 1(1 )n k+ 1 Beta(k + ;n k +b) (4.7) This property is called conjugacy, and the model is called a beta-binomial model. From this model, the posterior mean, mode, and, variance are calculated as following: E( jx;n; ; ) = +x + +n (4.8) Mode( jx;n; ; ) = +x 1 + +n 2 (4.9) Var( jx;n; ; ) = ( +x)( +n x)( + +n+ 1)( + +n)2 (4.10) 57 4.2.1 Prior Distribution of EMFR The beta distribution for random variable is de ned as f( j ; ) = ( + ) ( ) ( ) 1(1 ) 1; 0 < < 1; > 0; > 0 (4.11) E( ) = + (4.12) Mode( ) = 1 + 2 (4.13) Var( ) = ( + )2( + + 1) (4.14) The standard beta distribution is bounded between 0 and 1 and the shape of the distri- bution is determined by the parameter values of and . To utilize the beta function as a distribution of the EMFR value, and parameters are calculated from the U;L, and Mo values. The rst step is to transform the most likely estimate value (Mo) of the PERT to the standardized mode value ( ) of the beta distribution. = Mo LU L = 1 + 2 (4.15) Next, if we use the PERT assumption that the variance is one-sixth of the range, Var( ) = (1=6)2, then equations 4.13 and 4.14 can be solved for , giving ( 1)3 + (7 36 2 + 36 3)( 1)2 20 2( 1) 24 3 = 0 (4.16) Once the standardized mode value ( ) is computed from the values of PERT, and parameters and of beta distribution can be determined from equations 4.15 and 4.16. The initial estimated EMFR values used in chapter 3 are summarized in Table 4.1. Using these values and above calculation process, we can derive prior beta distributions of EMFR measures as shown in Figure 4.3. 58 Table 4.1: Initial Three{point estimates of EMFR ("i;i = 1;2;3;4). Phase 1 2 3 4 Optimistic 1 1 1 1 Most Likely 0.85 0.87 0.92 0.95 Pessimistic 0.60 0.7 0.80 0.90 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.5 1.0 1.5 2.0 theta f(theta) Prior Beta(4.558,3.135) (a) Phase 1: "1 PERT(0.6, 0.85, 1) 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.5 1.0 1.5 2.0 theta f(theta) Prior Beta(4.351,3.562) (b) Phase 2: "2 PERT(0.7, 0.87, 1) 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.5 1.0 1.5 2.0 theta f(theta) Prior Beta(4.481,3.320) (c) Phase 3: "3 PERT(0.8, 0.92, 1) 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.5 1.0 1.5 2.0 theta f(theta) Prior Beta(4.0,4.0) (d) Phase 4: "4 PERT(0.9, 0.95, 1) Figure 4.3: Prior Beta Distribution for the Initial EMFR at Each Phase. 59 4.2.2 Sampling and Posterior Distribution of EMFR If a beta distribution describes the prior beliefs. then a Bernoulli or Binomial sampling is needed to make a natural conjugate framework. A Bernoulli trial has two outcomes, described as \Success" or \Failure". However, since EMFR data is a three{point estimate, we will satisfy the Bernoulli description by categorizing the EMFR value as a success if its most likely estimate exceeds the standard value de ned for each phase, and a failure if it does not. Given the prior belief and sampling to be perfrmed, estimates of EMFR at each phase are identi ed as shown in Table 4.2. EMFR is measured by defense acquisition experts? assessment of how secure the required technology readiness level(TRL)is at a given point in time and is given in the form of optimistic, most likely, and pessimistic estimates. Ten experts or project management teams evaluate each EMFR(assumed) so there are ten di erent data samples. Table 4.2: Sampling Information of EMFR ("i;i = 1;2;3;4). Phase 1 Phase 2 Phase 3 Phase 4 n L M H L M H L M H L M H 1 0.62 0.85 1.00 0.74 0.87 1.00 0.68 0.86 1.00 0.71 0.87 1.00 2 0.68 0.75 0.95 0.68 0.80 1.00 0.68 0.78 0.98 0.68 0.79 0.99 3 0.45 0.65 0.80 0.75 0.85 1.00 0.60 0.75 0.90 0.68 0.80 0.95 4 0.70 0.85 0.95 0.69 0.79 0.95 0.70 0.82 0.95 0.69 0.81 0.95 5 0.75 0.88 1.00 0.58 0.69 0.85 0.67 0.79 0.93 0.62 0.74 0.89 6 0.55 0.70 0.90 0.69 0.82 0.96 0.62 0.76 0.93 0.66 0.79 0.95 7 0.75 0.85 0.95 0.75 0.87 0.96 0.75 0.86 0.96 0.75 0.87 0.96 8 0.80 0.90 1.00 0.80 0.90 1.00 0.80 0.90 1.00 0.80 0.90 1.00 9 0.75 0.88 1.00 0.75 0.88 1.00 0.75 0.88 1.00 0.75 0.88 1.00 10 0.69 0.81 0.95 0.80 0.92 1.00 0.75 0.87 0.98 0.77 0.89 0.99 The range of estimates is a re ection of the uncertainty associated with the technological maturity whether the technologies of each phase have acceptable levels of maturity in a 60 relevant investment costs and defense environment. Thus, the estimate of EMFR is consider as a success if its most likely estimate exceeds the standard value de ned for each phase, and a failure if it does not. The standard values are suggested EMFR of 0.80, 0.83, 0.85, and 0.87 from phase 1 to phase 4, respectively. From the sampling information, the parameters of the beta prior are revised by the equation 4.7 as shown in Table 4.3. Table 4.3: Posterior Beta Distribution of EMFR ("i;i = 1;2;3;4). Phase 1 Phase 2 Phase 3 Phase 4 Sampling Binomial (10, 7) Binomial (10, 6) Binomial (10, 5) Binomial (10, 5) Parameters Beta (11.558, 6.135) Beta (10.351, 7.562) Beta (9.481, 8.320) Beta (9.0, 9.0) ( ,,,)E xn? ?? 0.653 0.577 0.533 0.5 ( ,,,)Mode x n? ?? 0.672 0.587 0.537 0.5 ( ,,,)Var x n? ?? 0.012 0.013 0.013 0.013 Above posterior mean, best point estimate for , is a weighted mean of the prior mean and sample mean. Here, parameters and can be determined from prior information and \ " is the equivalent number of \Successes" and \ + " is the number of \Trials". E( jx;n; ; ) = +x + +n = + +n + x + +n = + + +n + + n + +n xn = + + +n prior mean + n + +n sample mean Thus, the use of the conjugate prior has the advantage of compatibility to a form of posterior distribution and parameter estimation of the posterior distribution becomes easier when new data is obtained. 61 Figure 4.5 represents prior and posterior beta distribution of EMFR. This posterior distribution of each phase is updated technical uncertainties for Monte Carlo simulation and used to re{estimate the project value distribution and volatility at each phase. 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 theta post.theta Prior Posterior (a) Phase 1: "1 Beta(11.56, 6.14) 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 theta post.theta Prior Posterior (b) Phase 2: "2 Beta(10.35, 7.56) 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.5 1.0 1.5 2.0 2.5 3.0 theta post.theta Prior Posterior (c) Phase 3: "3 Beta(9.48, 8.32) 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.5 1.0 1.5 2.0 2.5 3.0 theta post.theta Prior Posterior (d) Phase 4: "4 Beta(9.0, 9.0) Figure 4.4: Posterior Beta Distribution for the EMFR at Each Phase. 62 4.3 Log{normal Conjugate and Sampling In chapter 3, we used the Monte Carlo simulation with time frame method(MCST) for estimating the KHP volatility. While a multitude of uncertainties may exist in defense projects, these uncertainties must be quanti ed and reduced because it is exceedingly di cult to develop analytical models with more than two sources of uncertainty. To solve this prob- lem, MCST, the consolidated approach, can be used to combine the multiple uncertainties that drive the value of a defense project via Monte Carlo simulation into a single uncertainty: the project volatility. Thus we can calculate phase{speci c volatility parameters using the equation 3.16, which represents mathematical relationship between the project volatility( ) and the gross project value distribution, LN( T; T). Notice that the present value distri- bution of each phase must be log{normal distribution to use the equation 3.16. Therefore, if we estimate new project value distribution with new information, we can make Bayesian inferences about parameters of the log{normal distribution using conjugate properties. A log{normal distribution is a continuous probability distribution of a random variable whose logarithm is normally distributed. Thus a random variable Y is said to have a log{ normal distribution if X = ln(Y) has a normal distribution, X N( ; 2). In this case must be nonnegative. The shape of the log{normal probability density function is similar to that of the gamma distribution, with a long tail to the right. The log{normal density function is given by f(y) = 8> < >: 1 yp2 e (ln(y) )2=(2 2); > 0 0; elsewhere Thus probabilities for random variables with a log{normal distribution can be ob- tained by transforming them into probabilities that can be computed using the normal distribution[58]. 63 Consider a log{normal distribution with known parameter and unknown parameter . S =fy1;:::;yngis the sample data set taken from a log{normal distribution andV is the expected value of the log{normal variable. The likelihood can be de ned as: f(yjV) = 1 y q 2 2=n exp ( [ y (lnV 2=2)]2 2 2=n ) (4.17) The natural conjugate prior has the form (V) = 1 V q 2 20=n exp ( (lnV 0 2=2)2 2 20 ) (4.18) Thus, posterior distribution of future project value( ) can be derived as: (See [61] for detail process) ( jS) / LN( jV ; 2 ) (4.19) where; V = exp ( 2lnV 0 +n 20lnV n 20 + 2 ) ; 2 = 2 0 2 n 20 + 2 (4.20) If n0 represents the prior belief and n denotes the sample size, a relatively high n0 versus n would represent the prior belief containing more precision than the sample, and lead to the posterior distribution being skewed (or weighted) more to the prior. Conversely, a relatively high n versus n0 means that the sample contained more precision than the prior and lead to the posterior being skewed more to the sample statistic. Therefore, the posterior is in uenced by the amount of sampling and the relative precision of the prior and sample[61]. 4.3.1 Sampling and Posterior Distribution of the Project Value In chapter 3, We estimated the KHP project value distribution at each phase for esti- mating phase{speci c volatility measures. Each distribution is the prior of the log{normal conjugate for the Bayesian analysis. 64 Table 4.4: Prior Log{Normal Distribution of PVn(n = 1;2;3;4). Phase 1 Phase 1 Phase 1 Phase 1 Prior 2 (449.1,70.95 )LN 2 (615.4,115.1 )LN 2 (221.0,51.2 )LN 2 (9,802, 2,904.3 )LN In order to estimate the posterior, we need to obtain the new present value distribution of KHP as sample information using the Monte Carlo simulation. As mentioned before, a Monte Carlo simulation is the consolidated approach that allows a consideration of the uncertainties of multiple input variables speci c to KHP such as technical and market uncertainties. Estimates of these separate uncertainties are taken either from historical data, or from the subjective estimates of management. Table 4.5 shows the parameter values for the simulation. EMFR values are obtained from the results in beta conjugate Bayesian analysis in previous section. Table 4.5: Parameter Values for the Monte Carlo Simulation. Phase 1 Phase 2 Phase 3 Phase 4 Option Life 1 year 2 year 2 year 2 year EMFR (11.5,6.14)Beta (10.35,7.56)Beta (9.48,8.32)Beta (9.0,9.0)Beta Revenue Growth Rate 2 (0.033,0.0329)N 2 (0.031,0.0309)N 2 (0.023,0.0143)N 2 (0.040,0.0299)N Costs Growth Rate 2 (0.041,0.0429)N 2 (0.035,0.0362)N 2 (0.037,0.0312)N 2 (0.045,0.0411)N Market Share (3.5,7.5)U (3.5,7.5)U (3.5,7.5)U (7.5,10.5)U The gross project value of each phase is determined by the discounted value of future cash ows(excluding the investment costs). Using 10,000 simulation trials for each phase, we estimate the project value as sample information, S = (V1;V2;V3;V4). Then we can obtain the posterior distribution of the project value at each phase using equation 4.20 as shown in Table 4.6. 65 Table 4.6: Sample Information and Posterior Distribution of Each Phase. Phase 1 Phase 2 Phase 3 Phase 4 Sample Information 2 (476.5, 52.7)LN 2 (663.4, 108.6)LN 2 (270.1, 56.3)LN 2 (10,104.8, 2,786.1)LN Posterior 2 (466.6, 42.3)LN 2 (640.3, 79.0)LN 2 (242.0, 37.9)LN 2 (9,958.6, 2,010.6)LN 01234 0.0 0.2 0.4 0.6 0.8 1.0 1.2 PV f(PV) Posterior Sample Prior (a) Phase 1: V1 LN(466:6;42:32) 1.0 1.5 2.0 2.5 3.0 3.5 4.0 0.0 0.5 1.0 1.5 PV f(PV) Posterior Sample Prior (b) Phase 2: V2 LN(640:3;79:02) 0.5 1.0 1.5 2.0 2.5 0.0 0.5 1.0 1.5 2.0 2.5 3.0 PV f(PV) Posterior Sample Prior (c) Phase 3: V3 LN(242:0;37:93) 02468 0.0 0.2 0.4 0.6 PV f(PV) Posterior Sample Prior (d) Phase 4: V4 LN(9;958:6; 2;010:62) Figure 4.5: Bayesian Postaudit Distribution of the Project Value at Each Phase. 66 4.4 Bayesian Real Option Model for KHP Until now, we obtain the present value distribution of KHP using the Monte Carlo simulation and Bayesian revision method for updating the initial estimated project volatility at each phase. From the re{estimated project value distributions, we can calculate phase{ speci c volatility parameters using equation 3.16. To calculate the SNPW, major parameters for each investment phase are de ned in Table 4.7. Then, using the extended binomial lattice compound real options model as shown in chapter 3, we re{evaluate the strategic net present worth(SNPW) of KHP as shown in Figure 4.6. Table 4.7: Bayesian Postaudit Parameters for the Binomial Lattices. [Phase 1] Design [Phase 2] System & Prototype Production [Phase 3] Testing [Phase 4] Mass Production & Deployment Project Values 1 466.6V = 2 640.3V = 3 242.0V = 4 9,958.6V = Volatility 1 9.05%? = 2 8.69%? = 3 11.0%? = 4 14.13%? = Investment Cost 1 601.6I = 2 754.3I = 3 268.3I = 4 411.9I = Upside Potential 1 1.095u = 2 1.091u = 3 1.116u = 4 1.152u = Downside Potential 1 0.913d = 2 0.917d = 3 0.896d = 4 0.868d = Risk-free probability of upside change 1 0.6545q = 2 0.6628q = 3 0.6181q = 4 0.5779q = This calculation results in a compound option value of $7.98 billion and the SNPW of $15.15 billion for KHP. The Bayesian postaudit indicates that the KHP performance will be improved after taking project 1 because the SNPW increases by $170M. The Bayesian revision process can be repeated after undertaking the phase 3 investment project to get a revised estimate of volatility for phase 4 investment project and compare it with the forecast used in the real options model once phase 3 project is implemented. 67 V1 466.6 V2 640.3 V3 242.0 Lattices for the Gross Project Values V4 9,958.6 0 1 2 3 4 5 6 7 ??? ??? I1 601.6 466.6 510.8 I2 754.3 426.2 I3 268.3 640.3 698.4 761.8 I4 411.9 587.0 640.3 ?T 1.00 538.1 r 3.21% 242.0 270.1 301.6 u1 1.095 216.8 242.0 d1 0.913 194.2 u2 1.091 9,958.6 11,470.0 13,210.8 d2 0.917 8,646.3 9,958.6 u3 1.116 7,507.0 d3 0.896 u4 1.152 Real Call Option Values d4 0.868 0 1 2 3 4 5 6 7 ??? ??? Volatility 1 9.05% 7,982.2 8,267.6 Volatility 2 8.69% 8,183.1 Volatility 3 11.00% 8,358.4 8,664.3 8,983.4 Volatility 4 14.13% 8,552.9 8,861.9 q(1) 0.6545 8,759.8 1-q(1) 0.3455 8,975.9 9,284.4 9,605.2 q(2) 0.6628 9,231.1 9,545.6 1-q(2) 0.3372 9,497.8 q(3) 0.6181 9,571.9 11,070.9 12,798.9 1-q(3) 0.3819 8,247.3 9,546.7 q(4) 0.5779 7,095.1 1-q(4) 0.4221 Phase 1 Design Phase 2 System & Prototype Production Phase 3 Test Phase 4 Mass Production & Deployment Phase 0 Conception Figure 4.6: Bayesian Postaudit Gross Project Value and Real Call Value. 4.5 Conclusion Remarks Although various methods are developed and utilized to predict volatility of a real option, it is well known that a reliable estimation of uncertainty is di cult to achieve due to limitations of sample data, unrealistic models, and an inaccurate measure of parameters. In order to overcome such limitations, this chapter has presented procedures of updating volatility by using the Bayesian theory. As prior data of parameters for the initial estimated volatility relies mostly on expert opinion, uncertainty still exists. In other words, when an expert has made a wrong decision, project feasibility can be skewed. However, if the evaluation process of an option value and a project is conducted repeatedly by updating the initial estimated volatility with highly reliable data acquired from the actual market, accurate results of project feasibility assessment can be achieved and the Bayesian model can be used as the most e ective approach during this process. 68 Thus, this study gains meaning from two perspectives. First, it has utilized the Bayesian model to e ciently and accurately estimate project volatility that changes over time, and has presented such estimation procedures. Second, although most volatility estimation methods have focused on short-term projects and xed volatility prediction, this volatility estimation method, measuring the ever-changing volatility of long-term projects, has contributed to enhancing a secure environment and responding quickly and e ectively to medium and long- term defense environment changes. This study has assessed only beta and log{normal distribution of a conjugate prior dis- tribution. In the future, an applied method of the Bayesian model for other probability distributions that are relevant to an estimation of project volatility should be studied. 69 Chapter 5 Analytical Hierarchy Process(AHP) Model for Defense Acquisition Project As mentioned in chapter 1, the two main parts of preliminary feasibility assessment of a defense acquisition project are economic analysis and policy analysis. While economic feasi- bility can be analyzed through real options, policy analysis requires considering qualitative characteristics whose e ects cannot be quanti ed. Then, as the nal step, the results from economic and policy analysis are synthesized to reach a conclusion about implementing the project. In Korea, government guidelines require that the nal step of the preliminary feasibility analysis must be carried out using the AHP method to determine whether the program should be pursued. AHP helps defense acquisition decison{makers organize their values and preferences and translate these qualitative values into quantitative ratios. Thus it can combine both quantitative and qualitative elements together in measurement units and rank alternatives according to weighted criteria to determine the desirability of one alternative over the others. Therefore, in this chapter, analytical hierarchy process (AHP), one of the multi-criteria decision analysis (MCDA) methods, is applied to identify the optimum alternative to integrate the results of economic and policy feasibility analysis for KHP. Also, we discuss limitations of the AHP method and ways to improve on it. 5.1 Introduction There are di culties in integrating the results from economic and policy analysis. One major di culty is the problem of combining the results from quantitative and qualitative analysis. Speci cally, the results of economic analysis can be represented in currency or numerical units, but the qualitative results of policy analysis cannot easily be quanti ed. 70 Therefore, a comprehensive evaluation that integrates results from the two analyses is not a simple matter. Also, decision-makers must consider the details of the particular project, make consistent evaluations, and take into account the overall opinions of various stakehold- ers. To overcome these di culties, a multi-criteria decision analysis (MCDA) method can be utilized. This analysis mechanism is a decision{making tool which takes various attributes and multi-dimensional objectives into account and selects the optimum alternative with the highest economic and political feasibility. In other words, MCDA provides an analytical approach to evaluate and rank alternatives and helps the decision maker (DM) make a decision which re ects his/her preferences and desires. There are various MCDA methods and most of these have a similar process to nd the optimal solution. However, they evaluate the alternatives in di erent ways using di erent types of information and algorithms to make the ultimate decision. Di erent methods may lead to di erent results. MCDA methods are commonly grouped as discrete or continuous methods. Discrete methods attempt to identify the most desirable alternative from a nite set of alternatives, while continuous methods try to identify an optimal alternative from an in nite number of feasible alternatives[74]. Table 5.1 shows various MCDA methods divided into discrete and continuous methods. The Analytical Hierarchy Process (AHP) introduced by Saaty is a representative MCDA method and is used in various area. In Korea, government guidelines require that program feasibility analysis must be carried out using the AHP method. A critical feature of AHP is that it creates a hierarchy which consists of criteria and sub-criteria as assessment elements and measures the level of relative importance of these criteria through pairwise{comparison. It helps DMs organize their values and preferences and translate these qualitative values into quantitative ratios[68]. Thus it can combine both quantitative and qualitative elements together in measurement units other than monetary and rank alternatives according to weighted criteria to determine the desirability of one alternative over the others. The AHP 71 Table 5.1: Classi cation of MCDA Techniques [74]. Continuous ? Linear Programming ? Goal Programming Discrete Qualitative ? Concordance analysis ? Lexicographic ordering ? Linear assignment ? Frequency method Quantitative ? Multi-Attribute utility theory ? Weighted summation ? Compromise programming ? ELECTRE (Elimination and choice translating reality) Mixed ?Analytic hierarchy process (AHP) ?Value-focused thinking (VFT) method has been used to assess various public investment projects and calculate a compre- hensive conclusion through empirical analysis and mathematical veri cation. Therefore, in this chapter, AHP considered the representative method of \alternative-focused thinking" is applied to develop a MCDA model for the preliminary feasibility assessment of the Korean Helicopter R&D Program (KHP). 5.2 Analytical Hierarchy Process (AHP) to evaluate for the KHP AHP uses various objectives as assessment criteria and ranks preferred alternatives for each criterion. AHP is an intuitive and relatively easy method for formulating and analyzing decision problems and comparing alternatives to select the best alternative. Typically, to produce and assess criteria and alternatives for decision making, AHP is carried out according to the following steps: 1) conceptualizing and selecting alternatives, 2) structuring hierarchy, 3) weighting assessment criteria, 4) scoring and synthesizing, 5) acquiring feedback, and 6) drawing conclusions and recommending policy[Figure 5.1]. 72 Conceptualizing and Selecting Alternatives Structuring Hierarchy Weighting Assessment Criteria Scoring and Synthesizing Drawing Conclusions and Recommending Policy F e e d bac k Figure 5.1: Steps of the Analytical Hierarchy Process (AHP). 5.2.1 Conceptualizing and Selecting Alternatives The rst step of AHP is to deduce possible alternatives and assessment criteria through accurately de ning the given issue and situation. Alternatives can be found through various processes. The alternatives need to be capable of improving the results of the project im- plementation. AHP considers a set of evaluation criteria to nd the best alternatives, which represents the most suitable trade{o among the various criteria. 5.2.2 Structuring Hierarchy Once the subject for analysis has been conceptualized and alternatives have been se- lected, the hierarchical structure of decisions must be determined. The criteria and the relevant factors are decomposed hierarchically to analyze complex decision problems. Typi- cally, the ultimate goal of decision{making is put on the top level of the hierarchy and the selected alternatives on the bottom. Once these top and bottom levels are established, the rest of the hierarchy can be constructed by inserting sub{criteria between the parent criteria and the alternatives as shown in Figure 5.2. 73 Ultimate Goal Criterion 1 Criterion 2 Criterion n Sub-Criteria Alternative 1 Sub-Criteria Sub-Criteria Alternative m Level 1 Level 2 Level 3 Level k Figure 5.2: Standard Hierarchy of Analytical Hierarchy Process (AHP). 5.2.3 Weighting of Assessment Criteria For AHP analysis, weights of assessment criteria are measured to determine the relative importance. The main methodology for weighting is based on pairwise-comparison to de ne how important criterion i is with respect to criterion j. The evaluator makes pairwise- comparisons of all criteria and assigns a numeric value to each criterion using a 9{point scale proposed by Saaty. Table 5.2: Pairwise-Comparison Scales of AHP. Intensity of Importance Definition 1 Equal importance of both elements 3 Moderate importance of one element over another 5 Strong importance of one element over another 7 Demonstrated importance of one element over another 9 Extreme importance of one element over another 2, 4, 6, 8 Intermediate values between two adjacent judgments Reciprocals If activity i has one of the preceding numbers assigned to compare with activity j, then j has the reciprocal value when compared with i. To assign relative importance of one element over another among n elements (criteria), n(n 1) 2 pairwise-comparisons are needed for a n n matrix. Then, the pairwise-comparison data can be analyzed by the eigenvalue matrix technique in order to determine weights and 74 relative ranks of priorities. Consider matrix Ashown below, which is a reciprocal matrix in which all elements of the principal diagonal become 1. Element aij is an estimation of wi=wj, which is the relative weight of i to j. The n is the number of assessment criteria considered and the maximum eigenvalue of matrix A. A= [aij] = 0 BB BB BB BB @ a11 a12 ::: a1n a21 a22 ::: a2n ... ... ::: ... an1 an2 ::: ann 1 CC CC CC CC A = 0 BB BB BB BB @ w1=w1 w1=w2 ::: w1=wn w2=w1 w2=w2 ::: w2=wn ... ... ::: ... wn=w1 wn=w2 ::: wn=wn 1 CC CC CC CC A (5.1) If matrix A is multiplied by column vector w = [w1;w2;:::;wn]T, which refers to the preference weights of criteria as shown in Equation 5.2, this is eigenvalue problem for cal- culating non-zero value of simultaneous equations. Thus, the value of w calculated from Equation 5.3 is used as the weight vector. 0 BB BB BB BB @ w1=w1 w1=w2 ::: w1=wn w2=w1 w2=w2 ::: w2=wn ... ... ::: ... wn=w1 wn=w2 ::: wn=wn 1 CC CC CC CC A 0 BB BB BB BB @ w1 w2 ... wn 1 CC CC CC CC A = 0 BB BB BB BB @ nw1 nw2 ... nwn 1 CC CC CC CC A (5.2) A w = n w or (A nI)w = 0 (5.3) However, in practice, it is di cult to satisfy the above relationship exactly. When many pairwise{comparisons are performed, some inconsistencies can arise. For example, assume that three criteria are considered, and that decision{maker evaluates the rst criterion as more important than the second, and the second criterion as more important than the third. If the decision{maker mistakenly evaluates the third criterion as equal to or more important than the rst criterion, an inconsistency arises. Thus if the evaluator estimates matrix A 75 with a small inconsistency, the problem becomes A0 w0 = max w0 (5.4) where; max : the largest eigenvalue of A0: max is obtained from the summation of products between each element of eigenvector and the sum of columns of the reciprocal matrix. The matrix A is completely consistent if and only if max = n. Therefore, the important consideration during pairwise-comparison step is consistency. Consistency is presented as a ratio (CR: Consistency Ratio), and a zero ratio value means that the evaluator has performed pairwise-comparison with complete consistency. The CR can be obtained by the below relationships. CR = CIRI (5.5) CI(Consistency Index) = max nn 1 (5.6) Table 5.3: RI(Random Index). n 1 2 3 4 5 6 7 8 9 RI 0 0 0.58 0.9 1.12 1.24 1.32 1.41 1.45 If the value of CR is smaller than or equal to 0.1, the inconsistency is acceptable. On the other hand, if the CR is greater than 0.1, the evaluator needs to re-evaluate the subjective judgment and revise the pairwise-comparison matrix[75]. Once the matrix A0 is estimated, the normalized pairwise-comparison matrix A0norm is derived by making the sum of the elements on each column of A0 equal to 1. Each element 76 a0ij of the matrix A0norm is calculated by a0ij = a 0 ij nX k=1 a0kj (5.7) Then, the weight of criteria w, n-dimensional column vector, is obtained by averaging the elements on each row of A0norm. wi = nX k=1 a0ik n (5.8) 5.2.4 Scoring and Synthesizing The next step is to synthesize the local priorities across all criteria in order to determine the global priority. This procedure can be explained as a simple weighted average technique. Normally, the AHP adopts an additive aggregation with normalization of the sum of the local priorities to unity: pi = X j wj lij (5.9) where, pi : global priorty of the alternative i lij : local priorty wj : weight of the criterion j This calculation procedure ranks alternative courses of action based on the decision maker?s (or evaluator?s) judgments concerning the importance of the criteria and extent to which each criterion is met by each alternative. Sensitivity analysis is then performed as the last step of the decision{making process to observe how modifying data input a ects the analysis results. This step examines the sensitivity of the results to changes in the priorities (weights) of the criteria. This process is an important step of AHP because all pairwise{ comparison results are based on the decision{maker?s or evaluators? subjective assessments. 77 5.2.5 The AHP Approach for the KHP Decision Problem Conceptualizing and Selecting Alternatives The armed forces of the Republic of Korea (ROK) aim to implement KHP to replace de- teriorating airmobile helicopters, secure technology for key components, and achieve various other bene ts such as reducing maintenance costs and accumulating domestic technology. In order to achieve these objectives, the ROK Armed Forces consider the following possible alternatives and try to select the best alternative using AHP. Alternative 1: Research and development to produce 70% of key helicopter parts and components domestically Alternative 2: Research and development to produce 50% of key helicopter parts and components domestically (excluding some parts and components from Alternative 1 that would require a large initial expenditure and investment costs) Alternative 3: Research and development to produce 30% of key helicopter parts and components domestically Alternative 4: Do not R&D (produce key helicopter parts and components by technical transfer or direct purchase from overseas) Structuring Hierarchy Figure 5.3 shows the AHP hierarchy of KHP for the preliminary feasibility assessment. The assessment can be classi ed into three major categories: technical, economic, and policy. Each of these categories can then be subdivided into more detailed classi cations. On the hierarchy, two axioms need to be satis ed. The rst is homogeneity, meaning that each criterion should be comparable to the others on the same level. The second is dependency, meaning that the elements of each level need to be subordinate to the elements 78 KHP Feasibility Assessment Technical Feasibility Economic Feasibility (NPV+ROV) Policy Feasibility Adequacy of Technology Development Planning Success Possibility of Technology Development Overlapping with Existing Technology Necessity and Consistency of Defense-Policy Risk factors of Project Implementation Adequacy of Planning Process / Schedule Adequacy of Development Goal / Contents Adequacy of Support System Technology Development Trend Standard Technology / Interoperability / Gap Overall Project Level Overlapping Review Individual Task Level Overlapping Review Necessity for strengthening national defense Consistency / Connectivity with Defense Policy Require Operational Capabilities Changing Steadiness of Related Private Enterprises Financial Resource Procurement Restriction of Legal / Infringing Patent Rights Alternative 1: 70% R&D Alternative 2: 50% R&D Alternative 3: 30% R&D Alternative 4: Do not R&D Total Life-Cycle Cost Economic Effects (Total Revenue) R&D Cost (T ~ T+7) Production Cost (T+7 ~ T+27) Maintenance Cost (T+7 ~ T+27) Real Option Value (Consider Uncertainties) Effect on Production Inducement Productivity of Added Value Ripple Effect in the Aviation Industry Uncertainty of Acquisition / Maintenance Cost Uncertainty of World Aircraft Market Demand 1 st Level 2 nd Level 3 rd Level 4 th Level 5 th Level Figure 5.3: Hierarchy of KHP for AHP. of the next level up. A hierarchy that satis es these requirements can be broken down into as many levels as necessary to properly represent the analysis situation. Here we should state that while chapter 1 mentions only two main items for prelimi- nary feasibility assessment, economic and policy analysis, a third critical element for R&D project cases is technical feasibility. Unlike in general public projects, technical consideration elements for R&D projects can determine the project?s destiny, necessity, and range. 79 Weighting of Assessment Criteria In order to determine the weights for criteria, let us start by creating each pairwise- comparison matrix, assuming that experts in related elds have determined the relative importance of the criteria through an appropriate method such as a questionnaire survey. The relative importance between two criteria is measured according to a 9{point scale as shown in Table 5.2. For example, if decision{makers or evaluators believe that a criterion is \equally to moderately" more important than another criterion, a value of 2 expresses this judgment. If a criterion is \moderately" more important than another criterion, a value of 3 is appropriate. The results of the AHP analysis can be di erent if the pairwise{comparisons are done by di erent people with di erent preferences and background. First, the 2nd{level criteria are compared with respect to the ultimate goal on the 1st level. Table 5.4 represents the pairwise-comparison matrix for the three 2nd{level criteria with respect to the goal. The normalized weights (priorities) for criteria are given in the last column of each table. The evaluators have the same preferences for the three main considerations (technical, economic, and, political feasibilities) to analyze the preliminary feasibility assessment. Thus each of the three main considerations is weighted equally at 0.333 to consider each element of KHP feasibility analysis equally important. Table 5.4: Pairwise-Comparison between Main Considerations (2nd{Level). Equal Preference (CR: 0.0) Technical Feasibility Economic Feasibility Policy Feasibility Priority (Weight) Technical Feasibility 1 1 1 0.333 Economic Feasibility 1 1 1 0.333 Policy Feasibility 1 1 1 0.333 Next, pairwise{comparisons are performed on the 3rd{level criteria with respect to each 2nd{level criterion and 4th{level criteria are pairwise compared with respect to the 3rd level 80 criteria. Finally, all alternatives except the economic feasibility sub{criteria are compared with respect to the 4th{level criteria. All sub{criteria on the 3rd and 4th levels with re- spect to the criterion of \economic feasibility" on the 2nd{level are evaluated and expressed quantitatively. Then all these values are synthesized as monetary units to determine the \Strategic Net Present Worth (SNPW)" of KHP through real option analysis for the eco- nomic feasibility assessment. Thus, the alternatives are pairwise{compared directly with respect to the economic feasibility criterion on the 2nd{level. Economic Feasibility (NPV+ROV) Alternative 1: 70% R&D Alternative 2: 50% R&D Alternative 3: 30% R&D Alternative 4: Do not R&D Total Life-Cycle Cost Economic Effects (Total Revenue) R&D Cost (T ~ T+7) Production Cost (T+7 ~ T+27) Maintenance Cost (T+7 ~ T+27) Real Option Value (Consider Uncertainties) Effect on Production Inducement Productivity of Added Value Ripple Effect in the Aviation Industry Uncertainty of Acquisition / Maintenance Cost Uncertainty of World Aircraft Market Demand Figure 5.4: Pairwise-Comparison of alternatives with respect to the criterion of \economic feasibility". The cash ow of the each alternative is estimated by analyzing total life-cycle cost presented in the assessment data(input) and the economic impact of R&D(output), and dividing by the year. The total life-cycle cost in the assessment data includes R&D, mass production, and maintenance for 20 years. The economic impact was calculated by the sum of 1) e ect on production inducement, 2) productivity of added value, and 3) technology ripple e ect in the aviation industry. The economic impact is de ned as bene ts generated from conducting the project. Next, we can calculate phase{speci c volatility parameters from the present value dis- tribution using the Monte Carlo simulation. Then the compound option value of the each alternative is obtained by binomial compound options model as shown in chapter 3. Table 81 5.5 shows the pairwise{comparison results based on alternatives? SNPW with respect to the criterion of economic feasibility. Table 5.5: Pairwise-Comparison of Alternatives with respect to the Criterion of \Economic Feasibility" based on the Strategic NPW. CR: 0.00 Alt 1 (70%) Alt 2 (50%) Alt 3 (30%) Alt 4 (Do Not R&D) Priority (Local Weight) Alt 1 (70%) 1 1.046 1.032 1.513 0.279 Alt 2 (50%) 0.956. 1 0.987 1.447 0.267 Alt 3 (30%) 0.969 1.013 1 1.465 0.270 Alt 4 (Do Not R&D) 0.661 0.691 0.682 1 0.184 Strategic NPW of Each Alternative $ Million Total Life Cycle Cost Total Revenue Real Option Premium Strategic NPW Alt 1 (70%) 4,413.7 5,914.2 1,622.7 3,123.2 Alt 2 (50%) 4,491.0 5,921.6 1,556.4 2,987.0 Alt 3 (30%) 4,865.3 6,317.1 1,574.1 3,025.9 Alt 4 (Do Not R&D) 3,116.9 4,170.0 1,011.6 2,064.7 All other pairwise-comparison matrices are shown in the Appendix. Scoring and Synthesizing After creating all pairwise-comparison matrices with respect to upper level criteria, the AHP analysis is summarized as the overall scores synthesized with the relative weights. This procedure can be explained by the weighted average technique. Figure 5.5 is the result of AHP analysis providing a summary of the normalized relative weights for the four alternatives with respect to the three 2nd{level criteria. The above analysis result shows that \Alternative 1" has the highest weight of 0.272 and is closely followed by \Alternative 4" with a weight of 0.261. It should be noted that although \Alternative 4" came in a close second, it has the highest local weights in technical 82 .00 Technical (0.333) Economic (0.333) Policy (0.333) OVERALL Alt 1 (70%) Alt 4 (Do not R&D) Alt 2 (50%) Alt 3 (30%) .333 .333 .10 .20 .30 .40 .50 .60 .70 .80 .90 Obj% Alt% .10 .00 .20 .30 .40 Alt 1 Alt 4 Alt 2 Alt 3 Overall CR: 0.01 Technical Feasibility (0.333) Economic Feasibility (0.333) Policy Feasibility (0.333) Priority (Weight) Alt 1 (70%) 0.240 0.279 0.298 0.272 Alt 2 (50%) 0.248 0.268 0.207 0.241 Alt 3 (30%) 0.228 0.269 0.182 0.226 Alt 4 (Do not R&D) 0.284 0.184 0.313 0.261 Figure 5.5: The Overall Results of the KHP Analysis. and policy feasibility criteria. However, \Alternative 1", which creates many good economic e ects in the long-term view, has a much higher local weight of the economic feasibility criterion than, does Alternative 4; thus, it has the highest overall weight. Sensitivity Analysis on the weights of the Criteria The nal step of decision{making is sensitivity analysis, where weights of the key criteria are modi ed in order to observe the impact on the overall results. The current results of the model are analyzed according to the pairwise{comparisons, based on the assumption that the DMs have equal preference for the three critical criteria on the 2nd{level. First, 83 consider the technical feasibility criterion. Figure 5.6 shows how the overall weights for the four alternatives vary when the weight for this criterion is changed from 0 to 1. Alt% .10 .00 .20 .30 Alt 1 Alt 4 Alt 2 Alt 3 .0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1 Technical Feasibility Figure 5.6: Sensitivity of the \Technical Feasibility" Weight. \Alternative 1 (70% R&D)" dominates all other alternatives for the range from 0 to 0.47. However, if the weight of the technical feasibility criterion is greater than 0.47, \Alternative 4 (Do not R&D)" is the best choice for KHP. This nding means that direct purchase from overseas is more suitable for KHP in that case. \Alternative 3 (30% R&D)" is always ranked last irrespective of the weight change. Next, consider the economic feasibility criterion. Figure 5.7 shows how the overall weights for the four alternatives vary when the weight for the \Economic Feasibility" is changed from 0 to 1. As before, \Alternative 1 (70% R&D)" dominates all other alternatives for the range from 0.24 to 1. However, if the weight of economic feasibility criterion is less than 0.24, \Alter- native 4 (Do not R&D)" is the best choice for KHP. This nding is due to the fact that \Alternative 4" has the highest local weights in the technical and policy feasibility criteria but the lowest local weight in the economic feasibility criterion. Thus, if the weight of the 84 Alt% .10 .00 .20 .30 Alt 1 Alt 4 Alt 2 Alt 3 .0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1 Economic Feasibility Figure 5.7: Sensitivity of the \Economic Feasibility" Weight. economic feasibility criterion is increased, the overall weight of \Alternative 4" will tend to decrease rapidly. Finally, Figure 5.8 shows how the overall weights for the four alternatives vary when the weight for the policy feasibility criterion is changed from 0 to 1. The best choice for KHP Alt% .10 .00 .20 .40 Alt 1 Alt 4 Alt 2 Alt 3 .0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1 Policy Feasibility .30 Figure 5.8: Sensitivity of the \Policy Feasibility" Weight. changes from \Alternative 1" to \Alternative 4" when the weight for the policy feasibility 85 criterion is greater than about 0.63. But these two alternatives do not have large di erences with respect to the policy feasibility criterion. 5.2.6 Conclusion Remarks Until now, the AHP method has been used to decompose the decision problem for KHP and create a hierarchical structure for considering multiple and even con icting objectives. Pairwise{comparison allows decision maker(s) to consider each individual trade{o by check- ing the consistency to reduce bias in the decision{making process. In doing so, the qualitative and quantitative criteria are converted into subjective or objective evaluation measures and synthesized into overall priorities. As the result, AHP determines the ranking of alternatives to select the best option for KHP. However, despite its popularity and usefulness, AHP has some disadvantages. If the number of pairwise-comparisons to be made, n(n 1)2 , is very large, the accuracy and reliability of the estimation results are decreased. Another arti cial limitation of AHP is the use of the 9{point scale. It is not clear why 9{point scale is utilized. Decision{maker may nd it di cult to distinguish among criteria due to the scale restriction. Finally, the AHP generates criteria for analyzing decisions through \alternative-focused thinking". This approach impedes the consideration of criteria that have important values for the decision{maker. [76] Therefore, our next aim is to study how the \value-focused thinking" (VFT) method produces objectives and how it di ers from AHP, a representative method of \alternative- focused thinking." 86 Appendix Table 5.6: Pairwise-Comparison of sub-criteria (3rd Level) w.r.t the \Technical Feasibility". CR: 0.02 Adequacy of Technology Development Planning Success Possibility of Technology Development Overlapping with Existing Technology Priority (Weight) Adequacy of Technology Development Planning 1 1/3 2 0.238 Success Possibility of Technology Development 3 1 4 0.625 Overlapping with Existing Technology 1/2 1/4 1 0.137 Table 5.7: Pairwise-Comparison of sub-criteria (3rd Level) w.r.t the \Policy Feasibility". CR: 0.00 Necessity for strengthening national defense Consistency /Connectivity with Defense Policy Priority (Weight) Necessity for strengthening national defense 1 5 0.833 Consistency /Connectivity with Defense Policy 1/5 1 0.167 Table 5.8: Pairwise-Comparison of sub-criteria (4th Level) w.r.t the \Adequacy of Technol- ogy Development Planning". CR: 0.10 Adequacy of Planning Process / Schedule Adequacy of Development Goal / Contents Adequacy of Support System Priority (Weight) Adequacy of Planning Process / Schedule 1 1/4 2 0.218 Adequacy of Development Goal / Contents 4 1 3 0.630 Adequacy of Support System 1/2 1/3 1 0.152 87 Table 5.9: Pairwise-Comparison of sub-criteria (4th Level) w.r.t the \Success Possibility of Technology Development". CR: 0.00 Technology Development Trend Standard Technology / Interoperability / Gap Priority (Weight) Technology Development Trend 1 1/4 0.200 Standard Technology / Interoperability / Gap 4 1 0.800 Table 5.10: Pairwise-Comparison of sub-criteria (4th Level) w.r.t the \Overlapping with Existing Technology". CR: 0.00 Overall Project Level Overlapping Review Individual Task Level Overlapping Review Priority (Weight) Overall Project Level Overlapping Review 1 1 0.500 Individual Task Level Overlapping Review 1 1 0.500 Table 5.11: Pairwise-Comparison of sub-criteria (4th Level) w.r.t the \Necessity and Con- sistency of Defense-Policy". CR: 0.00 Necessity for Strengthening National Defense Consistency /Connectivity with Defense Policy Priority (Weight) Necessity for Strengthening National Defense 1 2 0.667 Consistency /Connectivity with Defense Policy 1/2 1 0.333 Table 5.12: Pairwise-Comparison of sub-criteria (4th Level) w.r.t the \Risk factors of Project Implementation". CR: 0.04 Restriction of Legal / Infringing Patent Rights ROC Changing Steadiness of Related Private Enterprises Financial Resource Procurement Priority (Weight) Restriction of Legal / Infringing Patent Rights 1 2 3 1 0.335 ROC Changing 1/2 1 3 1/3 0.187 Steadiness of Related Private Enterprises 1/3 1/3 1 1/3 0.097 Financial Resource Procurement 1 3 3 1 0.381 88 Table 5.13: Pairwise-Comparison of Alternatives w.r.t the \Adequacy of Planning Process and Schedule". adequacy of planning /schedule CR: 0.02 Alt 1 (70%) Alt 2 (50%) Alt 3 (30%) Alt 4 (Do Not) Priority (Weight) Alt 1 (70%) 1 1/2 1/3 1/3 0.110 Alt 2 (50%) 2 1 1 1/2 0.230 Alt 3 (30%) 3 1 1 1 0.302 Alt 4 (Do Not) 3 2 1 1 0.358 Table 5.14: Pairwise-Comparison of Alternatives w.r.t the \Adequacy of Development Goal and Contents". CR: 0.02 Alt 1 (70%) Alt 2 (50%) Alt 3 (30%) Alt 4 (Do Not) Priority (Weight) Alt 1 (70%) 1 2 2 4 0.447 Alt 2 (50%) 1/2 1 2 2 0.263 Alt 3 (30%) 1/2 1/2 1 1 0.159 Alt 4 (Do Not) 1/4 1/2 1 1 0.131 Table 5.15: Pairwise-Comparison of Alternatives w.r.t the \Adequacy of Support System". CR: 0.00 Alt 1 (70%) Alt 2 (50%) Alt 3 (30%) Alt 4 (Do Not) Priority (Weight) Alt 1 (70%) 1 1 1 1 0.250 Alt 2 (50%) 1 1 1 1 0.250 Alt 3 (30%) 1 1 1 1 0.250 Alt 4 (Do Not) 1 1 1 1 0.250 Table 5.16: Pairwise-Comparison of Alternatives w.r.t the \Technology Development Trend". Technology Trend Gap CR: 0.01 Alt 1 (70%) Alt 2 (50%) Alt 3 (30%) Alt 4 (Do Not) Priority ( Weight) Alt 1 (70%) 1 1 1 3 0.312 Alt 2 (50%) 1 1 1 2 0.280 Alt 3 (30%) 1 1 1 2 0.280 Alt 4 (Do Not) 1/3 1/2 1/2 1 0.128 89 Table 5.17: Pairwise-Comparison of Alternatives w.r.t the \Technology Interoperability and Gap". Technology Trend Gap CR: 0.02 Alt 1 (70%) Alt 2 (50%) Alt 3 (30%) Alt 4 (Do Not) Priority (Weight) Alt 1 (70%) 1 1/2 1/2 1/2 0.140 Alt 2 (50%) 2 1 1 1/2 0.232 Alt 3 (30%) 2 1 1 1/2 0.232 Alt 4 (Do Not) 2 2 2 1 0.396 Table 5.18: Pairwise-Comparison of Alternatives w.r.t the \Overall Overlapping Review". overall overlapping each task overlapping Economy CR: 0.00 Alt 1 (70%) Alt 2 (50%) Alt 3 (30%) Alt 4 (Do Not) Priority (Weight) Alt 1 (70%) 1 1 1 1 0.250 Alt 2 (50%) 1 1 1 1 0.250 Alt 3 (30%) 1 1 1 1 0.250 Alt 4 (Do Not) 1 1 1 1 0.250 Table 5.19: Pairwise-Comparison of Alternatives w.r.t the \Individual Overlapping Review". overall overlapping each task overlapping Economy CR: 0.01 Alt 1 (70%) Alt 2 (50%) Alt 3 (30%) Alt 4 (Do Not) Priority (Weight) Alt 1 (70%) 1 2 3 4 0.467 Alt 2 (50%) 1/2 1 2 3 0.277 Alt 3 (30%) 1/3 1/2 1 2 0.160 Alt 4 (Do Not) 1/4 1/3 1/2 1 0.096 Table 5.20: Pairwise-Comparison of Alternatives w.r.t the \Necessity for strengthening na- tional defense ". consistency stengthen defense CR: 0.00 Alt 1 (70%) Alt 2 (50%) Alt 3 (30%) Alt 4 (Do Not) Priority (Weight) Alt 1 (70%) 1 1 1 1/2 0.200 Alt 2 (50%) 1 1 1 1/2 0.200 Alt 3 (30%) 1 1 1 1/2 0.200 Alt 4 (Do Not) 2 2 2 1 0.400 90 Table 5.21: Pairwise-Comparison of Alternatives w.r.t the \Consistency and Connectivity with Defense Policy". consistency stengthen defense CR: 0.02 Alt 1 (70%) Alt 2 (50%) Alt 3 (30%) Alt 4 (Do Not) Priority (Weight) Alt 1 (70%) 1 3 4 5 0.546 Alt 2 (50%) 1/3 1 2 3 0.232 Alt 3 (30%) 1/4 1/2 1 2 0.138 Alt 4 (Do Not) 1/5 1/3 1/2 1 0.084 Table 5.22: Pairwise-Comparison of Alternatives w.r.t the \Restriction of Legal and Infring- ing Patent Rights". Legal ROC change/ CR: 0.02 Alt 1 (70%) Alt 2 (50%) Alt 3 (30%) Alt 4 (Do Not) Priority (Weight) Alt 1 (70%) 1 1 1 2 0.204 Alt 2 (50%) 1 1 1 2 0.204 Alt 3 (30%) 1 1 1 1 0.246 Alt 4 (Do Not) 1/2 1/2 1 1 0.346 Table 5.23: Pairwise-Comparison of Alternatives w.r.t the \Require Operational Capabilities Changing". Legal ROC change/ CR: 0.06 Alt 1 (70%) Alt 2 (50%) Alt 3 (30%) Alt 4 (Do Not) Priority (Weight) Alt 1 (70%) 1 1/2 1/2 1/3 0.118 Alt 2 (50%) 2 1 1/3 1/2 0.171 Alt 3 (30%) 2 3 1 1/2 0.301 Alt 4 (Do Not) 3 2 2 1 0.410 91 Table 5.24: Pairwise-Comparison of Alternatives w.r.t the \Steadiness of Related Private Enterprises". steadiness private comp fund procure CR: 0.02 Alt 1 (70%) Alt 2 (50%) Alt 3 (30%) Alt 4 (Do Not) Priority (Weight) Alt 1 (70%) 1 1/2 1/2 1/3 0.119 Alt 2 (50%) 2 1 1 1/3 0.201 Alt 3 (30%) 2 1 1 1/2 0.220 Alt 4 (Do Not) 3 3 2 1 0.460 Table 5.25: Pairwise-Comparison of Alternatives w.r.t the \Financial Resource Procure- ment". steadiness private comp fund procure CR: 0.05 Alt 1 (70%) Alt 2 (50%) Alt 3 (30%) Alt 4 (Do Not) Priority (Weight) Alt 1 (70%) 1 2 3 1/2 0.278 Alt 2 (50%) 1/2 1 3 1/3 0.183 Alt 3 (30%) 1/3 1/3 1 1/3 0.096 Alt 4 (Do Not) 2 3 3 1 0.443 92 Chapter 6 Value-Focused Thinking (VFT) Model for Defense Acquisiton Project There are rarely best practice cases as possible alternatives for the new R&D project. Therefore, preliminary feasibility assessment focused on choosing among alternatives through AHP is a limited way to think through decision situations. This kind of decision{making process is called \alternative{focused thinking"(AFT). AFT only selects an optimal alterna- tive that mostly satis es the given criteria and constraints. It does not provide a way to nd the criteria (objectives) for an e ective assessment in the rst place. Thus, decision{makers? actual values and fundamental objectives may not be consistent with the evaluation results. In this chapter, \value-focused thinking" (VFT) is used to develop a more suitable MCDA model than AHP for the preliminary feasibility assessment of the Korean Helicopter R&D Program (KHP). VFT is similar to AHP but accesses an issue through di erent ways in accordance with the type of decisions and fundamental objectives of assessment. VFT is one such technique that seeks to identify important aspects of a decision and lead the decision maker to the most valuable alternatives. Rather than simply comparing available alternatives, VFT unveils what value an alternative truly has to the decision maker. There- fore, we show that VFT is a more re ned analysis to assess the defense acquisition decision problems and determine improvement directions for decisions. 6.1 Introduction In order to consider all the important strategic objectives during the decision process, it is necessary to make explicit what the decision{maker(s) or the organization really wants to achieve from the new project. In other words, the value which is to be obtained from R&D needs to be speci ed and systematized to develop a decision model for evaluating the 93 criteria and selecting the optimal alternative. Keeney [69] de nes this value as \what we care about" or as the \driving force" which leads decision{making. Thus, he suggests that the rst step in decision{making should be specifying the values to be obtained, followed by assessing the values. This process, which may yield a higher value for e ective decisions, is called \value{focused thinking." VFT is one of the multi-criteria decision making tools to overcome the drawbacks of \alternative-focused thinking." Leon [76] veri ed that VFT generates a more extensive structure of objectives than AFT for the same decision prob- lem.VFT concentrates on determining the core values of the decision rather than choosing among predetermined alternatives. An understanding of the values can provide important insights for all aspects of a decision problem, and these insights make it possible to achieve much better consequences[69]. 6.2 Value Focused Thinking(VFT) to evaluate for the KHP Typically, VFT involves the following steps: 1) identifying the problem, 2) creating a value hierarchy and developing evaluation measures, 3) creating value functions and weight value hierarchy, 4) generating and scoring alternatives, 5) performing deterministic and sen- sitivity analysis, and 6) drawing conclusions and recommending policy[Figure 6.1]. This process is similar to AHP but the fundamental di erence is the step of generating alter- natives. Whereas the basis of AHP is selecting an alternative, in VFT, alternatives are considered only after determining the core values and structuring objectives. 6.2.1 Identifying the Problem An important rst step of VFT is identifying the problem. The main purpose of this step is to create a better understanding of the decision situation[72]. In this step, the decision context is framed by the critical elements related to the decision problem. Figure 6.2 shows the main factors and questions specifying the decision context. By de ning the decision context and carefully establishing the nature of the decision problem, the treatment of the 94 Identifying the Problem Creating a value hierarchy / Developing Evaluation measures Creating Value Functions / Weight Value Hierarchy Generating and Scoring Alternatives Performing Deterministic and Sensitivity Analysis Drawing Conclusions and Recommending Policy F e e d bac k Figure 6.1: \Value-Focused Thinking" (VFT) Process. real problem can be ensured. The decision context and corresponding fundamental objectives are closely related and they frame the decision situation.[21] Decision Context Political Structures Social Context Administrative Structures Stakeholders Values Purpose of the Analysis Information Available Decision Problem Decision Alternative Decision Context ? What is the nature of the problem and its environment? ? Who is responsible for the decision? ? Who is the decision-maker? ? What are the decision-maker?s values? ? What are the decision alternatives? ? What groups are affected by the decision? ? What is the purpose of the alternative? Figure 6.2: Decision Context. The most obvious way to identify objectives is to ask a group of decision-makers or stakeholders. Several techniques can be used to stimulate the identi cation of possible objectives[69]. 1. A wish list: What do you want? What do you value? What should you want? 95 2. Alternatives: What is a perfect alternative, a terrible alternative, some reasonable alternative? What is good or bad about each? 3. Problems and shortcomings: What is wrong or right with your organization? What needs xing? 4. Consequences: What has occurred that was good or bad? What might occur that you care about? 5. Goals, constraints, and guidelines: What are your aspirations? What limitations are place upon you? 6. Di erent perspectives: What would your competitor or your constituency be concerned about? At some time in the future, what would concern you? 7. Strategic objectives: What are your ultimate objectives? What are your values that are absolutely fundamental? 8. Generic objectives: What objectives do you have for your customers, your employees, your shareholders, yourself? What environmental, social, economic, or health and safety objectives are important? 9. Structuring objectives: Why is that objective important? How can you achieve it? What do you mean by this objective? 10. Quantifying objectives: How would you measure achievement of this objective? Why is objective A three times as important as objective B? Also, Parnell [79] de ne three standards for developing a multi{objective value model: platinum, gold, and silver. A platinum standard is based on interviews with decision-makers and stakeholders to determine the objectives. A gold standard is based on approved vision, policy, strategy, planning, or doctrine documents. A silver standard uses interviews with subject-matter experts and data provided by stakeholder representatives[78]. When real 96 decision-makers and stakeholders are not available, the silver standard is a useful way to identify the values and objectives. This research uses the silver standard in conjunction with a review of relevant literature to develop the notional value hierarchy. 6.2.2 Creating Value Hierarchy Structuring and hierarchical modeling of the objectives results in a deeper and more ac- curate understanding of what one should care about in the decision context[69]. Hierarchical modeling of objectives can be performed by the following steps[21]. Separating means objectives from fundamental objectives Fundamental and means objectives have di erent roles in the analysis. The fundamental objectives refer to the end objectives which the decision{maker really wants to achieve. These objectives characterize the reason for interest in a decision situation, and thus are an essential part of the problem structuring. Means objectives are helpful for creating alternatives and developing models to analyze the decision problem. Both fundamental and means objectives are often displayed in objective structures. The fundamental objectives can be speci ed with a hierarchy structure similar to AHP, while the means objectives can be speci ed in the form of means-ends networks. The distinctions between the objective structures are shown in Table 6.1[69]. Constructing Value Hierarchy There are two ways to construct value hierarchy: top{down and bottom{up. Generally, the top{down approach is better for constructing a fundamental objectives hierarchy and the bottom{up approach is better for generating a means-ends objectives network. The steps of the top-down approach are as follow: 1) Identify the overall fundamental objective, and 2) Subdivide the objectives until the lowest level is su ciently well de ned that a measurable attribute can be associated with it. 97 Table 6.1: Comparison of Fundamental and Means Objective Structures. Structure Characteristics Fundamental Objectives Hierarchy ? The hierarchy includes only fundamental objectives ? A higher-level objective is defined by the set of lower-level objectives ? The lower-level objectives are mutually exclusive and provide an exhaustive characterization of the higher-level objective ? A higher-level objective has at least two lower-level objectives connected to it Means-ends Objectives Network ? The network may include both fundamental and means objectives ? A lower-level objective is a means to the higher-level objectives ? The set of means objectives under a higher-level objective does not necessarily provide an exhaustive representation of the means leading to the higher-level ? A higher-level objective may have only one lower-level objective connected to it Checking the Hierarchy When constructing the value hierarchy, the analyst should check for the following characteristics[67]: Completeness: All relevant objectives should be included in the hierarchy. Non-redundancy: The set of attributes should be non-redundant to avoid double count- ing of the consequences. Decomposability: Attributes should be independent, that is, it should be possible to analyze one attribute at time. Operability: Attributes should be meaningful and assessable. 6.2.3 Development of Evaluation Measures The degree to which objectives are achieved in di erent decision alternatives is measured with attributes. There are four characteristics of attributes scales[67][69]. 98 Natural vs. Constructed: Natural attributes can be measured in natural scales (i.e. numbers, monetary value, etc.) Constructed scales are used in a variety of situa- tions where natural scales are not appropriate. Careful development of constructed attributes scales may promote thinking and describe the consequences in a decision situation much better than the natural attributes scales. Direct vs. Proxy: Direct attributes directly measure the degree of attainment of ob- jectives, while proxy attributes should be valued only for their perceived relationship to the achievement of the corresponding fundamental objective. Desirable evaluation measures describe the extent to which an objective is achieved in terms of an attribute in a non{ambiguous way. They also make it possible to assess the decision-maker?s preferences for di erent levels of the attribute without excessive amounts of time, money, or e ort. 6.2.4 Creating Value Function To conduct a VFT analysis, it is essential to elicit a value function which combines mul- tiple attributes into a single measure of the overall value of each alternative. When assessing a decision{maker?s preferences over the set of evaluation measures, individual evaluation measure scales must be converted to common scores with values between 0 and 1. There are two di erent procedures for determining a value function. One is a \single dimensional value function" (SDVF) that is made up of segments of straight lines that are joined together into a piecewise linear function, while the other uses an \exponential" form of the SDVF. When the value measure has a small number of possible di erent scoring levels, a piece- wise linear function is generally used and the relative value increments are speci ed between each of the possible evaluation measure scores. For example, for the value increments for the evaluation of strengthening the national defense power, between 0% and 50% and between 50% and 100% are equal, as each additional piece of evaluation helps better measure an 99 upper level objective. This assessment creates linearly increasing SDVF with equal value increments for each unit increase in the evaluation measure. First, the piecewise linear SD- VFs are elicited by identifying the least and most desirable levels of the evaluation measure; next, a value increment procedure is calculated. The other procedure uses \exponential SDVFs having a particular form which depends on the range of the evaluation measure and an \exponential constant"( ). If is greater than zero, the SDVF has a concave curve, and if less than zero, the SDVF has a convex curve. For monotonically increasing measures, the exponential function is v(x) = 8> >>< >>> : 1 exp[ (x Low)= ] 1 exp[ (High Low)= ]; 6= In nity x Low High Low; otherwise (6.1) There is no closed form solution of an exponential constant ( ). Therefore, if a mid-value is given between High and Low, it must be estimated using the Excel Solver or Table shown by Kirkwood [67]. For a detailed look at values and preference functions, see Kirkwood [67] and Keeney and Rai a [22]. 6.2.5 Determining Weights of Value Hierarchy In the additive value model, each evaluation measure within the hierarchy must be weighted based on relative importance and variation[78]. The purpose of weighting the value hierarchy is to identify the importance that each value contributes to the overall goal. The weight is associated with the change in total value, when the evaluation measure changes from the worst to the best level. In most cases, the weights are normalized, in such a way that the sum of the weights equals 1. There are several methods to determine weights, such as, AHP, swing{weighting, direct assessment, and so on. AHP requires the decision-maker to make pairwise-comparisons across all combinations of sub-criteria and alternatives. But swing{weighting does not need pairwise-comparisons and the procedure for determining weights is similar to that for nding 100 the piecewise linear SDVF. Therefore, the decision{maker can understand the method more easily and evaluate more consistently. In this research, each objective is weighted using the swing{weighting process described by Kirkwood [67]. The swing{weighting procedure is as follows: 1) Rank the evaluation measures in the order of importance based on the swing in each measure from the worst score to the best score. 2) Determine the relative increase in value for each of the swings compared to either the smallest or largest swing. 3) Continue until all evaluation measures have been assessed. 4) Normalize the weights. 6.2.6 Alternatives Generating and Scoring Alternatives are investigated according to the speci ed values. In doing so, fundamental and means objectives can be utilized as the criteria for seeking alternatives. For alternative assessment, the decision{maker needs to de ne the attributes for evaluating each objec- tive, focusing on fundamental objectives, and then perform an assessment based on these attributes. The value of an alternative is created by the additive value model: vj(x) = Xwivi(xij); i = 1;2;:::;n: j = 1;2;:::;m: (6.2) where; vj(x) : multiobjective value for alternative j wi : weight of the measure i vi(xij) : SDVF of measure i for alternative j The nal score is the amount of the alternative value. For example, if the nal value is 0.8, the alternative provides 80% of the total possible value that could be achieved by implementing this decision. Also, there is a 20% \value gap" between the perfect alternative and the current alternative. Thus, value gap is a more re ned analysis that can be used to 101 assess and identify the problem areas of the decision and to determine improvement direction for the current decision. 6.2.7 The VFT Approach to KHP Decision Problem Identifying Objectives and Creating Value Hierarchy As mentioned in the AHP section, the current decision problem is the preliminary feasibility assessment of KHP and its various objectives such as replacing deteriorating he- licopters, securing technology, creating national economic e ects, and so on. Many of these objectives are \means objectives" that lead to the ful lment of other fundamental objectives. For example, \replacing deteriorating helicopters" may be an important objective of KHP, but it is a means objective that leads to improve the fundamental objective of \reinforcing national defense power". Thus, for an e ective assessment, fundamental objectives should be identi ed rst, while means objectives are retained for use in identifying and creating decision alternatives later. Figure 6.3 shows the value hierarchy of KHP for the VFT analysis. The value hierarchy can be broken down into as many levels as necessary to properly represent the decision situation. It contains fundamental objectives which are identi ed with key considerations of KHP. In order to compare VFT and AHP, objectives similar to AHP criteria are considered. Development Evaluation Measures Evaluation measures are used to measure the degree to which the objectives are at- tained. The evaluation measures should provide an unambiguous rating of how well an alternative does with respect to each objective[67]. Also, it must be understandable to other KHP management teams. Thus, simple evaluation measures were sometimes chosen for this analysis, rather than the more complex evaluation measures used only by experts in a given eld[78]. 102 Maximize Effect on Achievement Technology Maximize Effect on National Economy Maximize Effects on Defense Policy Improve Technology Development Planning Improve national scientific and technological power Minimize Overlapping with Existing Technology Improve Defense Power / Maximize Consistency with Defense Policy Minimize Risk factors of Project Implementation Improve Planning Process / Schedule Satisfy Program Goal / Contents Improve Support System Gratify World Technology Trend Gratify Interoperability / Reduce Technology Gap Minimize Overall Project Level Overlapping Minimize Individual Task Level Overlapping Improve National Defense Power Maximize Consistency with Defense Policy Effective Respond to ROC Change Improve Steadiness of Private Enterprises Maximize Procurement of Financial Resource Minimize Restriction of Legal Minimize Total Life-Cycle Cost Maximize Economic Effects Minimize R&D Cost (T ~ T+7) Minimize Production Cost (T+7 ~ T+27) Minimize Maintenance Cost (T+7 ~ T+27) Improve Strategic NPW ( Dealing with Uncertainty) Maximize Effect on Production Inducement Maximize Productivity of Added Value Maximize Ripple Effect on Aviation Industry Actively respond to Cost Uncertainty Actively respond to Market Demand Uncertainty 1 st Level 2 nd Level 3 rd Level 4 th Level Maximize Effectiveness of KHP Figure 6.3: Value Hierarchy of KHP. 103 Simple evaluation measures, for example percentage (%), are understood easily by most everyone. For instance, the percentage (%) of the contribution rate to achieve the objective of satisfy the program goal and contents will be increased if some alternative has a strong possibility of reaching the goal. Table 6.2 shows the evaluation measures of each objective. It should be noted that the evaluation measure of maximizing the e ect on the national economy is a monetary (natural) value synthesized for SNPW of KHP through real option analysis, as it was in AHP. $2.0(bil) is the lowest value from real option analysis and $3.1(bil) is the highest as shown in Table 6.3. Table 6.2: The Evaluation Measures. Objectives Evaluation Measures Range Improve Planning Process / Schedule (%) contribution rate to achieve objective 0-100 Satisfy Program Goal / Contents (%) contribution rate to achieve the objective 0-100 Improve Support System (%) contribution rate to achieve the objective 0-100 Gratify World Technology Trend (%) contribution rate to achieve the objective 0-100 Gratify Interoperability / Reduce Technology Gap (%) contribution rate to achieve the objective 0-100 Minimize Overall Project Level Overlapping 0 : Overlapping, 1: No Overlapping 0 or 1 Minimize Individual Task Level Overlapping (%) Overlapping Rate 0-100 Maximize Effect on National Economy ($) Strategic Net Present Worth 2.0-3.1 bil $ Improve National Defense Power (%) contribution rate to achieve the objective 0-100 Maximize Consistency with Defense Policy (%) contribution rate to achieve the objective 0-100 Minimize Restriction of Legal (%) Restriction Rate of Legal 0-100 Effective Respond to ROC Change (%) contribution rate to achieve the objective 0-100 Improve Steadiness of Private Enterprises (%) contribution rate to achieve the objective 0-100 Maximize Procurement of Financial Resource (%) contribution rate to achieve the objective 0-100 Development of the SDVFs and Weights In the additive value function, SDVFs formalize the subjective preferences for the attain- ment of di erent levels of an evaluation measure. The piecewise-linear SDVFs are elicited by identifying the least and most desirable levels of the evaluation measure and value increment procedure is then followed. For instance, to determine the SDVF over percentage (%) of 104 Table 6.3: Strategic NPW of Each Alternative. CR: 0.00 Alt 1 (70%) Alt 2 (50%) Alt 3 (30%) Alt 4 (Do Not R&D) Priority (Local Weight) Alt 1 (70%) 1 1.046 1.032 1.513 0.279 Alt 2 (50%) 0.956. 1 0.987 1.447 0.267 Alt 3 (30%) 0.969 1.013 1 1.465 0.270 Alt 4 (Do Not R&D) 0.661 0.691 0.682 1 0.184 Strategic NPW of Each Alternative $ Million Total Life Cycle Cost Total Revenue Real Option Premium Strategic NPW Alt 1 (70%) 4,413.7 5,914.2 1,622.7 3,123.2 Alt 2 (50%) 4,491.0 5,921.6 1,556.4 2,987.0 Alt 3 (30%) 4,865.3 6,317.1 1,574.1 3,025.9 Alt 4 (Do Not R&D) 3,116.9 4,170.0 1,011.6 2,064.7 the contribution rate to achieve the objective to satisfy the program goal and contents, it is necessary to nd that the total value increment between the lowest possible level and the highest possible level is 1, and the fact that this value increment is the sum of the increments going from 0% to 100% can be used to determine the necessary values. Figure 6.4 represents examples of SDVFs of evaluation measures. 2.0 3.1 SNPW (bil $) 0 V alu e ( %) 20 40 60 80 100 20 40 60 80 100 Restriction Rate (%) (a) Linear SDVF Effect on National Economy Restriction Rate of Legal 0 V alu e ( %) 20 40 60 80 100 0 Figure 6.4: Examples of Increasing and Decreasing SDVFs. Next, the hierarchy must be weighted based on relative importance and variation. As mentioned before, the weights are assessed by swing{weighting technique. The procedure for determining weights is similar to that for nding the piecewise linear SDVF. Determining 105 weights with the swing{weighting technique involves considering swings in each objective, from the worst score of its evaluation measure to the best. The swings are then ranked in importance, and the decision-maker determines the relative increase in value for each of the swings compared to either the smallest or largest swing. The weights are then found by normalizing the assessments[78]. Determining the Overall Values for the Alternatives In order to achieve the various objectives of KHP, assume that the ROK Armed Forces determine the same possible alternatives as for AHP and try to select the best alternative using the VFT method. Alternative 1: Research and development to produce 70% of key helicopter parts and components domestically Alternative 2: Research and development to produce 50% of key helicopter parts and components domestically (excluding some parts and components from Alternative 1 that would require a large initial expenditure and investment costs) Alternative 3: Research and development to produce 30% of key helicopter parts and components domestically Alternative 4: Do not R&D (produce key helicopter parts and components by technical transfer or direct purchase from overseas) As explained before, additive value function is used for nding the overall values for the alternatives. vj(x) = Xwivi(xij); i = 1;2;:::;n: j = 1;2;:::;m: (6.3) Thus, after calculating vj(x) for each alternative (j = 1;2;3;4) we compare di erences in 106 values as shown in Table 6.4. The amount of value for each measure contributes to the overall value. Table 6.4: Analysis of Alternatives. 0 W e ig h t e d V alu e 0.2 0.4 0.6 0.8 1 (a) Linear SDVF Weighted Single Dimensional Values Alt 1 Alt 2 Alt 3 Alt 4 Objectives on 2 nd Level Alt 1 (70%) Alt 2 (50%) Alt 3 (30%) Alt 4 (No R&D) Maximize Effects on Achievement Technology 0.164 0.181 0.182 0.258 Maximize Effect on National Economy 0.333 0.320 0.322 0.061 Maximize Effects on Defense Policy 0.186 0.126 0.112 0.234 () j vx 0.683 0.627 0.616 0.553 The above analysis result shows that \Alternative 1" has the highest overall value of 0.683 and \Alternative 4" has the lowest overall value of 0.553. It should be noted that \Alternative 4" receives the majority of its value from maximizing e ects on technology and defense policy but almost no value from maximizing e ects on the national economy. Unlike AHP, VFT analysis gives a much higher weight and value to the objective of maximizing 107 e ects on the national economy, creating many good economic e ects through R&D in the long{term view. In this method, higher values are more preferred; thus, \Alternative 1" is the optimum selection. However, what does the value number for each alternative mean? The value number for each alternative gives the proportion of the distance, in a value sense, that the alternative is from the alternative with an overall value of zero to the alternative with an overall value of 1. Consider two hypothetical alternatives. One has the least preferred levels for all the evaluation measures and the other has the most preferred levels for all the evaluation measures. Then the value of each alternative means the distance in a value sense from the worst to best possible alternative. In above results, \Alternative 1" is 68.3% of the distance in a value sense from the worst to the best possible alternative. Another way to express this value is that if we select \Alternative 1", we obtain 63.8% of the value improvement, relative to the worst possible alternative[67]. Sensitivity Analysis on the Weights of the Objectives The current results of the model are analyzed according to the equal weight of the three critical objectives on the 2nd{level. First, consider maximizing e ects on technology objective. Figure 5.6 shows how the overall weights for the four alternatives vary when the weight for the this objective is changed from 0 to 1. \Alternative 1 (70% R&D)" dominates all other alternatives for the range from 0 to 0.55. However, if the weight of the objective is greater than 0.55, \Alternative 4 (Do not R&D)" is the best choice for the KHP. This nding means that direct purchase from overseas is more suitable for the KHP in that case. Next, consider maximizing e ects on the national economy objective. Figure 6.6 shows how the overall weights for the four alternatives vary when the weight for the \Economic Feasibility" is changed from 0 to 1. 108 Alt% .40 .20 .60 .80 Alt 1 Alt 4 Alt 2 Alt 3 .0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1 Maximize Effects on Technology Achievement Figure 6.5: Sensitivity of the \Maximize E ects on Technology" Weight. Alt% .25 .20 .50 1 Alt 1 Alt 4 Alt 2 Alt 3 .0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1 Maximize Effects on National Economy .75 Figure 6.6: Sensitivity of the \Maximize E ects on National Economy" Weight. As before, \Alternative 1 (70% R&D)" dominates all other alternatives for the range from 0.21 to 1. However, if the weight of the objective is less than 0.21, \Alternative 4 (Do not R&D)" is the best choice for KHP. This nding is due to the fact that \Alternative 4" has higher value in maximizing technology achievement and defense policy objectives but the lowest value in maximizing the national economy objective. Thus, if the weight of the 109 economic feasibility criterion is increased, the overall value of \Alternative 4" will tend to decrease rapidly. Finally, the Figure 6.7 shows how the overall weights for the four alternatives vary when the weight for the maximizing e ects on defense Policy objective is changed from 0 to 1. The best choice for KHP is varied from \Alternative 1" to \Alternative 4" when the weight Alt% .2 .0 .4 .8 Alt 1 Alt 4 Alt 2 Alt 3 .0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1 Maximize Effects on Defense Policy .6 Figure 6.7: Sensitivity of the \Maximize E ects on Defense Policy" Weight. is greater than about 0.71. Alternative 2 and 3 do not have large di erences with respect to the this objective. 6.2.8 Conclusion Remarks In this chapter, VFT methods have been applied for preliminary feasibility assessment for KHP. The di erences between AHP, a representative method of alternative{focused think- ing, and VFT for solving decision problems can be summarized as shown in Table 6.5. Applying the proposed two methods is simple and pro table for KHP assessment. How- ever, VFT considers all the important strategic objectives during the decision process and 110 Table 6.5: Di erences between AHP and VFT. AHP VFT Critical Criteria Alternatives Objectives Fundamental Objectives Means Objectives Alternatives Values (Evaluation Measures) Analysis Results Sum of Relative Weights Sum of Relative Weights ? Values (Evaluation Measures) Main Weighting Method Pairwise-Comparison Swing Benefits Evaluate Alternatives Verify Evaluation Consistency Evaluate Alternatives Uncovers Hidden Values Improves Communications Facilitate Involvement (Value Gap Analysis) Determine Critical Leverage Point (Means-Objectives Network) makes explicit what the decision{maker really wants to achieve from the new project. Rather than simply comparing available alternatives, VFT unveils what value an alternative truly has to the decision{maker. There are rarely best practice cases as possible alternatives for new investments. Thus, an assessment focused on selecting one alternative among several alternatives is a limited way to make strategic decisions. Since VFT overcomes this limita- tion, we conclude that VFT may be a more practical methodology than AHP for preliminary feasibility assessment. Further, VFT is much easier to implement and more understandable to decision{makers. Also, VFT is a more re ned analysis that can be used to assess and identify the problem areas of the decision and as a guide to determine improvement directions for current decisions. In this study, objectives were identi ed and analyzed based on a limited literature review. Accordingly, a more detailed value model needs to be developed through the per- spectives and data provided by relevant experts in the defense eld. However, this research can support strategic decision{making and provide a guideline for creatively seeking and assessing alternatives by developing value models based on VFT. 111 Chapter 7 Conclusions To successfully pursue a large scale defense acquisition project which requires a signif- icant nancial commitment by the government, it is necessary to assess potential nancial risk along with a wide variety of factors such as technology, industry ripple e ect, strategy, and security environment. This assessment enables decision{makers to optimize the appro- priation of budgets and to raise the e ciency of nancial operations as well as to achieve military strategic objectives. This preliminary feasibility assessment consists of two main parts: economic analysis and policy analysis of the project. The nal step is considering the whole results of the economic and policy analyses to determine whether the program should be pursued. A multi{criteria decision analysis is often used for this purpose in an attempt to identify the best alternative that meets the multiple objectives. Therefore, this disserta- tion develops an evaluation framework and decision{making model using real options and multi{criteria decision analysis (MCDA) methods to overcome the limitations of traditional evaluation methodologies for defense acquisition projects. First, we explore the rational initial volatility estimation method for real options. The most important yet di cult variable to estimate among real options input variables is volatil- ity. Therefore, many scholars have presented various methods for nding accurate volatility parameters. However, in estimating volatility of the initial analysis time, it is desirable to use methods that are faithful to the given circumstances and easy to estimate rather than methods that require complicated and unrealistic assumptions. Therefore, we compare the Monte Carlo simulation considered time frame method to other methods and demonstrate that it is the most suitable way to estimate the initial volatility in terms of accuracy and e ciency. 112 Second, we evaluate the economic value of the Korea Helicopter R&D Program (KHP), a multi{phased investment decision problem, using a compound real option model and em- pirical data to validate the practicality. Even when a project is selected and executed, it oc- casionally fails upon facing unexpected situations along the way. Thus, to solve this problem, it is highly important to draw a proper valuation and strategic and exible decision-making methodology, especially when considering the national defense environment of Korea, which is facing an enormous task of defense reform according to the change in wartime operational control. For this purpose, the compound real option model is the most appropriate approach. The results of the evaluation verify that there is a considerable amount of real option value in KHP, while conventional estimation methods fail to capture the potential positive value. Third, we examine an appropriate way to update the volatility estimates based on the arrival of new pieces of information during the acquisition process. A Bayesian revision process is presented to analyze given data and provide post{audit information that can be used for updating the volatility parameter of the real option model. The Bayesian analysis can calculate the most reliable and newest volatility according to project progression as it revises volatility with actual data acquired from the market. Project uncertainty analysis is a process of acquiring and analyzing various data to overcome initial analysis results that are limited by a lack of available information, and the Bayesian Revision Process is the most tting approach for this analysis. Finally, this dissertation describes a multi{objective decision{making model to integrate the results of economic and policy analysis for a nal determination. A preliminary feasibility assessment of KHP is carried out using both AHP, a representative method of alternative{ focused thinking, and the VFT method, and their di erences for solving decision problems is examined. Applying the proposed two methods is simple and pro table for assessing KHP. However, VFT considers all the important strategic objectives during the decision process and make explicit what the decision{maker really wants to achieve from the new project. The VFT process generates fundamental{objectives as the primary goals of KHP and related 113 means{objectives rather than simply comparing available alternatives. VFT unveils what value an alternative truly has to the decision{maker. This strength leads us to conclude that VFT may be a more practical methodology than AHP for preliminary feasibility assessment. Also, VFT is a more re ned analysis that can be used to assess and identify the problem areas of the decision and serve as a guide to determine improvement directions for the current decision. In conclusion, this research developed a new tool and methodology to improve the cur- rent preliminary feasibility assessment(PFA) system for defense projects. The new model estimates the probability distribution of technical uncertainty in real option framework us- ing technology readiness level(TRL) concepts and expected mission ful llment rate(EMFR) approach. Also, a value-focused thinking model was developed for the rst time to merge a real option framework in PFA of the defense project. Through a demonstration of modeling of PFA for KHP, the methodology is shown to be highly e ective for capturing the full values created by a new project. Bayesian learning real options and value-focused thinking help the defense planner in two major ways: to formulate an economic evaluation tool to capture the strategic net present worth(SNPW) and the impact of new information in a tractable form, and to create a better decision-making system and develop an enduring set of guiding principles for the defense acquisition project. However, in this study, nancial data and objectives were collected based on a limited literature review. Thus, to develop a more accurate value model, defense organizations and experts should interact to estimate a correct distribution of the cash ow and volatility based on real data and to improve all notional objectives, measures, SDVFs, and weights. Also, although this research model is certainly an improvement over traditional methods, the model must be applied to various defense acquisition projects to verify its superiority. 114 Bibliography [1] KDI, (2008), A Study on General Guidelines for Pre-feasibility Study , 5th Edition, Korea Development Institute, Seoul. [2] National Assembly Budget O ce, (2005), KHP Pre-Evaluation, Seoul [3] Ministry of National Defense, (2012), Defense White Paper, Seoul. [4] U.S. Government Accountability O ce(GAO), (2012), DEFENSE ACQUISITIONS As- sessments of Selected Weapon Programs, Report to Congressional Committees. [5] John V. Farr, William R. Johnson, and Robert P. Birmingham, (2004), A Multitiered Approach to Army Acquisition, Defense Acquisition Review Journal, pp. 234{246. [6] A.T.Kearney, (2012), The Value of Time for the Department of Defense, A.T Kearney Inc. [7] T. A. Luehrman, (1997, Investment Opportunities as Real Options: Getting Started on the Numbers, Harvard Business Review, pp.132{142. [8] G. Glaros, (2003), Real Options for Defense, USDOD. [9] Christine L. Komoroski, Thomas Housel, Sandra Hom, and Johnathan Mun, (2006), A methodology for improving the shipyard planning process: using KVA analysis, risk simulation and strategic real options, California, Naval Postgraduate School. [10] Albert Olagbemiro, Man-Tak Shing, Johnathan Mun, (2009) Application of Real Op- tions Theory to Software-Intensive System Acquisitions, California, Naval Postgraduate School. [11] J. Y. Park, (2004), The Study on Decision Factors of Optimal Timing of New Weapon System Introduction as a New Investment Opportunity from real option approach, The- sis, Seoul National University. [12] W. J. Jang, (2005), The Application of real options theory in defense acquisition projects, Dissertation, Seoul National University. [13] C.S. Park, (2010), Contemporary Engineering Economics, 5th Editon, Prentice Hall, NJ. [14] B. J. Park, (2009), Real Options and Evaluation under the Uncertainty-Case Study on IT Investment Project, SigmaPress, Seoul. 115 [15] R. Geske, (1979), The valuation of compound options, J. Financ, Econ, 7: pp. 63{81. [16] H. Herath and C.S. Park, (2002), Multi-stage capital investment opportunities as com- pound real options, The Engineering Economist, Vol. 47, No.1, pp. 1{27. [17] W. H. Sung, (2002), Valuation of Two-Stage Technology Investment Using Double Real Option, Journal of Korea technology innovation society, July, pp. 141{151. [18] D. Cassimon, P.J. Engelen, L. Thomassen, and M. Van Wouwe, (2004), The valuation of a NDA using a 6-fold compound option, Research Policy 33(1), pp. 41{51. [19] D. Cassimon, P.J. Engelen, and V. Yordanov, (2011), Compound real option valuation with phase-speci c volatility: A multi-phase mobile payments case study, Technovation Volume 31, Issues 5{6, May{June, pp. 240{255. [20] T.L. Saaty and L.G. Vargas, (1991), Prediction, Projection and Forecasting, Boston: Kluwer Academic. [21] H. Raimo, E. Harri, and S. Ahti, (2002), Multiple Criteria Decision Analysis e-Learning Site, Value Tree Analysis, http://MCDA.hut. . [22] R.L. Keeney and H. Rai a, (1976), Decisions with Multiple Objectives: Preferences and Value Tradeo s, Wiley, New York, NY. [23] Multi{Criteria analysis: a manual, (2009), Department for Communities and Local Government: London. [24] Stefan Hajkowicz and Andrew Higgins, (2008), A Comparison of Multiple Criteria Anal- ysis Techniques for Water Resource Management, European Journal of Operational Research 184, pp. 255{265 [25] Ren Estrella Maldonado, Ward Delabastita, AnjaWij els, and Jos Van Orshoven, (2012), Comparison of Discrete Multi-Criteria Decision Making Methods for Selection of A orestation Sites, Working Paper, Conference SAGEO 2012. [26] Igor Linkov, F. Kyle Satterstrom, and George P. Fenton, (2009), Prioritization of Ca- pability Gaps forJoint Small Arms Program Using Multi{criteria Decision Analysis, Journal of Multi{Criteria Decision Analysis, 16: pp. 179-185. [27] T. Copeland and V. Antikarov, (2001), Real Options: A Practitioner?s Guide, Thomson Texere, NY [28] K. Lee, Y. Kwon, and J. Shin, (2006), Derivatives Modeling : Using Matlab, Ajin, Seoul. [29] J. Mun, (2006), Real Options Analysis: Tools and Techniques for Valuing Investments and Decisions, John Wiley & Sons, NJ. [30] B.R. Cobb and J.M. Charnes, (2004), Real options volatility estimation with correlated inputs, The Engineering Economist, (49), pp. 119{137. 116 [31] A.K. Dixit and R.S. Pindyck, (1994), Investment under Uncertainty, Princeton Univer- sity Press, Princeton, NJ. [32] J. Smith, (2005), Alternative Approaches for Solving Real-Options Problems (Comments on Brand~ao et al.(2005), Vol. 2, No. 2, June, pp. 89{102. [33] L. Brand~ao, J. Dyer, and W. Hahn, (2005a), Using Binomial Decision Trees to Solve Real-Option Valuation Problems, Decision Analysis, Vol. 2, No. 2, June, pp. 69{88. [34] L. Brand~ao, J. Dyer, and W. Hahn, (2005b), Response to Comments on Brand~ao et al. (2005), Decision Analysis, Vol. 2, No. 2, June, pp. 103{109. [35] P. Godinho, (2006), Monte Carlo Estimation of Project Volatility for Real Options Anal- ysis, Applied Finance, Vol. 16, No. 1. [36] F. Longsta and E. Schwartz, (2001), Valuing American options by simulation: A simple least-squares approach, Review of Financial Studies, Vol. 14, No. 1, pp. 113{147. [37] Tero J. Haahtela, (2011), Estimating Changing Volatility in Cash Flow Simulation Based Real Option Valuation with Regression Sum of Squares Error Method, Working Paper, Aalto University. [38] L. Brand~ao, J. Dyer, and W. Hahn, (2012), Volatility estimation for stochastic project value models, European Journal of Operational Research 220, pp. 642{648. [39] D.S. Jung, (2007), How Good is the Volatility Smile Technique When Volatility Follows a Stochastic Process?, Industrial economic research, Vol. 20, No.3, June, pp. 1087{1104. [40] G. Bakshi, C. Cao, and Z. Chen, (1997), Empirical Performance of Alternative Option Pricing Models, Journal of Finance, 52, pp. 2003{2409. [41] Gill Je , (2002). Bayesian Methods - A Social and Behavioral Sciences Approach, Chap- man&Hall/CRC, Boca Raton, Florida. [42] Luke T. Miller and Chan S. Park, (2005), A Learning Real Options Framework with Application to Process Design and Capacity Planning, Production and Operations Man- agement Society, Vol. 14, No. 1, Spring 2005, pp. 5{20. [43] H. Herath and T. Herath, (2009), Investments in Information Security: A Real Options Perspective with Bayesian Postaudit, Journal of Management Information Systems, Vol. 25, No. 3, pp. 337{375. [44] Hugh McManus and Daniel Hastings, (2005), A Framework for Understanding Un- certainty and its Mitigation and Exploitation in Complex Systems, MIT Engineering Systems Symposium. [45] Arnd Huchzermeier and Christoph Loch, (2001), Project Management under Risk: Us- ing the Real Options Approach to Evaluate Flexibility in R&D, Manage Science, Vol.47, No.1, pp. 85{101. 117 [46] Jeremy M. Eckhause, Danny R. Hughes, and Steven A. Gabriel, (2009), Evaluating Real Options for Mitigating Technical Risk in Public Sector R&D Acquisitions, International Journal of Project Management, 27, pp. 85{101. [47] ASD(R&D), (2011), DOD Technology Readiness Assessment(TRA) Guidance, http://www.acq.osd.mil/chieftechnologist/publications/docs/TRA2011.pdf. [48] Robert Ferguson, Dennis Goldenson, James McCurley, Robert Stoddard, David Zubrow, and Debra Anderson, (2011), Quantifying Uncertainty in Early Lifecycle Cost Estima- tion(QUELCE), Technical Report, Software Engineering Measurement and Analysis (SEMA), Cost Estimation Research Group. [49] U.S. Government Accountability O ce(GAO), (2004), DEFENSE ACQUISITIONS As- sessments of Major Weapon Programs, Report to Congressional Committees. [50] S.Y. Sohn and H.J. Kim, (2008), A Study of COO model for Successful Defense Acqui- sition Programs, Journal of KIIE 2008 Fall Conference, pp. 954{963. [51] David W. Jimenez, (2009), Cost of Ownership and Overall Equipment E ciency: a Photovoltaics Perspective, Photovoltaics International Journal, pp. 16{22. [52] Daren L. Dance, Thomas DiFloria, and David W. Jimenez, (1996), Modeling the Cost of Ownership of Assembly and Inspection, IEEE Transactions on components, packaging, and manufacturing technology{Part C, Vol. 19(1), pp. 57{60. [53] http://www.riskamp.com/beta-pert. [54] http://www.globalsecurity.org/military/world/rok/khp.htm. [55] Jenifer Piesse, C Presiaux, and A Van de Putte, (2005), Economic Valuation of Complex Projects Exhibiting Both Technical and Economic Uncertainty, Journal of International Financial Analysts, Vol. 6. [56] Ministry of Knowledge Economy, (2008), Development Policy of Rotary{Wing Aircraft Industry, Report by Korea Aerospace Research Institute. [57] C. Hwang, J. Lee, B. Chang, and I. Hwang, (2010), Civil Helicopter Market Analysis, Report by Korea Aerospace Research Institute, http://library.kari.re.kr. [58] Dennis D. Wackerly, William Mendenhall, and Richard L. Schea er, (2002), Mathemat- ical Statistics with Applications, 6th Edition, Duxbury, CA. [59] http://www.bayesian-inference.com/samplesize. [60] Luke Miller and Chan S. Park, (2004), Economic Analysis in the Maintenance, Repair, and Overhaul Industry: An Options Approach, The Engineering Economist, 49, pp. 21{41. [61] Luke T. Miller, (2010), PMA License Valuation: A Bayesian Learning Real Options Approach, Review of Financial Economics, 19, pp. 28{37. 118 [62] Murphy Kevin, (2007), Conjugate Bayesian analysis of the Gaussian distribution, http://www.cs.ubc.ca/ murphyk/Papers/bayesGauss.pdf. [63] STEPI, (2010), Methodological Improvement for the Economic Assessment of Public R&D Programs, Science & Technology Policy Institution, Seoul. [64] M.S. Oh, (2012), Bayesian Inference with R monte carlo, Free Academy. [65] Navarro Dan and Perfors Amy, An Introduction to the Beta{Binomial Model, http://www.cs.cmu.edu/ 10701/lecture/technote2-betabinomial.pdf. [66] George Prueitt and C.S. Park, (1997), Phased Capacity Expansion{Using Continuous Distributions to Model Prior Beliefs, The Engineering Economist, Vol. 42, No. 2, 1997, pp. 91-110. [67] C. W. Kirkwood, (1997), Strategic Decision Making: Multiobjective Decision Analysis with Spreadsheets, Duxbury Press. [68] Thomas L. Saaty, (1994), Highlights and Critical Points in the Theory and Application of the Analytic Hierarchy Process, European Journal of Operational Research,(74), pp. 426{447. [69] Ralph L. Keeney, (1992), Value-Focused Thinking: A Path to Creative Decision making, Harvard University Press. [70] J. Malczewski, (1999), GIS and Multicriteria Decision Analysis, John Wiley and Sons, New York, NY. [71] Department for Communities and Local Government, (2009), Multi-criteria analysis: a manual, Website: www.communities.gov.uk. [72] Robert T Clemen and Reilly Terence, (2001), Making Hard Decisions with Decision Tools, Duxbury. [73] Minh-Tuan Nguyen, (2003), Some Prioritisation Methods for Defence Planning, Defence Systems Analysis Division Information Sciences Laboratory. [74] Stefan Hajkowicz, Mike Young, Sarah Wheeler, Darla H. MacDonald, and Doug Young, (2000), Supporting Decisions: Understanding Natural Resource Management As- sess- ment Techniques, Technical report, Land and Water Resources Research and Develop- ment Corporation. [75] Alonso J. Antonio and Lamata Teresa, (2006), Consistency in the Analytic Hierar- chy Process: A New Approach, International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, Vol. 14, No. 4, pp. 445-459. [76] Orfelio G. Leon, (1999), Value-Focused Thinking versus Alternative-Focused Thinking: E ects on Generation of Objectives, Organizational Behavior and Human Decision Pro- cesses, Vol. 80, No. 3, pp. 213{227. 119 [77] Stephen Chambal, Mark Shoviak, and Alfred E. Thal, (2003), Decision Analysis Method- ology to Evaluate Integrated Solid Waste Management Alternatives, Environmental Modeling and Assessment, pp. 25{32. [78] Jason R.W. Merrick, Gregory S. Parnell, Jamison Barnett, and Margot Garcia, (2005), A Multiple-Objective Decision Analysis of Stakeholder Values to Identify Watershed Improvement Needs, Decision Analysis, Vol.2, pp. 44{57. [79] G. Parnell, H. Conley, J. Jackson, L. Lehmkuhl, and J. Andrew, (1998), Foundations 2025: A framework for evaluating future air and space forces., Management Sci, 44(10), pp. 1336{1350. 120