Test Planning and Validation through an Optimized Kriging Interpolation Process in a Sequential Sampling Adaptive Computer Learning Environment by Jeremy Lamar Barnes A dissertation submitted to the Graduate Faculty of Auburn University in partial fulfillment of the requirements for the Degree of Doctor of Philosophy Auburn, Alabama August 4, 2012 Keywords: Kriging, Validation, Design of Experiments, Budget, Space Filling Designs, Variogram Copyright 2012 by Jeremy Lamar Barnes Approved by Saeed Maghsoodloo, Chair, Professor Emeritus of Industrial and System Engineering Alice Smith, Professor of Industrial and System Engineering Mark Carpenter, Professor of Statistics ii Abstract This dissertation explores Kriging in an adaptive computer learning environment with sequential sampling. The idea of this Design for Kriging (DFK) process was first mentioned in [1] titled ?Kriging for Interpolation in Random Simulation?. The idea presented by [1], paved the way for continued research in not only applications of this new methodology, but for many additional opportunities to optimize and expand research efforts. The author proposes several advancements to the above process by introducing a novel method of interpolation through an advanced Design for Kriging (DFK) process with cost considerations, advanced initial sample size and position determination, search techniques for the pilot design, and standardized variogram calculations. We use the terminology variogram over semivariogram as described by [2]. The resulting applications in this research are two-fold. One is to use this process in the upfront experimental design stage in order to optimize sample size and Factor Level Combination (FLC) determination while considering the overall budget. The other application is the use of sampled empirical and interpolated data to form a representative response dataset in order to perform statistical analyses for validation of Monte Carlo simulation models. The DFK process is defined as: 1) Define factor space, boundaries, and dimensions 2) Determine initial sample size through cost considerations and estimation variance analysis. Determine FLCs by a space filling design performed by an augmented simulated annealing algorithm 3) Observe responses, with replication if required, at the initial sample size and FLC selection 4) After sample responses have been observed, perform Kriging interpolation at where is some number of unobserved FLCs 5) Calculate the estimation variance 6) Based on the results from steps 3-5, identify the next candidate input combination set, , based on budget considerations and variance reduction, and repeat steps 3-5 using 7) After an acceptable prescribed accuracy measurement level is achieved or budget is exhausted, Krige the observations to achieve a representation of the underlying response function After DFK process is completed, statistically compare a verified Monte Carlo estimated response dataset ( ) with the combination of the Kriging metamodel response dataset ( ) and the actual response data , to assess the model against the combined response dataset . Acknowledgements The author is grateful to Dr. Saeed Maghsoodloo for his guidance, meticulous attention to detail, insight, and agreement to oversee this work. The author owes this very dissertation effort and Ph.D. completion to Professor Emeritus Saeed Maghsoodloo?s generosity and wisdom. The author must also recognize Dr. Alice Smith for her demanding, but continuous support for this effort. Dr. Alice Smith has produced a major contribution in the academic growth of the author through the efforts, assignments, and opportunities that she provided to the author. The author also wants to thank Dr. Mark Carpenter for his agreement and cooperation on this research. In addition, the author must recognize Dr. David Umphress for his positive support and guidance. This work is dedicated to the following: Carol Furman (mother), Rickie Barnes (father), Hannah Harris (sister), Mary Alice Barnes (fraternal grandmother), Pauline Thrower (maternal grandmother) and wife to be, Dilcu Helvaci. Without each of these people, this work is not possible. The author shares his deepest gratitude to all of these people. The author also wants to appreciate the MITRE Corporation and the Missile Defense Agency (MDA) for supplying the author with supplemental support that was critical for the completion of this degree. The author wants to add a debt of gratitude for all the friends and colleagues that encouraged the continuation of this research, despite all odds and setbacks. Finally, the author must appreciate Dr. Wim van Beers and Dr. Jack Kleijnen. Their work of Kriging in simulation in conjunction with the author?s personal interests inspired this research effort. Table of Contents Abstract ........................................................................................................................................... ii Acknowledgements ........................................................................................................................ iv List of Tables ................................................................................................................................. ix List of Figures ............................................................................................................................... xii 1 Introduction .................................................................................................................................. 1 1.1 Background and Motivation ............................................................................................. 1 1.2 Research Objectives ......................................................................................................... 3 1.3 Research Methods ............................................................................................................ 5 1.4 Dissertation Layout .......................................................................................................... 6 2 Literature Review......................................................................................................................... 9 2.1 Metamodels ...................................................................................................................... 9 2.2 Origins of Kriging .......................................................................................................... 10 2.2 Kriging Applications ...................................................................................................... 13 2.3 Kriging in Simulation ..................................................................................................... 14 2.4 The Use of Kriging in Optimization .............................................................................. 16 2.5 Monte Carlo Simulation ................................................................................................. 17 2.6 Validation ....................................................................................................................... 18 2.7 Validation Process .......................................................................................................... 21 2.8 Validation throughout the Model Life-Cycle ................................................................. 23 2.9 Validation Methodologies .............................................................................................. 24 2.10 Data Availability ............................................................................................................ 25 2.11 Validation Techniques.................................................................................................... 26 2.12 Literature Review Summary .......................................................................................... 27 3 Methodology .............................................................................................................................. 28 3.1 Kriging Methodology ..................................................................................................... 28 3.2 Ordinary Kriging ............................................................................................................ 36 3.3 Variogram and Covariance in Kriging ........................................................................... 39 3.3.1 Lag .................................................................................................................................. 46 3.3.2 Neighborhood ................................................................................................................. 47 3.4 The Bootstrap ................................................................................................................. 47 3.5 The Estimation Variance through Bootstrapping ........................................................... 49 3.6 Mean Square Error ......................................................................................................... 50 4 The Original Design for Kriging Process .................................................................................. 52 4.1 Step 1: The pilot design is selected ......................................................................... 55 4.2 Step 2: For gather pilot design response data ................................................... 56 4.3 Step 3: Estimate the mean of pilot data observations..................................................... 57 4.4 Step 4: Compute Kriging predictions ............................................................................. 57 4.5 Step 5: Perform non-parametric bootstrapping per input based on for 62 4.6 Step 6: Calculate the Kriging variance for candidate inputs .......................................... 64 4.7 Step 7: Determine the input with the largest variance ................................................... 65 4.8 Step 8: Repeat steps 4-7 until the stopping criteria has been reached............................ 65 5 Advancements in the Design for Kriging Process ..................................................................... 69 vii 5.1 Estimation Variance with No Empirical Data ................................................................ 69 5.2 Initial Size Based on Estimation Variance, Expected Improvement, and Budget .. 84 5.3 DFKS - Selection of the input FLCs through augmented spatial sampling ................... 86 5.4 Dynamic Array Slicing of the Covariance Matrix ......................................................... 94 5.5 The Covariance Function by Use of a Standard Method ............................................... 96 5.6 Reduction in Sampling Iterations and Stopping Criterion Based on Budget ................. 99 5.7 Summary of Advancements ......................................................................................... 100 6 Complete Methodology for the Advanced DFK Process ........................................................ 102 6.1 Application Area 1: Standard Variogram versus Traditional Models.......................... 102 6.2 Application Area 2: Sample Size Selection during Test Planning ............................... 108 6.3 Application Area 3: Additional Budget Determination ............................................... 117 6.4 Application Area 4: Validation of MC Simulation against Limited Empirical Data ... 120 6.5 Application Summary .................................................................................................. 125 7 The Advanced DFK Application Software .............................................................................. 126 7.1 Introduction .................................................................................................................. 126 7.2 Overview ...................................................................................................................... 126 7.3 Initial Sample Size and Location Selection.................................................................. 129 7.4 Sequential Kriging........................................................................................................ 133 7.5 Data Analysis and Recording ....................................................................................... 136 7.6 Conclusions .................................................................................................................. 137 8 Conclusions and Proposed Future Research ............................................................................ 138 8.1 Conclusions .................................................................................................................. 138 8.2 Future Work ................................................................................................................. 139 viii References ................................................................................................................................... 141 Appendix ..................................................................................................................................... 150 Appendix A: Software Code Base ........................................................................................ 150 Appendix B: Remaining Bootstrapped Data for Original DFK Process .............................. 225 Appendix C: Varying Sill and Nugget Variogram Tables and Plots .................................... 230 Appendix D: Supplemental Data from Application Area 2: Sample Size Selection ............ 237 Appendix E: Supplemental Data from Application Area 2: and FLC Selection ........... 248 Appendix F: MC Response Data Set and Augmented data set ......................................... 251 Appendix G: Additional Initial Sample Size and FLC Selection Data ................................. 265 List of Tables Table 1: Validation Methodology ................................................................................................. 24 Table 2: Variogram Models .......................................................................................................... 41 Table 3: Experimental Variogram Squared Calculations ............................................................. 42 Table 4: Experimental Variogram ................................................................................................ 43 Table 5: Fitted Variogram Models ................................................................................................ 45 Table 6: Summation of Squared Fitted Model Differences .......................................................... 46 Table 7: The Complete Lag Matrix - Original DFK Process........................................................ 55 Table 8: Pilot Design - Original DFK Process.............................................................................. 55 Table 9: Pilot Design Matrix of Lags ........................................................................................ 56 Table 10: Pilot Design Initial Response Data - Original DFK Process ........................................ 56 Table 11: Estimated Average Response Data - Original DFK Process ........................................ 57 Table 12: Partial Experimental Variogram Calculations - Original DFK Process ....................... 58 Table 13: Experimental Variogram Calculations - Original DFK Process ................................... 58 Table 14: Fitted Variogram Calculations - Original DFK Process ............................................... 59 Table 15: Minimum Squared Distance Variogram Calculations - Original DFK Process ........... 60 Table 16: Matrix - Original DFK Process ................................................................................. 60 Table 17: Matrix - Original DFK Process.................................................................................. 61 Table 18: Vector - Original DFK Process ................................................................................. 61 Table 19: Kriging Predictors - Original DFK Process .................................................................. 62 Table 20: Partial Bootstrap Data - Original DFK Process ............................................................ 63 Table 21: Bootstrapped Kriging Predictors - Original DFK Process ........................................... 64 Table 22: Bootstrapped Kriging Variance Estimates - Original DFK Process ............................. 65 Table 23: Experimental Design Iteration 1 - Original DFK Process ............................................ 66 Table 24: Pilot Design Initial Response Data - Original DFK Process ........................................ 66 Table 25: Estimated Average Response Data Iteration 1 - Original DFK Process ...................... 66 Table 26: Fitted Variogram Calculations Iteration 1 - Original DFK Process ............................. 67 Table 27: Kriging Predictors Iteration 1 - Original DFK Process ................................................ 67 Table 28: Updated Variance Estimates Iteration 1 - Original DFK Process ................................ 67 Table 29: Variogram Fitted Functions of Interest......................................................................... 71 Table 30: Reduced Variogram Fitted Functions of Interest.......................................................... 72 Table 31: Initial Set ................................................................................................................. 74 Table 32: Linear Variogram ............................................................................................. 75 Table 33: Spherical Variogram ........................................................................................ 75 Table 34: Exponential Variogram .................................................................................... 76 Table 35: Gaussian Variogram ........................................................................................ 76 Table 36: Varying Nugget ? Absolute Variance Estimation Difference ...................................... 80 Table 37: Varying Sill - Percent Variance Estimation Difference ............................................... 81 Table 38: Variogram Testing for Initial Selection Summary ....................................................... 82 Table 39: Parameter Values for Fitted Variogram ...................................................................... 104 Table 40: Optimized Variogram Values for Traditional Functions ............................................ 104 Table 41: Application Area 2 ? 28 Pilot Designs ....................................................................... 108 Table 42: Summary Data for Pilot Design Studies ..................................................................... 112 Table 43: Summary of and for all Unobserved FLCs ................................................ 114 Table 44: Recommended Pilot Sample Size and FLC Selection ................................................ 116 Table 45: Addition Budget Determination: Dimension and Factor Space Selection ................. 117 xi Table 46: Observed Values from System 1 and System 2 .......................................................... 121 Table 47: Black Box: ............................................................................................. 122 Table 48: Black Box: utilizing DFK Software ...................................................... 123 Table 49: Test Planning User Input Table .................................................................................. 131 Table 50: Pilot Design Estimated Response Data....................................................................... 237 Table 51: Pilot Design Estimated Variance Data ....................................................................... 240 Table 52: Pilot Design Squared Residual Data ........................................................................... 244 Table 53: Application Area 2 Variance Reduction Data ............................................................ 248 Table 54: Monte Carlo Data Sets ................................................................................................ 251 Table 55: Augmented Data Sets to Include Interpolated Data ................................................... 258 Table 56: Initial Sample Size and FLC Selection Data Compilation ......................................... 265 List of Figures Figure 1: Two-Dimensional Block with Estimation Location ...................................................... 12 Figure 2: Sample Variogram ......................................................................................................... 13 Figure 3: Overall Validation ......................................................................................................... 21 Figure 4: Observational Results Validation .................................................................................. 23 Figure 5: Validation through the Life-cycle ................................................................................. 26 Figure 6: Variogram Cloud ........................................................................................................... 43 Figure 7: Plotted Experimental Variogram ................................................................................... 44 Figure 8: Fitted Variogram vs. Experimental Variogram ............................................................. 45 Figure 9: Factor space - Original DFK Process Example ............................................................. 53 Figure 10: Fitted Variogram - Original DFK Process .................................................................. 59 Figure 11: Linear Variogram with Varying Range ....................................................................... 77 Figure 12: Spherical Variogram with Varying Range .................................................................. 77 Figure 13: Exponential Variogram with Varying Range .............................................................. 78 Figure 14: Gaussian Variogram with Varying Range ................................................................... 78 Figure 15: Factor space Array ....................................................................................................... 89 Figure 16: Spatial Sampling Design Matrix ................................................................................. 90 Figure 17: Noisy Empirical Variogram ...................................................................................... 103 Figure 18: Graphical Variogram Summary ................................................................................ 105 Figure 19: Standard Model Variogram ....................................................................................... 107 Figure 20: Response Data for Sample Size Selection ................................................................. 111 Figure 21: Variance Reduction in Pilot Designs......................................................................... 119 Figure 22: K-S Cumulative Probability Plots of MC vs. System Responses ............................. 123 Figure 23: K-S Cumulative Probability Plots of MC vs. Augmented System Responses .......... 124 Figure 24: Advanced DFK Software Introduction Screen .......................................................... 128 Figure 25: DFK Software - Kriging Introduction ....................................................................... 129 Figure 26: Advanced DFK Software - Test Planning ................................................................. 130 Figure 27: Advanced DFK Software - Sequential Kriging ......................................................... 134 Figure 28: Advanced DFK Software ? Processing ..................................................................... 135 Figure 29: Advanced DFK Software ? Output ........................................................................... 136 List of Abbreviations BoK Body of Knowledge CI Confidence Interval CPU Central Processing Unit DoD Department of Defense DFK Design for Kriging DFKS Design for Kriging Sampling DOX Design of Experiments EA Evolutionary Algorithm EI Expected Improvement FLC Factor Level Combination ( ) GoF Goodness of Fit GRG Generalized Reduced Gradient GUI Graphical User Interface IID Independent and Identically Distributed I/O Input/Output K-S Kolmogorov-Smirnov LC Linear Combination LHS Latin Hypercube Sampling LOS Level of Significance M&S Modeling and Simulation MC Monte Carlo MDA Missile Defense Agency MIL-STD Military Standard MSE Mean Square Error OK Ordinary Kriging SLREG Simple Linear Regression V&V Verification and Validation VBA Visual Basic for Applications Nomenclature Borel Set Correlogram Square neighborhood structure The dimensional set of real numbers Linear cost scalar Confidence coefficient Random error term A parameter Population mean Degrees of freedom Population standard deviation ? Ordinary Kriging estimated variance ? Simple Kriging estimated variance Variogram function Kriging weight vector The Kriging weight Ratio of lag over range: Lagrangian multiplier Fitted variogram value Covariance matrix Variogram matrix for a single unobserved FLC Variogram matrix for the selected FLCs Position vector p position in Position vector q position in Euclidean distance (lag) matrix Vector of Euclidean distances between observed FLCs and a single unobserved FLC Response vector Nugget Partial sill Distance per variogram function Euclidean distance Lag Lower bound Total number of FLCs in the factor space Number of unique lags in A subset of the possible FLCs xviii FLCs at which Kriging interpolation is performed The number of unobserved Factor Level Combinations (FLCs) at which Kriging interpolation is performed Sample size for Monte Carlo model Number of replications The square root of the unbiased estimator of population variance The variable is defined as an element of set Upper bound Factor space Dimension number The FLC all of whose elements are at minimum Next observational candidate point Inputs for the initial observational set Inputs for the next observational candidate set ? } Next observational candidate point with highest Kriging variance Response dataset generated from Monte Carlo models Interpolation dataset generated from DFK Observed sample response dataset Combined response dataset and ? Point estimate of mean response ? Point estimate of the response at a specific unobserved FLC Random function at location xix The response of a random process ? A Predicted response of a random process 1 CHAPTER 1 1 Introduction This chapter introduces the research background and motivation for the dissertation, outlines the objectives that the research intends to achieve, and then gives the research methods adopted. The introduction provides the framework for the research that follows. The chapter concludes by describing the layout of the dissertation. 1.1 Background and Motivation Kriging has traditionally been used in the realm of geostatistics since its inception in the 1960s and it has continued to gain importance in the deterministic and stochastic simulation community [3] and in the machine learning community [5]. The iterative nature of learning a predictive model is also known as active learning [6]. Kriging was developed by Georges Matheron and named in honor of Danie Krige, a South African mining engineer. Kriging is used extensively in geostatistics which is the mapping of surfaces from limited sample data and estimation of values at the unsampled locations [7],[8],[9]. There are several methods of Kriging, all with the intent to estimate through interpolation a continuous, spatial attribute at an unsampled site. Kriging is a form of generalized linear regression that forms an optimal (or best) linear estimator in a minimum mean square error sense [9] through partial differentiation. Kriging is formally defined as a random process described by where is a fixed subset of with a positive -dimension [10] and is a random function at location . 2 This dissertation introduces a novel advanced DFK process and utilizes that process for test planning and simulation model validation. The DFK process is a general approach that can be used in many areas where data interpolation is needed assuming, as in any case, that the experiment meets the criteria and assumptions needed for the DFK algorithms to work. The application of Kriging can range from unexploded ordnance location prediction through the use of the logistic probability density function, topography resolution enhancement, image sharpening, computer graphical enhancements, and statistical validation and response data estimations. This advanced DFK process has applications in experiments that have a need to optimize interpolation accuracy while minimizing cost. The advanced DFK process starts with the identification of an experiment/system. The experiment obviously contains a sample space, boundaries, inputs, outputs, and other relevant characteristics. After experiment identification and the establishment of factor space, the boundaries and dimensions are formalized. Utilizing a unique space filling design, along with cost considerations and estimation variance ( ? ), the initial sample size and FLCs are formalized. Next, empirical observed responses with the initial sample size and at the FLCs are collected from the experiment. Inputs that are independent cannot be used in Kriging since the Kriging coefficient matrix will become singular resulting in a zero determinant with no feasible solution for Kriging. After the responses from the sample observations with replication, if required, have been established, a fit of the covariance matrix is performed. This results in the function which is used to create the variogram ( ) where , where is defined as a correlogram. Next, Kriging is performed at unobserved points. The estimation variance is 3 again calculated with the use of the variogram to determine if the current interpolations meet a prescribed accuracy level to determine where the next worst interpolation occurs. Kriging is performed again, within budget constraints, for another sample set until an acceptable accuracy level is reached. Kriging is performed a final time to achieve the remaining unobserved responses . With the use of and where , a surface map of the representative underlying function can be constructed. Next, a verified Monte Carlo model is run a sufficient number of samples each of size , with replication as required, to generate . The datasets and are statistically compared using the non-parametric Kolmogorov- Smirnov (K-S) Goodness-of-Fit (GoF) Test to decide if the cumulative distribution function of the empirical sample differs from the cumulative distribution function produced from the Monte Carlo Model. These methods are presented in Chapter 6. 1.2 Research Objectives The research objectives mainly include three aspects. The research objectives are a unique combination of mathematical methods along with distinct applications for the justification of the establishment of such an interpolation method process. The first objective explores the level of accuracy that can be attained between a simulation model and empirical limited data by using sequential Kriging in an adaptive computer learning environment with associated cost considerations. The response dataset generated through the interpolation capabilities of Kriging are compared with that of an associated Monte Carlo simulation model to infer validation statements of the Monte Carlo model. 4 The second objective determines techniques in augmented space filling designs, estimation variance calculations, and cost constraints to identify an initial sample size and FLCs for this application. This contribution allows one to determine not only the number of data points to sample or tests to be conducted, but also what FLCs of inputs should be selected to maximize the application of Kriging interpolation model. The second objective addresses using interpolation methods for test planning prior to executing any tests or collecting sample data. The reader should note that the calculation methods are presented in Chapter 5. The third objective, although seeming to be multiple objectives, all have the intent of optimizing the DFK process. The context of optimization has two meanings herein. The first meaning is to optimize the estimation variance by selecting FLCs with the highest estimation variance and the second is to optimize the software in terms of Central Processing Unit (CPU) time. The third objective is the determination of an augmented simulated annealing process for sequential sampling providing an adaptive computer learning concept for the process by selecting a sample size weighted to where the interpolation estimates are the worst, i.e. the unknown response is the noisiest. In addition, a single method is evaluated for a covariance function for use when developing the variogram that will allow for increased accuracy of the variogram calculations. This is accomplished through the combination of iterative regression analysis combined with Generalized Reduced Gradient (GRG) or Evolutionary Algorithm (EA) as required. Additionally, the gamma ( ) values of the traditional variograms currently have to be recalculated for each additional input. We optimize the traditional variogram calculation 5 process during the test planning phase through dynamic array slicing that eliminates the need for recalculations of the variogram for previous inputs. 1.3 Research Methods The dissertation presented here improves the utility of Kriging and establishes an advanced DFK process with cost constraint considerations. The research develops and enhances an overarching process based on multiple mathematical theories that will improve interpolation accuracy, define minimal required observational samples with a goal of minimizing cost, and provide an advanced statistical measure to perform validation between computer model datasets and limited empirical datasets. As is inherent in Kriging, the models are fully and clearly expressed and are differentiable, which will allow sensitivity analyses in the responses through partial differentiation to assist in selection of . This approach combined with FLC selection aims to reduce the number of total iterations required. The empirical research in this dissertation is important. The dissertation separates out each step of the process and uses several cases to investigate the improvement of the proposed step in the process. The dissertation then puts together the findings from DFK optimization and performs validation case studies based on physical systems (or representations thereof). The final results of the DFK process are presented in an effective manner. 6 Finally, comparison and contrast is widely used throughout the research. For the presented test cases, models or methods are used to implement, analyze and compare their advantages and disadvantages. 1.4 Dissertation Layout The dissertation is divided into eight chapters including this first chapter entitled ?Introduction?. The layout and organization of remaining chapters are as follows. Chapter 2 presents the literature review on Kriging. This includes the origins, applications, and its early development. The application review of Kriging pertains to Kriging in simulation, empirical sampling, and sequential sampling. Chapter 2 continues by expanding on the current Body of Knowledge (BoK) for validation. The purpose of this portion of the chapter is to express explicitly where this advanced interpretation process benefits validation research. The chapter continues with some review of simulation modeling including definitions for important and relevant statistical measures. The chapter finalizes with general information regarding the Monte Carlo method. In Chapter 3, the Kriging methodology is presented. The principles, mathematical development, and structure are identified and introduced. Chapter 3 continues with discussion of the variogram or covariance functions. Next, an introduction to the terminology and ideology and assumptions behind the lag and the neighborhood structure when used in spatial interpolations is presented. Chapter 3 concludes with a brief summary. Chapter 4 describes in detail the current DFK process. The layout for describing this process is through a detailed discussion of each step while utilizing an empirical example for clarity and 7 demonstration of concept. The included demonstration pertains to an experiment in soil sampling for a certain area around Toomer?s corner in Auburn, AL to determine the amount of soil contamination by the use of Spike 80DF. This chapter is critical for understanding the current approach. The reading of this chapter will provide the reader with the information needed in understanding the remaining chapters. Chapter 5 describes specific advancements that pertain to the associated steps in the advanced DFK process. Chapter 5 will empirically investigate the effectiveness of each of the following: 1) Determination of when cost constraints are present along with estimation variability 2) Space filling designs to maximize the information observed from each input combination 3) Standard variogram fitted function 4) Stopping criteria based on budget considerations are investigated Chapter 6 reassembles the seven step process into a comprehensive and cohesive interpolation method for generating . Iterative Kriging is performed through software to demonstrate augmentation of . Chapter 6 empirically analyzes how the process aides in the overall test planning and validation efforts. The systems (or representations) used were independently developed and given as black boxes in which to gather response data. This actual data (or the terminology ?truth? data used by DoD) are augmented through the advanced DFK process to provide data to compare with the corresponding Monte Carlo simulations data. Chapter 6 concludes by evaluation of against . This assessment yields an overall effectiveness of the proposed process. 8 Chapter 7 introduces the DFK software in order to satisfy the application needs. The software provides a user-friendly graphical user interface (GUI), contains built-in help, and is presented as an add-in to Microsoft Excel?. The detailed description about usage and manipulation of this functional application software will be discussed with graphical aides throughout. In addition, chapter 7 provides concise information in order to obtain and install the software. Chapter 8 concludes the dissertation and lays the foundation for future research directions. 9 CHAPTER 2 2 Literature Review Simulation is the experimenting with or exercising a representation of an entity, a system or an idea under a multitude of objectives including, but far from limited to, acquisition and analysis [11]. Models begin in the conceptual stage which supports the development of the underlying theories and assumptions such that the structure, logic, and mathematical and casual relationships match a system that the model is set to represent [12]. 2.1 Metamodels Originally metamodels were constructed using regression analysis as discussed in [13]. Regression metamodels are categorized by order. First order metamodels have a response variable Y and are modeled as where are the model parameters, each Xi is a model input, and ? is residual error. Second order metamodels include both pure quadratic and two-variable interaction terms and are modeled as follows ? ? with similar variable representations as the first order metamodel. Metamodeling techniques have evolved to include such methods as neural networks, Kriging, experimental designs, along with the traditional regression methods [14]. Metamodels allow improved understanding, a quicker turnaround time for data generation used in sensitivity analysis, optimization, and decision analysis. Metamodels provide lower fidelity of the full 10 simulation model with a simpler model that can usually be constructed within a spreadsheet [15]. A metamodel approximates the Input/Output (I/O) transformation of simulation such that the metamodel relates its outputs to that of a system. Therefore, metamodel output approximations can help in determining a simulation model?s validity. In summary, metamodels will continue to be utilized as simulations become increasingly complex as they provide rapid information about the underlying simulation model. This information can be used in decision making, validation, or optimization of simulation models. Optimization of simulation models through the use of metamodels is not discussed as it is left for future research. 2.2 Origins of Kriging Kriging, originally named ?krigeage? by Pierre Carlier [10], was developed by Georges Matheron [16] and named in honor of Danie Krige, a South African mining engineer. The approach was developed to ascertain gold mining valuation problems. Initially, through exploration research, Kriging can be tracked back to the research of Wold [17], Kolmogorov [18],[19], and Wiener [20]. The process of Kriging is used extensively in geostatistics which is the mapping of surfaces from limited sample data and estimation of values at the unsampled locations [7],[8],[9]. Matheron [16] describes Kriging as a weighted average of available samples that can be used for predictions and with suitable weights the variance should be minimized [21]. Daniel Krige defined the term as a multiple regression which is the best linear weighted moving average of the grade of a type of rock that contains mineral for a block (or area) of any size by assigning an optimum set of weights to all the available and relevant data 11 inside and outside the block [22]. Ord [23] states that Kriging is an interpolation method for random spatial process, and similarly Hemyari and Nofziger [24] define Kriging as a weighted average, where the weights are dependent upon location and structure of covariance of observed points [24]. The author argues against the later definition and instead states that the weighted average may depend upon location and structure of covariance of observed or unobserved points as long as the problem has a clearly defined bounded region and the normality assumption holds. The above statement is clarified in Chapter 5. Originally, Kriging was a linear predictor. Later developments in geostatistics, Kriging was extended to nonlinear spatial prediction called indicator Kriging. As stated in [21], the origins of Kriging given by Cressie [25]. Cressie addresses Kriging from different disciplines and states the conclusion that Kriging is equal to spatial optimal linear prediction. There are several forms of Kriging, all initially formulated for the estimation of a continuous, spatial attribute at an unsampled point. Figure 1 shown below is a pictorial of a block for a two- dimensional sample with the open circle being the estimation location [9]. Extrapolations outside the block are possible, but are unreliable. The designs presented in this research are limited to blocks with some positive . 12 Figure 1: Two-Dimensional Block with Estimation Location Kriging uses a weighted average of sample values to yield a linear estimate of the unknown value at a given location. This estimator is in the form of ? where the weights sum to 1 (in Ordinary/Punctual Kriging) [26], [27] and is the response data. The results or interpolations are unbiased with an estimation variance. Typically, weights are optimized using the variogram model describing the location of the samples and all the relevant inter-relationships between known and unknown values. A variogram describes the covariance structure of the difference between two observations and is the backbone in ordinary Kriging [28]. A sample variogram is shown below in Figure 2. 13 Figure 2: Sample Variogram This section provides a brief introduction and history about Kriging which is formally introduced and discussed in Chapter 3. 2.2 Kriging Applications Kriging now covers many research areas and disciplines. The original applications were that of ore and gold mining and quickly expanded to soil sciences [29],[30], geology [31],[32],[33], meteorology [34], and hydrology [35],[36]. Today Kriging applications continue to grow and can be seen in wireless network analysis, cost estimation, engineering design, simulation studies, and optimization. The literature demonstrates the growth of Kriging applications in many new areas. These areas include biomechanical engineering [37],[38], wireless wave propagation [39],[40], material sciences [41],[42], engineering design [43],[44], economic analysis [45],[46], simulation 0 5 10 15 20 25 30 35 40 45 50 0 5 10 15 20 25 Emperical Variogram Emperical Variogram Linear Spherical Exponential Gaussian 14 interpolation [47],[48][3], and optimization [49],[50],[51]. The use of Kriging in simulation interpolation has given rise to the term Design for Kriging (DFK) as mentioned by Klein [52]. DFK was first introduced by Beers [3] and, arguably, was originally titled ?Customized Sequential Designs for Random Simulation Experiments: Kriging Metamodeling and Bootstrapping?. The introduction of DFK further increases the opportunity for application in design, planning, optimization, and simulation. 2.3 Kriging in Simulation Random simulations are typically run with different combinations of simulation inputs while responses are observed, generally with replication. These I/O data are analyzed through statistical analysis such as low-order regression metamodels. Since a metamodel is an approximation of the I/O transformation implied by the underlying simulation model, the metamodel can be applied to the output of all types of simulation models. Even though Kriging metamodels have been applied extensively in discrete event simulation, Kriging has hardly been applied to random simulation. As with regression, in deterministic simulation, Kriging has been applied frequently and is attractive since this technique ensures that the metamodel?s prediction has exactly the same value as the observed simulation output. When used in random simulation, Kriging produces an unbiased mean estimator of the simulation responses at unsampled locations [28]. For the introduction of Kriging in simulation, the classic references by Sacks et al. [53],[54] utilize Kriging interpolation as an inexpensive but efficient predictor for cost reduction in computer experiments that are computationally expensive to run. Mitchell and Morris [55] 15 suggested that modifications can be made to handle simulation interpolation with random inputs while investigating the use of Kriging to evaluate the importance of input parameters in the deterministic simulation of a groundwater flow model. Barton [47] acknowledges the application of Kriging in random simulation, but points out that there is only a small set of samples available at this time. Barton [47] states that the current availability of computer code is a limiting factor. Kriging metamodels have been shown to yield more accurate predictions than low-order polynomial regression models [28] and also have been shown through the male gypsy moth flight phenology study that Kriging accuracy is nearly as precise as a complicated 6th order polynomial regression model [56]. Kriging in random simulation environments is perhaps most represented by the work of van Beers and Kleijnen [57],[1],[4],[3]. They explored Kriging in three main areas. The first area investigated was Kriging in random simulation when variances of simulation outputs are not constant. The second investigation was the introduction of DFK in a deterministic simulation environment. That led to the third area of investigation and the basis behind this dissertation: DFK in random simulations. The DFK in random simulation study led to an alternative and more accurate variance measure as discussed by Kleijnen [58]. Physical simulation, like computationally expensive simulations, has similar difficulties in obtaining data with an adequate sample size because of cost and schedule constraints. Exploring Kriging interpolation for physical systems has been very few in the literature [59],[60]. This research provides more work in this area as it is warranted. 16 2.4 The Use of Kriging in Optimization The literature describes that Kriging has recently been applied to optimization in a two areas. The first area is the assistance of Kriging in evolutionary optimization. The second area is in sequential Kriging optimization. Evolutionary algorithms are used to solve optimization problems where exact fitness functions may not exist. Kriging is used as a temporal or progressive fitness function to assist in evolutionary optimization. Research in this area has been conducted by Ratle [61]. Sequential Kriging optimization uses Kriging to approximate the objective function with a Kriging model, and then uses the Kriging model to determine points for sequential sampling [21]. Sequential Kriging optimization is similar to DFK in that a Kriging model is used to determine points to be sequentially sampled. Additionally, Biles et al introduced Kriging for constrained simulation optimization. Constrained simulations are simulations that impose the additional constraint of being related by some relation [62]. The research in [51], modeled an inventory system with the objective of finding the optimal values of reorder point and maximum inventory. Optimal solutions were found through the results of their experiments. Their results indicate that Kriging offers opportunities for solving constrained optimization problems in stochastic simulation. Although Kriging offers potential improvements in optimization, little literature exists on improving the DFK process in a sequential sampling adaptive computer learning environment. This research addresses optimization in areas of initial sample size selection, variogram modeling, and reductions in the number of iterations in the sequential sampling environment. 17 2.5 Monte Carlo Simulation Monte Carlo Simulation involves the use of pseudo random numbers to model systems where time plays no substantive role (i.e., static models). ?The Monte Carlo method provides approximate solutions to a variety of mathematical problems by performing statistical sampling experiments on a computer,? [63]. Monte Carlo allows generation of artificial data through the use of a random number generator and utilizing the underlying probability law of interest. To generate Monte Carlo inputs the cumulative distribution function ( ) of the input(s) must be inverted. After inverting the , uniform random numbers between 0 and 1 are generated and put onto a one-to-one correspondence with the inverted , thus resulting in random inputs that can be used in Monte Carlo simulation. This procedure is on sound statistical ground because it can be proven that the of all continuous variates have the distribution. Primary components of a Monte Carlo simulation are as follows: 1. ? the density function describing the system to be modeled 2. Random number generator ? source of ?random? numbers 3. Sampling rule(s) ? method for generating random samples from the specified (generally based on the random numbers but can be transformed) 4. Scoring ? recording the ?outcome? of each sample A secondary (?optional?) component that is sometimes found in Monte Carlo models is variance reduction techniques ? methods for reducing variance in the estimated solution (to reduce the required computer time). 18 2.6 Validation Many industries ranging from commercial to military require the use of multifaceted representations to closely imitate complex real-world processes, functions, systems, and entities. These representations are more commonly known as models with the process known as modeling. While models can range from mathematical approximations, physical representations, diagrams, etc., the models in this dissertation are developed and utilized in a controlled test case environment with a limited initial empirical dataset [64]. Simulation models, in general, are extensively used to gain insight into advanced configurations of systems and sub-systems that interact together with some common goal. The users and stakeholders of these simulation models are rightfully concerned whether the information derived from these models are valid, which means a viable application that satisfies the metrics of simulation models? objectives or intended use [13]. Also note that the term confidence here does not coincide with the statistical definition of confidence but instead is defined as self-assurance. Benefits from valid simulation models and simulation models in general are far-reaching and range from decision making [65], predicting (forecasting) through interpolation and extrapolation, response surface mapping, operating procedure policies, to evaluating/defining performance characteristics. Although there are great benefits from valid simulation models, the reader should be aware that there are many difficulties in gaining the confidence level in these models to claim an agreed upon statement of validity. A major issue that complicates validation of all simulation models is the wide variety of quantitative and qualitative approaches that contain uncertainty, risk, and the 19 typical high cost involved in gaining a required confidence level from the stakeholder(s) that the model(s) at hand are valid. This is reiterated by the quote [12] ?there is no set of specific tests that can be easily applied to determine the validity of the model.? Validation costs not only involve life-cycle model development, testing, capital resources including human capital, and other cost factors, but also the price of a commonly accepted and regarded standard [66] known as independent (or third party) validation. Independent validation is not only desirable but offers benefits, such as the lack of bias, when used as an integral part of the overall validation effort [67]. The validation issue compounds when models are accompanied by very limited amounts of response data that are available, or can be realistically obtained. For clarity, response data also called truth or empirical data is regarded as any actual system data, performance characteristics, and/or model parameters that have been collected and is in an acceptable state to analyze the simulation model against. It is extremely important to notice the words ?acceptable state?. Response data need to be rigorously analyzed to ensure that it represents an accurate representation when estimating parameters. All simulation models are validated to some level depending on many factors such as cost, schedule, criticality and data availability. Simulation models can be categorized into three types of validation modeling environments: models with no real data, models with limited available data, and models with a multitude of available data. Each of these situations calls for specific validation methods. The categorized type of validation for this dissertation focuses on simulation models using limited data sets. Since model validation is specific to the intended 20 purposes of the models, all boundaries, accuracies, parameters, and conditions will be clearly identified as described in [12] prior to model development. Validation is not to be performed at any specific stage of the model development life-cycle but at increments in between and throughout. Generally accepted life-cycle descriptions are found in MIL-STD-499B [68] and can be reviewed for clarification at the reader?s digression. As stated, validation of a simulated model should be completed at increments during model development. Validation therefore should be an integral part of a simulation model?s development. Different methodologies and techniques apply depending on where the model is in the development cycle, the availability of data, and the type of model at hand. Even with different methodologies and techniques available, programmatically five primary areas are involved for simulation model validation: requirements validation, conceptual model validation, design verification, implementation verification/validation, and operational validation [69]. This dissertation from a simulation model point of view focuses on operational validity with the assumption that the representative system or experiment exists in which to draw limited response data through sequential sampling. Sequential sampling limited response data yields points that allow for estimations of the underlying response function. As response data are obtained throughout the sequence, the response functions become more accurate and precise. The precision and accuracy is not due to sequential Kriging, but due to increasing sample size associated from sequential sampling. Using this given response function (set of weights for Kriging) information, interpolation can be used to estimate the unsampled response points using algorithms such as regression and Kriging. 21 Validation requires three steps. The first is the process of validation. The second is the methodologies of validation that are available and can be applied throughout the life-cycle of a model. The final step is the validation techniques. Many techniques (prescriptive and descriptive) can be used that overlap the different methodologies while overarching validation process remaining relatively the same. Validation is depicted by Figure 3. Figure 3: Overall Validation The overarching approach to validation appears to be straightforward, but the details involved are very dynamic and complex depending on the application. 2.7 Validation Process The validation process begins by obtaining pre-test prediction data through methods such as MC. Next, test data are gathered and compared with the MC data, and results are determined to either be consistent or different. If the results are consistent, they should be formally recorded 22 and therefore should be deliverable. This information is categorized as the current state of validation. The states will vary from initial, to ongoing, to final as the life-cycle of a model matures. If the results are different when compared, an analysis should be conducted to determine if the discrepancy is in the test condition, the model, human error, and/or the interface/component performance. If it is determined that it is in the model, then the model should be altered and the process should be repeated. If the discrepancy is in the test condition then one has to determine if the test condition met all the objectives. If all objectives have been met, then formally record the results. If the objectives were not met, then replan the test. Finally if the discrepancy was caused due to system interface/component performance (or human error) then the governing Validation and Verification (V&V) authority has to decide whether the error was due to an anomaly, reevaluate the system and document the results or to modify the Modeling and Simulation (M&S) and start the process over. The general validation process described by [69] is shown below in Figure 4. 23 Figure 4: Observational Results Validation This process obviously assumes that the model has been developed and partially undergone verification tasks. 2.8 Validation throughout the Model Life-Cycle The life-cycle of a simulation model generally goes through five stages. These stages are the pre-concept and concept definition, defining mathematical relationships, model development including metamodels, operational stage, and the support stage. These stages do not have to occur linearly as in a waterfall development and may contain feedback loops between them. Verification is considered complete after the models have been developed and are at a level of maturity in order to proceed to validation. 24 2.9 Validation Methodologies Validation occurs through various types of methodologies and varies throughout the life- cycle development of a simulation model [66] which was previously described. The overall validation process should be completed by using one or more of the methods described in [12] and summarized in Table 1: Validation Methodology. Table 1: Validation Methodology Validation Methodology Description Comparison to Other Models Comparing the result of one model to that of a valid model Degenerate Tests Removing portions of the model and observing the behavior Event Validity The events of occurrences of the simulation model are compared to the real system Extreme-Conditional Tests Testing a model at its boundaries and observing the behavior Face Validity Asking experts to view the I/O stream Fixed Values Allows the developer to check the output against calculated values Historical Data Validation Allows usage of data to build and test a model Multistage Validation Develop assumptions, empirically test where plausible, and compare input output relationship Operational Graphics Use graphical images or renderings to demonstrate the model Sensitivity Analysis Change a value such as the input of the model and determine the effect of the output Predictive Validation Run the model in advance of the actual system to be tested and review results after the system has completed the same cycle Traces Tracing an entity through a system Turing Tests Ask experts to discriminate the difference between the model and the actual system Various techniques can be used in each of the methodologies described. 25 2.10 Data Availability The nature of data are; unavailable, limited available, and a large amount of available data. Data originate from either the model or the referent test. Despite where data originate, it will always come in one of the above three forms. Validation methodologies and techniques of models differ greatly depending on the nature of available data. Statistical theory shows that as the amount of available data increases, the confidence level in validity will increase. This assumption is based on parameters such as system complexity, but should be considered a reasonable assumption under most nominal conditions. Figure 5: Validation through the Life- cycle as shown below is a summary of the previous topics discussed. Pr e - c o n c e p t an d Co n ce p t De fin i ti o n M a th e ma ti c al R e l a ti o n s h i p s De fin e d Mo d e l an d A l t e rn a t e Mo d e l De v e l o p me n t ( i f r e q u i r e d ) Phas e Da t a A v ail ab i l i ty V ali d a ti o n T e c h n i q u e N o n e L i mit e d Mu l ti tu d e R e q u i r e me n ts an d S o W ( Do c u me n t a ti o n ) N o n e L i mit e d Mu l ti tu d e F ac e V ali d i ty w i th S u b je c t Ma t t e r E x p e rt N o n e L i mit e d M u l ti tu d e F ac e V ali d i ty , Co mp o n e n t Bas e d F ac e V ali d i ty , Co mp o n e n t Ba s e d Do c u me n t a ti o n F ac e V ali d i ty , Co mp o n e n t Ba s e d V e ri fic a ti o n c o m p l e t e d i n th i s s t ag e 26 Figure 5: Validation through the Life-cycle Validation can be incrementally performed or performed in one comprehensive undertaking. Despite the approach, the overall process of validation will remain the same. For achieving successful levels of validity, a combination of different validation methodologies and techniques, depending on data availability and model maturity, should be considered carefully. Expertise in applying the validation process along with a well-balanced and managed schedule and budget will ensure successful validation on the M&S of interest. 2.11 Validation Techniques Various techniques are used in validation of M&S. These validation techniques are separated into two categories: descriptive and prescriptive. Descriptive techniques are considered ?what to do? approaches for validation and the prescriptive techniques are considered Ac cre dit ati on 27 ?how to do? approaches for validation processes [70]. Typically, the techniques described as prescriptive are focused more on mathematical analysis, typically statistical in nature. The descriptive techniques focus more on the programmatic portion of validation. This research introduces an advancement of the prescriptive interpolation technique of Kriging through a novel DFK with cost constraint process. This new process will be compared to other common statistical measurement used in validation to provide an empirical measure of improvement of this advanced technique over standard methods. 2.12 Literature Review Summary The literature describes a multitude of research in areas of Kriging and validation. Kriging has been used extensively in traditional applications such as geology and soil sampling. The literature supports increasing applications of Kriging in simulation, design, and optimization in today?s modern technological environments. In lieu of the literature, research is needed in the DFK process and utilization of Kriging models for physical experimentation. Improving the DFK process will allow for test planning, decreased computational intensity in sequential sampling, and decreased cost in physical sampling. The improvements also set the stage for even further advancements and applications for this unique process in future. 28 CHAPTER 3 3 Methodology This chapter describes the mathematical concepts that are presented throughout Chapters 4, 5, and 6. The other mathematical methodologies that accompany the advanced DFK process are presented in Chapter 5. 3.1 Kriging Methodology First, it is important to establish the definition of Euclidean distance. Given points and , the length of the line segment connecting them is ???? . The distance in any space from to can be stated as: ?? where is the separation distance between two points. There are several forms of Kriging, all with the intent to estimate a continuous, spatial attribute at an unsampled site. According to [9], ?Kriging is a form of generalized linear regression for the formulation of an optimal estimator in the minimum mean square error sense?. Simple Kriging provides a gateway into more detailed methods of Kriging. Simple Kriging is limited due to its simplicity and embedded assumptions. Ordinary Kriging, the method used in this research, is the most widely used Kriging method and is derived from the principles founded 29 in simple Kriging. The acronym B.L.U.E is associated with ordinary Kriging. The acronym stands for Best Linear Unbiased Estimator [8]. Ordinary Kriging is linear since it estimates weighted linear combinations of data. It is unbiased in the sense that it attempts to have zero mean residual. Finally, ordinary Kriging is considered ?best? since it tries to minimize the variance of errors. Practically speaking, the goal of ordinary Kriging is unattainable as the true mean error and variance are almost always unknown. This implies that it cannot guarantee zero mean error or that variance is minimized. The best attempt is to build a model from available data and work with the average error and the error variance. In ordinary Kriging, a probability model is used such that the bias and error variance can both be calculated. Weights should be chosen such that the average error for the model is exactly zero and that the error variance is minimized. To develop the Kriging methodology, the following must be defined in order to begin the mathematical development. Let be a random function consisting of the collection of random variables where is a fixed subset of in the positive dimension d and where is the sample space. makes up a Borel set , the collection of all subsets including . This stochastic function can be written as with the fitted value at an unobserved point X written as ? ? taken from a linear combination of observed values. 30 To develop the details of the Kriging mathematical model a quick review of the fundamental properties of linear combinations of random variables is provided. The derivations and proofs can be found in [9]. Let be a random variable of a continuous random function, where the FLC then for any constant coefficient ? the following holds true This is derived from the distributive property of the linear mathematical first moment operator , ? , where is the probability density function, is a constant and can therefore be pulled out of the integral resulting in ? where the integral is the expected value of . Then for any coefficient the following summation holds true ( ?) [? ] ? ? Next the second moment needs to be determined. Let be a random function of location. Then for any coefficient the following holds true ( ?) [{? } ] {? } ?? [ ( )] which reduces to 31 ( ?) ?? According to [9], the simple Kriging mathematical model is based on the following three assumptions followed by two definitions: 1. The sampling is a partial realization of a random function where x denotes spatial location. 2. The random function is second order stationary. This implies that moments involving up to two variates are insensitive to any joint spatial translation, depending on the Euclidean distance. 3. The assumption that the mean is known. This assumption is unique to simple Kriging. There are definitions that must be stated in order to accurately develop the Kriging mathematical model. Let be a second order stationary random function with mean . The estimator ? at input ?location? is given by the following linear combination of random variables at sites , where respresents a FLC, considered in the sampling ? ? ? Now let be the covariance of a second order stationary random function ; then the general expression for the variance at unsampled site written as for convenience is equal to 32 [ ? ] ? ? ? [? ] ?? Separating the term and using ( ) ( ) , then ?? According to [9], the purpose of simple Kriging is to find a set of weights for the estimator that yields the minimum mean square error. The solution to this system of equations is known as a set of normal equations. Now, let be a positive integer and be a set of real numbers, and let be a set of points in an -dimensional Euclidean space. Then the continuous function is said to be positive definite if ?? ( ) Let be the weights, written in matrix notation as , for the simple Kriging estimator and let be the covariance for the random function. If the covariance is positive definite then two 33 conclusions can be made according to [9]. The first conclusion is the weights produce the minimum estimation variance and are the solution to the covariance matrix below: ? ? ?????????? ? and the second conclusion is that the estimation variance is positive definite. To minimize the above system of equations, the partial derivatives of for must be taken and set to zero. This will minimize the mean square error given by the weights. This can be shown through the following Recall the following to demonstrate the nonnegative variance ?? With the unbiased estimator established, the second or stationary assumption proven to hold, and the variance minimized in a mean square error sense through differentiation, it?s important to note the simple estimation variance, , which is given by 34 ? The when the nugget = 0 which is the assumption within this research. Thus, the resulting estimation variance is ? ? The simple Kriging estimation variance is turned into a constrained optimization problem by introducing a LaGrange multiplier for ordinary Kriging. This addition will be seen in the ordinary Kriging section to follow. This variance derivation allows for the developments in step 1 of the advanced DFK process to aide in the initial sample size selection. For simplicity and completeness the simple Kriging system of equations will be discussed using matrix notation. First, let ?s be the sampling sites of a discreet sample subset of size and let ?s be covariances. The covariance matrix, is defined as [ ( ) ( ) ( ) ( ) ( )] The above equation can also be written with the following notation where represents the matrix of covariances, shown in the next section, is derived from the variogram function . [ ( ) ( ) ( ) ] 35 Let be the optimal weights for the estimator and let stand for the transpose of the matrix. Then is the matrix For the third matrix development, let be the covariance of the random function, be the estimation location, and the ?s be sampling sites of a discreet sample subset of size . Then is the vector ( ) The final matrix definition as explained in [9], states that be random variables of a random function with mean and let be sampling sites, . Then the matrix is [ ( ) ] The developments in this section now provide the reader with an algorithm to perform simple Kriging. In summary, the algorithm consists of five steps. The steps are as follows: 1. Calculate each term in matrix C 2. Calculate each term in vector c 3. Solve the system of equations C = c where 4. Compute the estimate(s). ? where 36 5. Calculate the estimation variance ? ? Under the consideration that the formulation is independent from the physical nature of the spatial attribute, the algorithm is completely general and applies to the characterization of any spatial attribute satisfying the assumptions in [9]. The reader should note that cannot be a singular matrix. If is singular, then there will be no unique solution that will exist for the problem. 3.2 Ordinary Kriging Ordinary Kriging is the most widely used form of Kriging and is the method of Kriging used in this dissertation. Simple Kriging requires information about the mean in order to solve the system of equations while minimizing the variance of the estimation error. Ordinary Kriging does not have the requirement of knowing information about the mean. This changes the problem from an unconstrained optimization problem into a constrained optimization problem. In order to solve the constrained optimization problem, a Lagrange method of multipliers is introduced [9]. In order to predict a response at the data values from neighboring samples points are combined linearly with weights resulting in the following ? ? 37 As described in the literature, the sum of the weights must sum to one and the assumption is that the data are part of a realization of an intrinsic random function with a variogram . Collecting the variances with the variogram is warranted for ordinary Kriging but not simple Kriging. This is due to the fact that simple Kriging does not include a constraint on the weights. Further proof that the weights sum to one and the variogram is authorized and shown in [9]. The estimation variance is ( ? ) and through linear combination can be calculated by ?? Ordinary Kriging is an exact interpolator in the sense that if is identical with a data location then the estimated value is identical with the data value at that point ? [9]. Minimizing the estimation variance with the constraint on the weights the ordinary Kriging system is obtained. The system is as described below ( )( ) ( ) where are weights to be assigned to the data values and is the Lagrange multiplier. The above equation can be rewritten as 38 Although the data contained in these sets of matrices is different than the data contained in the simple Kriging formulation, we maintain the same notation for simplicity. The purpose of Lagrange multiplier is to covert an unconstrained minimization problem into a constrained one [8]. This is accomplished by setting the partial first derivative of to zero. This produces equations and unknowns without adding any more unknowns. The following steps are taken 1. Calculate each term in matrix C through the fitted variogram function 2. Calculate each term in vector c through the fitted variogram function 3. Solve the system of equations where 4. Compute the estimate(s) ? where 5. Calculate the Ordinary Kriging estimation variance ? ?? In summary, the ordinary Kriging methodology is similar to that of simple Kriging. Without knowledge of the population mean, the problem is turned into a constrained optimization problem which varies from simple Kriging by the addition of a Lagrange multiplier. 39 3.3 Variogram and Covariance in Kriging The variogram provides the underlying calculations that allow the Kriging system of equations to be generated and solved. The variance increases as the distance increases until at a certain distance away from a point the variance will equal the variance around the average value, and will therefore no longer increase, causing a flat region to occur on the variogram called a sill. From the point of interest to the distance where the flat region begins is termed the range or span of the regionalized variable. Within this range, locations are related to each other, and all known samples contained in this region, also referred to as the neighborhood, and must be considered when estimating the unknown point of interest. This can be visualized on a two dimensional graph where the x-axis represents the lag (or bins) with a maximum value located at the range and the y-axis represents the variance with a maximum value located at the sill (the nugget + the partial sill). The variogram describes the variance of the difference between two observations and is the backbone in ordinary Kriging [28]. The variogram is obtained through fitted functions that most closely resemble the experimental variogram. The experimental variogram is the array of calculations based on sampled response data. After the fitted variogram has been plotted, then the fitted functions are compared through minimum mean square error to determine the closest fit. The function with the closest fit is used to calculate the matrix as previously defined. The traditional fitted functions ensure that variogram calculations result in a non-decreasing function. This holds the Kriging assumption that the covariance is non-decreasing as a function of spatial 40 separation. Without using the fitted function for variograms, the Kriging estimates will be unreliable. The empirical variogram is calculated by ? where is the number of distinct lag pairs and represents the difference square of sampled data at FLCs and . As stated, for Kriging, we need to replace the empirical variogram with an acceptable variogram model. One reason is that the Kriging algorithm needs variogram values for lag distances other than the ones used in the empirical variogram. Another reason is the variogram models used in the Kriging algorithm need to be positive definite, in order the system of Kriging equations to be non-singular. Therefore, it is generally accepted that the application of Kriging must choose from a list of acceptable variogram models. A list of four frequently used models and the ones that are used in the test planning portion of this research are shown in Table 2. 41 Table 2: Variogram Models Linear ( ) Spherical ( ( ) ( ) ) Exponential ( ( )) Gaussian ( ( )) Other variogram models, although not as common, exist and can be explored as future research as required. To illustrate the variogram methodology completely, it is conducted through an example. Here we are drilling and taking a hypothetical soil sample value at distances of one foot apart from 1 foot to 10 feet. The values at each lag (increment) are calculated by the difference of the sampled values squared. The table below shows a summary of I/O in the first two columns, the lag in the top row, and in the columns associated with each lag . 42 Table 3: Experimental Variogram Squared Calculations Observation (Z) Lag=1 2 3 4 5 6 7 8 9 1 5 2 6 1 3 4 4 1 4 7 9 1 4 5 7 0 9 1 4 6 4 9 9 0 4 1 7 2 4 25 25 4 16 9 8 1 1 9 36 36 9 25 16 9 3 4 1 1 16 16 1 9 4 10 1 4 0 1 9 36 36 9 25 16 For proof of correlation, all pairs are calculated as described above and plotted below in the variogram cloud. 43 Figure 6: Variogram Cloud The next calculation is to generate the experimental variogram values. This is done for each lag increment and is calculated by ? where has previously been calculated. The calculations are shown in the table below. Table 4: Experimental Variogram Experimental Var Lag (h)=1 2 3 4 5 6 7 8 9 2 3.4375 4.8571 6.0833 7.8 8.875 5.6666 7.25 8 This allows for the experimental variogram to be plotted with the lag on the abscissa and the calculated values on the ordinate. The graph is shown below. 0 5 10 15 20 25 30 35 40 0 2 4 6 8 10 12 Semivariogram Cloud Series1 Series2 Series3 Series4 Series5 Series6 Series7 Series8 Series9 44 Figure 7: Plotted Experimental Variogram The next objective is to fit a model which is then used to calculate the final values. This dissertation provides software that models variogram in three ways (1) to aid in the process of user selection of the appropriate model and user defined nugget, sill, and range values, (2) automatic selection of a model through sill and range parameter tweaking, and (3) through a standardized variogram model that is explained later in the research. The nugget value is associated with measurement error or variation at distances smaller than the smallest lag. For the purpose of this research the nugget value is assumed zero by default but can be altered by the user after specific variogram selection. The range, , is generally selected near the abscissa of the variogram where data no longer exhibit spatial autocorrelation, or at the maximum lag-value. The sill is the corresponding ordinate at ? ?. Autocorrelation is similarity of observations based on a separation unit. The software code base is in Appendix A and can be referenced for precise calculations. The resulting sill and range are 0 1 2 3 4 5 6 7 8 9 10 0 2 4 6 8 10 Experimental Variogram Experimental Variogram 45 , and respectively. Using the parameters , the following results are obtained for : Table 5: Fitted Variogram Models h=1 2 3 4 5 6 7 8 9 Linear 1.013 2.026 3.039 4.052 5.065 6.078 7.091 8.104 9.117 Sphere 1.511 2.970 4.327 5.530 6.527 7.267 7.699 7.771 7.432 EXPO 2.516 4.220 5.373 6.154 6.682 7.040 7.282 7.446 7.557 Gaus 0.385 1.431 2.855 4.330 5.599 6.536 7.141 7.487 7.662 Plotting these values and comparing them to the experimental variogram shows that the spherical model is a good fit for the sample data. Figure 8: Fitted Variogram vs. Experimental Variogram To prove the spherical model is the closet fit to the experimental variogram, the squared distances at each lag point are summed for each fitted model and shown in the table below. The spherical model has the lowest squared difference summation out of the four models. 0 1 2 3 4 5 6 7 8 9 10 0 2 4 6 8 10 Experimental Semi Sphere Model Expo Model Gaus Model Linear 46 Table 6: Summation of Squared Fitted Model Differences Squared Differences per lag Summation Linear 0.974 1.992 3.306 4.126 7.480 7.823 2.029 0.729 1.248 29.707 SPH 0.239 0.218 0.281 0.306 1.621 2.586 4.129 0.271 0.323 9.974 EXPO 0.267 0.612 0.266 0.005 1.249 3.367 2.610 0.039 0.196 8.611 Gaus 2.607 4.027 4.007 3.073 4.847 5.472 2.175 0.056 0.114 26.378 Now that the values have been determined, this information is used to generate the covariance matrix ( ) also referred to as simply with the included Lagrangian multiplier. In addition, is also used to generate . 3.3.1 Lag The lag or bin selection is a critical component in the formation of the variogram. The determination of the lag size can help smooth out outlier data and reduce the computational intensity of the variogram calculations. The lag is generally and initially set to the smallest spatial difference or increment. Depending on the amount of FLCs, this can lead to a large set of data points that need to be calculated in order to generate the variogram model. Certain approaches and strategies are discussed throughout the literature, but with current computational capabilities and the scope of this research, the lag is set to the smallest increment of all the FLCs and remains unchanged throughout the DFK process. 47 3.3.2 Neighborhood Another important decision has to be made about neighborhood selection. The neighborhood is the group(s) of data that exhibit similar characteristics. Distributing data into neighborhoods allows for multiple variogram models to be developed and for the use of multiple ?instances? of Kriging to be performed and analyzed independently gaining a more accurate global interpolation. Although, numerous strategies have been discussed in the literature, this research places the entire factor space into one ?global? neighborhood. This approach does not change the scope of the research and allows for future research as described in the final chapter. 3.4 The Bootstrap Bootstrap methods allow for generation of data using resampling techniques [71]. The bootstrap is a data resampling simulation technique that is used to make estimations. It originates from the phrase ?to pull oneself up by one?s bootstrap.? The bootstrap characteristics lend itself to I/O modeling in simulation experiments. The parametric bootstrap assuming normality is generally used in deterministic simulations, while distribution free bootstrapping is used in discrete-event random simulations and is used in Chapter 4. In the literature presented by [71], a parametric bootstrap allows a method to apply a goodness-of-fit test to be performed on real data to determine the fit of different statistical distributions based upon calculating probabilities resulting from the bootstrap approximations. Applying a goodness-of-fit test in this manner should occur only when a wide array of known data has already been gathered from the system. Bootstrapping has the potential to aide in determining unknown statistical distributions in conjunction with goodness-of-fit tests against 48 real system data. Other literature describes that the bootstrap method, a resampling technique, is also used in validating models with one data point when another data point is not feasible. This technique is described in [72] as the validity of the Bay of Biscay model and was tested against the historic documented encounter of the Allies against the German U-Boat threat just off the Bay of Biscay. The validation used two six month intervals of the battle and assumed a confidence bound. The parameters of interest were U-Boat sightings and U-Boat kills. Questions about the model validity arose after 20 replications were executed and resultant data were analyzed using two tailed t-tests. These questions led [72] to consideration of non- parametric bootstrapping for model validation since this technique results in an estimated statistical distribution against which the model can be compared. The resampling was accomplished using Monte Carlo runs and the two scenario results were statistically more significant using bootstrapping. Recently nonparametric bootstrapping has been combined with Kriging to generate more accurate and effective metamodels. Bootstrapping allows estimation of prediction variances for inputs that are yet to be simulated; therefore, it resamples and replaces outputs for each previous scenario already simulated under the assumption of independent and identically distributed data (IID). Promising results using bootstrapping have also been shown in these trace-driven simulations or simulations that have at least one of the same inputs as the real system [73]. This application is typically seen in traditional simulation, but has been shown to provide better estimates of variance when using Kriging as defined in [58]. 49 Bootstrapped observations are generally denoted by a superscript *. For notation, bootstrap observations are formally written as ( ) ( ) where is the number of bootstrapped observations selected randomly from the replicated sample observations. Next, the bootstrapped averages are computed from the bootstrapped samples. This is formally written as ? ? After the bootstrapped averages have been calculated, then the process is repeated for number of replications where is the bootstrap run size. 3.5 The Estimation Variance through Bootstrapping It was proposed in [58] that the ?correct? Kriging variance is estimated through the use of bootstrapping as the traditional Kriging variance was proven to underestimate the true variance. We return to the bootstrap sampled data ? ? The Kriging predictors are then calculated by ? ( ) ? ? 50 This is repeated for the bootstrap sample size . Now the average of the Kriging predictors can be calculated. This is written formally as ?? ? ? With the bootstrapped Kriging interpolations calculated along with the averaged bootstrapped Kriging predictions, we now can proceed to calculate the ?correct? Kriging variance. This is formally written as ( ? ) ? ? ?? The purpose for bootstrapping section is only to give the reader familiarity with the technique as an example of the current DFK process is given in the next chapter. The scope of this research is to augment limited empirical data with an interpolation response set. Due to limited empirical data, the bootstrapping technique is not an appropriate method to be used in this research. Thus we use the traditional Kriging variance as stated in the literature and are able to derive useful and acceptable information with this technique. 3.6 Mean Square Error Mean squared error (MSE) is a way to quantify the difference between an estimator and the true value of the quantity being estimated. Mean squared deviation or error is generally presented in the form of ? ? where represents a parameter of interest. 51 MSE is a commonly measure in statistics and is mentioned here since it plays an important part in the validation analyses. 52 CHAPTER 4 4 The Original Design for Kriging Process This chapter describes the current DFK process step by step and is illustrated by an example to solidify the work. The experiment must first be generally defined as a set of combinations with FLCs. In defining an experiment, it must be noted that the experimental region is bounded by where and [74]. can take the form of a unit cube of positive dimension . The goal of this original DFK process is to find the design where the Kriging interpolation accuracy is maximized while minimizing the amount of sampled input FLCs. The following example provides a detailed procedure of the original DFK process. It is meant only to illustrate the combination of mathematical methods involved in DFK steps that generates satisfactory results. The original DFK process as described by [4], structures the problem statement to address interpolations of expansive simulation models where empirical data are realizable. The following example structures the problem such that the empirical dataset requires interpolation and simulation data are readily obtainable. Soil samples are to be taken of a predefined area around the Toomer?s oaks to determine the % concentration of Spike 80DF, , contained in the soil. A minimal amount of samples must be taken because the testing cost for the Spike 80DF concentration is prohibitive. Based on the initial sample set, estimation through Kriging interpolation of the % concentration throughout the 53 remaining unsampled locations of the factor space will be conducted. The variance must be less than 0.065 before continued sampling is no longer required. The observations at each location are to be replicated times. In practice, a common method of determining replications is to use confidence intervals with the sample interval defined by [75] as: ? ( ) ? ? ( ) ? where ? is the response mean, is the t-distribution with degrees of freedom and confidence coefficient ), and is the pooled sample standard deviation. A graphical representation of the factor space is shown below: Figure 9: Factor space - Original DFK Process Example The boundaries of the problem are and . The factor space needs to be formally defined for consistency and accuracy. This is a two dimensional grid with the following factor space 54 ? The lag (or bin) of the factor space is to be {( )} which is the just minimum unit spacing of the Euclidean differences of therefore maximizing the number of points to be used in the variogram model. This is shown in step four of the current DFK process. The Euclidean distances for each lag must be determined. These distances are shown below in the table below. 55 Table 7: The Complete Lag Matrix - Original DFK Process 1 1 1 2 2 2 3 3 3 1 2 3 1 2 3 1 2 3 1 1 0.000 1.000 2.000 1.000 1.414 2.236 2.000 2.236 2.828 1 2 1.000 0.000 1.000 1.414 1.000 1.414 2.236 2.000 2.236 1 3 2.000 1.000 0.000 2.236 1.414 1.000 2.828 2.236 2.000 2 1 1.000 1.414 2.236 0.000 1.000 2.000 1.000 1.414 2.236 2 2 1.414 1.000 1.414 1.000 0.000 1.000 1.414 1.000 1.414 2 3 2.236 1.414 1.000 2.000 1.000 0.000 2.236 1.414 1.000 3 1 2.000 2.236 2.828 1.000 1.414 2.236 0.000 1.000 2.000 3 2 2.236 2.000 2.236 1.414 1.000 1.414 1.000 0.000 1.000 3 3 2.828 2.236 2.000 2.236 1.414 1.000 2.000 1.000 0.000 With the problem defined and the necessary preliminary calculations complete, the process as defined in the literature proceeds to the first step. 4.1 Step 1: The pilot design is selected For simplicity and as used in the literature, a design is selected which will maximize the minimum distance between two points of the design [53]. Utilizing this type of design leads to an initial pilot design of with . A graphical representation of the pilot design is shown below. Table 8: Pilot Design - Original DFK Process Pilot Design 3 x x 2 x 1 x x 1 2 3 56 Table 9: Pilot Design Matrix of Lags 1 1 2 3 3 1 3 2 1 3 1 1 0.000 2.000 1.414 2.000 2.828 1 3 2.000 0.000 1.414 2.828 2.000 2 2 1.414 1.414 0.000 1.414 1.414 3 1 2.000 2.828 1.414 0.000 2.000 3 3 2.828 2.000 1.414 2.000 0.000 Now, that the experimental pilot design been determined, the process continues to the second step. 4.2 Step 2: For gather pilot design response data Observe independent and identically distributed (IID) replicates for . The observation matrix for this example is generated and is shown below. [ ] Table 10 shows the response data values for each factor level combination and for each replicate. Table 10: Pilot Design Initial Response Data - Original DFK Process Observed Responses per replicate 1 2 3 4 5 6 1 1 0.09119 0.16054 0.09678 0.06663 0.1251 0.09119 1 3 0.25698 0.29018 0.29232 0.65918 0.28086 0.20269 2 2 0.26352 0.322 0.35138 0.8048 0.13238 0.19329 3 1 0.39189 0.4025 0.26776 0.55254 0.41169 0.21622 3 3 0.36294 0.42085 0.97557 1.10043 0.21919 0.18774 57 The sampled observation of this data completes the second step of the DFK process. 4.3 Step 3: Estimate the mean of pilot data observations The purpose for step three in the original DFK process is to estimate the sample mean of the responses at each factor level combination. The estimate of the mean is shown through ? ? . By following the above formulation the estimated means were calculated and are shown in the table below. Table 11: Estimated Average Response Data - Original DFK Process ? 1 1 0.106801343 1 3 0.330369242 2 2 0.344561425 3 1 0.373768358 3 3 0.544453992 The process proceeds to step four to determine the Kriging predictions. 4.4 Step 4: Compute Kriging predictions Based on ? for inputs the Kriging predictors are computed for the expected output set using inputs for the set . The candidate input set is selected using a space-filling method. As described, the candidate inputs are selected halfway between the neighboring FLCs. For this example, = 2 and ; . Kriging, which uses a weighted linear combination of all observed outputs, is used to observe the 58 unobserved responses. First the experimental variogram must be calculated. This is performed through the following formula. ? Performing the calculation of the above formula leads to the following Table: Table 12: Partial Experimental Variogram Calculations - Original DFK Process Lag ( ) ? 1.41421 0.05653 0.000201 0.000853 0.039957 0.097541 2 0.049983 0.071271 0.045832 0.029134 0.19622 2.82843 0.001883 0.19154 - - 0.193423 Finishing the experimental variogram calculation result in the following Table: Table 13: Experimental Variogram Calculations - Original DFK Process Gamma ? ?(1.41421) 0.01219267 ?(2) 0.024527482 ?(2.82843) 0.048355831 Now that the experimental variogram is estimated, we must now determine which of the four standard functions represents the experimental variogram the best. This is done by using the standard functions described in the literature such as the spherical, exponential, Gaussian, and linear functions. A nugget of was used. A sill of 0.047 and a range of 2.828 were 59 selected by analyzing the graph of the experimental variogram and determining that autocorrelation was still present at the maximum lag . The following Table outlines these calculations along with the sill and range identification. Table 14: Fitted Variogram Calculations - Original DFK Process Sill = 0.047 Range = 2.828 Gamma Linear Sphere EXPO Gauss ?(1.414) 0.023503549 0.032316493 0.036515258 0.024803802 ?(2) 0.033239038 0.041546287 0.041367762 0.036517633 ?(2.828) 0.047007099 0.046999998 0.044661068 0.044662127 A plot of the fitted variogram functions can be seen in the figure below. Figure 10: Fitted Variogram - Original DFK Process To determine the best fit the following least squares selection criteria was used. ? 0 0.01 0.02 0.03 0.04 0.05 0.06 0 0.5 1 1.5 2 2.5 3 Experimental Sphere Expo Gaus Linear 60 where is the fitted variogram value from Table 13 and is the square distance between the experimental and fitted variogram values. These calculations can be seen in the following Table. Table 15: Minimum Squared Distance Variogram Calculations - Original DFK Process Squared Differences Sphere Expo Gaussian Linear ?(1.414) 0.000404968 0.000591588 0.000159041 0.000127936 ?(2) 0.00028964 0.000283595 0.000143764 7.58912E-05 ?(2.828) 1.83828E-06 1.36513E-05 1.36434E-05 1.81908E-06 ? 0.000696446 0.000888835 0.000316448 0.000205646 ? 0.000205646 => Linear Now that it has been determined that the linear variogram should be used, the Kriging calculations can be conducted. This begins by constructing the fitted variogram matrix from using the linear function. This results in the following Table. Table 16: Matrix - Original DFK Process Matrix with LaGrangian Multiplier 0.0000 0.0332 0.0235 0.0332 0.0470 1 0.0332 0.0000 0.0235 0.0470 0.0332 1 0.0235 0.0235 0.0000 0.0235 0.0235 1 0.0332 0.0470 0.0235 0.0000 0.0332 1 0.0470 0.0332 0.0235 0.0332 0.0000 1 0.0000 0.0332 0.0235 0.0332 0.0470 1 1 1 1 1 1 0 61 In addition must be found for and . These calculations are completed by using the same approach as to calculate . Table 17: Matrix - Original DFK Process 2 3 for and 1 2 1 2.236068 0.0166 0.0372 2.236068 2.236068 0.0372 0.0372 1 1 0.0166 0.0166 1 1 0.0166 0.0166 2.236068 1 0.0372 0.0166 - - 1 1 Solving for , and results in the following Table. Table 18: Vector - Original DFK Process for for 0.396017 -0.041 -0.041 -0.041 0.289962 0.289962 0.396017 0.396017 -0.041 0.396017 -8.3E-05 -8.3E-05 Finally the Kriging estimators for and are calculated as ?( ) . The Kriging estimates are shown in the table below. 62 Table 19: Kriging Predictors - Original DFK Process ? for ? for 0.254357 0.445619 This concludes step four of the original DFK process. Now the process continues to step five out of the eight total steps. 4.5 Step 5: Perform non-parametric bootstrapping per input based on for The responses from the first three steps in the process will be resampled with replacement to estimate the Kriging variance when response data are known with all bootstrapped observations denoted with a superscript * [76]. The reader should note that this example does not contain enough data to perform bootstrapping in the correct statistical sense. The use of bootstrapping here is completed for illustration of the current DFK process only. Performing bootstrapping generates a response dataset of ( ) for where is the initial sample size and is the bootstrap sample size. The averages are computed again for number of resamples at each replicate. The original DFK literature uses , which is used in this example. The literature suggests that range from 25 to 200, thus this example uses 25 for illustration purposes [76], [3]. The partial bootstrapped replications are shown in the table below. Please refer to Appendix B: Remaining Bootstrapped Data for Original DFK Process for the omitted data. 63 Table 20: Partial Bootstrap Data - Original DFK Process Bootstrap Replicate 1 ? 0.067 0.067 0.125 0.067 0.067 0.091 0.091 0.067 0.091 0.091 0.082 0.659 0.257 0.281 0.290 0.281 0.257 0.659 0.659 0.257 0.281 0.388 0.322 0.322 0.132 0.193 0.805 0.805 0.193 0.322 0.322 0.805 0.422 0.268 0.216 0.268 0.268 0.553 0.392 0.268 0.403 0.553 0.412 0.360 0.421 1.100 0.363 0.976 1.100 0.421 0.219 0.188 0.219 0.363 0.537 . . . Bootstrap Replicate 25 ? 0.091 0.091 0.091 0.067 0.091 0.097 0.125 0.161 0.161 0.091 0.107 0.257 0.290 0.203 0.290 0.203 0.203 0.290 0.290 0.257 0.290 0.257 0.132 0.322 0.322 0.805 0.132 0.322 0.132 0.132 0.805 0.351 0.346 0.392 0.412 0.268 0.392 0.216 0.392 0.392 0.392 0.553 0.216 0.362 0.188 1.100 0.363 1.100 0.976 0.976 0.421 1.100 0.976 0.421 0.762 For ? with the bootstrapped Kriging predictor is calculated for each of . The table below shows the list of Kriging predictors. ? ? ? 64 Table 21: Bootstrapped Kriging Predictors - Original DFK Process ? ? 1 0.25957 0.45829 2 0.26962 0.43914 3 0.23993 0.39593 4 0.27430 0.46518 5 0.27681 0.51430 6 0.25528 0.45402 7 0.24318 0.45435 8 0.21866 0.46675 9 0.25237 0.43650 10 0.28397 0.45508 11 0.22468 0.42190 12 0.26024 0.45855 13 0.21054 0.47351 14 0.23881 0.37866 15 0.23796 0.51686 16 0.27062 0.47716 17 0.23588 0.38166 18 0.27018 0.44092 19 0.27309 0.44625 20 0.24346 0.50728 21 0.25143 0.44396 22 0.24521 0.40514 23 0.23509 0.36838 24 0.23464 0.45664 25 0.24414 0.53060 This concludes step five of the process. The process proceeds to step six for variance calculations. 4.6 Step 6: Calculate the Kriging variance for candidate inputs For each with sample size the Kriging variance is to be calculated by applying the following formula. 65 ? ? ? ? ? ? The calculations which are shown in the table below conclude step six of the process. Table 22: Bootstrapped Kriging Variance Estimates - Original DFK Process ? ? ? ? 0.060332175 0.195948707 It is determined from the table above that all of the variances do not meet the stopping criterion of less than 0.065. Therefore we proceed to step seven of the process. 4.7 Step 7: Determine the input with the largest variance The FLC with the largest estimated variance as a result of bootstrapping is added to the set . To formally choose the maximum input, it is defined with the following argument. ? The result from this step of the process is to add to original set of selected 5 FLCs, and observe the responses per replicates as before to obtain the mean response ? . 4.8 Step 8: Repeat steps 4-7 until the stopping criteria has been reached The problem statement requires the variance be less than 0.065 so that steps four through seven are repeated. The updated design is shown in the table below. 66 Table 23: Experimental Design Iteration 1 - Original DFK Process Experimental Design Iteration 1 3 x x 2 x x 1 x x 1 2 3 The sampled response data now contain the replication data along with ? . The set for is determined as before. Table 24: Pilot Design Initial Response Data - Original DFK Process Observed Responses per replicate 1 2 3 4 5 6 1 1 0.09119 0.16054 0.09678 0.06663 0.1251 0.09119 1 3 0.25698 0.29018 0.29232 0.65918 0.28086 0.20269 2 2 0.26352 0.322 0.35138 0.8048 0.13238 0.19329 3 1 0.39189 0.4025 0.26776 0.55254 0.41169 0.21622 3 2 0.56767 0.40032 0.68785 0.24624 0.40058 0.21369 3 3 0.36294 0.42085 0.97557 1.10043 0.21919 0.18774 Table 25: Estimated Average Response Data Iteration 1 - Original DFK Process ( , ) ? 1 1 0.106801343 1 3 0.330369242 2 2 0.344561425 3 1 0.373768358 3 2 0.419392835 3 3 0.544453992 67 Table 26: Fitted Variogram Calculations Iteration 1 - Original DFK Process Squared Differences Sphere Expo Gaussian Linear ?(1) 0.000400132 0.000720546 0.000116945 0.000162119 ?(1.414) 0.000404968 0.000591588 0.000159041 0.000127936 ?(2) 0.00028964 0.000283595 0.000143764 7.58912E-05 ?(2.236) 0.000313897 0.000262628 0.000179205 0.000115621 ?(2.828) 1.83828E-06 1.36513E-05 1.36434E-05 1.81908E-06 ? 0.00109474 0.00159573 0.00041975 0.000365946 ? 0.000365946 => Linear Table 27: Kriging Predictors Iteration 1 - Original DFK Process ? for ? for 0.253757162 0.236128133 Table 28: Updated Variance Estimates Iteration 1 - Original DFK Process ? ? ? ? 0.06427691 0.071514567 Our process has reached the precision as stated in the problem statement of a Kriging variance less than 0.065. The process is now concluded. 4.9 Summary of the Current DFK Process The current DFK process consists of eight steps all with the intent to perform incremental interpolation based on the identification of the factor space and stopping criteria. As outlined very methodically, one can see that the process although novel, has many potential 68 improvements and applications. This research proposes improvements in the current DFK process and extends this powerful nonlinear interpolation method into areas of applications such as experiment planning and simulation validation. This is accomplished by incorporating cost considerations along with an augmented space filling design to provide unique interpolation solutions based on dimensionality through a standardized covariance algorithm. The process as a whole is optimized by utilizing budget constraints to reduce the number of iterations as this becomes critical as the number of FLCs increase. Unless it is cost prohibitive, a one-by-one sequential sampling becomes unrealistic in high dimensional environments as the number of factor level combinations increase dramatically. 69 CHAPTER 5 5 Advancements in the Design for Kriging Process This chapter focuses on specific improvements in the DFK process that were outlined in Chapter 4. It discusses detailed mathematical developments, assumptions, and examples that we present in order to advance the DFK process and to utilize the process in test planning and validation type environments. 5.1 Estimation Variance with No Empirical Data One goal in this research is to derive a method to estimate interpolation variance prior to obtaining response data. The purpose is to utilize this information in order to determine through some general heuristics an initial pilot design including sample size and FLC selection. The interpolation variance allows for an optimized determination of initial sample sizes for use in the experimentation process. Literature describes taking some number of pilot samples, computing a variogram and then estimating a fitted function. The sample size selection described in the literature varies greatly. The methods described in the literature are not presented in an effective manner to reduce total iterations in a DOX environment. To calculate estimation variance with no empirical data, three assumptions are made. The first assumption is that inputs or dimensions are equally spaced integer values or can be represented as such through a data transformation. This allows for automatic and computationally efficient generation of factor spaces without a user requirement to input all 70 FLCs individually. The second assumption is that the factor space for each input or dimension is bounded by the same values including minimum, maximum, and incremental or separation distance, i.e., a block design. These designs are common in DOX literature. Further research should be conducted to relax each of these assumptions. The third assumption is that data are normally distributed. If required a test for normality can be conducted or the central limit theorem may be assumed if the sample size is sufficiently large. We must first develop a mathematical relationship between the commonly used variogram fitted functions such that the estimation variance calculations are completely independent of the sample data. This is reasonable since that the Kriging weights are based on spatial separation of the input data and these Kriging weights determine the Kriging model. Our methodology utilizes the spherical, exponential, Gaussian, and linear variogram functions as these are the most commonly used functions as described in the literature to represent experimental variograms. To develop an algorithm to estimate initial variances without response data, the fitted functions must be related. We reiterate the functions of interest in the table below. 71 Table 29: Variogram Fitted Functions of Interest Linear ( ) Spherical ( ( ) ( ) ) Exponential ( ( )) Gaussian ( ( )) One can see that and are simply linear and scalar factors of the above four equations assuming that . The assumption holds with a good neighborhood selection as autocorrelation will be present at the maximum lag . Replacing with and rewriting each equation leads to the table below. 72 Table 30: Reduced Variogram Fitted Functions of Interest Linear Spherical Exponential ( ) Gaussian ( ) Since the linear fitted variogram model is equal to , it is clear to see now that each function is directly related to the linear variogram. This simplification carries over into the Kriging weight calculation of . The result is four systems of equations that are directly related. The value of used below varies depending on the iteration of DFK process. For example during the pilot design phase . during the pilot design phase of the example presented in Chapter 4. After selecting , we added the candidate input with the highest estimation variance to the design, thus . The notation is generically presented here as the number of FLCs in the design and varies throughout the DFK iteration process. ( )( ) ( ) 73 ( )( ) ( ) ( ( ) ( ) ( ) ( ) )( ) ( ( ) ( ) ) ( ( ) ( ) ( ) ( ) )( ) ( ( ) ( ) ) From , the relationship carries furthermore into variance calculations. Each variance carries a direct relationship that depends on the ratio of lag to range . ? ? ? ? 74 ? ? ? ? Demonstrating the relationship alone, doesn?t allow for selection of a fitted variogram function without knowledge of the empirical variogram. To address this issue, we look at descriptive statistical information in order to present the user of the advanced DFK software with worst case initial assessments of the estimation variance in order for determining initial sample size and FLC selection. To demonstrate the method above, we look at the following datasets, graphs, and associated descriptive statistical information. The following sets of lags are given for a one dimensional or single input sample space. Table 31: Initial Set values 1 2 3 4 5 6 7 8 9 10 Four tests were carried out to demonstrate the relationship between the variogram models: 1. Alter the range ( ), while holding , and constant 2. Alter the sill ( ), while holding , and constant 3. Alter the nugget ( ), while holding , and constant 4. Alter and analyze absolute and percentage difference between each variogram The formulations of each variogram using Table 30 were computed for given lags and variograms are plotted for tests one through three based on . The results are 75 shown in the variogram tables and plots that follow below for test one. Test two and three data can be found in the appendix. Table 32: Linear Variogram 2 3 4 5 6 7 8 9 1 1 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1 2 0.5000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1 3 0.3333 0.6667 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1 4 0.2500 0.5000 0.7500 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1 5 0.2000 0.4000 0.6000 0.8000 1.0000 1.0000 1.0000 1.0000 1.0000 1 6 0.1667 0.3333 0.5000 0.6667 0.8333 1.0000 1.0000 1.0000 1.0000 1 7 0.1429 0.2857 0.4286 0.5714 0.7143 0.8571 1.0000 1.0000 1.0000 1 8 0.1250 0.2500 0.3750 0.5000 0.6250 0.7500 0.8750 1.0000 1.0000 1 9 0.1111 0.2222 0.3333 0.4444 0.5556 0.6667 0.7778 0.8889 1.0000 Table 33: Spherical Variogram 2 3 4 5 6 7 8 9 1 1 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1 2 0.6875 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1 3 0.4815 0.8519 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1 4 0.3672 0.6875 0.9141 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1 5 0.2960 0.5680 0.7920 0.9440 1.0000 1.0000 1.0000 1.0000 1.0000 1 6 0.2477 0.4815 0.6875 0.8519 0.9606 1.0000 1.0000 1.0000 1.0000 1 7 0.2128 0.4169 0.6035 0.7638 0.8892 0.9708 1.0000 1.0000 1.0000 1 8 0.1865 0.3672 0.5361 0.6875 0.8154 0.9141 0.9775 1.0000 1.0000 1 9 0.1660 0.3278 0.4815 0.6228 0.7476 0.8519 0.9314 0.9822 1.0000 76 Table 34: Exponential Variogram 2 3 4 5 6 7 8 9 1 1 0.9502 0.9975 0.9999 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1 2 0.7769 0.9502 0.9889 0.9975 0.9994 0.9999 1.0000 1.0000 1.0000 1 3 0.6321 0.8647 0.9502 0.9817 0.9933 0.9975 0.9991 0.9997 0.9999 1 4 0.5276 0.7769 0.8946 0.9502 0.9765 0.9889 0.9948 0.9975 0.9988 1 5 0.4512 0.6988 0.8347 0.9093 0.9502 0.9727 0.9850 0.9918 0.9955 1 6 0.3935 0.6321 0.7769 0.8647 0.9179 0.9502 0.9698 0.9817 0.9889 1 7 0.3486 0.5756 0.7235 0.8199 0.8827 0.9236 0.9502 0.9676 0.9789 1 8 0.3127 0.5276 0.6753 0.7769 0.8466 0.8946 0.9276 0.9502 0.9658 1 9 0.2835 0.4866 0.6321 0.7364 0.8111 0.8647 0.9030 0.9305 0.9502 Table 35: Gaussian Variogram 2 3 4 5 6 7 8 9 1 1 0.9502 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1 2 0.5276 0.9502 0.9988 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1 3 0.2835 0.7364 0.9502 0.9952 0.9998 1.0000 1.0000 1.0000 1.0000 1 4 0.1710 0.5276 0.8150 0.9502 0.9908 0.9988 0.9999 1.0000 1.0000 1 5 0.1131 0.3812 0.6604 0.8534 0.9502 0.9867 0.9972 0.9995 0.9999 1 6 0.0800 0.2835 0.5276 0.7364 0.8755 0.9502 0.9831 0.9952 0.9988 1 7 0.0594 0.2172 0.4236 0.6245 0.7836 0.8896 0.9502 0.9801 0.9930 1 8 0.0458 0.1710 0.3442 0.5276 0.6902 0.8150 0.8994 0.9502 0.9776 1 9 0.0364 0.1377 0.2835 0.4471 0.6038 0.7364 0.8371 0.9066 0.9502 The relationship from the four tables above becomes clear in the four graphs below. 77 Figure 11: Linear Variogram with Varying Range Figure 12: Spherical Variogram with Varying Range 0.0000 0.2000 0.4000 0.6000 0.8000 1.0000 1.2000 1 2 3 4 5 6 7 8 9 Linear Series2 Series3 Series4 Series5 Series6 Series7 Series8 Series9 Series10 0.0000 0.2000 0.4000 0.6000 0.8000 1.0000 1.2000 1 2 3 4 5 6 7 8 9 Sphere Series12 Series13 Series14 Series15 Series16 Series17 Series18 Series19 Series20 78 Figure 13: Exponential Variogram with Varying Range Figure 14: Gaussian Variogram with Varying Range Each of the above four fitted variogram functions simply display a shift in the curves as the range increases up to the maximum lag value. The appendix shows information in the same 0.0000 0.2000 0.4000 0.6000 0.8000 1.0000 1.2000 1 2 3 4 5 6 7 8 9 Expo Series2 Series3 Series4 Series5 Series6 Series7 Series8 Series9 Series10 0.0000 0.2000 0.4000 0.6000 0.8000 1.0000 1.2000 1 2 3 4 5 6 7 8 9 Gaussian Series2 Series3 Series4 Series5 Series6 Series7 Series8 Series9 Series10 79 format for varying the sill and nugget as described in test two and three. More interestingly, is how the estimation variance is related between the four fitted variogram functions. The fourth test evaluated how the absolute difference and percent difference changed between each variogram model as the nugget and the sill vary. It was found that the absolute difference in the variance estimates did not change as the nugget value varied as expected. In addition, it was found that the percentage difference in the variance estimates did not change as the sill varied. This reiterates the relationship formulations presented earlier. The pertinent information is shown in the tables below. The tables were below were constructed by the following steps. Step 1) Create a pilot design of FLCs in the pertinent factor space Step 2) Calculate the lag matrix and the RHS lag vector for the pilot design of each candidate FLC Step 3) Use the results from step 2 to obtain and Step 4) Obtain to calculate the prediction variance at each candidate FLC Step 5) Repeat steps 1-4 adding the FLC with the highest prediction variance until only one unsampled FLC remains Step 6) While conducting steps 1-5, alter from during each iteration The columns in Table 36 represent the differences in variances for the fitted variogram functions listed in the header-row, while the rows represent iterations of the steps discussed above. The columns in Table 37 represent the percent differences in variances for the fitted variogram functions listed in the header, while the rows represent iterations of the steps discussed above. 80 Table 36: Varying Nugget ? Absolute Variance Estimation Difference Spherical vs. Linear Absolute Difference 0.306429 0.639306 0.935541 1.106539 1.106539 0.935541 0.639306 0.306429 0.451834 0.910264 1.258831 1.425296 1.385754 1.165628 0.837552 0.430943 0.830027 1.092439 1.165212 1.049539 0.799686 0.40978 0.750073 0.930732 0.922292 0.756282 0.388195 0.670094 0.774729 0.703147 0.366372 0.591838 0.632645 0.346939 0.52865 0.350336 Spherical vs. Exponential Absolute Difference 2.313099 3.039183 3.124317 3.060789 3.060789 3.124317 3.039183 2.313099 2.307736 3.056868 3.204878 3.209759 3.206988 3.059568 2.309114 2.305049 3.07687 3.26705 3.268351 3.079581 2.306676 2.300171 3.078186 3.249915 3.07988 2.301494 2.283753 3.016416 3.017045 2.284591 2.234084 2.784468 2.234477 2.093243 2.093349 1.672668 Spherical vs. Gamma Absolute Difference 1.533124 1.595178 0.916754 0.31965 0.31965 0.916754 1.595178 1.533124 2.064983 3.025081 2.768716 2.044696 1.698081 1.790431 1.543276 2.110336 3.399144 3.624909 3.0996 2.435846 1.658772 2.056569 3.289461 3.550938 2.975659 1.844279 1.965832 2.966895 2.920537 1.902855 1.832771 2.454796 1.82443 1.619611 1.61906 1.207473 81 Table 37: Varying Sill - Percent Variance Estimation Difference Spherical vs. Linear % Difference 13.6559 15.6815 17.4118 18.3079 18.3079 17.4118 15.6815 20.5962 23.4725 25.5018 26.9113 28.0850 30.0784 38.2027 20.1880 22.7727 24.6836 26.3356 28.8166 37.4908 19.8734 22.3318 24.4910 27.4731 36.7015 19.7340 22.4421 25.9518 35.7598 19.9871 24.0731 34.5209 21.4204 Spherical vs. Exponential % Difference 103.082558 74.54815 58.14802 50.64145 50.64145 58.14802 74.54815 105.195019 78.82576 64.92535 60.60421 64.99587 78.95038 105.324 107.98273 84.41712 73.81874 73.86986 84.55437 108.1413 111.552974 91.64626 85.51739 91.74298 111.6889 116.094957 101.0226 101.0649 116.187 121.878713 113.2586 121.9263 129.23916 129.2542 138.525575 Spherical vs. Gamma % Difference 68.3232161 39.12814 17.06211 5.288677 5.288677 17.06211 39.12814 94.1294465 78.0061 56.08945 38.60638 34.41493 46.20104 70.3924 98.8611404 93.25903 81.90454 70.05582 66.8797 77.76635 99.7388627 97.93651 93.43844 88.63846 89.50075 99.9333554 99.36407 97.83207 96.77315 99.9853926 99.84915 99.55171 99.9966018 99.96913 99.9995129 As a result of these tests, the differences in the relationship of the variogram functions are reduced to the ratio . As stated earlier, our defined factor spaces are limited to a single 82 neighborhood. Since a single neighborhood is used, we are under the initial assumption that the lag distances still have some influence on the point being estimated. Under this assumption, we set the initial range, , to be the maximum lag distance. This allows for the ratio to range from . An iterative computational model was developed for the four fitted variogram functions to determine which model produces the highest estimation variance, therefore which model should be used for sample size selection. The estimation points were taken at all locations that were not part of the evolving input matrix. The maximum or worst variance was taken following the iteration of the model. The results are shown in the table below. Table 38: Variogram Testing for Initial Selection Summary L 0.444 0.889 1.333 1.778 2.222 2.667 3.111 3.556 4.000 S 0.756 1.511 2.267 3.022 3.778 4.533 5.289 6.044 6.800 E 1.006 2.012 3.018 4.024 5.030 6.036 7.042 8.048 9.054 G 0.441 0.882 1.323 1.764 2.206 2.647 3.088 3.529 3.970 As shown from Table 38, we select the exponential as the initial fitted variogram as it provides the highest estimation variances out of the four common fitted variogram functions that were chosen. In order to further validate the exponential selection, a simple 3 FLC MC model was developed in Microsoft Excel?. The nugget was set to zero in the MC model since we have no direct evidence of micro-variability or measurement based error prior to data sampling. The MC model was executed with 100 replications by varying from 83 where and . The exponential model resulted in the highest estimation variance in all replications. This allows us to choose the exponential fitted variogram model for test planning purposes. It is important to discuss the parameter values for . Recall the assumption that the data are normal. Under this assumption, the response data, although unknown but normal, can be actually standardized according to However, in OK ? and ? have to be estimated as their values are unknown. It can be shown that the random variable has a Student-t distribution with degrees of freedom, and hence we make the approximation ? Since the standard normal variance is one, we therefore set the sill, which represents data variability, also equal to one. Recall that the nugget is assumed to be zero and that the range equal to due to the single neighborhood selection. Utilizing these parameter settings allows for generic block pilot designs of dimensions to be produced with no knowledge of the response data as described in Section 5.3. For reiteration, the reader should recall that the Kriging model is a linear weighting method where the weights are based solely on separation distance or lag . This along with the single neighborhood assumption allows for these parameter setting and fitted variogram selection through 84 ? ? Examples of constructing these designs and their feasibility are discussed in the next chapter. 5.2 Initial Size Based on Estimation Variance, Expected Improvement, and Budget Initial sample size recommendations are displayed in the list box on the test planning module of the software application. The expected improvement is considered as the percent reduction in variance as increases. The calculation is based on an incremental variance ratio summarized by ? ? ? Specifically, as the number of FLCs are incremented an improvement (estimated variance reduction) is realized and the amount of that improvement is based on the FLC that was chosen to be incorporated into the sequential design during the iteration. ? is divided by total amount of variance contained within ? ? . The amount given in the list box is the percentage of total variance that was reduced by sampling at of the current . As one would expect, the total amount reduced by the addition of the all FLCs of the factor space is 100%. The incremental cost is calculated in one of two methods. The first method is when no linear scale is incorporated into the pilot design. This method states that cost per sample remains the same. Under this method the budget remaining column in the left list box of the test planning portion of the software and is calculated by: 85 ? If a linear cost scale is added then the equation becomes: ? The addition of the linear scale can be positive or negative. This allows for users of DFK software to adjust the cost per test as the cost to obtain each sample increases or decreases. Unless specified, the software continues to calculate the pilot design after the budget is exhausted in order to give what-if analysis capability. This can be changed by users of DFK software by selecting to stop the program when the budget reaches zero. For further clarification on how these calculations are applied, the reader should refer to Section 7.3. The software application continues to calculate estimation variances as is incremented until the upper bound of the factor space is reached. The information gathered from ? and the test budget are analyzed to produce a set of heuristics used in aiding the user in initial sample size selection. The results are discussed in Chapter 6 and a table is presented to summarize recommendations on sample size selection. The results from the block designs as discussed in this chapter and analyzed in Chapter 6 are located in the appendix. Further analyses can be conducted in future studies to redefine the heuristics developed herein. The reader should note that a single value that can be applied in all situations and environments does not exist. We present a generalized approach to determine . 86 5.3 DFKS - Selection of the input FLCs through augmented spatial sampling In this section, we describe an augmented spatial sampling scheme that lends itself to Kriging methodology. Terminology that we use to describe this technique is Design for Kriging Sampling (DFKS). Based on , is selected randomly using a spatial sampling scheme based on the set of lags as defined in the factor space. The idea behind our algorithm is taken from the well-known simulating annealing algorithm. The simulated annealing algorithm pseudo code as taken directly from [77] is found below. { Step 1: Parameter Initialization; 1.a) Set the annealing parameters; ; 1.b) Initialize the iteration counter; ; /* el: outer loop counter */ 1.c) GENERATE the initial solution (generate the solution randomly or start with a known solution). Calculate the objective function, Get; ; Step 2: Annealing schedule; ?Execute steps 2.a-2.g until conditions in 2.g are met? 2.a) Inner loop initialization; ; /* ; inner loop counter */ 87 2.b) Initialize solution for the inner loop; ; ; 2.c) Achiving equilibrium at every temperature. Execute inner loop stes 2.c.1-2.c.5 until conditions in 2.c.5 are met; 2.c.1) ; 2.c.2) Generate a neighboring solution and calculate the new objective function 2.c.3) ; 2.c.4) IF ( ) OR Random(0,1) ; THEN accept ; ELSE reject ; ; ; 2.c.5) IF ( ) THEN terminate the inner loop GOTO step 2.d ELSE continue the inner loop GOTO step 2.c.1 2.d) ; 2.e) ; ; 2.f) ; 88 2.g) IF ( ) THEN terminate the outer loop GOTO Step 3 ELSE continue outer loop GOTO Step 2.a Step 3: Terminate the best solution obtained and stop } Prior to presenting the pseudo code for our selection process, it is important to show the developmental steps of the sampling process and to formally define the problem. The mathematical model is defined by: Objective: ? Subject to: ? Where: 89 To devise a solution to the problem stated, we begin by creating a two dimensional factor space array. The first dimension uniquely identifies the element from is the total FLCs contained in the factor space. The second dimension contains the FLC elements themselves. A pictorial of this array is shown below. Figure 15: Factor space Array Next, a lag array is assigned two unique again as a two dimensional array. The first dimension is the unique identifier and the second dimension is the and lag elements of the Euclidean distance matrix . Similarly, the covariance matrix is defined in the same manner. This is made clear in the diagram below. The reader should note that the Distance/COV in the figure 90 below does not mean divided by, instead it is simply showing the two arrays have the same composition in one figure. Figure 16: Spatial Sampling Design Matrix After the array assignments have been completed and the initial lags have been calculated and populated, the DFKS process begins. Prior to beginning the augmented simulated annealing process for the estimation process we must state our software initialization requirements. The factor space must be known and is defined upfront by the user. The factor space consists of three user inputs: (1) , (2) , and (3) . With the user definition of the factor space, we proceed to the initial sampling rule: 91 ? The sample locations at are all permutations of , i.e., all FLCs located at the factor space boundaries. Kriging is an interpolation process, not useful for extrapolation and therefore the boundaries need to be sampled as a minimum, therefore we set as the minimum FLC selection for pilot design ? After FLC selection, the center point, , rounded up of the factor space is selected. and are the first and last index values for the factor space. This allows for the highest possible estimation variance as the initial starting point of the search algorithm Since the initial FLCs and the initial estimated variance have been defined as in traditional DOX literature, the DFKS process begins. This search process is described through the use of pseudo code. Recall our array definitions as they will be referenced throughout the pseudo code explanation. { Step 1: Parameter Initialization; 1.a) Set variogram parameters; 1.b) Gather user defined data; Minimum lag increment = 1; 92 1.c) Initial Dimensioning of Arrays; Problem_Space_Array; Distance_Array; Covariance_Array 1.d) GENERATE initial FLC selection; Select through all permutations of ; 1.e) GENERATE initial candidate input; Select center point of factor space; 1.f) Initial Dimensioning of Arrays; Candidate_Distance_Array; Candidate_COV_Array; 1.g) GENERATE initial solution. Calculate initial estimation variance at the design midpoint, Get; ; 1.h) SET direction_flag = 1 (next search will be an increasing FLC search); Step 2: Search Schedule; 2.a) Outer Loop Initialization ; ?Execute until FLC LISTCOUNT = or Budget if selected? 2.b) Perform Array Slicing (see next section); 2.c) Inner Loop Initialization; 93 ?Execute WHILE FLC LISTCOUNT <> ? Step 3: Solution Search; 3.a) Select next input candidate based on direction_flag; 3.a.1) IF direction_flag = 1 THEN search increasing FLCs 3.a.2) IF direction_flag = 0 THEN search decreasing FLCs 3.a.3) where is the index of the most recently added FLC and the is based on the direction_flag (Generate a neighboring solution); 3.b) Check against the current (prevent duplicate FLC selection); 3.c) Calculate the new object function 3.c.1) If THEN accept tenative as ; 3.c.2) ELSEIF and inner loop <> OR THEN CONTINUE; 3.c.3) ELSEIF inner loop = OR THEN EXIT inner loop; Step 4: Continue inner loop 4.a) FOR EACH THEN accept tenative as ; 94 4.b) IF inner loop = and factor space search exhausted THEN EXIT inner loop; Step 5: Accept ; 5.a) Add candidate as the next FLC; 5.b) Add and to the GUI; 5.c) IF outer loop = THEN EXIT outer loop; Step 6: Terminate the best solution obtained; 6.a) Add to the GUI; 6.b) and STOP. } 5.4 Dynamic Array Slicing of the Covariance Matrix With sequential DFK, the fitted variogram functions must be completely recalculated during each iteration to incorporate the introduction of new FLCs into and . This can occur with random sampling, minimax sampling, Latin Hypercube Sampling (LHS) sampling, and DFKS process proposed here. To avoid recalculating the distance matrix and the covariance matrix altogether during each iteration we introduce array slicing into the software package. Visual Basic for Applications (VBA) does not include multidimensional array slicing commands. We therefore created our own method of performing this operation. The logic behind the array slicing technique is based solely on sequential sampling. For clarity, we again present the logic in form of pseudo code. 95 { Step 1: Gather Initial State; Distance_Array; Covariance_Array; Step 2: Initialize Outer Loop ( ); FOR NEXT through array rows; Step 3: Initialize Inner Loop ( ); FOR NEXT through array columns up to ; (the reason for looping up to is to gain computational efficiency by only looking at half of the matrix due to its symmetrical nature) Step 4: Determine if slice is required; IF then 0 since diagonals are 0 with ; ELSEIF THEN data remains in ; (where is the current design and is the candidate input that was chosen to be added through DFKS) ELSEIF THEN perform slice; Step 5: Determine Slice Method; IF THEN move all subsequent data down a row ( ); ELSEIF THEN move all subsequent data down a row ( ) and across a column ( ). } 96 After the slicing operation has occurred, the only blank elements left in the arrays are elements that have yet been calculated. A subsequent dual row and column loop in the software calculates the new Euclidean distances and values based on the FLC chosen from the DFKS algorithm. The result of this process is that only the new elements of the array are calculated through the use of the slicing logic that is inherently incorporated into the software. This technique allows for CPU time optimization. 5.5 The Covariance Function by Use of a Standard Method Fitted variogram functions in the literature range from user interpretation of graphs under subjective judgment to calculations, through parameter tweaking, of the empirical variogram versus the fitted variogram. We aim to develop a standard model for determining variograms. This standard model aims to optimize the variogram function, not necessarily the parameters . In fact, the method we present is based on the assumption that . We aim to present a standardized model for generating the fitted variogram based on the empirical semivariogram values during sequential Kriging, not the test planning portion of the software as that part of the software is relevant prior to data sampling. For clarity, the test planning portion of the software uses the exponential fitted function due to lack of empirical data as previously described. Our standardized model is the default choice for variogram modeling in DFK software. In order to develop a standardized model some number of empirical variogram points and the lags associated with generating the variogram are determined. Prior to 97 performing our iterative regression approach, we first determine the shape of the empirical variogram curve. If the software determines the empirical variogram curve does not exhibit spatial correlation, then the iterative regression model is limited to simple linear regression which is the optimal solution for performing Kriging under these assumption violations. After, determination of the empirical variogram curve a simple linear regression model is initialized: where represents the empirical variogram calculated by ? and represents the individual lags. The regression model that results may not meet the non- negative requirement for the variogram. In order to eliminate any decreasing ? values, we smooth the regression model. This is accomplished by defining and solving the following problem. Objective: ? ? Subject to: FOR = 1 to STEP 1 ? ? NEXT 98 By changing: This problem is solved through the use of DFK software by first calling the regression analysis pack in Excel? and secondly by calling the solver add-in in Excel?. After the software sets the parameters for the solver, the software initially sets the solver algorithm to GRG which solves non-linear but smooth problems. This method generates solutions with very high computational efficiency. If the empirical variogram points are non-linear non-smooth, then software switches over to EA to provide a solution to the model. In this event, the computational effort may be high and generating an optimized solution possibly requires CPU times on the order of minutes. After the solver operations are finished, an iteration is considered complete. Iterations continue until either one of the two requirements are met: (1) the reduction in the sum of squares of the residuals of the current iteration is the same or greater than the previous iteration or (2) a and at least a 3rd order model has been obtained. The stopping criterion of the 2nd requirement was observed through multiple testing. The program terminates without finding a solution utilizing the standard model if the software has completed an 8th order model without satisfactory results or as results become unreliable. Based on this approach, the sequential Kriging software generates a standard and optimized fitted variogram solution. These calculations are all performed in the background of the software. There is one limitation to this approach. GRG is a local optimizer thus may not always produce acceptable results. The software recognizes when this event happens and 99 automatically uses the EA algorithm. Results from this process are shown in the following chapter. 5.6 Reduction in Sampling Iterations and Stopping Criterion Based on Budget Sequential sampling in the sense of one sample at a time may not be a realistic achievement in the physical testing environment. Physical test environments may experience many limitations such as personnel availability, range availability, schedule, repeatability, and environmental change. The iterations in this research are based on budget availability. Users can use the test planning portion of the software to get initial estimates of sample size based on user inputs of budget constraints. This method of planning gives users advance knowledge prior to gathering sample responses. After sample responses are gathered, Kriging is performed with knowledge of the sample response data. After analyzing the estimation variance, additional sample data is gathered strictly based on remaining/additional budget. In the event that additional budget is required, utilization of the test planning to determine an updated and sequential Kriging for interpolation can give the user good indications additional budget requirements in order to perform tests necessary to gather the adequate sample data. The idea behind this approach aims to reduce the number of sampling iterations and to allow for careful consideration when selecting a sample size during these individual iterations if required. Utilizing test budgets in this process allows for a basic, yet realistic approach in iteration reduction. The following chapter provides a demonstration of utilizing budgets for 100 iteration reduction and how the combined use of the test planning and sequential Kriging software can be used as a tool for determining additional budgets when required. 5.7 Summary of Advancements This chapter developed mathematical methodologies and algorithms that are required in order to develop advanced DFK software and to test the software in the test planning and validation environments that are presented in the next chapter. In this chapter we introduced a novel approach at selecting the fitted variogram function without the presence of empirical data. We also introduced the concept of cost constraints into the sequential sampling process. Since our focus is on the interpolation of physical systems, it is imperative that cost constraints be taken into consideration as it affects the overall amount of samples that are collected. This presents users with a trade-off decision of interpolation accuracy versus additional cost. We also introduced a unique random spatial sampling scheme that focuses on sampling directly from lag information instead of FLC information after the problem boundaries have been established. This technique is combined with the sample size selection process to yield information to users of the advanced DFK software about initial sample size and associated FLCs. After the initial samples are taken, users can begin the sequential Kriging process. In this process, a standard model for fitting the variogram through the use of iterative regression and GRG or EA is developed. This method holds to variogram assumptions and allows for optimization in this area along with accurate Kriging predictions. Finally, we introduced a methodology to decrease the number of sequential samples by first, optimizing the sample size and sample locations selections during the sequential sampling process, and second, basing the iterative samples on an 101 overall test budget. The next chapter discusses applications of the research through utilization of the software. 102 CHAPTER 6 6 Complete Methodology for the Advanced DFK Process This chapter focuses on test planning and validation applications that are possible through the use of DFK software. It presents results of the software in four distinct areas or demonstrations. The first is the difference in MSE using standard fitted variogram models versus the standard model as defined previously. The second demonstration examines developing general heuristics for initial sample size selection based on expected improvement (EI) from the test planning portion of the software. The third discusses the combined use of the test planning and sequential Kriging aspects of the software in determining additional budget required. The final demonstration is a study on validation of black box simulations against verified MC models using all aspects of the DFK software. The chapter concludes with a brief summary. 6.1 Application Area 1: Standard Variogram versus Traditional Models In the first applications area we compare, through the use of , our standard variogram model approach against the linear, spherical, exponential, and Gaussian literature models. It is of note to remind the reader that a single optimal neighborhood selection was used in the standard model development. To demonstrate results, we used a noisy empirical variogram with 20 unique lag distances. The empirical variogram is shown below. 103 Figure 17: Noisy Empirical Variogram The first objective is to fit the above empirical variograms with linear, spherical, exponential, and Gaussian models while running GRG or EA on and setting . This is defined through the following model: Objective: ? Subject to: 0 5 10 15 20 25 30 35 40 45 0 5 10 15 20 Emperical Variogram Emperical Variogram 104 By changing: The optimized models were found to be: Table 39: Parameter Values for Fitted Variogram Linear 0 20 41.5 Spherical 0 20 39.131 Exponential 0 20 37.432 Gaussian 0 20 41.5 Table 40: Optimized Variogram Values for Traditional Functions Linear Residuals Spherical Residuals Exponen Residuals Gaussian Residuals 2.0750 1.16 2.9324 3.73 5.2139 17.76 0.3101 0.48 4.1500 34.22 5.8501 17.22 9.7016 0.09 1.2265 76.97 6.2250 189.75 8.7385 126.82 13.5642 41.42 2.7088 298.99 8.3000 32.49 11.5828 5.84 16.8887 8.34 4.6928 86.62 10.3750 2.64 14.3685 5.61 19.7502 60.07 7.0953 24.06 12.4500 30.80 17.0807 0.85 22.2131 17.75 9.8198 66.92 14.5250 41.93 19.7050 1.68 24.3329 11.11 12.7628 67.85 16.6000 54.76 22.2265 3.15 26.1575 4.65 15.8205 66.90 18.6750 86.96 24.6306 11.35 27.7279 0.07 18.8945 82.91 20.7500 39.06 26.9026 0.01 29.0796 4.32 21.8968 26.04 22.8250 38.13 29.0280 0.00 30.2429 1.54 24.7534 18.03 24.9000 26.01 30.9918 0.98 31.2443 1.55 27.4068 6.72 26.9750 8.85 32.7796 77.08 32.1061 65.71 29.8163 33.83 29.0500 1.10 34.3767 40.66 32.8479 23.50 31.9581 15.67 31.1250 15.02 35.7683 0.59 33.4864 2.29 33.8233 1.38 105 33.2000 23.04 36.9398 1.12 34.0360 15.71 35.4158 6.68 35.2750 13.88 37.8765 1.26 34.5090 20.17 36.7498 5.06 37.3500 7.02 38.5637 2.06 34.9161 25.85 37.8465 4.64 39.4250 1.16 38.9868 2.29 35.2665 27.39 38.7318 3.13 41.5000 0.00 39.1311 5.61 35.5681 35.19 39.4338 4.27 SS(RES) 647.97 SS(RES) 307.93 SS(RES) 384.49 SS(RES) 897.16 Figure 18: Graphical Variogram Summary From the experimental trials above, the spherical variogram had the lowest of all the models. We will now compare the with , where is the standard model proposed in this dissertation. We start by defining the new model: Objective: 0 5 10 15 20 25 30 35 40 45 0 5 10 15 20 25 Emperical Variogram Emperical Variogram Linear Spherical Exponential Gaussian 106 ? ? Subject to: FOR = 1 to STEP 1 ? ? NEXT by changing: Solving the above problem, allows us to smooth the regression model to obtain a non-decreasing function. We begin by performing iterative regression analysis. The first order simple linear regression resulted in an . After smoothing the 1st iteration, the where . The reader should be cognizant of the fact that is not to be confused with in a general regression analysis because is the sum of the square of the residuals after the GRG algorithm has been applied to smooth the function. Next, the stopping criterion were analyzed and since but the iterations were the iterations continued. Adding the second regressor, we obtain and , where . Since our aim was to show we manually stop the regression iterations. A diagram of the standard model titled ?y-hat? included into the previous figure shows a visual comparison and contrast. 107 Figure 19: Standard Model Variogram A more accurate variogram model, leads to better Kriging predictions. We propose that using this standardized approach, taking advantage of modern computer systems, is a viable alternative to the traditional methods of attempting to fit multiple variogram functions under the current assumptions. Additional variogram models along with various model testing confirm our approach. If , GRG may yield results that are not feasible. Note that an is statistically attainable but we had to constraint because GRG wil not yield a usable solution. The software is designed such that it recognizes this requirement and prevents the next iteration of the regression model in the event that . Future research may 0 5 10 15 20 25 30 35 40 45 0 5 10 15 20 25 Emperical Variogram Emperical Variogram Linear Spherical Exponential Gaussian Yhat 108 be conducted to ensure the lowest is presented using the standard model while simplifying the complexity of the regression model. 6.2 Application Area 2: Sample Size Selection during Test Planning The test planning portion of the software is designed to aid in the determination of initial sample size and FLC selection that is adequate based on the designed experiment. The approach we use is to examine the total amount of variance reduction by increasing sample size. Based on various percentages of reduction, we compare our results to actual response data to determine heuristics for initial sample size and initial FLC selection. We generated 28 pilot designs with user inputs in the DFK software as shown in the table below. Table 41: Application Area 2 ? 28 Pilot Designs # Inputs/Dimensions # of FLCs % VAR Reduction 1 25 10 1 25 25 1 25 33 1 25 50 1 25 66 1 25 75 1 25 90 2 25 10 2 25 25 109 2 25 33 2 25 50 2 25 66 2 25 75 2 25 90 1 125 10 1 125 25 1 125 33 1 125 50 1 125 66 1 125 75 1 125 90 3 125 10 3 125 25 3 125 33 3 125 50 3 125 66 3 125 75 3 125 90 110 These inputs were selected to represent a varying quantity of FLCs, to cover dimensionality impacts and common percentages that users would generally specify. Each of the pilot designs were tested against three response models. The first response model portrays a smooth curve. The second model portrays an increasing response but ?rough? curve, while the third model depicts a noisy response curve. The user input settings and model designs span similar situations as would be encountered in field studies or industry. Further testing should be conducted for models that display large noise in certain areas such as the tails, models with large variability given a small variability in FLCs, or other various models that are commonly encountered in practice. The three response curves that were selected are presented in the figures below. 0 1 2 3 4 5 6 0 50 100 150 Smooth Polynomial Smooth Polynomial 111 Figure 20: Response Data for Sample Size Selection For each of the 28 pilot designs, we selected two FLCs at random and performed Kriging based on each of the three response data models. After collecting the Kriged data along with the 0 2 4 6 8 10 12 14 16 18 20 0 50 100 150 Rough Polynomial Rough Polynomial 0 10 20 30 40 50 60 70 0 50 100 150 Noisy Response Noisy Data 112 prediction variances, we determined the actual observed response. This allowed for the calculation of the squared error between the response data and the Kriging model, , where is the intial sample size based on the selected % variance reduction. The following tables summarize the findings of the prediction variance and . A brief discussion of the analysis of the results follows these tables. The complete table of data used to generate the summary tables below is found in Appendix D. Table 42: Summary Data for Pilot Design Studies Summary of Random FLC Selection for Selection Smooth Response Candidate FLC Variance Va ria nc e Re duc tion (Pil ot De sign) 25% 5 0.4362048 0.000256 33% 4 0.3785 0.00018225 50% 8 0.120273272 7.92176E-06 66% 3 0.071747605 2.5E-07 75% 6 0.070012506 3.1783E-06 90% 6 0.069653811 2.5E-07 Summary of Random FLC Selection for Selection Smooth Response Candidate FLC Variance Va ria nc e Re duc tion (Pil ot De sign) 25% (1, 2) 0.258012564 0.000981459 33% (2, 2) 0.30082494 0.002663559 50% (1, 4) 0.010271132 2.37865E-05 66% (2, 1) 0.05349981 0.000500417 75% (2, 3) 0.005026733 2.62449E-05 90% (3, 2) 0.162164651 2.43318E-05 Summary of Random FLC Selection for Selection Rough Response Candidate FLC Variance Va ria nce Re du cti on (Pil ot De sig n) 25% 93 1.404718222 0.004019103 33% 120 3.571362043 2.851391695 113 50% 120 3.571362043 0.3842741 66% 90 1.33942208 0.012527153 75% 80 0.609687765 0.001007153 90% 70 0.386930139 8.99955E-05 Summary of Random FLC Selection for Selection Rough Response Candidate FLC Variance Va ria nc e Re duc tion (Pil ot De sign) 25% (4, 4, 1) 0 0.080620736 33% (5, 1, 2) 0.689085195 0.350832228 50% (4, 5, 5) 0.763073107 0.050146604 66% (3, 5, 4) 0.422687635 0.061600135 75% (3, 4, 5) 0.374146817 0.30880224 90% (4, 4, 2) 0.373281973 0.059318233 Analysis of these data did not show any clear methods for selection due to the randomness of the FLC selection. The random FLCs have a wide range of estimated variability based on the FLC selected and . Candidate FLC variance would be much smaller when taken directly next to observed FLC responses and would increase as the candidate FLC would be further away (spatially) from observed responses due to the linear weight assignment method in Kriging. A more definitive technique had to be established. Again, we conducted all the tests as previously described except that we studied all unobserved FLCs in the factor space instead of random FLCs throughout the factor space. The same data of interest, and ? were collected except for each unsampled FLC in the factor space. The tables below summarize the finding from the secondary study. The first and third table show and ? values while the second and fourth table show the percent reduction in and ? as the percent of variability was reduced in the pilot design as generated from the test planning portion of the software. 114 Table 43: Summary of and ? for all Unobserved FLCs Sum of Variability for all Unobserved FLCs in Factor Space Smooth Response Rough Response Noisy Response Va ria nc e Re duc tion (Pil ot De sign) 25% 7.7971608 64.7506721 3.289102713 33% 5.470734589 49.16320671 3.091723569 50% 2.453574742 3.737812542 2.304255731 66% 1.411036229 2.924561928 1.089949344 75% 0.910162579 1.561493688 0.904060708 90% 0.348269054 0.43705094 0.276952231 Smooth Response Rough Response Noisy Response Va ria nc e Re duc tion (Pil ot De sign) 25% 99.01041773 928.9869839 81.41313689 33% 29.0301759 512.1710515 71.06256482 50% 24.4743451 512.1710515 30.50388668 66% 4.178449566 76.19619544 14.20431279 75% 8.807956097 48.77145576 22.71659314 90% 2.943178333 15.8641357 9.839428735 % Reduction in Variability for all Unobserved FLCs in Factor Space Smooth Response Rough Response Noisy Response Va ria nc e Re duc tion (Pil ot De sign) 25% 42.39675433 52.82543656 30.02089645 33% 29.74690356 40.10873976 28.21934163 50% 13.34121589 3.049413587 21.03182197 66% 7.672453844 2.385940648 9.948383875 75% 4.948973125 1.273910881 8.251707313 90% 1.893699248 0.356558564 2.527848768 Smooth Response Rough Response Noisy Response Va ria nc e Re duc tion (Pil ot De sign) 25% 58.77924442 44.3608223 35.43708721 33% 17.23426528 24.45710155 30.93174398 50% 14.52961765 24.45710155 13.2775733 66% 2.480608748 3.63850726 6.182779466 75% 5.228995253 2.328925937 9.887960628 90% 1.747268647 0.757541405 4.282855415 115 for all Unobserved FLCs in Factor Space Smooth Response Rough Response Noisy Response Va ria nc e Re duc tion (Pil ot De sign) 25% 0.004147 9.07303615 56.59759201 33% 0.002203 9.07531689 36.29107218 50% 0.0001375 2.988934423 3.79253055 66% 1.425E-05 0.20451854 0.715526668 75% 7.25E-06 0.208055907 0.320130762 90% 1.25E-06 0.144014964 0.069381913 Smooth Response Rough Response Noisy Response Va ria nc e Re duc tion (Pil ot De sign) 25% 0.954304 122.8576312 366.9507408 33% 0.795937645 33.28298388 382.1229022 50% 0.030811673 33.28298388 396.7815453 66% 0.00102 10.27969455 79.2147675 75% 0.001134845 2.858423796 47.26833224 90% 0.000719842 0.072907515 13.89661911 % Reduction in for all Unobserved FLCs in Factor Space Smooth Response Rough Response Noisy Response Va ria nc e Re duc tion (Pil ot De sign) 25% 63.69955071 41.82302777 57.87889527 33% 33.83894628 41.83354106 37.11265959 50% 2.112054069 13.77777905 3.878389003 66% 0.218885603 0.942747769 0.731725354 75% 0.111362851 0.9590536 0.327378148 90% 0.019200492 0.66385075 0.070952638 Smooth Response Rough Response Noisy Response Va ria nc e Re duc tion (Pil ot De sign) 25% 53.49453551 60.6301274 28.52906096 33% 44.61713943 16.42512177 29.70864031 50% 1.727181417 16.42512177 30.84829553 66% 0.057177195 5.073019755 6.158654773 75% 0.063614948 1.410629501 3.674937757 90% 0.040351507 0.035979791 1.080410664 Analysis of the data from the secondary study yields conclusive results. It is clear from the tables above that there are distinct breaking points in the reduction of and ? at the 50-66% pilot design variance reduction. With this knowledge we generate sample size guidelines in 116 Table 44 below. The complete table of FLC selections during the DFK software pilot design data generation process based on variance reduction is contained in Appendix E. An additional table for varying dimensions and is contained in Appendix G. Even further studies to expand the table in Appendix G is recommended to yield a detailed list of recommended sample size and FLC locations based on and dimension . Table 44: Recommended Pilot Sample Size and FLC Selection Recommended FLC Selection Smooth Response 7 (Based off 50% Pilot Design Variance Reduction) 1, 4, 7, 13, 19, 22, 25 Rough Response 10 (Based off 66% Pilot Design Variance Reduction) 1, 2, 4, 7, 10, 13, 16, 19, 22, 25 Noisy Response 10 (Based off 66% Pilot Design Variance Reduction) 1, 2, 4, 7, 10, 13, 16, 19, 22, 25 Recommended Smooth Response 14 (Based off 50% Pilot Design Variance Reduction) 1, 16, 24, 32, 47, 55, 63, 71, 79, 94, 102, 110, 118, 125 Rough Response 28 (Based off 66% Pilot Design Variance Reduction) 1, 8, 12, 16, 20, 24, 28, 32, 39, 43, 47, 51, 55, 63, 67, 71, 75, 79, 83, 87, 94, 98, 102, 106, 110, 114, 118, 125 Noisy Response 28 (Based off 66% Pilot 1, 8, 12, 16, 20, 24, 28, 32, 39, 117 Design Variance Reduction) 43, 47, 51, 55, 63, 67, 71, 75, 79, 83, 87, 94, 98, 102, 106, 110, 114, 118, 125 This section established a methodology for selecting initial sample size and FLC selection. With use of DFK software, additional designs can be replicated and studied with little effort to expand the above table. 6.3 Application Area 3: Additional Budget Determination Based on the analysis of previous section, we pursue, through the use of DFK software, a method of determining additional budget in the event that a pilot design sample size did not produce adequate results. A result of this portion of research is to determine a subsequent sample size in the event that another iteration of experimental sampling is required. To address this issue, we focus on variance reduction results from the test planning portion of the software. We studied four designs for this application area. The four designs are shown below in Table 45. Table 45: Addition Budget Determination: Dimension and Factor Space Selection Dimensions - Factor Space - 1 25 2 25 1 125 3 125 118 For each of the four designs, a graph of variance reduction as increased are plotted and shown below. 0 0.2 0.4 0.6 0.8 1 1.2 0 10 20 30 1-d 25 FLC Estimated Variance 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 10 20 30 2-d 25 FLC Estimated Variance 119 Figure 21: Variance Reduction in Pilot Designs All four graphs display distinct breaking points in variance estimation as increased. A comparison of the dips in the and graphs against the selection 0 0.2 0.4 0.6 0.8 1 1.2 0 50 100 150 1-d 125 FLC Estimated Variance 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0 50 100 150 3-d 125 FLC Estimated Variance 120 described in the previous section displays a direct correlation between selection and a major dip in the graph. Based on analysis of the graphs and the studies in the previous section, it is determined that in the event additional samples are required, users of DFK software should base an additional sample size selection where the next dip in the graph occurs. For example, the second large dip in the design occurs as which corresponds to our initial sample size selection in Table 44. Users should examine the next dip in the graph which occurs at and thus request additional test budget for the three additional samples if required. 6.4 Application Area 4: Validation of MC Simulation against Limited Empirical Data This section presents two distinct black box representations of physical systems that were independently generated by Dilcu Helvaci and provided for analysis. The test planning portion of the software is used to determine initial sample size and sample location positions. The sequential sampling portion of the software is used to complete analysis for sampling and to produce interpolations over the entire factor space. Monte Carlo simulations are used to generate the system responses. We analyze validity statements of the Monte Carlo model with and without the use of the advanced DFK software to demonstrate software effectiveness. The results are shown in the case below. We begin by establishing an approach in which to compare two datasets. The chosen statistic is the Kolmogorov-Smirnov as it is a nonparametric test that determines if there exists a significant difference between two datasets. The null hypothesis for the Kolmogorov-Smirnov statistic is . The result of this test will determine whether our data sets 121 cumulative distributions are practically the same at a 5% Level of Significance (LOS). In the event that we cannot reject we will consider that MC model a good representation of the actual data. If is rejected, we will augment with the Kriging interpolations and perform the test again except the hypotheses will now use instead of . The factor space for our two black box problems are both determined to be with , , and dimensionality of . The actual observed values are shown in the table below. The MC values are found in Appendix F. Table 46: Observed Values from System 1 and System 2 - System 1 - System 2 1 6.8 32.90095391 30 108722 37.0198053 60 866940 38.21338068 90 2922600 41.40984528 93 3224527.2 45.94042492 99 3889222.8 47.23850124 102 4253353.2 49.78464302 106 4773256.8 52.10657163 109 5189832.8 54.35796892 112 5629965.2 54.36469918 115 6094304 55.80191681 118 6583497.2 56.80046242 121 7098192.8 60.31219199 122 7275541.2 63.14901965 132 9214055.2 63.40949893 141 11229052.8 63.46611498 180 23354202 67.56625352 189 27034012.8 67.74433258 203 33495053.2 70.68682079 243 57443273.2 73.14682206 122 We proceed to perform the K-S test on MC each data set . A table of the s is provided below. Table 47: Black Box: System 1: ( ) System 2: ( ) 0.275 0.089 Since the s for both systems are > 0.05, we cannot reject the null hypothesis that the datasets are statistically the same. We draw evidence from the K-S test to state that the MC models for either system is an accurate representation of the sampled data sets . Standard K-S graphs are provided for visual clarification. The first figure represents System 1 and the second figure represents System 2. 123 Figure 22: K-S Cumulative Probability Plots of MC vs. System Responses Next, we repeat the process after we perform sequential Kriging on both data sets to interpolate at all unsampled FLCs in order to augment the data sets . The augmented data sets are found in Appendix F. After performing Kriging, the were again obtained and shown below. Table 48: Black Box: utilizing DFK Software System 1: ( ) System 2: ( ) 0.311 The K-S plots are also shown in the following figures. 124 Figure 23: K-S Cumulative Probability Plots of MC vs. Augmented System Responses The results show that for System 1, after the Kriging data was gathered, we more strongly cannot reject that the MC model is an adequate representation of the physical system. The MC model representing System 1 is considered valid. In System 2, the null hypothesis was not rejected initially. After generating the Kriging data, the null hypothesis now must be rejected, thus the MC model is not a good representation of System 2. The MC model representing System 2 is not considered valid. 125 6.5 Application Summary This chapter demonstrates results of the methodologies presented in Chapter 5. We began the chapter by discussing the results of standardizing the variogram model based solely on necessary variogram requirements. No standard variogram model analyzed in this research demonstrated more accurate Kriging predictions in terms of versus the four common variogram functions found in Geostatistical literature even after optimizing the parameters ( ) through GRG or EA as required. Our method allows for a dynamic smoothing model which minimizes the sum of squares of residuals based on an empirical variogram model. Our second objective was to demonstrate heuristics for initial sample size and sample location selection. We used an effective method based on percent of total estimated variance reduction along with percent total reduction in . Our presented tables accurately demonstrate a distinct capability in determining sample size and FLC location selections when Kriging interpolation is performed. Our third objective was to determine the possibility of additional sample size selection in the event that the initial sample size did not produce adequate results. We were able to clearly demonstrate a trend through the graphical use of pilot design variance reduction data that directly corresponded to our results in Section 6.2. This allows users of DFK software a tool to determine additional cost requirements to obtain the required number of additional samples. Our final objective was to demonstrate validation capabilities while DFK software tool. We performed a K-S test on two systems in which we had no/little knowledge of the underlying response curve. The K-S tests in conjunction with DFK software, did in fact allow us to make claims about the validity of the corresponding MC models that were used. 126 CHAPTER 7 7 The Advanced DFK Application Software This chapter introduces the DFK software. The attainment, installation, and use of the software are described in complete detail throughout the chapter. The chapter also includes many graphical aides that can be referenced during the use of the software. In addition, help features are available throughout the software GUI and can be accessed by simply clicking the appropriate help icon. The research conclusions immediately follow this chapter. 7.1 Introduction The advanced DFK software is an add-in for Microsoft Excel? version 2007 and higher. The software implements all the advancements and mathematical derivations contained in this dissertation. It automates all calculations into two major steps allowing the process developed in this research to be used for data analysis. Without this unique software tool, this research would remain impractical for use in its intended applications and beyond. 7.2 Overview The advanced DFK software is written in VBA. This platform is a high level language that incorporates versatility and the convenience of using Excel? while maintaining a satisfactory level of performance for the scale of problems presented herein. For major industrial purposes and analysis, the software application should be introduced into mainstream statistical software such as Minitab? or MATLAB?. To obtain a copy of the software for personal use, 127 please request a copy from jlb0014@aubrn.edu. After receiving the file, copy it to one of two places depending on your Windows based operating system - :\Users\\AppData\Roaming\Microsoft\AddIns (Windows 7) - :\WINDOWS\Application Data\Microsoft\AddIns\ (Windows XP). - Open Excel? - Go to File (or the Windows bubble in the top left corner if Office 2007 is installed) - Options > Add-Ins > Go? > Check ? DFK? and press OK - Restart Excel? The software has been successfully installed. To execute the software application, select the ?Add-Ins? ribbon menu item, then select ?Kriging?. The user may also request the automated installer to simplify the above steps. The software application was designed to be intuitive and user friendly. Upon opening the software, ?Kriging Start?, the user is presented the following screen: 128 Figure 24: Advanced DFK Software Introduction Screen From the introduction screen, users can perform the test planning tools as discussed and derived in this research or skip directly to performing Kriging depending on user needs. From this screen there is also a brief introduction to Kriging. This introduction is shown in the figure below. 129 Figure 25: DFK Software - Kriging Introduction The primary purpose of the above figure is to introduce users into the software application?s common help feature. Anytime the help icon is seen, users can click on it to open the above figure with pertinent help information. After returning back to the beginning GUI, a selection from one of the three buttons must be made, then the start-up screen will be unloaded and the program must be executed again in order to load it. 7.3 Initial Sample Size and Location Selection To launch the test planning portion of the software application, execute the software which leads to the introduction GUI and then click on the ?Test Planning? button across the bottom. This action brings users to the following screen: 130 Figure 26: Advanced DFK Software - Test Planning This screen requires user input prior to executing the ?Plan/Replan Test Scenario? or the ?Agree and Post Data To Worksheet for Sequential Kriging? with the latter button requiring that data in the two lists directly above it be populated by executing the code behind the ?Plan/Replan Test 131 Scenario?. There are a number of inputs that are required by the user. They are described in the table below. Table 49: Test Planning User Input Table Input Description Dimensions ( ) This input is required. This is the number of inputs to a system or the number of spatial dimensions. Min Value This input is required. This is the minimum value or starting point/location of the inputs. This must be greater than zero. For this software the minimum value for each input must be the same or balanced. Max Value This input is required. This is the maximum value or starting point/location of the inputs. This must be greater than zero. For this software the maximum value for each input must be the same or balanced. Lag ( ) This input is optional and disabled. Test Budget This input is required. This input constrains the output from the analysis against a monetary figure (in dollars). Cost per Test This input is required. This input box provides the user to the ability to specific a monetary amount associated with each test/sample (in dollars). Linear Scale ( ) This input is optional. If left blank, then it is assumed that each test/sample cost an equal amount. This input is essentially a linear multiplier and can be positive or negative. 132 After the user has identified and entered the appropriate input information as described above, then the ?Plan/Replan Test Scenario? button is pressed to execute calculations described throughout Chapters 3-6. Upon completion of the processing, two list boxes will be populated with data. The left-most list box will contain the following data: - sample size - variance Estimation - remaining budget (including over budget scenarios) - and EI. The right-most list box includes a list of initial FLCs, which is dimensional dependent. Users make a decision at this point as to output the data onto the spreadsheet for use in sequential Kriging or to adjust inputs and execute the planning code until satisfactory outputs are generated. If users choose to output data onto the spreadsheet, one of three outcomes is possible. The first possible outcome is the program will output all data. The second possible outcome is that the program will output only data that results in the test scenarios that do not exceed the total budget. The third option is to only output data until a certain amount of total variability has been negated. Users should be aware that just because the budget has not been exhausted, this doesn?t imply that the correct sample size has been determined. The optimal sample size selection should be based more on the estimated variance. Users should also be aware that this initial variance estimation can underestimate the true variance of data. Therefore, users have to use judgment before proceeding to actual physical testing of a system. This type of analysis is 133 consistent with most software applications as users should not assume results are always accurate, understand where they come from, and be able to make judgments based on data presented. After appropriate data have been obtained, users should execute the program again and proceed to the ?Sequential Kriging?. 7.4 Sequential Kriging To perform this portion of the advanced DFK software application, users should execute the software from the ribbon menu, and select the ?Sequential Kriging? button. This brings users to the following GUI: 134 Figure 27: Advanced DFK Software - Sequential Kriging After this portion of the program is loaded, users are required to select the inputs from a spreadsheet, either from the test planning portion of the software or from the user?s own supplied 135 data. The observed response data that accompanies the pilot sample is also required. Finally users must select the Kriging estimation point and the resolution (optional) for sequential Kriging. The software will perform an individual iteration where it estimates the next candidate input set and records them on a program generated spreadsheet. Users can also generate values to use as long as the format used in the program?s self-generating sheet tool is not altered in any way. Confidence intervals are also given. The data is exported to a program generated sheet. The user may notice the following screen after the ?Predict? button is pressed: Figure 28: Advanced DFK Software ? Processing This is the nominal screen to indicate that code is being processed. After code execution has completed, users are presented with the following output screen: 136 Figure 29: Advanced DFK Software ? Output The user can utilize this information as needed to perform analysis. 7.5 Data Analysis and Recording Data calculations are recorded. This allows users to perform further analyses based on the generated datasets. In this research, these data are used to perform MC validation in such a 137 manner as to make inferences about use of the advanced DFK process for the application of validation. 7.6 Conclusions The software developed for this research presents users with a unique capability to perform analyses currently unavailable or overly time-consuming by hand-calculations. The software is easily understandable and user-friendly. Help is presented throughout the program to assist users in understanding all capabilities that are presented. A streamlined GUI was designed and developed to further make the software application available to a larger audience. Methodology and formulation presented in this dissertation are utilized as the underlying code. The code base is presented in Appendix A for review and understanding. 138 CHAPTER 8 8 Conclusions and Proposed Future Research Results of this research have shown the extension, expansion, and augmentation of the ?Customized Sequential Designs for Random Simulation Experiments: Kriging Metamodeling and Bootstrapping? process introduced by van Beer?s. Introduction of Design of Experiments (DOX) with Kriging is an expanding field of research with many areas of optimization and application. This thesis reiterates the phrase Design for Kriging (DFK) recently coined by [52] as it accurately and precisely describes the intent of the process. It summarizes extensions of the DFK process including initial sample size determination, space filling designs for Kriging, optimization of sequential sampling methods, and inclusion of a standard variogram model. It also introduces test planning and validation applications that take advantage of DFK methodology. Summarized details of our conclusions are found in the following section. 8.1 Conclusions Based on the data analysis and results presented in Chapter 6, a number of conclusions are drawn. Through our standardized variogram model, we are able to minimize the error between an experimental variogram model and our model thus allowing for accurate Kriging predictions. Secondly, we presented a method of utilizing the DFK software in order to determine initial sample size and sample location determination. Multiple summarized tables of data were constructed to demonstrate these results. Additionally, we demonstrated a trend based 139 on pilot design variance reduction for additional budget considerations that directly correspond to our results in Section 6.2. As an important but closing note, we discuss validation utilizing DFK software when limited empirical data are present. It was found, through the use of the K-S test along with DFK software, that the additional data provided by the software allowed us to more closely scrutinize the differences between two datasets. This allowed us to make claims about MC models that were designed to represent such systems. 8.2 Future Work The work herein and especially development of the advanced DFK software has opened further avenues for future research, which can determine the feasibility of a permanent computer learning model adaptation such as in neural networks or central database storage of pilot designs. This would highly benefit the overall process and software. The software could recognize exact or similar problems as has been performed previously and perform decision analysis on the factor space without having to perform all required DFK calculations. Another research topic is determining the feasibility of the process to be multi-threaded into processing individual Kriging instances for increased speed and accuracy. Porting the software into a platform that is more suitable for multi-threading is almost certainly required. Future research may include advanced selection of FLCs including non-integer values. A potential future advancement is to expand the software inputs to include distributions as inputs and incorporation of separate ranges per input thus introducing non-blocked designs. Kriging with multiple response variables can also be studied for feasibility. This exploratory study could open avenues for future Kriging research and software advancement. Additionally, the incorporation of weight standardization to correct 140 for negative Kriging weights would allow the DFK software to be more robust. Finally, Kriging with real-time determination of neighborhoods and separation of neighborhoods (possibly into different threads) would allow the overall process to be more efficient and produce more accurate results. The study of multiple simultaneous neighborhood processing would be a large leap forward in the current software design and would require a large research effort. A mathematical study to precisely identify the next FLC during the test planning phase would eliminate the search algorithm used in the code base, thus tremendously decreasing the computational time in creating new designs. Further, a more detailed study into the standard variogram model such as nonparametric regression could provide a more robust method of arriving at accurate variograms. 141 References [1] J. Kleijnen and W. van Beers, ?Customized Sequential Designs for Random Simulation Experiments: Kriging Metamodeling and Bootstrapping,? CentER, vol. 2005?55, p. 33, Mar. 2005. [2] M. Bachmaier and M. Backes, ?Variogram or semivariogram? Understanding the variances in a variogram,? Precision Agriculture, vol. 9, no. 3, pp. 173?175, Feb. 2008. [3] W. van Beers, ?Kriging Metamodeling for Simulation,? Tilburg, Netherlands, 2004. [4] J. Kleijnen and W. van Beers, ?Kriging for Interpolation in Random Simulation,? CentER, vol. 2001-74, 2001. [5] D. Dahlem and W. Harrison, ?Globally Optimal Multi-agent Reinforcement Learning Parameters in Distributed Task Assignment,? in Web Intelligence and Intelligent Agent Technologies, 2009. WI-IAT ?09. IEEE/WIC/ACM International Joint Conferences on, 2009, vol. 2, pp. 28?35. [6] R. B. Gramacy, H. K. H. Lee, and W. G. Macready, ?Parameter space exploration with Gaussian process trees,? Proceedings of the twenty-first international conference on Machine learning, p. 45?, 2004. [7] R. Webster and M. Oliver, Geostatistics for Environmental Scientists. John Wiley & Sons, LTD., 2001. [8] E. Isaaks and R. Srivastava, An Introduction to Applied Geostatistics. Oxford University Press, Inc., 1989. [9] R. Olea, Geostatistics for Engineers and Earth Scientists. Kluwer Academic Publishers, 1999. [10] N. Cressie, Statistics for Spatial Data. Wiley-Interscience, 1993. [11] O. Balci, ?A Methodology for Certification of Modeling and Simulation Applications,? TOMACS, vol. 7, no. 4, pp. 352 ? 377, 2001. 142 [12] R. Sargent, ?A Tutorial on Verification and Validation of Simulation Models,? in Proceedings of the 39th conference on Winter simulation, Washington, D.C., United States, 2007. [13] J. Kleijnen and W. Van Groenendaal, Simulation: A Statistical Perspective. Baffins Lane, Chichester, West Sussex PO19 1UD, England: John Wiley & Sons, LTD., 1992. [14] A. Forrester, ?Black-Box Calibration for Complex-System Simulation,? Philosophical Transactions of the Royal Society, vol. 368, pp. 3567?3579, Jul. 2010. [15] B. Ankenman, B. Nelson, and J. Staum, ?Stochastic Kriging for Simulation Metamodeling,? in Proceedings of the 40th Conference on Winter Simulation, Miami, Florida, United States, 2008, pp. 362 ? 370. [16] G. Matheron, ?Principles of Geostatistics,? Economic Geology, vol. 58, no. 8, pp. 1246 ? 1266, Dec. 1963. [17] H. O. A. Wold, A Study in the Analysis of Stationary Time Series, 2nd ed. Almqvist & Wiksell, 1954. [18] R. H. Bishop and I.-T. I., Systems, and Automation Society, Sur 1?interpolation et Extrapolation des Suites Stationnaires. CRC Press, 2002. [19] A. N. Kolmogorov, ?Dissipation of Energy in the Locally Isotropic Turbulence,? Proceedings: Mathematical and Physical Sciences, vol. 434, no. 1890, pp. 15?17, Jul. 1991. [20] N. Wiener, Extrapolation, Interpolation, and Smoothing of Stationary Time Series: With Engineering Applications. The MIT Press, 1964. [21] H. Liu, ?Taylor Kriging Metamodeling for Simulation Interpolation, Sensitivity Analysis and Optimization,? Dissertation, Auburn, Auburn, AL, 2009. [22] D. G. Krige, Lognormal-De Wijsian Geostatistics for Ore Evaluation, 2nd Revised ed. South African Institute of Mining and Metallurgy, 1981. [23] J. K. Ord, ?Kriging, Entry,? Encyclopedia of Statistical Sciences, vol. 4. John Wiley & Sons, Inc., New York, pp. 411?413, 1983. [24] P. Hemyari, ?Analytical Solution for Punctual Kriging in One Dimension.?. 143 [25] N. Cressie, ?The Origins of Kriging,? Mathematical Geology, vol. 22, no. 3, pp. 239?252, Apr. 1990. [26] I. Clark, Practical Geostatistics. Appliced Science Publishers LTD, 1979. [27] I. Clark and W. Harper, Practical Geostatistics 2000. Ecosse North America LLC., 2000. [28] W. van Beers, ?Kriging Metamodeling in Discrete-Event Simulation: An Overview,? in Proceedings of the 37th conference on Winter simulation, Orlando, Florida, United States, 2005, pp. 202 ? 208. [29] P. Goovaerts, ?Factorial Kriging Analysis: A Useful Tool for Exploring the Structure of Multivariate Spatial Soil Information,? Journal of Soil Science, vol. 43, no. 4, pp. 597?619, Dec. 1992. [30] A. P. Reis, A. J. Sousa, E. Ferreira da Silva, C. Patinha, and E. C. Fonseca, ?Combining Multiple Correspondence Analysis with Factorial Kriging Analysis for Geochemical Mapping of the Gold?Silver Deposit at Marrancos (Portugal),? Applied Geochemistry, vol. 19, no. 4, pp. 623?631, Apr. 2004. [31] M. W. Davis and P. G. Culbane, ?Contouring Very Large Data Sets Using Kriging,? Geostatistics for Natural Resources Characterization, vol. 2, 1984. [32] A. G. Royle, F. L. Clausen, and P. Frederiksen, ?Practical Universal Kriging and Automatic Contouring,? Elsevier scietific publishing company, vol. 1, pp. 377?394, 1981. [33] F. Trochu, ?A Contouring Program Based on Dual Kriging Interpolation,? Engineering with Computers, vol. 9, no. 3, pp. 160?177, Sep. 1993. [34] S. De Iaco, D. E. Myers, and D. Posa, ?Space-time Variograms and a Functional Form for Total Air Pollution Measurements,? vol. 41, no. 2, pp. 311?328, 2002. [35] R. L. Bras and I. Rodriguez-Iturbe, Random Functions and Hydrology. Dover Publications, 1994. [36] S. Cheng and R. Wang, ?An Approach for Evaluating the Hydrological Effects of Urbanization and its Application,? Hydrological Processes, vol. 16, no. 7, pp. 1403?1418, May 2002. 144 [37] B. Andr?, F. Trochu, and J. Dansereau, ?Approach for the Smoothing of Three- Dimensional Reconstructions of the Human Spine Using Dual Kriging Interpolation,? Med Biol Eng Comput, vol. 34, no. 3, pp. 185?191, May 1996. [38] S. Delorme, Y. Petit, J. A. de Guise, C.-E. Aubin, H. Labelle, C. Landry, and J. Dansereau, ?Three-Dimensional Modelling and Rendering of the Human Skeletal Trunk from 2D Radiographic Images,? in Second International Conference on 3-D Digital Imaging and Modeling, 1999. Proceedings, 1999, pp. 497?505. [39] P. De Doncker, X. Cognet, J. Dricot, R. Meys, M. H?lier, and W. Tabbara, ?Electromagnetic Wave Propagation Prediction Using Spatial Statistics: Experimental validation,? in Proc. of the 9th Symp. on Communications and Vehicular Technology in the Benelux, 2002. [40] J. Dricot, R. Meys, M. H?lier, and W. Tabbara, ?Kriging the Fields: A New Statistical Tool for Wave Propagation Analysis,? in Proc. of the Int. Conf. on Electromagnetics in Advanced Applications, 2003. [41] J. Echaabi, F. Trochu, and R. Gauvin, ?A General Strength Theory for Composite Materials Based on Dual Kriging Interpolation,? Journal of Reinforced Plastics and Composites, vol. 14, no. 3, pp. 211 ?232, Mar. 1995. [42] G. Kamanayo, F. Trochu, and B. Sanschagrin, ?Prediction of Shrinkage by Dual Kriging for Injection Molded Polypropylene Plaques,? Advances in Polymer Technology, vol. 13, no. 4, pp. 305?314, Dec. 1994. [43] A. Limaiem and H. A. ElMaraghy, ?Automatic Inference of Parametric Equations in Geometric Modeling Using Dual Kriging,? in , 1996 IEEE International Conference on Robotics and Automation, 1996. Proceedings, 1996, vol. 2, pp. 1499?1504 vol.2. [44] P. McLean, P. L?ger, and R. Tinawi, ?Post-Processing of Finite Element Stress Fields Using Dual Kriging Based Methods for Structural Analysis of Concrete Dams,? Finite Elements in Analysis and Design, vol. 42, no. 6, pp. 532?546, Mar. 2006. [45] R. A. V. Rossel, P. Goovaerts, and A. B. McBratney, ?Assessment of the Production and Economic Risks of Site-Specific Liming Using Geostatistical Uncertainty Modelling,? Environmetrics, vol. 12, no. 8, pp. 699?711, 2001. [46] R. Chaveesuk and A. E. Smith, Dual kriging: an Exploratory Use in Economic Metamodeling: An Article from: Engineering Economist. Thomson Gale, 2005. 145 [47] R. R. Barton, ?Metamodeling: A State of the Art Review,? in Simulation Conference Proceedings, 1994. Winter, 1994, pp. 237? 244. [48] J. Kleijnen and W. van Beers, ?Robustness of Kriging when interpolating in random simulation with heterogeneous variances: Some experiments,? ejor, vol. 165, no. 2005, pp. 826?834, 2005. [49] H. Gunes, H. E. Cekli, and U. Rist, ?Data Enhancement, Smoothing, Reconstruction and Optimization by Kriging Interpolation,? in Proceedings of the 40th Conference on Winter Simulation, Miami, Florida, United States, 2008, pp. 379 ? 386. [50] T. Simpson, T. Mauery, J. Korte, and F. Mistree, ?Comparison of Response Surface and Kriging Models for Multidisciplinary Design Optimization,? Penn State. [51] W. Biles, J. Kleijnen, W. van Beers, and I. van Nieuwenhuyse, ?Kriging Metamodeling in Constrained Simulation Optimization: An Explorative Study,? in Proceedings of the 39th conference on Winter simulation, Washington, D.C., United States, 2007, pp. 355 ? 362. [52] J. Kleijnen, Design and analysis of simulation experiments. New York: Springer, 2008. [53] J. R. Koehler and A. B. Owen, Computer Experiments. Handbook of Statistics, vol. 13. S. Gosh and C.R. Rao, 1996. [54] J. Sacks, W. Welch, T. Mitchell, and H. Wynn, ?Design and Analysis of Computer Experiments,? JSTOR, vol. 4, no. 4, pp. 409?423, 1989. [55] T. J. Mitchell and M. D. Morris, ?The Spatial Correlation Function Approach to Response Surface Estimation,? in Proceedings of the 24th conference on Winter simulation, New York, NY, USA, 1992, pp. 565?571. [56] J. R?gni?re and A. Sharov, ?Simulating Temperature-Dependent Ecological Processes at the Sub-Continental Scale: Male Gypsy Moth Flight Phenology as an Example,? International Journal of Biometeorology, vol. 42, no. 3, pp. 146?152, Feb. 1999. [57] J. Kleijnen and W. van Beers, ?Application-Driven Sequential Designs for Simulation Experiments: Kriging Metamodeling,? CentER, vol. 2003?33. [58] D. den Hertog, J. Kleijnen, and A. Siem, ?The Correct Kriging Variance Estimated by Bootstrapping,? JORS, vol. 57, no. 4, pp. 400?409, 2006. 146 [59] N. Flipo, N. Jeann?e, M. Poulin, S. Even, and E. Ledoux, ?Assessment of nitrate pollution in the Grand Morin aquifers (France): Combined use of geostatistics and physically based modeling,? Environmental Pollution, vol. 146, no. 1, pp. 241?256, Mar. 2007. [60] W. Ren, M. Steurer, and S. Woodruff, ?Accuracy evaluation in power hardware-in-the- loop (PHIL) simulation center for advanced power systems,? in Proceedings of the 2007 summer computer simulation conference, San Diego, CA, USA, 2007, pp. 489?493. [61] A. Ratle, ?Kriging as a Surrogate Fitness Landscape in Evolutionary Optimization,? AI EDAM, vol. 15, no. 01, pp. 37?49, 2001. [62] D. de Frutos Escrig and C. Gregorio Rodr?guez, ?Constrained Simulations, Nested Simulation Semantics and Counting Bisimulations,? Electronic Notes in Theoretical Computer Science, vol. 206, pp. 41?58, Apr. 2008. [63] G. S. Fishman, Monte Carlo: Concepts, Algorithms, and Applications. Springer, 1996. [64] R. Hill, J. Miller, and G. McIntyre, ?Applications of Discrete Event Simulation Modeling to Military Applications,? in Proceedings of the 33rd Winter Simulation Conference, Arlington, Virginia, United States, 2001, pp. 780 ? 788. [65] A. Law, Simulation Modeling and Analysis, Third ed. United States: McGraw-Hill, 2000. [66] O. Balci, W. Ormsby, R. Nance, and J. Arthur, ?Expanding our Horizons in Verification, Validation, and Accreditation Research and Practice,? in Proceedings of the 34th Winter Simulation Conference, San Diego, California, United States, 2002, pp. 653 ? 663. [67] J. Arthur and R. Nance, ?Independent Verification and Validation: A Missing Link in Simulation Methodology?,? in Proceedings of the 28th conference on Winter simulation, Coronado, California, United States, 1996, pp. 230 ? 236. [68] Department of Defense, ?Military Standard 499 System Engineering.? Department of Defense, 06-May-1994. [69] D. Caughlin, ?An Integrated Approach to Verification, Validation, and Accreditation of Models and Simulations,? in Proceedings of the 32nd conference on Winter simulation, Orlando, Florida, United States, 2000, pp. 872 ? 881. [70] L. Chwif, P. S?rgio Muniz Silva, and L. Mitio Shimada, ?A Prescriptive Technique for V&V of Simulation Models When No Real-Life Data are Available,? in Proceedings of the 147 38th Winter Simulation Conference, Monterey, California, United States, 2006, pp. 911 ? 918. [71] R. Cheng, W. Holland, and N. Hughes, ?Selection of Input Models Using Bootstrap Goodness-of-Fit,? in Proceedings of the 28th conference on Winter simulation, Coronado, California, United States, 1996, pp. 199 ? 206. [72] L. Champagne and R. Hill, ?Agent-model Validation Based on Historical Data,? in Proceedings of the 39th conference on Winter simulation, Washington, D.C., United States, 2007, pp. 1223 ? 1231. [73] J. Kleijnen, R. Cheng, and B. Bettonvil, ?Validation of Trace-Driven Simulation Models: More on Bootstrap Tests,? Management Science - INFORMS, vol. 47, no. 11, pp. 1533? 1558, Nov. 2001. [74] W. van Beers and J. Kleijnen, ?Kriging Interpolation in Simulation: A Survey,? in Proceedings of the 36th conference on Winter simulation, Washington, D.C., United States, 2004, pp. 113 ? 121. [75] D. Montgomery, Design and Analysis of Experiments, 6th ed. Arizona State University: Wiley and Sons, Inc., 2005. [76] B. Efron and R. Tibshirani, An Introduction to the Bootstrap. United States: Chapman & Hall, 1993. [77] A. Baykaso?lu and G. Nabil, ?A simulated annealing algorithm for dynamic layout problem,? Computers & Operations Research, vol. 31, no. 2, pp. 313?315, Feb. 2004. [78] P. Bratley, B. L. Fox, and L. E. Schrage, A Guide to Simulation, 2nd ed. Springer, 1987. [79] J. Kleijnen, Handbook of Simulation. New York: Wiley and Sons, Inc., 1998. [80] O. Balci, Methodology and Validation, vol. 19. Simulation Councils, Inc., 1987. [81] W. Kelton, R. Sadowski, and D. Sturrock, Simulation with Arena, Third ed. McGraw-Hill, 2003. [82] A. C. BajPai, I. M. Calus, and J. A. Fairley, Statistical Methods for Engineers and Scientists. Great Britain: John Wiley & Sons, Inc., 1978. 148 [83] J. Kleijnen, Statistical Techniques in Simulation: Part 1, vol. 9. United States: Marcel Dekker, Inc., 1974. [84] J. Kleijnen, Statistical techniques in simulation: Part II. New York: M. Dekker, 1974. [85] B. Nelson, Stochastic Modeling: Analysis & Simulation. United States: McGraw-Hill, 1995. [86] W. Cauer, Theorie der linearen Wechselstromschaltungen, vol. 1. Leipzig: Akad. Verlags- Gesellschaft Becker und Erler, 1941. [87] B. Zeigler, Theory of modelling and simulation. New York: Wiley, 1976. [88] A. Bluman, Probability demystified. New York: McGraw-Hill, 2005. [89] A. Bajpai, Statistical methods for engineers and scientists?: a students? course book. Chichester?;;New York: Wiley, 1978. [90] B. Efron, The jackknife, the bootstrap, and other resampling plans. Philadelphia Pa.: Society for Industrial and Applied Mathematics, 1982. [91] M. Bartlett, The statistical analysis of spatial pattern. London ?;New York: Chapman and Hall?;Wiley, 1975. [92] H. Lambert, S. Vogl, A. Brewster, and K.-P. Dunn, ?On Validating the Modeling and Simulation of a Generic Radar,? MIT Lincoln Laboratory, Technical Report 1134, Feb. 2009. [93] O. Balci, W. Ormsby, J. Car, and S. Saadi, ?Planning for verification, validation, and accreditation of modeling and simulation applications,? in Proceedings of the 32nd conference on Winter simulation, Orlando, Florida, United States, 2000, pp. 829 ? 839. [94] L. Champagne and R. Hill, ?Simulation validation with historic outcomes,? in Proceedings of the 37th conference on Winter simulation, Orlando, Florida, United States, 2005, pp. 1138 ? 1147. [95] J. Chew and C. Sullivan, ?Verification, Validation, and Accreditation in the Life Cycle of Models and Simulations,? in Proceedings of the 32nd conference on Winter simulation, Orlando, Florida, United States, 2000, pp. 813 ? 818. 149 [96] D. Hartley, ?Verification and Validation in Military Simulations,? in Proceedings of the 29th conference on Winter simulation, Atlanta, Georgia, United States, 1997, pp. 925 ? 932. [97] J. Carr, Data Visualization in the Geosciences. Prentice-Hall, Inc., 2002. [98] J.-P. Chiles and P. Delfiner, Geostatistics Modeling Spatial Uncertainty. John Wiley & Sons, Inc., 1999. [99] G. Box, Empirical model-building and response surfaces. New York: Wiley, 1987. [100] D. Birkes, Alternative methods of regression. New York: Wiley, 1993. [101] M. Schimek, Smoothing and regression?: approaches, computation, and application. New York: Wiley, 2000. [102] J. Fox, Regression diagnostics. Newbury Park Calif.: Sage Publications, 1991. [103] S. Karlin, Studies in spline functions and approximation theory. New York: Academic Press, 1976. [104] L. Sakhnovich, Interpolation theory and its applications. Dordrecht?;;Boston?;;London: Kluwer academic publ., 1997. [105] S. Amari, Methods of information geometry. Providence RI: American Mathematical Soc., 2000. [106] F. Chaitin-Chatelin, Spectral approximation of linear operators. New York: Academic Press, 1983. [107] N. Cressie, ?Fitting Variogram Models by Weighted Least Squares,? Journal of the International Association for Mathematical Geology, vol. 17, no. 5, pp. 563?586, Jul. 1985. 150 Appendix Appendix A: Software Code Base Appendix A contains the software application code that was used to automate the DFK process. The appendix is divided up into multiple sections. The sections include the main Kriging processing module, the test planning module, and the associated code that runs behind all the forms that is seen through the GUI while using the software. A description of how to use the software and how to install the software was previously presented in Chapter 7. Appendix A.1: Main Kriging Module '**************************************************************** 'Advanced Design for Kriging 'Created by: Jeremy L. Barnes 'Contact Information: jeremylbarnes@hotmail.com 'Auburn University 'Date: 20 June 2012 'Revision 1.1 'Revision Log: 'Revision 1.1 Added the standard model selection for the variogram 'Revision 1.0 Cleaned up code to remove universal Kriging. A backup copy with the code is maintained 'Revision 0.9 Added the entire test planning module to the software 'Rev 0.8 Fixed Multi Dimension Universal Kriging and Kriging with resolution 'Rev 0.7 Added Lag and Neighborhood 'Rev 0.6 Added Data Recording and Manual Selection of Fitted Variograms 'Rev 0.5 Added Help 'Rev 0.4 Multi Dimension Kriging with code clean up 'Rev 0.3 Multi Dimension Kriging 'Rev 0.2 Single Dimension Kriging with Stochastic bug fix 'Rev 0.1 Single Dimension Kriging 'Beta Module 'This code contains 9 steps. The steps are explained throughout the code. 'No calculations are rounded, only the displayed results. Option Explicit Public resolution As Double Public test_diff_v_counter As Integer Public cnt_reg As Integer Public model_name As String 151 Public ss_res_comp As Double Sub pd_analyze_start() pd_analysis.version.Value = "1.0" Dim fill_var_box As Integer For fill_var_box = 5 To 95 Step 5 pd_analysis.ComboBox1.AddItem fill_var_box Next fill_var_box pd_analysis.ComboBox1.ListIndex = 0 pd_analysis.Show End Sub Sub kriging_start() Dim err_msg_1 If Application.version < "12.0" Then err_msg_1 = MsgBox("The minimum required version of Excel is 2007. The program will now terminate.", vbOKOnly, "Error Handler") Exit Sub End If Kriging_Intro.Show End Sub Sub ShowSemiForm() 'Populate the Kriging Type Box in the GUI SemiForm.KrigType.Clear SemiForm.KrigType.AddItem "Ordinary Kriging" SemiForm.KrigType.Text = SemiForm.KrigType.List(0) 'Populate the Exp_Variogram Calculation Box in the GUI SemiForm.exp_var_select.Clear SemiForm.exp_var_select.AddItem "Standard Model" SemiForm.exp_var_select.AddItem "Traditional with Parameter Tweaking" SemiForm.exp_var_select.AddItem "Linear" SemiForm.exp_var_select.AddItem "Exponential" SemiForm.exp_var_select.AddItem "Gaussian" SemiForm.exp_var_select.AddItem "Spherical" SemiForm.exp_var_select.Text = SemiForm.exp_var_select.List(0) 'Version Dim rev As String rev = "1.1" SemiForm.ver_txt.Text = rev 'Populate the Kriging Assumptions in the GUI SemiForm.KrigAssum.Text = "-Data is spatially correlated in Euclidean space." & Chr(13) & "-Data is normally distributed." & Chr(13) & "-Second Order Stationary." '************************************************************************* 'Step 1 - Utilize interface to gather data '************************************************************************* CheckSolver SemiForm.Show End Sub Sub calc_semiv_model_value(OriginalSampleRange As String) 152 '********Set Variables*************** cnt_reg = 0 test_diff_v_counter = 0 model_name = "" ss_res_comp = 0 '************************************ SemiForm.Frame4.Visible = False SemiForm.Frame5.Visible = False SemiForm.Frame6.Visible = False SemiForm.Frame7.Visible = False SemiForm.Frame8.Visible = False SemiForm.Frame9.Visible = False SemiForm.Frame10.ZOrder (0) SemiForm.Frame10.Visible = True SemiForm.prog_label.Caption = "Initializing" DoEvents 'SemiForm.Repaint 'Determine start time Dim sngStart As Single, sngEnd As Single Dim sngElapsed As Single sngStart = Timer ' Get start time. 'Set max_inp_output array equal to the range of the input/output values Dim max_inp_out_array_size As Integer max_inp_out_array_size = Range(SemiForm.RefEditOriginalSampleRange.Value).Rows.Count '*************** Err Handling***************** Dim err_msg_1 If Range(SemiForm.RefEditOriginalSampleRange.Value).Rows.Count <> Range(SemiForm.Refoutdata.Value).Rows.Count Then err_msg_1 = MsgBox("Your input and output values should have the same number of rows. The program will now terminate.", vbOKOnly, "Error Handler") Exit Sub End If If SemiForm.Predict_Input.Value = "" Then err_msg_1 = MsgBox("You did not select a Kriging Prediction input. The program will now terminate.", vbOKOnly, "Error Handler") Exit Sub End If If Range(SemiForm.RefEditOriginalSampleRange.Value).Columns.Count <> Range(SemiForm.Predict_Input.Value).Columns.Count Then err_msg_1 = MsgBox("Your input dimensions do not match the dimensions of your Kriging estimate. The program will now terminate.", vbOKOnly, "Error Handler") Exit Sub End If If SemiForm.Mult_Pred.Value <> "" Then If Range(SemiForm.Mult_Pred.Value).Rows.Count > 1 Or Range(SemiForm.Mult_Pred.Value).Columns.Count > 1 Then 153 err_msg_1 = MsgBox("Your resolution dimensions exceeded one or you selected more than a single number. The program will now terminate.", vbOKOnly, "Error Handler") Exit Sub End If Dim Ins As Long Ins = InStr(1, SemiForm.RefEditOriginalSampleRange.Value, ":") SemiForm.Predict_Input.Value = Left(SemiForm.RefEditOriginalSampleRange.Value, Ins - 1) End If '***************End Err Handling***************** 'Create a new sheet for data dumps and calculation results Dim NewBook1 As New Worksheet Application.DisplayAlerts = False On Error Resume Next Worksheets("Krig_Dump").Delete Worksheets("Kriging_Regression").Delete Application.DisplayAlerts = True Set NewBook1 = Worksheets.Add NewBook1.Name = "Krig_Dump" '************************************************************************ 'Step 2 - Sort Data 'Parse Input Data before using to Sort Dim Addx, fAddx, outAddx, lAddx, Addx_final As String Dim Wkb As String Dim Wks As String Ins = InStr(1, SemiForm.RefEditOriginalSampleRange.Value, "]") If Ins = 0 Then Wkb = ActiveWorkbook.Name Else Wkb = Mid(SemiForm.RefEditOriginalSampleRange.Value, 2, Ins - 1) End If Ins = InStr(1, SemiForm.RefEditOriginalSampleRange.Value, "!") Wks = Left(SemiForm.RefEditOriginalSampleRange.Value, Ins - 1) Addx = Mid(SemiForm.RefEditOriginalSampleRange.Value, Ins + 1, Len(SemiForm.RefEditOriginalSampleRange.Value) - Ins) outAddx = Mid(SemiForm.Refoutdata.Value, Ins + 1, Len(SemiForm.Refoutdata.Value) - Ins) Dim rng, RngO As Range Dim r, c, co Dim LastRow As String Set rng = Workbooks(Wkb).Worksheets(Wks).Range(Addx) Set RngO = Workbooks(Wkb).Worksheets(Wks).Range(outAddx) r = rng.Row c = rng.Column LastRow = Last(1, RngO) co = RngO.Column Dim intI As Integer Dim cstring, cstringO As String 154 'Limitation is that the column for the input can not be greater than the Z coulumn in excel For intI = 0 To 25 If c - 1 = intI Then cstring = Chr$(97 + intI) Exit For End If Next 'Create the partial input range for use fAddx = cstring & r For intI = 0 To 25 If co - 1 = intI Then cstringO = Chr$(97 + intI) Exit For End If Next 'Create the partial output range lAddx = cstringO & LastRow 'Create the final range for sorting Addx_final = fAddx & ":" & lAddx 'Create the entire range to use for sorting fAddx = fAddx & ":" & cstring & co 'Data has been parsed. Sort Data. This is an excel function. Range(Addx).Select ActiveWorkbook.Worksheets(Wks).Sort.SortFields.Clear ActiveWorkbook.Worksheets(Wks).Sort.SortFields.Add Key:=Range(fAddx), _ SortOn:=xlSortOnValues, Order:=xlAscending, DataOption:=xlSortNormal 'Loop to sort on multiple columns Ins = InStr(1, Addx, ":") If Mid(Addx, Ins + 2) <> UCase(cstring) Then For intI = c To (co - 1) cstring = Chr$(97 + intI) fAddx = cstring & r & ":" & cstring & co ActiveWorkbook.Worksheets(Wks).Sort.SortFields.Add Key:=Range(fAddx), _ SortOn:=xlSortOnValues, Order:=xlAscending, DataOption:=xlSortNormal Next End If 'Perform the final sort after the multiple keys have been generated With ActiveWorkbook.Worksheets(Wks).Sort .SetRange Range(Addx_final) .Header = xlNo .MatchCase = False .Orientation = xlTopToBottom .SortMethod = xlPinYin .Apply End With 'Populate array with sorted data from the selected I/O ranges 155 'Example inp_out_data(1,1) is the first input and inp_out_data(1,2) is the output Dim inp_out_data() As Double ReDim inp_out_data(1 To max_inp_out_array_size, 1 To 2) Dim ii As Integer Dim det_dim As Integer Dim dim_dup_inp As Integer Dim dup_count As Integer 'Determine dimensions det_dim = Range(SemiForm.RefEditOriginalSampleRange.Value).Columns.Count 'Input 'Check for duplicate input ii = 0 Dim refedit_result As Variant refedit_result = SemiForm.RefEditOriginalSampleRange.Value For ii = 1 To (max_inp_out_array_size * det_dim) Step det_dim dup_count = 0 For dim_dup_inp = 0 To (det_dim - 1) If Range(refedit_result).Item(ii + dim_dup_inp) = Range(refedit_result).Item(ii + det_dim + dim_dup_inp) Then dup_count = dup_count + 1 If dup_count = det_dim Then err_msg_1 = MsgBox("You have duplicate inputs. The program will now terminate.", vbOKOnly, "Error Handler") Exit Sub End If Else Exit For End If Next Next 'Populate Output Array. For ii = 1 To max_inp_out_array_size inp_out_data(ii, 2) = Range(SemiForm.Refoutdata.Value).Cells(ii).Value Next '*************Err handling********* Dim dim_err_count As Integer For dim_err_count = 1 To det_dim If Range(SemiForm.Predict_Input.Value).Cells(1, dim_err_count).Value > Range(SemiForm.RefEditOriginalSampleRange.Value).Cells(Range(SemiForm.RefEdit OriginalSampleRange.Value).Rows.Count, dim_err_count).Value Or Range(SemiForm.Predict_Input.Value).Cells(1, dim_err_count).Value < Range(SemiForm.RefEditOriginalSampleRange.Value).Cells(1, dim_err_count).Value Then err_msg_1 = MsgBox("The Kriging Prediction number can not be higher (or lower) than the largest (smallest) input value. The program will exit now.", vbOKOnly, "Error Handler") Exit Sub 156 End If Next '*************End Err handling***** '************************************************************************ 'Step 3 - Calculate distance number of pairs, distance bewteen pairs, and difference in measured value 'Logic developed by Sabahattin (Gohkan) Ozden in php and translated into vba by Jeremy Barnes 'Calculate distances between and group equal distances together Dim a_count As Integer Dim b_count As Integer Dim c_count As Integer Dim flag As Integer Dim max_array_size As Integer 'The plus 1 is added to account for the 0 difference which is always present. max_array_size = (Range(SemiForm.RefEditOriginalSampleRange.Value).Rows.Count * (Range(SemiForm.RefEditOriginalSampleRange.Value).Rows.Count - 1) / 2) + 1 Dim diff_array() As Double ReDim diff_array(1 To 1) 'Index is unique identifier of each set Dim num_diff_array() As Double ReDim num_diff_array(1 To 1) Dim out_array() As Double 'First index is unique identifier of each set. Second index is the different output values inside the set ReDim out_array(1 To max_array_size, 1 To max_inp_out_array_size) 'Three counters for use in determining the differnces in the data set a_count = 0 b_count = 0 c_count = 0 Dim inp_out_counter 'Inital set of diff_array will always be zero diff_array(1) = -1 'For next from 1 to selectd I/O range '************************************* Dim det_dim_s As String 'det_dim_s = Chr(65 + det_dim) & 1 Worksheets("Krig_Dump").Range("A1").Value = "Lag Frequency" Worksheets("Krig_Dump").Range("B1").Value = "FLC Lags (" & det_dim & "D)" Worksheets("Krig_Dump").Range("C1").Value = "RHS Lags" '************************************* Dim dim_count As Integer Dim dim_count_1 As Integer Dim euc_cal As Double For a_count = 1 To max_inp_out_array_size For b_count = 1 To a_count 'c_count has the unique identifier for the differences c_count = 1 157 'Flag is used to determine if a unique set of difference data already exists flag = 0 Do While c_count <= test_diff_v_counter 'If set exists add new input output variable 'Calculate all Euclidean distances by looping through each dimension 'Euclidean calculation sqrt(SUM(1 to n) (psubi - qsubi)^2 euc_cal = euc_values(a_count, b_count, det_dim) 'Floating comparision is impossible. Must convert to string for accurate comparision. If Str(diff_array(c_count)) = Str(euc_cal) Then 'Output difference array data to Krig Dump sheet 'ActiveCell.FormulaR1C1 = diff_array(c_count) 'Store number of pairs num_diff_array(c_count) = num_diff_array(c_count) + 1 'Store output difference to be used in the Exp- semiovariogram '************************************* 'Put Input differences onto krig_dump sheet Worksheets("Krig_Dump").Range("B1").Offset(c_count, 0) = diff_array(c_count) Worksheets("Krig_Dump").Range("A1").Offset(c_count, 0) = num_diff_array(c_count) '************************************* 'Dynamic REDIM If num_diff_array(c_count) >= WorksheetFunction.Max(num_diff_array) Then ReDim Preserve out_array(1 To max_array_size, 1 To WorksheetFunction.Max(num_diff_array)) End If out_array(c_count, num_diff_array(c_count)) = Abs(inp_out_data(a_count, 2) - inp_out_data(b_count, 2)) flag = 1 Exit Do End If c_count = c_count + 1 Loop If flag = 0 Then 'There is no set. Create one. euc_cal = euc_values(a_count, b_count, det_dim) 'Redim for only the used array elements therefore providing efficient code. ReDim Preserve diff_array(1 To c_count) ReDim Preserve num_diff_array(1 To c_count) diff_array(c_count) = euc_cal 'Store number of pairs num_diff_array(c_count) = 1 'Increment set counter test_diff_v_counter = test_diff_v_counter + 1 158 'Store output difference to be used in the Exp-semiovariogram out_array(c_count, num_diff_array(c_count)) = Abs(inp_out_data(a_count, 2) - inp_out_data(b_count, 2)) '************************************* 'Put Input differences onto krig_dump sheet Worksheets("Krig_Dump").Range("B1").Offset(c_count, 0) = diff_array(c_count) Worksheets("Krig_Dump").Range("A1").Offset(c_count, 0) = num_diff_array(c_count) '************************************* End If Next Next 'Memory management. This code Redims the inp_1, inp_2, and out_array therefore eliminating empty elements. 'Idea taken from http://www.xtremevbtalk.com/showthread.php?t=82476 'Counters Dim m As Integer Dim n As Integer Dim iTemp() As Double 'Temporary array ReDim iTemp(1 To test_diff_v_counter, 1 To WorksheetFunction.Max(num_diff_array)) 'Copy original array into temp array: For m = 1 To test_diff_v_counter 'Loop for 1st dimension For n = 1 To WorksheetFunction.Max(num_diff_array) 'Loop for output iTemp(m, n) = out_array(m, n) Next n Next m 'Put values back from temporary array ReDim out_array(1 To test_diff_v_counter, 1 To WorksheetFunction.Max(num_diff_array)) For m = LBound(iTemp, 1) To UBound(iTemp, 1) 'Loop for 1st dimension For n = LBound(iTemp, 2) To UBound(iTemp, 2) 'Loop for 2nd dimension out_array(m, n) = iTemp(m, n) Next n '************************************* 'Put output array differences into the new worksheet Worksheets("Krig_Dump").Range("C1").Offset(m, 0) = out_array(m, 1) '************************************* Next m 'This statements purges the original inp_out array since it is not used in the code anymore therefore making the code more efficient. Erase iTemp '************************************************************************* Dim emp_semiv() As Double Dim summation_var As Double Dim j As Integer SemiForm.prog_label.Caption = "Empirical Variogram" DoEvents 159 If SemiForm.exp_var_select.Text = "Standard Model" Then 'Error Check If test_diff_v_counter >= 100 Then err_msg_1 = MsgBox("The problem contains more than 100 unique lags, which will max out the Excel Solver. Switching to Traditional with Parameter Tweaking.", vbOKOnly, "Error Handler") SemiForm.exp_var_select.Text = "Traditional with Parameter Tweaking" End If End If If SemiForm.exp_var_select.Text = "Traditional with Parameter Tweaking" Then 'Determine lag prior to calculating the experimental semivariogram Dim bin As Double If SemiForm.ComboBoxlag.Value <> "(Default)" Then bin = CDbl(SemiForm.ComboBoxlag.Value) End If 'Step 4 - Calculate exp semi gamma values using 1/2n(h)*SUM(1 to n(h))[var(difference in values)] 'Set the emp_semiv array to the number of elements in the diff_array because for each individual difference there will be 1 exp-semi point ReDim emp_semiv(1 To UBound(diff_array, 1)) 'This variable is the SUM(1 to n(h))[var(difference in values)] part of the exp-semi equation 'Counters 'Start with 2 instead of 1 because 1 is simply the 0 difference array and all semi-variogram values would be calculated as 0. ii = 2 j = 0 Worksheets("Krig_Dump").Range(Chr(68) & ii - 1).Value = "Emp-Var Value" Do det_dim_s = Chr(68) & (ii + 1) 'Reset the summation variable for individual difference summation_var = 0 'SUM(1 to n(h))[var(difference in values)] For j = 1 To num_diff_array(ii) summation_var = summation_var + (out_array(ii, j)) ^ 2 Next 'This is the 1/2n(h)* summation_var part of the equation emp_semiv(ii) = (1 / (2 * num_diff_array(ii))) * summation_var If ii = 2 Then Worksheets("Krig_Dump").Range(Chr(68) & ii).Value = 0 Worksheets("Krig_Dump").Range(det_dim_s).Value = emp_semiv(ii) ii = ii + 1 Loop While ii <= test_diff_v_counter '************Add Variogram Chart***************************************** ActiveSheet.Range("A" & test_diff_v_counter + 3).Select ActiveSheet.Shapes.AddChart.Select 160 ActiveChart.ChartType = xlXYScatterSmoothNoMarkers ActiveChart.SeriesCollection.NewSeries ActiveChart.SeriesCollection(1).Name = "=""Emperical Variogram""" ActiveChart.SeriesCollection(1).XValues = "=Krig_Dump!$B$2:$B$" & test_diff_v_counter + 1 ActiveChart.SeriesCollection(1).Values = "=Krig_Dump!$D$2:$D$" & test_diff_v_counter + 1 '************************************************************************* '************************************************************************* 'Step 5 and 6 - Calculate the fitted variogram model 'This module calculates values of C0, C1, and a for the Spherical, Expon, Linear, and Guassian semi-variogram models and returns recommendations 'based on minimizing the distances between the obeservations and samples. 'C0 - nugget 'C1 - Sil 'a - range 'Calculate C0 (nugget) through linear extrapolation 'Eventually look into more advanced models of extrapolation which means this needs to be calculate after the fitted 'model has been calculated Dim nugget As Double 'This uses the 1st and 2nd actual numbers in the exp_semiv i.e. it excludes the first number which is always 0 'nugget = emp_semiv(3) + ((0 - 2) / (1 - 2)) * (emp_semiv(2) - emp_semiv(3)) nugget = 0 If nugget < 0 Then nugget = 0 'Calculate models values with sil and range beginning from .1 up to the max emperical value with a step of .1 'The max emperical value is used since no fitted model calculation will exceed that (prove this) 'Sil and range variables Dim sill As Double Dim range_1 As Double 'Fitted Model Variables Dim s_gamma() As Double ReDim s_gamma(1 To 1) Dim e_gamma() As Double ReDim e_gamma(1 To 1) Dim l_gamma() As Double ReDim l_gamma(1 To 1) 'Model variables for determining the minimum value between the exp_semivariogram and the fitted models Dim s_diff() As Double ReDim s_diff(1 To 1) Dim e_diff() As Double ReDim e_diff(1 To 1) 161 Dim l_diff() As Double ReDim l_diff(1 To 1) Dim s_diff_sum As Double Dim e_diff_sum As Double Dim l_diff_sum As Double 'Comp gets the min of the least squares fit of each model Dim comp, comp_1 As Double Dim i As Double 'This is the counter where we will loop from 1 to the total number of emp_semi elements Dim pair_incr As Integer 'There variables store the optimal sill and range Dim record_c1 As Double Dim record_a As Double comp_1 = -1 ' Arbitrary number Dim elements As Long elements = UBound(emp_semiv) 'Or this can be set to test_diff_v_counter. Will always be the same. 'Matrix counter Dim mat_count As Long mat_count = 1 SemiForm.prog_label.Caption = "Tweaking Parameters" DoEvents 'Increment sill and range by some step_val and determine the resulting minimal difference between the fitted and the experimental calculations 'The fitted model that results in the lowest minimal difference between it and the experimental semi-var is the selected model to use Dim fit_model As Integer For fit_model = elements To Int(elements * 0.75) Step -1 range_1 = diff_array(fit_model) sill = emp_semiv(fit_model) For pair_incr = 1 To elements 'Standard variogram calculations for spherical, exponential, gamma, and linear models If diff_array(pair_incr) <= range_1 Then s_gamma(mat_count) = nugget + sill * ((1.5 * (diff_array(pair_incr) / range_1)) - (0.5 * (diff_array(pair_incr) / range_1) ^ 3)) Else s_gamma(mat_count) = sill End If e_gamma(mat_count) = nugget + sill * (1 - Exp((-3 * diff_array(pair_incr)) / range_1)) If diff_array(pair_incr) <= range_1 Then l_gamma(mat_count) = nugget + (diff_array(pair_incr) * (sill / range_1)) Else l_gamma(mat_count) = sill End If 162 'Distance between calculated model point and exp_semi s_diff(mat_count) = Abs(s_gamma(mat_count) - emp_semiv(pair_incr)) ^ 2 e_diff(mat_count) = Abs(e_gamma(mat_count) - emp_semiv(pair_incr)) ^ 2 l_diff(mat_count) = Abs(l_gamma(mat_count) - emp_semiv(pair_incr)) ^ 2 s_diff_sum = s_diff_sum + s_diff(mat_count) e_diff_sum = e_diff_sum + e_diff(mat_count) l_diff_sum = l_diff_sum + l_diff(mat_count) Next 'Determine minimal distance comp = Application.WorksheetFunction.Min(s_diff_sum, e_diff_sum, l_diff_sum) 'Whichever fitted model has the lowest minimal distance then add that to the record_c1 and record_a variables If comp_1 > comp Or comp_1 = -1 Then If s_diff_sum < e_diff_sum And s_diff_sum < l_diff_sum Then model_name = "Spherical" record_c1 = sill record_a = range_1 ElseIf e_diff_sum < s_diff_sum And e_diff_sum < l_diff_sum Then model_name = "Exponential" record_c1 = sill record_a = range_1 Else model_name = "Linear" record_c1 = sill record_a = range_1 End If 'This sets the minimum difference number to compare against each time through the loop comp_1 = comp End If 'Reset variables s_diff_sum = 0 e_diff_sum = 0 l_diff_sum = 0 mat_count = mat_count + 1 ReDim Preserve s_gamma(1 To mat_count) ReDim Preserve e_gamma(1 To mat_count) ReDim Preserve l_gamma(1 To mat_count) ReDim Preserve s_diff(1 To mat_count) ReDim Preserve e_diff(1 To mat_count) ReDim Preserve l_diff(1 To mat_count) Next '************************************* 'Puts the exerpimental semivariogram parameters on the Krig_Dump sheet 163 det_dim_s = Chr(69) & 1 Worksheets("Krig_Dump").Range(det_dim_s).Value = "Experimental Semivariogram (Best Estimate: " & model_name & ") C0, C1, a" Worksheets("Krig_Dump").Range(det_dim_s).Offset(1, 0) = nugget Worksheets("Krig_Dump").Range(det_dim_s).Offset(2, 0) = record_c1 Worksheets("Krig_Dump").Range(det_dim_s).Offset(3, 0) = record_a '************************************* '***Sort the lag and exp semi data** Range("A2:D" & test_diff_v_counter + 1).Select ActiveWorkbook.Worksheets("Krig_Dump").Sort.SortFields.Clear ActiveWorkbook.Worksheets("Krig_Dump").Sort.SortFields.Add Key:=Range("B2"), _ SortOn:=xlSortOnValues, Order:=xlAscending, DataOption:=xlSortNormal With ActiveWorkbook.Worksheets("Krig_Dump").Sort .SetRange Range("A2:D" & test_diff_v_counter + 1) .Header = xlNo .MatchCase = False .Orientation = xlTopToBottom .SortMethod = xlPinYin .Apply End With 'Free up memory of unused data Erase s_gamma Erase e_gamma Erase l_gamma Erase s_diff Erase e_diff Erase l_diff Erase emp_semiv ElseIf SemiForm.exp_var_select.Text = "Standard Model" Then 'Regress(inpyrng, [inpxrng], [constant], [labels], [confid], [soutrng], [residuals], [sresiduals], [rplots], [lplots], [routrng], [nplots], [poutrng]) Dim r_sqrd As Double Dim NewBook_1 As New Worksheet Dim past_data_reg As Integer Dim past_d_r_1 As Integer Dim coef_store(1 To 8) As Double Dim coef_store_counter As Integer For coef_store_counter = 1 To 8 coef_store(coef_store_counter) = 0 Next coef_store_counter 'Step 4 - Calculate exp semi gamma values using 1/2n(h)*SUM(1 to n(h))[var(difference in values)] 'Set the emp_semiv array to the number of elements in the diff_array because for each individual difference there will be 1 exp-semi point ReDim emp_semiv(1 To UBound(diff_array, 1)) 'This variable is the SUM(1 to n(h))[var(difference in values)] part of the exp-semi equation 164 'Counters 'Start with 2 instead of 1 because 1 is simply the 0 difference array and all semi-variogram values would be calculated as 0. ii = 2 j = 0 Dim valid_var As Boolean Dim emp_var_count As Integer emp_var_count = 0 Dim reg_stop_crit As Integer Do det_dim_s = Chr(68) & (ii + 1) 'Reset the summation variable for individual difference summation_var = 0 'SUM(1 to n(h))[var(difference in values)] For j = 1 To num_diff_array(ii) summation_var = summation_var + (out_array(ii, j)) ^ 2 Next 'This is the 1/2n(h)* summation_var part of the equation emp_semiv(ii) = (1 / (2 * num_diff_array(ii))) * summation_var If ii = 2 Then Worksheets("Krig_Dump").Range("D2").Value = 0 Worksheets("Krig_Dump").Range(det_dim_s).Value = emp_semiv(ii) ii = ii + 1 Loop While ii <= test_diff_v_counter '*********************************** '***Sort the lag and exp semi data** Range("A2:D" & test_diff_v_counter + 1).Select ActiveWorkbook.Worksheets("Krig_Dump").Sort.SortFields.Clear ActiveWorkbook.Worksheets("Krig_Dump").Sort.SortFields.Add Key:=Range("B2"), _ SortOn:=xlSortOnValues, Order:=xlAscending, DataOption:=xlSortNormal With ActiveWorkbook.Worksheets("Krig_Dump").Sort .SetRange Range("A2:D" & test_diff_v_counter + 1) .Header = xlNo .MatchCase = False .Orientation = xlTopToBottom .SortMethod = xlPinYin .Apply End With Dim vario_check As Integer For vario_check = 1 To test_diff_v_counter If Worksheets("Krig_Dump").Range("D1").Offset(vario_check + 1, 0) >= Worksheets("Krig_Dump").Range("D1").Offset(vario_check, 0) Then emp_var_count = emp_var_count + 1 End If Next vario_check If emp_var_count + 1 = test_diff_v_counter Then valid_var = True Else valid_var = False 165 End If record_c1 = Worksheets("Krig_Dump").Cells(test_diff_v_counter + 1, 4).Value record_a = Worksheets("Krig_Dump").Cells(test_diff_v_counter + 1, 2).Value 'Puts the exerpimental semivariogram calculations on the Krig_Dump sheet det_dim_s = Chr(69) & 1 Worksheets("Krig_Dump").Range("D1").Value = "Exp-Var" Worksheets("Krig_Dump").Range("E1").Value = "C0, C1, a" Worksheets("Krig_Dump").Range(det_dim_s).Offset(1, 0) = nugget record_c1 = WorksheetFunction.Max(emp_semiv) Worksheets("Krig_Dump").Range(det_dim_s).Offset(2, 0) = record_c1 Worksheets("Krig_Dump").Range(det_dim_s).Offset(3, 0) = record_a '************************************* 'Find a reasonable model but not one with r^2 of 100%. GRG will not work properly with a model of r^2 100% If valid_var = True Then 'Set to arbitray number just to enter loop to perform regression once reg_stop_crit = 0 Else reg_stop_crit = cnt_reg + 2 End If Do While test_diff_v_counter <> reg_stop_crit SemiForm.prog_label.Caption = "Regression" DoEvents cnt_reg = cnt_reg + 1 reg_stop_crit = reg_stop_crit + 1 'Perform Regression If cnt_reg = 1 Then Application.Run "ATPVBAEN.XLAM!Regress", Worksheets("Krig_Dump").Range("D2:D" & test_diff_v_counter + 1) _ , Worksheets("Krig_Dump").Range("B2:B" & test_diff_v_counter + 1), False, False, , "Kriging_Regression", False _ , False, False, False, , False r_sqrd = Worksheets("Kriging_Regression").Range("B5").Value If r_sqrd <> 1 Then curve_fit End If 'If the regression iteration didn't produce a better result then stop If ss_res_comp = Worksheets("Kriging_Regression").Range("J2").Value Then For coef_store_counter = 1 To cnt_reg Worksheets("Kriging_Regression").Range("B" & 17 + coef_store_counter).Value = coef_store(coef_store_counter) Next coef_store_counter Exit Do Else 166 For coef_store_counter = 1 To cnt_reg coef_store(coef_store_counter) = Worksheets("Kriging_Regression").Range("B" & 17 + coef_store_counter).Value Next coef_store_counter ss_res_comp = Worksheets("Kriging_Regression").Range("J2").Value End If 'If the model is adequate, and there are a minimum of three regressors, then stop If r_sqrd >= 0.8 And cnt_reg >= 3 Then Exit Do If valid_var = True Then Exit Do ElseIf cnt_reg = 2 Then Application.DisplayAlerts = False On Error Resume Next Worksheets("Kriging_Regression").Delete Application.DisplayAlerts = True Worksheets("Krig_Dump").Columns("C:C").Select Selection.Insert Shift:=xlToRight, CopyOrigin:=xlFormatFromLeftOrAbove Worksheets("Krig_Dump").Range("C1").Value = "Lag11" For past_d_r_1 = 1 To test_diff_v_counter Worksheets("Krig_Dump").Range("C" & past_d_r_1 + 1).Value = (Worksheets("Krig_Dump").Range("B" & past_d_r_1 + 1).Value) ^ 2 Next past_d_r_1 Application.Run "ATPVBAEN.XLAM!Regress", Worksheets("Krig_Dump").Range("E2:E" & test_diff_v_counter + 1) _ , Worksheets("Krig_Dump").Range("B2:C" & test_diff_v_counter + 1), False, False, , "Kriging_Regression", False _ , False, False, False, , False r_sqrd = Worksheets("Kriging_Regression").Range("B5").Value If r_sqrd <> 1 Then curve_fit End If 'If the regression iteration didn't produce a better result then stop If Round(ss_res_comp, 5) <= Round(Worksheets("Kriging_Regression").Range("J2").Value, 5) Then For coef_store_counter = 1 To cnt_reg Worksheets("Kriging_Regression").Range("B" & 17 + coef_store_counter).Value = coef_store(coef_store_counter) Next coef_store_counter Exit Do Else For coef_store_counter = 1 To cnt_reg coef_store(coef_store_counter) = Worksheets("Kriging_Regression").Range("B" & 17 + coef_store_counter).Value Next coef_store_counter ss_res_comp = Worksheets("Kriging_Regression").Range("J2").Value End If 167 'If the model is adequate, and there are a minimum of three regressors, then stop If r_sqrd >= 0.8 And cnt_reg >= 3 Then Exit Do ElseIf cnt_reg = 3 Then Application.DisplayAlerts = False On Error Resume Next Worksheets("Kriging_Regression").Delete Application.DisplayAlerts = True Worksheets("Krig_Dump").Columns("D:D").Select Selection.Insert Shift:=xlToRight, CopyOrigin:=xlFormatFromLeftOrAbove Worksheets("Krig_Dump").Range("D1").Value = "Lag111" For past_d_r_1 = 1 To test_diff_v_counter Worksheets("Krig_Dump").Range("D" & past_d_r_1 + 1).Value = (Worksheets("Krig_Dump").Range("B" & past_d_r_1 + 1).Value) ^ 3 Next past_d_r_1 Application.Run "ATPVBAEN.XLAM!Regress", Worksheets("Krig_Dump").Range("F2:F" & test_diff_v_counter + 1) _ , Worksheets("Krig_Dump").Range("B2:D" & test_diff_v_counter + 1), False, False, , "Kriging_Regression", False _ , False, False, False, , False r_sqrd = Worksheets("Kriging_Regression").Range("B5").Value If r_sqrd <> 1 Then curve_fit End If 'If the regression iteration didn't produce a better result then stop If Round(ss_res_comp, 5) <= Round(Worksheets("Kriging_Regression").Range("J2").Value, 5) Then For coef_store_counter = 1 To cnt_reg Worksheets("Kriging_Regression").Range("B" & 17 + coef_store_counter).Value = coef_store(coef_store_counter) Next coef_store_counter Exit Do Else For coef_store_counter = 1 To cnt_reg coef_store(coef_store_counter) = Worksheets("Kriging_Regression").Range("B" & 17 + coef_store_counter).Value Next coef_store_counter ss_res_comp = Worksheets("Kriging_Regression").Range("J2").Value End If 'If the model is adequate, and there are a minimum of three regressors, then stop If r_sqrd >= 0.8 And cnt_reg >= 3 Then Exit Do ElseIf cnt_reg = 4 Then Application.DisplayAlerts = False On Error Resume Next Worksheets("Kriging_Regression").Delete Application.DisplayAlerts = True 168 Worksheets("Krig_Dump").Columns("E:E").Select Selection.Insert Shift:=xlToRight, CopyOrigin:=xlFormatFromLeftOrAbove Worksheets("Krig_Dump").Range("E1").Value = "Lag1111" For past_d_r_1 = 1 To test_diff_v_counter Worksheets("Krig_Dump").Range("E" & past_d_r_1 + 1).Value = (Worksheets("Krig_Dump").Range("B" & past_d_r_1 + 1).Value) ^ 4 Next past_d_r_1 Application.Run "ATPVBAEN.XLAM!Regress", Worksheets("Krig_Dump").Range("G2:G" & test_diff_v_counter + 1) _ , Worksheets("Krig_Dump").Range("B2:E" & test_diff_v_counter + 1), False, False, , "Kriging_Regression", False _ , False, False, False, , False r_sqrd = Worksheets("Kriging_Regression").Range("B5").Value If r_sqrd <> 1 Then curve_fit End If 'If the regression iteration didn't produce a better result then stop If Round(ss_res_comp, 5) <= Round(Worksheets("Kriging_Regression").Range("J2").Value, 5) Then For coef_store_counter = 1 To cnt_reg Worksheets("Kriging_Regression").Range("B" & 17 + coef_store_counter).Value = coef_store(coef_store_counter) Next coef_store_counter Exit Do Else For coef_store_counter = 1 To cnt_reg coef_store(coef_store_counter) = Worksheets("Kriging_Regression").Range("B" & 17 + coef_store_counter).Value Next coef_store_counter ss_res_comp = Worksheets("Kriging_Regression").Range("J2").Value End If 'If the model is adequate, and there are a minimum of three regressors, then stop If r_sqrd >= 0.8 And cnt_reg >= 3 Then Exit Do ElseIf cnt_reg = 5 Then Application.DisplayAlerts = False On Error Resume Next Worksheets("Kriging_Regression").Delete Application.DisplayAlerts = True Worksheets("Krig_Dump").Columns("F:F").Select Selection.Insert Shift:=xlToRight, CopyOrigin:=xlFormatFromLeftOrAbove Worksheets("Krig_Dump").Range("F1").Value = "Lag11111" For past_d_r_1 = 1 To test_diff_v_counter Worksheets("Krig_Dump").Range("F" & past_d_r_1 + 1).Value = (Worksheets("Krig_Dump").Range("B" & past_d_r_1 + 1).Value) ^ 5 Next past_d_r_1 169 Application.Run "ATPVBAEN.XLAM!Regress", Worksheets("Krig_Dump").Range("H2:H" & test_diff_v_counter + 1) _ , Worksheets("Krig_Dump").Range("B2:F" & test_diff_v_counter + 1), False, False, , "Kriging_Regression", False _ , False, False, False, , False r_sqrd = Worksheets("Kriging_Regression").Range("B5").Value If r_sqrd <> 1 Then curve_fit End If 'If the regression iteration didn't produce a better result then stop If Round(ss_res_comp, 5) <= Round(Worksheets("Kriging_Regression").Range("J2").Value, 5) Then For coef_store_counter = 1 To cnt_reg Worksheets("Kriging_Regression").Range("B" & 17 + coef_store_counter).Value = coef_store(coef_store_counter) Next coef_store_counter Exit Do Else For coef_store_counter = 1 To cnt_reg coef_store(coef_store_counter) = Worksheets("Kriging_Regression").Range("B" & 17 + coef_store_counter).Value Next coef_store_counter ss_res_comp = Worksheets("Kriging_Regression").Range("J2").Value End If 'If the model is adequate, and there are a minimum of three regressors, then stop If r_sqrd >= 0.8 And cnt_reg >= 3 Then Exit Do ElseIf cnt_reg = 6 Then Application.DisplayAlerts = False On Error Resume Next Worksheets("Kriging_Regression").Delete Application.DisplayAlerts = True Worksheets("Krig_Dump").Columns("G:G").Select Selection.Insert Shift:=xlToRight, CopyOrigin:=xlFormatFromLeftOrAbove Worksheets("Krig_Dump").Range("G1").Value = "Lag111111" For past_d_r_1 = 1 To test_diff_v_counter Worksheets("Krig_Dump").Range("G" & past_d_r_1 + 1).Value = (Worksheets("Krig_Dump").Range("B" & past_d_r_1 + 1).Value) ^ 6 Next past_d_r_1 Application.Run "ATPVBAEN.XLAM!Regress", Worksheets("Krig_Dump").Range("I2:I" & test_diff_v_counter + 1) _ , Worksheets("Krig_Dump").Range("B2:G" & test_diff_v_counter + 1), False, False, , "Kriging_Regression", False _ , False, False, False, , False r_sqrd = Worksheets("Kriging_Regression").Range("B5").Value If r_sqrd <> 1 Then curve_fit 170 End If 'If the regression iteration didn't produce a better result then stop If Round(ss_res_comp, 5) <= Round(Worksheets("Kriging_Regression").Range("J2").Value, 5) Then For coef_store_counter = 1 To cnt_reg Worksheets("Kriging_Regression").Range("B" & 17 + coef_store_counter).Value = coef_store(coef_store_counter) Next coef_store_counter Exit Do Else For coef_store_counter = 1 To cnt_reg coef_store(coef_store_counter) = Worksheets("Kriging_Regression").Range("B" & 17 + coef_store_counter).Value Next coef_store_counter ss_res_comp = Worksheets("Kriging_Regression").Range("J2").Value End If 'If the model is adequate, and there are a minimum of three regressors, then stop If r_sqrd >= 0.8 And cnt_reg >= 3 Then Exit Do ElseIf cnt_reg = 7 Then Application.DisplayAlerts = False On Error Resume Next Worksheets("Kriging_Regression").Delete Application.DisplayAlerts = True Worksheets("Krig_Dump").Columns("H:H").Select Selection.Insert Shift:=xlToRight, CopyOrigin:=xlFormatFromLeftOrAbove Worksheets("Krig_Dump").Range("H1").Value = "Lag1111111" For past_d_r_1 = 1 To test_diff_v_counter Worksheets("Krig_Dump").Range("H" & past_d_r_1 + 1).Value = (Worksheets("Krig_Dump").Range("B" & past_d_r_1 + 1).Value) ^ 7 Next past_d_r_1 Application.Run "ATPVBAEN.XLAM!Regress", Worksheets("Krig_Dump").Range("J2:J" & test_diff_v_counter + 1) _ , Worksheets("Krig_Dump").Range("B2:H" & test_diff_v_counter + 1), False, False, , "Kriging_Regression", False _ , False, False, False, , False r_sqrd = Worksheets("Kriging_Regression").Range("B5").Value If r_sqrd <> 1 Then curve_fit End If 'If the regression iteration didn't produce a better result then stop If Round(ss_res_comp, 5) <= Round(Worksheets("Kriging_Regression").Range("J2").Value, 5) Then For coef_store_counter = 1 To cnt_reg Worksheets("Kriging_Regression").Range("B" & 17 + coef_store_counter).Value = coef_store(coef_store_counter) 171 Next coef_store_counter Exit Do Else For coef_store_counter = 1 To cnt_reg coef_store(coef_store_counter) = Worksheets("Kriging_Regression").Range("B" & 17 + coef_store_counter).Value Next coef_store_counter ss_res_comp = Worksheets("Kriging_Regression").Range("J2").Value End If 'If the model is adequate, and there are a minimum of three regressors, then stop If r_sqrd >= 0.8 And cnt_reg >= 3 Then Exit Do ElseIf cnt_reg = 8 Then Application.DisplayAlerts = False On Error Resume Next Worksheets("Kriging_Regression").Delete Application.DisplayAlerts = True Worksheets("Krig_Dump").Columns("I:I").Select Selection.Insert Shift:=xlToRight, CopyOrigin:=xlFormatFromLeftOrAbove Worksheets("Krig_Dump").Range("I1").Value = "Lag1111111" For past_d_r_1 = 1 To test_diff_v_counter Worksheets("Krig_Dump").Range("I" & past_d_r_1 + 1).Value = (Worksheets("Krig_Dump").Range("B" & past_d_r_1 + 1).Value) ^ 8 Next past_d_r_1 Application.Run "ATPVBAEN.XLAM!Regress", Worksheets("Krig_Dump").Range("K2:K" & test_diff_v_counter + 1) _ , Worksheets("Krig_Dump").Range("B2:I" & test_diff_v_counter + 1), False, False, , "Kriging_Regression", False _ , False, False, False, , False r_sqrd = Worksheets("Kriging_Regression").Range("B5").Value If r_sqrd <> 1 Then curve_fit End If 'If the regression iteration didn't produce a better result then stop If Round(ss_res_comp, 5) <= Round(Worksheets("Kriging_Regression").Range("J2").Value, 5) Then For coef_store_counter = 1 To cnt_reg Worksheets("Kriging_Regression").Range("B" & 17 + coef_store_counter).Value = coef_store(coef_store_counter) Next coef_store_counter Exit Do Else For coef_store_counter = 1 To cnt_reg coef_store(coef_store_counter) = Worksheets("Kriging_Regression").Range("B" & 17 + coef_store_counter).Value Next coef_store_counter 172 ss_res_comp = Worksheets("Kriging_Regression").Range("J2").Value End If 'If the model is adequate, and there are a minimum of three regressors, then stop If r_sqrd >= 0.8 And cnt_reg >= 3 Then Exit Do End If If cnt_reg > 8 Then err_msg_1 = MsgBox("The regression model could not find an acceptable solution. Please choose another variogram option and run the program again.", vbOKOnly, "Error Handler") Exit Sub End If Loop '************Add Variogram Chart***************************************** Dim chart_title As String Dim chart_title_count As Integer For chart_title_count = 1 To cnt_reg If chart_title_count = cnt_reg Then chart_title = chart_title & "(" & Round(Worksheets("Kriging_Regression").Range("B" & 17 + chart_title_count).Value, 4) & ")*Lag^" & chart_title_count Else chart_title = chart_title & "(" & Round(Worksheets("Kriging_Regression").Range("B" & 17 + chart_title_count).Value, 4) & ")*Lag^" & chart_title_count & " + " End If Next chart_title_count Application.Sheets("Krig_Dump").Activate Range("A" & test_diff_v_counter + 3).Select ActiveSheet.Shapes.AddChart.Select ActiveChart.ChartType = xlXYScatterSmoothNoMarkers ActiveChart.HasTitle = True ActiveChart.ChartTitle.Text = chart_title ActiveChart.SeriesCollection.NewSeries ActiveChart.SeriesCollection(1).Name = "=""Emperical Variogram""" ActiveChart.SeriesCollection(1).XValues = "=Krig_Dump!$B$2:$B$" & test_diff_v_counter + 1 ActiveChart.SeriesCollection(1).Values = "=Krig_Dump!" & Chr(67 + cnt_reg) & "$2:$" & Chr(67 + cnt_reg) & "$" & test_diff_v_counter + 1 ActiveChart.SeriesCollection.NewSeries ActiveChart.SeriesCollection(2).Name = "=""Standard Model""" ActiveChart.SeriesCollection(2).XValues = "=Krig_Dump!$B$2:$B$" & test_diff_v_counter + 1 ActiveChart.SeriesCollection(2).Values = "=Kriging_Regression!$B$28:$B$" & 27 + test_diff_v_counter '************************************************************************* Else 173 SemiForm.prog_label.Caption = "User Defined Variogram" DoEvents 'SemiForm.Repaint 'Error Handling If SemiForm.nug.Value = "" Or CDbl(SemiForm.nug.Value) < 0 Then err_msg_1 = MsgBox("The nugget value can not be null or below zero if you have selected a specific experimental semivariogram. The program will exit now.", vbOKOnly, "Error Handler") Exit Sub End If If SemiForm.sil.Value = "" Or CDbl(SemiForm.sil.Value) < 0 Then err_msg_1 = MsgBox("The sill value can not be null or below zero if you have selected a specific experimental semivariogram. The program will exit now.", vbOKOnly, "Error Handler") Exit Sub End If If SemiForm.ran.Value = "" Or CDbl(SemiForm.ran.Value) < 0 Then err_msg_1 = MsgBox("The range value can not be null or below zero if you have selected a specific experimental semivariogram. The program will exit now.", vbOKOnly, "Error Handler") Exit Sub End If 'End Error Handling 'This data is gathered from the user input if the preferred calculation methods are not used. model_name = SemiForm.exp_var_select.Text nugget = CDbl(SemiForm.nug.Value) record_c1 = CDbl(SemiForm.sil.Value) record_a = CDbl(SemiForm.ran.Value) '************************************* 'det_dim_s = Chr(68) & 1 Worksheets("Krig_Dump").Range(Chr(69) & 1).Value = "Selected Semivariogram (" & model_name & ") Values (c0, c1, a)" Worksheets("Krig_Dump").Range(Chr(69) & 1).Offset(1, 1) = nugget Worksheets("Krig_Dump").Range(Chr(69) & 1).Offset(2, 1) = record_c1 Worksheets("Krig_Dump").Range(Chr(69) & 1).Offset(3, 1) = record_a '************************************* End If 'Dim for 1 additional element to take into account the lagrangian multiplier Dim Final_Gamma() As Double ReDim Final_Gamma(1 To (max_inp_out_array_size + 1), 1 To (max_inp_out_array_size + 1)) 'Calculate final gamma values Dim inp_inc_1 As Integer Dim inp_inc_2 As Integer Dim matrix_start_position As Integer Dim matrix_value As Double Dim fin_gamma_reg As Double matrix_start_position = 2 174 Dim msg As String For inp_inc_2 = 1 To max_inp_out_array_size 'Lagrangian multiplier Final_Gamma(inp_inc_2, (max_inp_out_array_size + 1)) = 1 Final_Gamma((max_inp_out_array_size + 1), inp_inc_2) = 1 Final_Gamma(inp_inc_2, inp_inc_2) = 0 For inp_inc_1 = matrix_start_position To (max_inp_out_array_size) msg = "" matrix_value = euc_values(inp_inc_2, inp_inc_1, det_dim) If model_name = "Spherical" Then 'Spherical calculation If matrix_value <= record_a Then Final_Gamma(inp_inc_2, inp_inc_1) = nugget + record_c1 * ((1.5 * (matrix_value / record_a)) - (0.5 * (matrix_value) ^ 3)) Else Final_Gamma(inp_inc_2, inp_inc_1) = record_c1 End If ElseIf model_name = "Exponential" Then 'Exponential calculation Final_Gamma(inp_inc_2, inp_inc_1) = nugget + record_c1 * (1 - Exp((-3 * matrix_value) / record_a)) ElseIf model_name = "Gaussian" Then 'Gaussian Calculation If matrix_value <= record_a Then Final_Gamma(inp_inc_2, inp_inc_1) = nugget + record_c1 * (1 - Exp((-3 * (matrix_value ^ 2)) / (record_a ^ 2))) Else Final_Gamma(inp_inc_2, inp_inc_1) = record_c1 End If ElseIf model_name = "Linear" Then 'Linear Calculation If matrix_value <= record_a Then Final_Gamma(inp_inc_2, inp_inc_1) = nugget + (matrix_value * (record_c1 / record_a)) Else Final_Gamma(inp_inc_2, inp_inc_1) = record_c1 End If Else 'This option is if the standard model is used 'The model depends on the order of the regression model For fin_gamma_reg = 1 To cnt_reg Final_Gamma(inp_inc_2, inp_inc_1) = Final_Gamma(inp_inc_2, inp_inc_1) + ((matrix_value ^ fin_gamma_reg) * Worksheets("Kriging_Regression").Range("B" & 17 + fin_gamma_reg).Value) Next fin_gamma_reg End If Final_Gamma(inp_inc_1, inp_inc_2) = Final_Gamma(inp_inc_2, inp_inc_1) Next matrix_start_position = matrix_start_position + 1 175 Next 'Lagrangian multiplier Final_Gamma((max_inp_out_array_size + 1), (max_inp_out_array_size + 1)) = 0 Dim f_msg() As String 'Use this to populate a text box in the results GUI ReDim f_msg(1 To max_inp_out_array_size) 'ReDim f_msg(1 To UBound(diff_array, 1)) 'This variable is to aide in displaying results '********************************** det_dim_s = Chr(70 + cnt_reg) & 1 Worksheets("Krig_Dump").Range(det_dim_s).Value = "Final LHS Gamma Calculations" Dim ofset_cal As Integer '********************************** Dim f_msg_inp As String For inp_inc_2 = 1 To max_inp_out_array_size msg = "" For inp_inc_1 = 1 To max_inp_out_array_size '********************************** ofset_cal = ofset_cal + 1 '********************************** 'Populate output to user If inp_inc_1 < max_inp_out_array_size Then msg = msg & Round(Final_Gamma(inp_inc_2, inp_inc_1), 3) & ", " Else msg = msg & Round(Final_Gamma(inp_inc_2, inp_inc_1), 3) End If '************************************* If inp_inc_1 = max_inp_out_array_size Then Worksheets("Krig_Dump").Range(det_dim_s).Offset(ofset_cal, 0) = Final_Gamma(inp_inc_2, inp_inc_1) ofset_cal = ofset_cal + 1 Worksheets("Krig_Dump").Range(det_dim_s).Offset(ofset_cal, 0) = "" Else Worksheets("Krig_Dump").Range(det_dim_s).Offset(ofset_cal, 0) = Final_Gamma(inp_inc_2, inp_inc_1) End If '************************************* Next 'Fill the final gamma GUI text box f_msg_inp = "" For dim_count = 1 To det_dim 'If (det_dim = 1 And inp_inc_2 <> max_inp_out_array_size) Or (det_dim <> 1 And inp_inc_2 = max_inp_out_array_size) Then If (det_dim = 1) Or (det_dim <> 1 And dim_count = det_dim) Then f_msg_inp = f_msg_inp & Range(SemiForm.RefEditOriginalSampleRange.Value).Cells(inp_inc_2, dim_count).Value 176 'ElseIf det_dim <> 1 And inp_inc_2 <> max_inp_out_array_size Then ElseIf (det_dim <> 1) And (dim_count <> det_dim) Then f_msg_inp = f_msg_inp & Range(SemiForm.RefEditOriginalSampleRange.Value).Cells(inp_inc_2, dim_count).Value & "," End If Next f_msg(inp_inc_2) = "Input " & f_msg_inp & ":: " & msg & Chr(13) & Chr(13) 'Populate Gamma Coeff results box Results_Form.Results_1.Text = Results_Form.Results_1.Text & f_msg(inp_inc_2) Next 'Populate fitted model details If model_name <> "Standard Model" Then Results_Form.Results_2.Text = "Your fitted model is " & model_name & "." & Chr(13) & Chr(13) & "Co is = " & Round(nugget, 3) & ", c1 is = " & Round(record_c1, 3) & ", and a is = " & Round(record_a, 3) & "." _ & Chr(13) & Chr(13) & "The resulting minimal difference in samples versues observations was " & Round(comp_1, 3) & "." Else Results_Form.Results_2.Text = "Your fitted model is " & model_name & "." & Chr(13) & "The resulting minimal difference in samples versues observations was " & Worksheets("Kriging_Regression").Range("J2").Value & "." End If '************************************************************************* 'Step 7 and 8 - Construct the system of equations 'Perform Matrix Transformation 'This variable is a flag to indicate whether to perform multiple kriging or a single point prediction. More less a dummy flag. Dim dimension_count As Integer Dim ArrInv() As Variant Dim ArrAns() As Variant Dim multi_dim_exit As Integer Dim msg1 As String If SemiForm.Mult_Pred.Value = "" Then Dim krig_inp_point As Double Else Dim krig_inp_point_2() As Double ReDim krig_inp_point_2(1 To det_dim) End If Dim krig_inp_point_1 As Double Dim point_est() As Double Dim matrix_inv As Integer Dim matrix_inv_1 As Integer Dim matrix_count As Integer Dim Z As Double Dim krig_count As Integer Dim krig_var As Double 177 Dim krig_std As Double Dim krig_conf As Double Dim krig_conf_l As Double Dim krig_conf_h As Double Dim arr_count As Integer Dim msg_2 As String Dim pred_value_lp As Integer Dim krig_result_point As Variant Dim MD_array_counter As Integer Dim col As Integer Dim reset_dim As Integer SemiForm.prog_label.Caption = "Weight Calculations" DoEvents 'RHS gamma values MD_array_counter = det_dim dimension_count = 1 resolution = 0 'determine the minimum and maximum number in each dimension to construct the MD stopping points If SemiForm.Mult_Pred.Value <> "" And det_dim > 1 Then Dim max_dim_var() As Integer Dim min_dim_var() As Integer ReDim max_dim_var(1 To det_dim) ReDim min_dim_var(1 To det_dim) For col = 1 To det_dim max_dim_var(col) = Application.WorksheetFunction.Max(Range(SemiForm.RefEditOriginalSampleRange.V alue).Columns(col)) min_dim_var(col) = Application.WorksheetFunction.Min(Range(SemiForm.RefEditOriginalSampleRange.V alue).Columns(col)) If col <> det_dim Then krig_inp_point_2(col) = Range(SemiForm.Predict_Input.Value).Cells(1, col).Value Else krig_inp_point_2(col) = Range(SemiForm.Predict_Input.Value).Cells(1, col).Value - Range(SemiForm.Mult_Pred.Value) End If Next End If Dim disp_count As Integer 'This loops based until all the values from the original point plus the resolution value are covered Dim get_weight As Interior Dim max_min_chk As Integer Dim MD_exit_chk As Integer Dim MD_array_counter_1 As Integer Dim MD_array_counter_2 As Integer MD_array_counter_2 = 1 178 Dim dimen_offset As Integer dimen_offset = 1 get_weight = 0 Do get_weight = get_weight + 1 'Need to calculate the RHS gamma values 'This is the input variable selected by the user 'SemiForm.Mult_Pred.Value = "" means single point kriging prediction else then some resoltion was given. If SemiForm.Mult_Pred.Value <> "" Then 'det_dim > 1 means there are more than 1 dimension to the problem If det_dim = 1 Then krig_inp_point = Range(SemiForm.Predict_Input.Value) + resolution '***Exit loop once the prediction point exceeds the maximum input value i.e. to prevent extrapolation*** If krig_inp_point > Range(SemiForm.RefEditOriginalSampleRange.Value).Cells(Range(SemiForm.RefEdit OriginalSampleRange.Value).Rows.Count, 1) Then Exit Do Else 'Need to increment each bound until the max, then reset it to the min value and start incrementing the next dimension 'Create a counter as we go from one dimension to the next. Once the counter exceeds the last dimension then exit the loop 'If krig_inp_point_2(MD_array_counter) + Range(SemiForm.Mult_Pred.Value) > max_dim_var(MD_array_counter) And dimension_count = 1 Then MD_array_counter_1 = 0 For MD_exit_chk = 1 To det_dim If krig_inp_point_2(MD_exit_chk) >= Range(SemiForm.Predict_Input.Value).Cells(Range(SemiForm.RefEditOriginalSampl eRange.Value).Rows.Count, MD_exit_chk).Value Then MD_array_counter_1 = MD_array_counter_1 + 1 End If If MD_array_counter_1 = det_dim Then Exit Do End If Next MD_exit_chk 'ElseIf krig_inp_point_2(MD_array_counter) + Range(SemiForm.Mult_Pred.Value) > max_dim_var(MD_array_counter) And dimension_count >= 1 Then If krig_inp_point_2(MD_array_counter) + Range(SemiForm.Mult_Pred.Value) > max_dim_var(MD_array_counter) And dimension_count >= 1 Then 'The prior statement captures when you hit a maximum point krig_inp_point_2(MD_array_counter) = min_dim_var(MD_array_counter) For max_min_chk = (det_dim - 1) To 1 Step -1 179 krig_inp_point_2(max_min_chk) = krig_inp_point_2(max_min_chk) + Range(SemiForm.Mult_Pred.Value) If krig_inp_point_2(max_min_chk) > max_dim_var(max_min_chk) Then krig_inp_point_2(max_min_chk) = min_dim_var(max_min_chk) Else Exit For End If Next max_min_chk Else krig_inp_point_2(MD_array_counter) = krig_inp_point_2(MD_array_counter) + Range(SemiForm.Mult_Pred.Value) End If End If End If ReDim point_est(1 To (max_inp_out_array_size + 1)) '************************************* det_dim_s = Chr(71 + cnt_reg) & 1 Worksheets("Krig_Dump").Range(det_dim_s).Value = "Final Right Hand Side Gamma Calculations" '************************************* 'Loop through each input value and calculate the distance between the input point and the kriging point along with the RHS gamma calculation For pair_incr = 1 To max_inp_out_array_size 'If det_dim = 1 Then If SemiForm.Mult_Pred.Value = "" Then krig_inp_point_1 = euc_values_rhs(pair_incr, 1, det_dim) ElseIf SemiForm.Mult_Pred.Value <> "" And det_dim = 1 Then krig_inp_point_1 = euc_values_rhs(pair_incr, 1, det_dim) Else krig_inp_point_1 = euc_values_rhs_MD_w_RES(pair_incr, 1, det_dim, krig_inp_point_2) End If If model_name = "Spherical" Then 'RHS gamma calculations If krig_inp_point_1 = 0 Then point_est(pair_incr) = 0 Else If Abs(krig_inp_point_1) <= record_a Then point_est(pair_incr) = nugget + record_c1 * ((1.5 * (Abs(krig_inp_point_1) / record_a)) - (0.5 * (Abs(krig_inp_point_1) / record_a) ^ 3)) Else point_est(pair_incr) = record_c1 End If End If ElseIf model_name = "Exponential" Then 'RHS gamma calculations If krig_inp_point_1 = 0 Then 180 point_est(pair_incr) = 0 Else point_est(pair_incr) = nugget + record_c1 * (1 - Exp((-3 * Abs(krig_inp_point_1)) / record_a)) End If ElseIf model_name = "Gaussian" Then 'RHS gamma calculations If krig_inp_point_1 = 0 Then point_est(pair_incr) = 0 Else If Abs(krig_inp_point_1) <= record_a Then point_est(pair_incr) = nugget + record_c1 * (1 - Exp(-3 * (krig_inp_point_1 ^ 2) / (record_a ^ 2))) Else point_est(pair_incr) = record_c1 End If End If ElseIf model_name = "Linear" Then 'RHS gamma calculations If krig_inp_point_1 = 0 Then point_est(pair_incr) = 0 Else If Abs(krig_inp_point_1) <= record_a Then point_est(pair_incr) = nugget + Abs(krig_inp_point_1) * (record_c1 / record_a) Else point_est(pair_incr) = record_c1 End If End If Else 'RHS gamma calculations If krig_inp_point_1 = 0 Then point_est(pair_incr) = 0 Else For fin_gamma_reg = 1 To cnt_reg point_est(pair_incr) = point_est(pair_incr) + (Abs(krig_inp_point_1 ^ fin_gamma_reg) * Worksheets("Kriging_Regression").Range("B" & 17 + fin_gamma_reg).Value) Next fin_gamma_reg End If End If 'Put RHS Gamma Values in Spreadsheet '************************************* If SemiForm.Mult_Pred.Value = "" Then Worksheets("Krig_Dump").Range(det_dim_s).Offset(pair_incr, 0) = point_est(pair_incr) Else Worksheets("Krig_Dump").Range(det_dim_s).Offset(1, 0) = "N/A" End If '************************************* 181 If pair_incr <> max_inp_out_array_size Then msg1 = msg1 & Round(point_est(pair_incr), 3) & Chr(13) Else msg1 = msg1 & Round(point_est(pair_incr), 3) & Chr(13) & Chr(13) End If Next 'RHS Lagrangian multiplier point_est(max_inp_out_array_size + 1) = 1 'Populate RHS gamma calculation results Results_Form.Results_4.Text = msg1 '************************************* Dim det_dim_s_weights As String det_dim_s_weights = Chr(72 + cnt_reg) & 1 Worksheets("Krig_Dump").Range(det_dim_s_weights).Value = "Final Weights" 'Calculate weights 'Only need to get the weights once If get_weight = 1 Then ArrInv() = Application.WorksheetFunction.MInverse(Final_Gamma) If ArrInv(1) = "" Then ArrInv = Excel.Application.WorksheetFunction.MInverse(Final_Gamma) End If End If ArrAns = Excel.Application.WorksheetFunction.MMult(ArrInv, Excel.Application.WorksheetFunction.Transpose(point_est)) If ArrAns(1) = "" Then ArrAns = Excel.Application.WorksheetFunction.MMult(ArrInv, Excel.Application.WorksheetFunction.Transpose(point_est)) End If '************************************* 'Put Gamma Inversion Values in Krig_Dump Spreadsheet If SemiForm.Mult_Pred.Value = "" Then Worksheets("Krig_Dump").Range(det_dim_s_weights).Offset(1, 0).Resize(UBound(ArrAns), 1) = ArrAns Else Worksheets("Krig_Dump").Range(det_dim_s_weights).Offset(1, 0) = "N/A" Worksheets("Krig_Dump").Range(det_dim_s).Offset(1, 0) = "N/A" End If '************************************** arr_count = 1 'Populate the weights to be displayed to the user Do If arr_count <> UBound(ArrAns, 1) Then msg_2 = msg_2 & "Lamda (" & arr_count & "): " & Round(ArrAns(arr_count, 1), 3) & Chr(13) Else 182 msg_2 = msg_2 & "Mu (1): " & Round(ArrAns(arr_count, 1), 3) & Chr(13) & Chr(13) End If arr_count = arr_count + 1 Loop While arr_count <= UBound(ArrAns, 1) Results_Form.Weight.Text = msg_2 'http://microsoft.allfaq.org/forums/t/155165.aspx '************************************************************************* 'Step 9 is to perform Ordinay Kriging to estimate point 'Reset the prediction value to 0 each time. 'This variable is called Z but it actually represents the t distribution, not the z-score of the standard normal Z = 0 krig_var = 0 krig_std = 0 'Perform the kriging algorithm. Z* = SUM(weights*original output values) For krig_count = 1 To max_inp_out_array_size Z = Z + (ArrAns(krig_count, 1) * inp_out_data(krig_count, 2)) Next For krig_count = 1 To (max_inp_out_array_size + 1) krig_var = krig_var + ArrAns(krig_count, 1) * point_est(krig_count) Next If krig_var < 0 Then krig_var = 0 krig_std = Sqr(Abs(krig_var)) krig_conf = 0 krig_conf = Application.WorksheetFunction.TInv((1 - CDbl(SemiForm.CI.Value)), max_inp_out_array_size - 1) * krig_std / (Sqr(max_inp_out_array_size)) krig_conf_l = Z - krig_conf krig_conf_h = Z + krig_conf '************************************* disp_count = disp_count + 1 det_dim_s = Chr(73 + cnt_reg) & disp_count If disp_count = 1 Then Worksheets("Krig_Dump").Range(det_dim_s).Value = "Kriging Prediction" End If Worksheets("Krig_Dump").Range(det_dim_s).Offset(dimension_count, 0) = Z det_dim_s = Chr(74 + cnt_reg) & disp_count If disp_count = 1 Then Worksheets("Krig_Dump").Range(det_dim_s).Value = "Prediction Error" End If Worksheets("Krig_Dump").Range(det_dim_s).Offset(dimension_count, 0) = krig_var det_dim_s = Chr(75 + cnt_reg) & disp_count 183 If disp_count = 1 Then Worksheets("Krig_Dump").Range(det_dim_s).Value = "Error Standard Deviation" End If Worksheets("Krig_Dump").Range(det_dim_s).Offset(dimension_count, 0) = krig_std det_dim_s = Chr(76 + cnt_reg) & disp_count If disp_count = 1 Then Worksheets("Krig_Dump").Range(det_dim_s).Value = "FLC" End If Dim FLC_wks_outpt As String Dim FLC_dim_count As Integer If SemiForm.Mult_Pred.Value = "" Then FLC_wks_outpt = "" If det_dim = 1 Then FLC_wks_outpt = Range(SemiForm.Predict_Input.Value) Else For FLC_dim_count = 1 To det_dim If FLC_dim_count <> det_dim Then FLC_wks_outpt = FLC_wks_outpt & Range(SemiForm.Predict_Input.Value).Cells(1, FLC_dim_count).Value & "," Else FLC_wks_outpt = FLC_wks_outpt & Range(SemiForm.Predict_Input.Value).Cells(1, FLC_dim_count).Value End If Next End If Else FLC_wks_outpt = "" If det_dim = 1 Then FLC_wks_outpt = krig_inp_point Else For FLC_dim_count = 1 To det_dim If FLC_dim_count <> det_dim Then FLC_wks_outpt = FLC_wks_outpt & krig_inp_point_2(FLC_dim_count) & "," Else FLC_wks_outpt = FLC_wks_outpt & krig_inp_point_2(FLC_dim_count) End If Next End If End If Worksheets("Krig_Dump").Range(det_dim_s).Offset(dimension_count, 0) = FLC_wks_outpt '************************************** '************************************************************************* 'Populate Kriging results box. 184 'SemiForm.Mult_Pred.Value = "" means single point kriging prediction else then some resoltion was given. pred_value_lp = 0 If SemiForm.Mult_Pred.Value = "" Then If det_dim = 1 Then Results_Form.Results_3.Text = "The predictive value at point " & Round(Range(SemiForm.Predict_Input.Value).Cells(1, 1).Value, 3) & " is: " & Round(Z, 3) & " with a Variance of " & Round(krig_var, 3) & " and a Std Dev of " & Round(krig_std, 3) & ". The 95% HCIL (+/- " & Round(krig_conf, 3) & ") is " & Round(krig_conf_l, 3) & "|" & Round(krig_conf_h, 3) & "." Else For pred_value_lp = 1 To det_dim If pred_value_lp = 1 Then krig_result_point = krig_result_point & Range(SemiForm.Predict_Input.Value).Cells(1, pred_value_lp).Value Else krig_result_point = krig_result_point & ", " & Range(SemiForm.Predict_Input.Value).Cells(1, pred_value_lp).Value End If Next Results_Form.Results_3.Text = "The predictive value at point " & krig_result_point & " is: " & Round(Z, 3) & " with a Variance of " & Round(krig_var, 3) & " and a Std Dev of " & Round(krig_std, 3) & ". The 95% HCIL (+/- " & Round(krig_conf, 3) & ") is " & Round(krig_conf_l, 3) & "|" & Round(krig_conf_h, 3) & "." End If Else If det_dim = 1 Then Results_Form.Results_3.Text = Results_Form.Results_3.Text & "The predictive value at point " & Round(krig_inp_point, 3) & " is: " & Round(Z, 3) & " Variance: " & Round(krig_var, 3) & " Std Dev: " & Round(krig_std, 3) & ". The 95% HCIL (+/- " & Round(krig_conf, 3) & ") is " & Round(krig_conf_l, 3) & "|" & Round(krig_conf_h, 3) & "." & Chr(13) Else krig_result_point = "" For pred_value_lp = 1 To det_dim If pred_value_lp = 1 Then krig_result_point = krig_result_point & krig_inp_point_2(pred_value_lp) Else krig_result_point = krig_result_point & ", " & krig_inp_point_2(pred_value_lp) End If Next Results_Form.Results_3.Text = Results_Form.Results_3.Text & "The predictive value at point " & krig_result_point & " is: " & Round(Z, 3) & " with a Variance of " & Round(krig_var, 3) & " and a Std Dev of " & Round(krig_std, 3) & ". The 95% CI (z +/- " & Round(krig_conf, 3) & ") is " & Round(krig_conf_l, 3) & "|" & Round(krig_conf_h, 3) & "." & Chr(13) End If 185 End If If SemiForm.Mult_Pred.Value <> "" Then resolution = resolution + Range(SemiForm.Mult_Pred.Value) End If Loop While SemiForm.Mult_Pred.Value <> "" '********************************** 'Autofit the Krig_Dump sheet Worksheets("Krig_Dump").Columns("A:Z").EntireColumn.AutoFit '********************************** Sheets("Krig_Dump").Activate If SemiForm.Mult_Pred.Value <> "" Then '************Add Response Curve***************************************** Dim last_used_cell As Variant last_used_cell = Range(Chr(73 + cnt_reg) & "65536").End(xlUp).Row + 1 Range("A" & test_diff_v_counter + 3).Select ActiveSheet.Shapes.AddChart.Select ActiveChart.ChartType = xlXYScatterSmoothNoMarkers ActiveChart.SeriesCollection.NewSeries ActiveChart.SeriesCollection(1).Name = "=""Response Curve""" If det_dim = 1 Then ActiveChart.SeriesCollection(1).XValues = "=Krig_Dump!$" & Chr(76 + cnt_reg) & "$2:$" & Chr(76 + cnt_reg) & "$" & last_used_cell - 1 End If ActiveChart.SeriesCollection(1).Values = "=Krig_Dump!$" & Chr(73 + cnt_reg) & "$2:$" & Chr(73 + cnt_reg) & "$" & last_used_cell - 1 '************************************************************************* End If Range("A1").Select 'Determine end time sngEnd = Timer ' Get end time. 'Determine time elapsed sngElapsed = Format(sngEnd - sngStart, "Fixed") ' Elapsed time. Results_Form.E_Time_Txt.Text = sngElapsed Unload SemiForm Results_Form.Show End Sub Function Last(choice As Long, rng As Range) 'Ron de Bruin, 5 May 2008 ' 1 = last row ' 2 = last column ' 3 = last cell Dim lrw As Long Dim lcol As Long Select Case choice Case 1: On Error Resume Next Last = rng.Find(What:="*", _ After:=rng.Cells(1), _ 186 Lookat:=xlPart, _ LookIn:=xlFormulas, _ SearchOrder:=xlByRows, _ SearchDirection:=xlPrevious, _ MatchCase:=False).Row On Error GoTo 0 Case 2: On Error Resume Next Last = rng.Find(What:="*", _ After:=rng.Cells(1), _ Lookat:=xlPart, _ LookIn:=xlFormulas, _ SearchOrder:=xlByColumns, _ SearchDirection:=xlPrevious, _ MatchCase:=False).Column On Error GoTo 0 Case 3: On Error Resume Next lrw = rng.Find(What:="*", _ After:=rng.Cells(1), _ Lookat:=xlPart, _ LookIn:=xlFormulas, _ SearchOrder:=xlByRows, _ SearchDirection:=xlPrevious, _ MatchCase:=False).Row On Error GoTo 0 On Error Resume Next lcol = rng.Find(What:="*", _ After:=rng.Cells(1), _ Lookat:=xlPart, _ LookIn:=xlFormulas, _ SearchOrder:=xlByColumns, _ SearchDirection:=xlPrevious, _ MatchCase:=False).Column On Error GoTo 0 On Error Resume Next Last = rng.Parent.Cells(lrw, lcol).Address(False, False) If Err.Number > 0 Then Last = rng.Cells(1).Address(False, False) Err.Clear End If On Error GoTo 0 End Select End Function Function euc_values(f_a_count As Integer, f_b_count As Integer, f_det_dim As Integer) 'Reset before calculating each time Dim f_euc_cal_0 As Double Dim f_dim_count_0 As Integer f_euc_cal_0 = 0 187 For f_dim_count_0 = 1 To f_det_dim f_euc_cal_0 = f_euc_cal_0 + (Range(SemiForm.RefEditOriginalSampleRange.Value).Cells(f_a_count, f_dim_count_0).Value - Range(SemiForm.RefEditOriginalSampleRange.Value).Cells(f_b_count, f_dim_count_0).Value) ^ 2 Next euc_values = Sqr(f_euc_cal_0) End Function Function euc_values_rhs(f_a_count As Integer, f_b_count As Integer, f_det_dim As Integer) 'Reset before calculating each time Dim f_euc_cal_1 As Double Dim f_dim_count_1 As Integer f_euc_cal_1 = 0 For f_dim_count_1 = 1 To f_det_dim f_euc_cal_1 = f_euc_cal_1 + (Range(SemiForm.RefEditOriginalSampleRange.Value).Cells(f_a_count, f_dim_count_1).Value - (Range(SemiForm.Predict_Input.Value).Cells(f_b_count, f_dim_count_1).Value + resolution)) ^ 2 Next euc_values_rhs = Sqr(f_euc_cal_1) End Function Function euc_values_rhs_MD_w_RES(f_a_count As Integer, f_b_count As Integer, f_det_dim As Integer, arr() As Double) 'Reset before calculating each time Dim f_euc_cal_2 As Double Dim f_dim_count_2 As Integer f_euc_cal_2 = 0 For f_dim_count_2 = 1 To f_det_dim f_euc_cal_2 = f_euc_cal_2 + (Range(SemiForm.RefEditOriginalSampleRange.Value).Cells(f_a_count, f_dim_count_2).Value - arr(f_dim_count_2)) ^ 2 Next euc_values_rhs_MD_w_RES = Sqr(f_euc_cal_2) End Function Function CheckSolver() As Boolean 'Adjusted for Application.Run() to avoid Reference problems with Solver ' Peltier Technical Services, Inc., Copyright 2007. All rights reserved. ' Returns True if Solver can be used, False if not. Dim bSolverInstalled As Boolean Dim bSolverInstalled_1 As Boolean Dim bSolverInstalled_2 As Boolean ' Assume true unless otherwise CheckSolver = True On Error Resume Next ' check whether Solver is installed bSolverInstalled = Application.AddIns("Solver Add-In").Installed bSolverInstalled_1 = Application.AddIns("Analysis ToolPak").Installed 188 bSolverInstalled_2 = Application.AddIns("Analysis ToolPak - VBA").Installed Err.Clear If Not bSolverInstalled Then ' (re)install Solver Application.AddIns("Solver Add-In").Installed = True bSolverInstalled = Application.AddIns("Solver Add-In").Installed End If If Not bSolverInstalled_1 Then Application.AddIns("Analysis ToolPak").Installed = True bSolverInstalled_1 = Application.AddIns("Analysis ToolPak").Installed End If If Not bSolverInstalled_2 Then Application.AddIns("Analysis ToolPak - VBA").Installed = True bSolverInstalled_2 = Application.AddIns("Analysis ToolPak - VBA").Installed End If If Not bSolverInstalled Or Not bSolverInstalled_1 Or Not bSolverInstalled_2 Then MsgBox "The required data analysis tools are not found. Please load the Analysis ToolPak, Analysis ToolPak - VBA, and Solver Add-In.", vbCritical CheckSolver = False End If On Error GoTo 0 End Function Function curve_fit() SemiForm.prog_label.Caption = "GRG" DoEvents ' reset Dim yhat_reg As Integer Dim yhat As String Dim res_reg As String Dim sum_sq_res As Integer Dim past_d_r_2 As Integer Dim err_msg_2 'After Regression we need to calculate yhat, eis, and SSres 'yhat Range("B28").Select For sum_sq_res = 1 To (test_diff_v_counter) For yhat_reg = 1 To cnt_reg + 1 If yhat_reg = 1 Then yhat = yhat + "=R[" & -10 - sum_sq_res & "]C" Else yhat = yhat + "+R[" & -11 - sum_sq_res + yhat_reg & "]C*Krig_Dump!R[-26]C[" & -2 + yhat_reg & "]" End If Next yhat_reg Worksheets("Kriging_Regression").Range("B" & 27 + sum_sq_res).FormulaR1C1 = yhat yhat = "" 189 Next sum_sq_res Range("C28").Select 'eis For sum_sq_res = 1 To (test_diff_v_counter) res_reg = "=Krig_Dump!R[-26]C[" & cnt_reg & "]-RC[-1]" Worksheets("Kriging_Regression").Range("C" & 27 + sum_sq_res).FormulaR1C1 = res_reg res_reg = "" Next sum_sq_res 'SSres For sum_sq_res = 1 To (test_diff_v_counter) Worksheets("Kriging_Regression").Range("D" & 27 + sum_sq_res).Formula = "=C" & 27 + sum_sq_res & "^ 2" Next sum_sq_res Worksheets("Kriging_Regression").Range("L1").Value = "Max Gamma Value" Worksheets("Kriging_Regression").Range("L2").Value = Worksheets("Krig_Dump").Cells(test_diff_v_counter + 1, cnt_reg + 3).Value Worksheets("Kriging_Regression").Range("J1").Value = "SS(Res)" Worksheets("Kriging_Regression").Range("J2").Formula = "=sum(D" & 28 & ":D" & 27 + test_diff_v_counter & ")" Application.Run "Solver.xlam!SolverReset" Dim sol_by_chang As String Dim last_const As String sol_by_chang = "$B$17:$B$" & 17 + (cnt_reg) Application.Run "Solver.xlam!SolverOk", "$J$2", 2, 0, sol_by_chang, 1, "GRG Nonlinear" Application.Run "Solver.xlam!SolverAdd", "$B$17", 2, "0" For past_d_r_2 = 1 To (test_diff_v_counter - 1) Application.Run "Solver.xlam!SolverAdd", "$B$" & 27 + past_d_r_2, 1, "$B$" & 28 + past_d_r_2 Next past_d_r_2 ' run the analysis Dim result As Integer result = Application.Run("Solver.xlam!SolverSolve", True) ' finish the analysis Application.Run "Solver.xlam!SolverFinish" ' report on success of analysis If result >= 4 Or (Round(ss_res_comp, 5) < Round(Worksheets("Kriging_Regression").Range("J2").Value, 5) And ss_res_comp <> 0) Then SemiForm.prog_label.Caption = "Switching to EA" DoEvents 'Resolve using EA engine Application.Run "Solver.xlam!SolverOk", "$J$2", 2, 0, sol_by_chang, 1, "Evolutionary" Dim result_1 As Integer ' run the analysis result_1 = Application.Run("Solver.xlam!SolverSolve", True) ' finish the analysis Application.Run "Solver.xlam!SolverFinish" 190 End If End Function Appendix A.2: Test Planning Module Public sum_var As Double '************************************* 'Simple load and unload form section Private Sub CommandButton3_Click() ' Put away the form Unload pd_analysis Exit Sub End Sub Private Sub CommandButton4_Click() ' Put away the form Unload Me 'Load form ShowSemiForm End Sub '************************************* '******************START HELP SECTION OF THE FORM**************************** Private Sub Image11_Click() Krig_Help.TextBox1.Text = "Enter the integer starting (min) value for the inputs. This can not be a negative number." Krig_Help.Show End Sub Private Sub Image12_Click() Krig_Help.TextBox1.Text = "Enter the integer ending (max) value for the inputs. This can not be a negative number and must be greter than the minimum number." Krig_Help.Show End Sub Private Sub Image13_Click() Krig_Help.TextBox1.Text = "Enter the total test budget allocated to collecting sample data." Krig_Help.Show End Sub Private Sub Image14_Click() Krig_Help.TextBox1.Text = "If a total test budget is entered, the user may the cost test test." Krig_Help.Show End Sub Private Sub Image15_Click() Krig_Help.TextBox1.Text = "This box allows the user to linearly scale the cost per test up or down by a factor between -9.9 to 9.9." Krig_Help.Show End Sub Private Sub Image16_Click() 191 Krig_Help.TextBox1.Text = "This list box shows the recommended FLCs that are associated with each sample size." Krig_Help.Show End Sub Private Sub Image18_Click() Krig_Help.TextBox1.Text = "This allows the user to select how much of the data should be posted into the worksheet. The default of 0 is to paste the recommnded pilot design into the worksheet. If altered, this value must be an integer greater than 1 and less than or equal to the available problem space sample size. Changing this value from the default results in an increase in processing time." Krig_Help.Show End Sub Private Sub Image19_Click() Krig_Help.TextBox1.Text = "This allows the user to enter a nugget value. The default is to 0 and must not be set to negative." Krig_Help.Show End Sub Private Sub Image4_Click() Krig_Help.TextBox1.Text = "The user can specify the increment size. The default covers all unique lags in the problem space." Krig_Help.Show End Sub Private Sub Image5_Click() Krig_Help.TextBox1.Text = "Enter a numerical integer that is greater than zero. Note: each input/dimension must be equal in size. The max is 8 dimensions" Krig_Help.Show End Sub Private Sub Image9_Click() Krig_Help.TextBox1.Text = "The list box below displays critical information depending on sample size. This information is used to allow the user to select a starting sample size." Krig_Help.Show End Sub '******************END HELP SECTION OF THE FORM**************************** Private Sub exp_var_confirm_Click() 'Determine start time Dim sngStart As Single, sngEnd As Single Dim sngElapsed As Single sngStart = Timer ' Get start time. Me.CommandButton1.Visible = False '******************START FORM ERROR CHECKING******************************* If CDbl(pd_analysis.num_dim.Value) < 1 Or pd_analysis.num_dim.Value = "" Or CDbl(pd_analysis.num_dim.Value) - Int(CDbl(pd_analysis.num_dim.Value)) <> 0 Or CDbl(pd_analysis.num_dim.Value) > 8 Then err_msg_1 = MsgBox("Required. The number of dimensions must be an integer greater than 0 but less than 8.", vbOKOnly, "Error Handler") Exit Sub 192 End If If CDbl(pd_analysis.min_val.Value) < 0 Or pd_analysis.min_val.Value = "" Or CDbl(pd_analysis.min_val.Value) - Int(CDbl(pd_analysis.min_val.Value)) <> 0 Then err_msg_1 = MsgBox("Required. The minimum starting value must be an integer greater than or equal to 0.", vbOKOnly, "Error Handler") Exit Sub End If 'Currently disabled with plans to enable listed as part of future research. The only lag selection is the default 'which is the minimum increment of 1 for this research. 'If CDbl(pd_analysis.ComboBoxlag.Value) <= 0 Or pd_analysis.ComboBoxlag.Value <> "(Default)" Then ' err_msg_1 = MsgBox("Required. The minimum lag distance must be greater than 0. The program will exit now.", vbOKOnly, "Error Handler") ' Exit Sub 'End If If CDbl(pd_analysis.max_val.Value) <= CDbl(pd_analysis.min_val.Value) Or pd_analysis.max_val.Value = "" Or CDbl(pd_analysis.max_val.Value) - Int(CDbl(pd_analysis.max_val.Value)) <> 0 Then err_msg_1 = MsgBox("Required. The maximum starting value must be an integer greater than the minimum value.", vbOKOnly, "Error Handler") Exit Sub End If If CDbl(pd_analysis.nug.Value) < 0 Or pd_analysis.nug.Value = "" Then err_msg_1 = MsgBox("Required. The nugget value must be greater than 0.", vbOKOnly, "Error Handler") Exit Sub End If If CDbl(pd_analysis.budget.Value) <= 0 Or pd_analysis.budget.Value = "" Then err_msg_1 = MsgBox("Required. The budget must be greater than 0.", vbOKOnly, "Error Handler") Exit Sub End If If CDbl(pd_analysis.cost_p_test.Value) > CDbl(pd_analysis.budget.Value) Or CDbl(pd_analysis.cost_p_test.Value) <= 0 Or pd_analysis.cost_p_test.Value = "" Then err_msg_1 = MsgBox("Required. The cost per test must be greater than 0 and less than the total budget and must not be blank.", vbOKOnly, "Error Handler") Exit Sub End If If CDbl(pd_analysis.lin_cost.Value) > 9.9 Or CDbl(pd_analysis.lin_cost.Value) < -9.9 Then err_msg_1 = MsgBox("Optional unless a budget and cost per test is specified. The value must range between -9.9 to 9.9. This box must remain blank if no budget is specified.", vbOKOnly, "Error Handler") Exit Sub End If 193 If (CDbl(pd_analysis.max_val) ^ CDbl(num_dim)) > 65000 Then err_msg_1 = MsgBox("The number of FLCs exceeds the amount allowed by Excel. Please adjust (lower) the Max Value and/or the Number of Dimensions.", vbOKOnly, "Error Handler") Exit Sub End If '******************END FORM ERROR CHECKING**************************** '*******************Create new sheet to display data****************** 'Create a new sheet for data dumps and calculation results Dim NewBook As New Worksheet Application.DisplayAlerts = False On Error Resume Next Sheets("Test_Planning").Delete Application.DisplayAlerts = True Set NewBook = Worksheets.Add NewBook.Name = "Test_Planning" '********************************************************************** '**************Display title and user input on new worksheet*********** Worksheets("Test_Planning").Range("A1").Value = "Test Planning Worksheet" With Worksheets("Test_Planning").Range("A1").Font .Name = "Calibri" .Size = 24 .Strikethrough = False .Superscript = False .Subscript = False .OutlineFont = False .Shadow = False .Underline = xlUnderlineStyleNone .ThemeColor = xlThemeColorLight1 .TintAndShade = 0 .ThemeFont = xlThemeFontMinor End With For hghl = 2 To 3 For hghl_1 = 2 To 6 Step 2 With Worksheets("Test_Planning").Cells(hghl, hghl_1).Interior .Pattern = xlSolid .PatternColorIndex = xlAutomatic .ThemeColor = xlThemeColorLight2 .TintAndShade = 0.799981688894314 .PatternTintAndShade = 0 End With Next hghl_1 Next hghl With Worksheets("Test_Planning").Range("A1").Font.Bold = True Worksheets("Test_Planning").Range("A2").Value = "# Dim =" Worksheets("Test_Planning").Range("B2").Value = CDbl(num_dim) Worksheets("Test_Planning").Range("C2").Value = "Min Value =" Worksheets("Test_Planning").Range("D2").Value = CDbl(min_val) Worksheets("Test_Planning").Range("E2").Value = "Max Value =" 194 Worksheets("Test_Planning").Range("F2").Value = CDbl(max_val) Worksheets("Test_Planning").Range("A3").Value = "Budget =" Worksheets("Test_Planning").Range("B3").Value = CDbl(pd_analysis.budget) Worksheets("Test_Planning").Range("C3").Value = "Cost/Test =" Worksheets("Test_Planning").Range("D3").Value = CDbl(pd_analysis.cost_p_test) Worksheets("Test_Planning").Range("E3").Value = "Cost Scale =" Worksheets("Test_Planning").Range("F3").Value = CDbl(pd_analysis.lin_cost) Worksheets("Test_Planning").Range("A4").Value = "FLCs" End With Columns("C:C").EntireColumn.AutoFit Columns("E:E").EntireColumn.AutoFit With Worksheets("Test_Planning").Range("A4").Font .Name = "Calibri" .Size = 18 .Strikethrough = False .Superscript = False .Subscript = False .OutlineFont = False .Shadow = False .Underline = xlUnderlineStyleNone .ThemeColor = xlThemeColorLight1 .TintAndShade = 0 .ThemeFont = xlThemeFontMinor End With '********************************************************************** '*****Convert Input into equally spaced array with dimension Xsupd***** 'Step 1 - Create Problem Space Array 'This routine creates a 2D array. The 1st dimension is the unique identifier and the second dimension houses 'the n-dimensional values of the problem space. Dim problem_space_array() As Integer ReDim problem_space_array(1 To CDbl(pd_analysis.max_val) ^ CDbl(num_dim), 1 To CDbl(num_dim)) Dim x_count As Integer Dim lag_inc As Integer Dim lag_inc_1 As Integer Dim x_count1 As Integer For x_count = 1 To CDbl(num_dim) lag_inc_1 = 0 If x_count = 1 Then lag_inc_count = 1 lag_inc = 0 dim_x_count = x_count Else lag_inc = 1 lag_inc_count = 1 195 dim_x_count = x_count - 1 End If For x1_count = 1 To CDbl(pd_analysis.max_val) ^ CDbl(num_dim) lag_inc_1 = lag_inc_1 + 1 If CDbl(pd_analysis.max_val) ^ dim_x_count >= lag_inc_count Then If x_count = 1 Then lag_inc = lag_inc + 1 End If Else If x_count = 1 Then lag_inc = 1 lag_inc_count = 1 Else If lag_inc = CDbl(pd_analysis.max_val) Then lag_inc = 1 Else lag_inc = lag_inc + 1 End If lag_inc_count = 1 End If End If lag_inc_count = lag_inc_count + 1 problem_space_array(lag_inc_1, (CDbl(num_dim) - x_count + 1)) = CDbl(pd_analysis.min_val) + (lag_inc - 1) 'The following two with statements provide highlighting for the FLCs With Worksheets("Test_Planning").Cells(x1_count + 4, CDbl(num_dim) - (x_count - 1)).Interior .Pattern = xlSolid .PatternColorIndex = xlAutomatic .ThemeColor = xlThemeColorLight1 .TintAndShade = 0.799981688894314 .PatternTintAndShade = 0 End With Next x1_count Next x_count [a5].Resize(UBound(problem_space_array), CDbl(num_dim)) = problem_space_array '******************End Input Arrray Creation*************************************** '******************Step 2 - Augmented simulated annealing process****************** 'This process consists of multiple steps that are defined by comment as required 'This appraoch, by nature, is required to be integrated together with most of the remaining code of this section. 'This is the reason why step 2 is so large in terms of lines of code. '***The first and second action is to take the min and max value of Xsupd and create a design and COV matrix*** 196 'Distance Matrix Creation Dim dist_matrix() As Double '2^num_dim will capture all the end points of the factor space ReDim dist_matrix(1 To 2 ^ num_dim, 1 To 2 ^ num_dim) 'Need to determine all FLCs to be added to pilot design 'Assign the FLCs for the pilot design and determine the Euclidean distance 'Assigns the diagonal of zeros Dim init_dsgn As Integer Dim init_dsgn_1 As Integer For init_dsgn = 1 To 2 ^ num_dim dist_matrix(init_dsgn, init_dsgn) = 0 Next init_dsgn 'Reset before calculating each time Dim f_euc_cal_0 As Double Dim f_dim_count_0 As Integer 'Need to determine the lags for each value in the permutation Dim per_check As Integer per_check = 0 Dim perm_matrix() As Variant ReDim perm_matrix(1 To 2 ^ num_dim, 1 To num_dim) Dim per_count As Integer per_count = 0 Dim lb_1 As MSForms.ListBox Set lb_1 = Me.sam_location lb_1.Clear ' clear the listbox content lb_1.ColumnCount = CDbl(num_dim) + 1 For init_dsgn = 1 To CDbl(pd_analysis.max_val) ^ CDbl(num_dim) 'Only get distance if the current FLC passes the permutation test For f_dim_count_0 = 1 To CDbl(num_dim) 'Need to determine which FLCs in the worksheet have a mix of all high or low values If Worksheets("Test_Planning").Cells(init_dsgn + 4, f_dim_count_0) = CDbl(pd_analysis.max_val) Or Worksheets("Test_Planning").Cells(init_dsgn + 4, f_dim_count_0) = CDbl(pd_analysis.min_val) Then per_check = per_check + 1 End If Next f_dim_count_0 If per_check = CDbl(num_dim) Then per_count = per_count + 1 For f_dim_count_0 = 1 To CDbl(num_dim) perm_matrix(per_count, f_dim_count_0) = Worksheets("Test_Planning").Cells(init_dsgn + 4, f_dim_count_0) 'Store the FLC ID Number Next f_dim_count_0 lb_1.AddItem init_dsgn End If per_check = 0 Next init_dsgn 197 'After permutation matrix is developed, now we need the lag matrix For init_dsgn = 2 To 2 ^ num_dim 'rows For init_dsgn_1 = 1 To (init_dsgn - 1) 'columns f_euc_cal_0 = 0 For f_dim_count_0 = 1 To CDbl(num_dim) f_euc_cal_0 = f_euc_cal_0 + (perm_matrix(init_dsgn, f_dim_count_0) - perm_matrix(init_dsgn_1, f_dim_count_0)) ^ 2 Next f_dim_count_0 'Due to Symmetry dist_matrix(init_dsgn_1, init_dsgn) = Sqr(f_euc_cal_0) dist_matrix(init_dsgn, init_dsgn_1) = dist_matrix(init_dsgn_1, init_dsgn) Next init_dsgn_1 Next init_dsgn 'Create the initial COV matrix 'The 1st dimension will be redimmed as the pilot design grows and is the ith index of the COV matrix 'The 2nd dimension will be redimmed as the pilot design grows and is the jth index of the COV matrix Dim cov_matrix() As Double ReDim cov_matrix(1 To (2 ^ num_dim) + 1, 1 To (2 ^ num_dim) + 1) Dim nugget As Double Dim c1 As Double Dim a As Double 'Set the range equal to the maximum lag a = dist_matrix(2 ^ num_dim, 1) 'Set c1 = 1. c1 or the sil is the variance portion of the graph. 'It is reasonable to set the sill equal to the sample variance since the data is evenly distributed. 'Under the assumption of normality, the data can be transformed into Z (standard normal) thus requiring 'the population variance = 1 c1 = 1 nugget = CDbl(pd_analysis.nug) For init_dsgn = 1 To (2 ^ num_dim) + 1 cov_matrix(init_dsgn, init_dsgn) = 0 Next init_dsgn For init_dsgn = 2 To 2 ^ num_dim 'row For init_dsgn_1 = 1 To (init_dsgn - 1) 'column cov_matrix(init_dsgn, init_dsgn_1) = nugget + c1 * (1 - Exp((-3 * dist_matrix(init_dsgn, init_dsgn_1)) / a)) 'Due to symmetry cov_matrix(init_dsgn_1, init_dsgn) = cov_matrix(init_dsgn, init_dsgn_1) Next init_dsgn_1 Next init_dsgn For init_dsgn = 1 To (2 ^ num_dim) 'row '****Lagrangian Multiplier***** cov_matrix(init_dsgn, ((2 ^ num_dim)) + 1) = 1 cov_matrix(((2 ^ num_dim)) + 1, init_dsgn) = 1 198 '****************************** Next init_dsgn '*****END building and defining the initial matricies*** '*****Get cost data************************************* Dim tot_budget As Double Dim cpt As Double Dim lin_scale As Double tot_budget = CDbl(pd_analysis.budget) cpt = CDbl(pd_analysis.cost_p_test) lin_scale = CDbl(pd_analysis.lin_cost) '*****End Get cost data************************************* '***********Start search algorithm************* 'set expected improvement, which will be used as the stopping criterion Dim ei As Double Dim ei_weight As Double 'Set EI to a starting value of 1 Dim flag As Integer 'This flag determines the search direction. 0 searches left and 1 searches right Dim dir_flag As Integer flag = 0 Dim index_count As Integer 'set the starting search point to the indexed value just below the max index_count = CDbl(pd_analysis.max_val) - 1 'Identify candidate input 'Need to store the input index and distance Dim RHS_lag_marix() As Integer ReDim RHS_lag_matrix(1 To 1, 1 To CDbl(num_dim)) Dim RHS_dis_matrix() As Double ReDim RHS_dis_matrix(1 To 2 ^ num_dim, 1 To 1) Dim lag_pointer As Integer Dim cand_input() As Double ReDim cand_input(1 To (2 ^ num_dim) + 1) lag_pointer = Application.WorksheetFunction.RoundUp(UBound(problem_space_array) / 2, 0) 'After center point is selected, calculate the RHS lag and COV matrix For f_dim_count_0 = 1 To CDbl(num_dim) RHS_lag_matrix(1, f_dim_count_0) = problem_space_array(lag_pointer, f_dim_count_0) Next f_dim_count_0 For init_dsgn = 1 To (2 ^ num_dim) 'row f_euc_cal_0 = 0 For f_dim_count_0 = 1 To CDbl(num_dim) f_euc_cal_0 = f_euc_cal_0 + (perm_matrix(init_dsgn, f_dim_count_0) - RHS_lag_matrix(1, f_dim_count_0)) ^ 2 Next RHS_dis_matrix(init_dsgn, 1) = Sqr(f_euc_cal_0) 199 cand_input(init_dsgn) = nugget + c1 * (1 - Exp((-3 * RHS_dis_matrix(init_dsgn, 1)) / a)) Next init_dsgn 'Lagrange multiplier cand_input(init_dsgn) = 1 'Perform search algorithm until either an acceptable expected improvement is reached or no better solutions are found 'Need to first calculate the initial variance based on the 2 point initial FLC. This is the starting point i.e. worst variance possible. 'This requires one additional point. (max+min)/2 will be used to calculate the worst variance. Dim ArrInv() As Variant Dim ArrAns() As Variant 'Calculate weights ArrInv() = Application.MInverse(cov_matrix) ArrAns() = Application.MMult(ArrInv, Application.Transpose(cand_input)) 'Calculate initial variance Dim krig_var As Double Dim var_best As Double krig_var = 0 For krig_count = 1 To CDbl(2 ^ num_dim) + 1 krig_var = krig_var + ArrAns(krig_count, 1) * cand_input(krig_count) Next 'The search direction is 1d based on distances (left -0 and right - 1) dir_flag = 1 'Set the initial variance equal to the best candidate and therefore add the third FLC to the design 'var_best = krig_var var_best = 0 Dim krig_var_old As Double krig_var_old = krig_var 'Fill the list boxes with the initial data Dim lb As MSForms.ListBox Set lb = Me.sam_selection Dim ini_samp_size As Integer Dim ini_samp_size_1 As Integer ini_samp_size = 0 'Fill the sample selection list box with the initial two samples Dim cost_tot_new As Integer 'ei = 100 With lb .Clear ' clear the listbox content .ColumnCount = 4 cost_tot = CDbl(pd_analysis.cost_p_test) bud_remain = CDbl(pd_analysis.budget) For ini_samp_size = 1 To (2 ^ num_dim) + 1 .AddItem ini_samp_size If ini_samp_size <> CDbl(2 ^ num_dim) + 1 Then .List(ini_samp_size - 1, 1) = "-" Else 200 .List(ini_samp_size - 1, 1) = krig_var End If If CDbl(pd_analysis.lin_cost) = 0 Then bud_remain = bud_remain - cost_tot Else If ini_samp_size = 1 Then bud_remain = bud_remain - cost_tot 'This variable increases each time by a linear amount. This variable tracks that. cost_tot_new = (CDbl(pd_analysis.lin_cost) * cost_tot) Else bud_remain = bud_remain - (CDbl(pd_analysis.lin_cost) * cost_tot_new) cost_tot_new = (CDbl(pd_analysis.lin_cost) * cost_tot_new) End If End If .List(ini_samp_size - 1, 2) = Round(bud_remain, 2) Next ini_samp_size sum_var = sum_var + krig_var .ListIndex = 0 'select the first item End With 'Fill the sample location list box with the initial samples With lb_1 Dim col_wid_num As String col_wid_num = "0;" Dim col_wid As Integer For col_wid = 1 To CDbl(num_dim) col_wid_num = col_wid_num & "20;" .ColumnWidths = col_wid_num Next col_wid Dim fill_FLC_box As Integer fill_FLC_box = 0 For ini_samp_size = 1 To 2 ^ CDbl(num_dim) For fill_FLC_box = 1 To pd_analysis.num_dim .List(ini_samp_size - 1, fill_FLC_box) = perm_matrix(ini_samp_size, fill_FLC_box) Next fill_FLC_box flc = "" Next ini_samp_size End With If bud_remain <= 0 Then err_msg_1 = MsgBox("The budget entered is not sufficient to begin sampling. The program will terminate.", vbOKOnly, "Error Handler") Exit Sub End If Erase perm_matrix 'Fill the sample location/selection list box with the initial two samples 'Also add location and COV to the matrix of data 'Counters 201 Dim m As Integer Dim n As Integer Dim array_inc_count As Integer array_inc_count = 0 Dim comp_count As Integer 'Set comp_count = 1 from the beginning since we will never insert a new line before the lower problem bound comp_count = 1 Dim iTemp() As Double 'Temporary array for distance matrix Dim iTemp_1() As Double 'Temporary array for COV matrix Dim find_inp_index As Integer Dim fill_next_can_inp_1 As Integer Dim dup As Integer Dim det_loc_1 As Integer Dim fill_FLC_box_1 As Integer 'This is the location counter Dim loc_int As Integer Dim RHS_lag_matrix_TEMP() As Variant ReDim RHS_lag_matrix_TEMP(1 To 1, 1 To CDbl(num_dim)) 'This variable is used to find the maximin difference after the variances don't differ for a given design matrix Dim ind_diff As Integer '*****This array tracks the order of the array Dim fp_inp_index_ordered() As Integer ReDim fp_inp_index_ordered(1 To 2 ^ num_dim) For init_desgn = 1 To (2 ^ num_dim) fp_inp_index_ordered(init_desgn) = CDbl(lb_1.Column(0, init_desgn - 1)) Next init_desgn Dim iloop As Integer Dim iloop2 As Integer Dim str1 As Integer Dim str2 As Integer Dim dup_found As Integer Dim hit_end As Integer Dim move_counter As Integer Dim past_data_ind As Integer Dim tot_design_check As Integer Dim design_check_1 As Integer design_check_1 = 0 Me.neighborhood.Visible = False Me.Frame10.Visible = False Me.Frame11.Visible = False Me.Frame12.Visible = False Me.Frame13.Visible = False Me.Frame18.Visible = False Me.iter.ZOrder (0) Me.iter.Visible = True Me.outof.Value = UBound(problem_space_array) Do While CDbl(lb.ListCount) < UBound(problem_space_array) 202 Me.strt.Value = lb.ListCount 'Determine end time sngEnd = Timer ' Get end time. 'Determine time elapsed sngElapsed = Format(sngEnd - sngStart, "Fixed") ' Elapsed time. Me.time_elap.Text = sngElapsed DoEvents 'Me.Repaint 'Need to perform dynamic array slicing to insert the new values while rearranging the current values in the dist and COV matrices 'Temp storage idea taken from http://www.xtremevbtalk.com/showthread.php?t=82476 ReDim iTemp(1 To CDbl(2 ^ num_dim) + array_inc_count, 1 To CDbl(2 ^ num_dim) + array_inc_count) ReDim iTemp_1(1 To ((CDbl(2 ^ num_dim) + 1) + array_inc_count), 1 To ((CDbl(2 ^ num_dim) + 1) + array_inc_count)) 'Copy original array into temp array: For m = 1 To (CDbl(2 ^ num_dim) + array_inc_count) 'Loop for 1st row For n = 1 To (CDbl(2 ^ num_dim) + array_inc_count) 'Loop for column If n <> (CDbl(2 ^ num_dim) + array_inc_count) Then iTemp(m, n) = dist_matrix(m, n) End If iTemp_1(m, n) = cov_matrix(m, n) Next n Next m 'Put values back from temporary array while adding the new design candidate in the necessary position and adjusting the matrices accordingly array_inc_count = array_inc_count + 1 'For each candidate that gets added during the annealing process, the matrix will continue to grow ReDim dist_matrix(1 To CDbl(2 ^ num_dim) + array_inc_count, 1 To CDbl(2 ^ num_dim) + array_inc_count) ReDim cov_matrix(1 To ((CDbl(2 ^ num_dim) + 1) + array_inc_count), 1 To ((CDbl(2 ^ num_dim) + 1) + array_inc_count)) 'Loop through the current list of FLCs to determine where to insert the candidate FLC 'Have to determine which row the cand_input must be inserted into 'After that, the appropriate distances and COV must be calculated relative to every other distance in the pilot design 'First determine which indexed input the candidate input is equal to If Me.end_cal = True Then If bud_remain <= 0 Then With Me.amt_to_paste For past_data_ind = CDbl(lb_1.ListCount) To 1 Step -1 .AddItem past_data_ind, 0 Next past_data_ind End With For ei_pop = 2 To CDbl(lb.ListCount) 203 lb.Column(3, ei_pop) = "-" Next ei_pop Me.CommandButton1.Visible = True 'Determine end time sngEnd = Timer ' Get end time. 'Determine time elapsed sngElapsed = Format(sngEnd - sngStart, "Fixed") ' Elapsed time. Me.time_elap.Text = sngElapsed Me.iter.Visible = False Me.neighborhood.Visible = True Me.Frame10.Visible = True Me.Frame11.Visible = True Me.Frame12.Visible = True Me.Frame13.Visible = True Me.Frame18.Visible = True Exit Sub End If End If For find_inp_index = 1 To CDbl(pd_analysis.max_val) ^ CDbl(num_dim) comp_count = 1 For det_loc_1 = 1 To CDbl(num_dim) If RHS_lag_matrix(1, det_loc_1) <= problem_space_array(find_inp_index, det_loc_1) And RHS_lag_matrix(1, det_loc_1) >= problem_space_array(find_inp_index, det_loc_1) Then comp_count = comp_count + 1 End If Next det_loc_1 If comp_count = det_loc_1 Then lb_1.AddItem find_inp_index For fill_FLC_box_1 = 1 To CDbl(pd_analysis.num_dim) With lb_1 .List(lb_1.ListCount - 1, fill_FLC_box_1) = problem_space_array(find_inp_index, fill_FLC_box_1) End With Next fill_FLC_box_1 Exit For End If Next find_inp_index If design_check_1 = 2 Then If CDbl(lb_1.Column(0, lb_1.ListCount - 1)) <= UBound(problem_space_array) / 2 Then dir_flag = 1 Else dir_flag = 0 End If End If 'The purpose of this is to set the proper direction if the design search was previously exhausted 204 design_check_1 = 0 ReDim Preserve fp_inp_index_ordered(1 To lb_1.ListCount) fp_inp_index_ordered(lb_1.ListCount) = find_inp_index 'Sort the array (http://www.ozgrid.com/VBA/sort-array.htm) For lLoop = 1 To UBound(fp_inp_index_ordered) For lLoop2 = lLoop To UBound(fp_inp_index_ordered) If fp_inp_index_ordered(lLoop2) < fp_inp_index_ordered(lLoop) Then str1 = fp_inp_index_ordered(lLoop) str2 = fp_inp_index_ordered(lLoop2) fp_inp_index_ordered(lLoop) = str2 fp_inp_index_ordered(lLoop2) = str1 End If Next lLoop2 Next lLoop 'Now the new input has been added to the location list box 'Next add the new elements to the dist and cov matrix m = 0 n = 0 move_counter = 0 For m = LBound(iTemp_1, 1) To (UBound(iTemp_1, 1) - 1) 'Loop through rows For n = 1 To m 'Loop through columns. This only loops through step-wise half of the matrix. After that, the rest can be filled through symmetry 'Need to accomplish two things. One move original elements to new locations and also need to calculate new data into the inserted elements 'This calculates new data elements 'And m <> n ignores the diagonal elements since the difference in distance between one position and itself is 0 'If m=n then do nothing as the elements are already 0 If (n < m) And fp_inp_index_ordered(m) < find_inp_index Then 'Nothing moves. Put back into the original array dist_matrix(m, n) = iTemp(m, n) cov_matrix(m, n) = iTemp_1(m, n) dist_matrix(n, m) = dist_matrix(m, n) cov_matrix(n, m) = cov_matrix(m, n) ElseIf (n < m) And fp_inp_index_ordered(m) < fp_inp_index_ordered(m + 1) Then 'The reason for this redundacy is Excel will accept a false True for the above statement when n+1 exceeds the last element of the list If (n < m) Then 'Move original data 'Move data down If move_counter = 0 Then move_counter = m If move_counter > n Then dist_matrix(m + 1, n) = iTemp(m, n) cov_matrix(m + 1, n) = iTemp_1(m, n) dist_matrix(n, m + 1) = dist_matrix(m + 1, n) 205 cov_matrix(n, m + 1) = cov_matrix(m + 1, n) 'Move data down and across ElseIf move_counter <= n And move_counter <> 0 Then dist_matrix(m + 1, n + 1) = iTemp(m, n) cov_matrix(m + 1, n + 1) = iTemp_1(m, n) dist_matrix(n + 1, m + 1) = dist_matrix(m + 1, n + 1) cov_matrix(n + 1, m + 1) = cov_matrix(m + 1, n + 1) End If End If End If Next n Next m 'After all elements have been moved, then calculate all new elements. The location is determined if the new element's current value is 0 and it is a non-diagnonal For m = LBound(iTemp_1, 1) To UBound(iTemp_1, 1) 'Loop through rows For n = 1 To m 'Loop through columns. This only loops through step-wise half of the matrix. After that, the rest can be filled If n <> m And dist_matrix(m, n) = 0 And find_inp_index >= fp_inp_index_ordered(m) Then 'This calculates and adds new data elements f_euc_cal_0 = 0 For f_dim_count_0 = 1 To CDbl(num_dim) f_euc_cal_0 = f_euc_cal_0 + (problem_space_array(find_inp_index, f_dim_count_0) - problem_space_array(fp_inp_index_ordered(n), f_dim_count_0)) ^ 2 Next dist_matrix(m, n) = Sqr(f_euc_cal_0) 'Due to Symmetry dist_matrix(n, m) = Sqr(f_euc_cal_0) 'Reset range to the max value a = WorksheetFunction.Max(dist_matrix) cov_matrix(m, n) = nugget + c1 * (1 - Exp((-3 * dist_matrix(m, n)) / a)) cov_matrix(n, m) = nugget + c1 * (1 - Exp((-3 * dist_matrix(m, n)) / a)) 'End new data element calculation ElseIf n <> m And dist_matrix(m, n) = 0 And find_inp_index < fp_inp_index_ordered(m) Then 'This calculates and adds new data elements f_euc_cal_0 = 0 For f_dim_count_0 = 1 To CDbl(num_dim) f_euc_cal_0 = f_euc_cal_0 + (problem_space_array(find_inp_index, f_dim_count_0) - problem_space_array(fp_inp_index_ordered(m), f_dim_count_0)) ^ 2 Next dist_matrix(m, n) = Sqr(f_euc_cal_0) 206 'Due to Symmetry dist_matrix(n, m) = Sqr(f_euc_cal_0) 'Reset range to the max value a = WorksheetFunction.Max(dist_matrix) cov_matrix(m, n) = nugget + c1 * (1 - Exp((-3 * dist_matrix(m, n)) / a)) cov_matrix(n, m) = nugget + c1 * (1 - Exp((-3 * dist_matrix(m, n)) / a)) 'End new data element calculation End If 'Lagrangian multiplier If (n = m) Then cov_matrix(m, UBound(iTemp_1, 1) + 1) = 1 cov_matrix(UBound(iTemp_1, 1) + 1, m) = 1 End If Next n Next m 'Reset the location counter to offset by one each time loc_int = 1 ReDim cand_input(1 To CDbl(lb_1.ListCount) + 1) ReDim RHS_lag_matrix(1 To 1, 1 To CDbl(num_dim)) ReDim RHS_dis_matrix(1 To CDbl(lb_1.ListCount), 1 To 1) 'Start searching for the worst input '************************************************************************* 'Set initial var_new variable var_new = 0 'Search for a better solution. If found add to the pilot design, redirect the directional flag and continue by exiting loop 'Stop looking on the last FLC and directly add that to the list. Do While (CDbl(lb.ListCount) <> UBound(problem_space_array)) dup_found = 1 hit_end = 0 'search by using dist_mat then set it to RHS For fill_next_can_inp = 1 To CDbl(lb_1.ListCount) If dir_flag = 0 Then f_euc_cal_0 = 0 For f_dim_count_0 = 1 To CDbl(num_dim) RHS_lag_matrix(1, f_dim_count_0) = problem_space_array(find_inp_index - loc_int, f_dim_count_0) f_euc_cal_0 = f_euc_cal_0 + (problem_space_array(fp_inp_index_ordered(fill_next_can_inp), f_dim_count_0) - RHS_lag_matrix(1, f_dim_count_0)) ^ 2 Next RHS_dis_matrix(fill_next_can_inp, 1) = Sqr(f_euc_cal_0) 'Need to check if RHS lag has already been used, if so increment loc_int 'The IF statement only checks the candidate input against the list once to see if it is a duplicate If dup_found = 1 Then 207 For fill_next_can_inp_1 = 1 To CDbl(lb_1.ListCount) dup = 0 For f_dim_count_0 = 1 To CDbl(num_dim) If RHS_lag_matrix(1, f_dim_count_0) = CDbl(lb_1.Column(f_dim_count_0, fill_next_can_inp_1 - 1)) Then dup = dup + 1 End If Next f_dim_count_0 If dup = CDbl(num_dim) Then If CDbl(lb_1.Column(0, fill_next_can_inp_1 - 1)) = LBound(problem_space_array) Or (CDbl(lb_1.Column(0, fill_next_can_inp_1 - 1) = UBound(problem_space_array))) Then If (krig_var = var_best And CDbl(lb_1.ListCount) <> UBound(problem_space_array)) Or (var_best = 0 And CDbl(lb_1.ListCount) <> UBound(problem_space_array)) Then 'Reset back to starting point For f_dim_count_0 = 1 To CDbl(num_dim) RHS_lag_matrix(1, f_dim_count_0) = CDbl(lb_1.Column(f_dim_count_0, CDbl(lb_1.ListCount - 1))) Next loc_int = 0 'If no better variance is found by searching the entire direction tot_design_check = tot_design_check + 1 'Change the direction dir_flag = 1 Else hit_end = 1 fill_next_can_inp = CDbl(lb_1.ListCount) loc_int = 1 End If RHS_dis_matrix(1, 1) = temp_rhs_var End If loc_int = loc_int + 1 'If the input is found to be a duplicate, then start the process over If hit_end = 0 And tot_design_check <> 2 Then fill_next_can_inp = 0 Else fill_next_can_inp = CDbl(lb_1.ListCount) End If dup_found = 1 Exit For Else dup_found = 0 End If Next fill_next_can_inp_1 208 End If ElseIf dir_flag = 1 Then f_euc_cal_0 = 0 For f_dim_count_0 = 1 To CDbl(num_dim) RHS_lag_matrix(1, f_dim_count_0) = problem_space_array(find_inp_index + loc_int, f_dim_count_0) f_euc_cal_0 = f_euc_cal_0 + (problem_space_array(fp_inp_index_ordered(fill_next_can_inp), f_dim_count_0) - RHS_lag_matrix(1, f_dim_count_0)) ^ 2 Next f_dim_count_0 RHS_dis_matrix(fill_next_can_inp, 1) = Sqr(f_euc_cal_0) 'Need to check if RHS lag has already been used, if so increment loc_int 'The IF statement only checks the candidate input against the list once to see if it is a duplicate If dup_found = 1 Then For fill_next_can_inp_1 = 1 To CDbl(lb_1.ListCount) dup = 0 For f_dim_count_0 = 1 To CDbl(num_dim) If RHS_lag_matrix(1, f_dim_count_0) = CDbl(lb_1.Column(f_dim_count_0, fill_next_can_inp_1 - 1)) Then dup = dup + 1 End If Next If dup = CDbl(num_dim) Then If CDbl(lb_1.Column(0, fill_next_can_inp_1 - 1)) = LBound(problem_space_array) Or CDbl(lb_1.Column(0, fill_next_can_inp_1 - 1)) = UBound(problem_space_array) Then 'Need to add provision for hitting the end of the design and not having found a better solution. Have to reverse the direction and start over from the previous starting point. If (krig_var = var_best And CDbl(lb_1.ListCount) <> UBound(problem_space_array)) Or (var_best = 0 And CDbl(lb_1.ListCount) <> UBound(problem_space_array)) Then 'Reset back to starting point For f_dim_count_0 = 1 To CDbl(num_dim) RHS_lag_matrix(1, f_dim_count_0) = CDbl(lb_1.Column(f_dim_count_0, CDbl(lb_1.ListCount - 1))) Next loc_int = 0 'If no better variance is found by searching the entire direction tot_design_check = tot_design_check + 1 'Change the direction dir_flag = 0 Else hit_end = 1 209 fill_next_can_inp = CDbl(lb_1.ListCount) loc_int = 1 End If RHS_dis_matrix(1, 1) = temp_rhs_var End If loc_int = loc_int + 1 'If the input is found to be a duplicate, then start the process over unless it is at the beginning or ending number and we have not searched the whole problem space If hit_end = 0 And tot_design_check <> 2 Then fill_next_can_inp = 0 Else fill_next_can_inp = CDbl(lb_1.ListCount) End If dup_found = 1 Exit For Else dup_found = 0 End If Next fill_next_can_inp_1 End If End If Next fill_next_can_inp For fill_next_can_inp = 1 To CDbl(lb_1.ListCount) cand_input(fill_next_can_inp) = nugget + c1 * (1 - Exp((-3 * RHS_dis_matrix(fill_next_can_inp, 1)) / a)) Next fill_next_can_inp cand_input(fill_next_can_inp) = 1 krig_var = 0 'Reset Design Check 'Calculate weights ArrInv() = Application.MInverse(cov_matrix) ArrAns() = Application.MMult(ArrInv, Application.Transpose(cand_input)) For krig_count = 1 To lb_1.ListCount + 1 krig_var = krig_var + ArrAns(krig_count, 1) * cand_input(krig_count) Next loc_int = loc_int + 1 'The first option contines to track the RHS_lag_matrix while better solutions are found If hit_end = 0 And tot_design_check <> 2 Then If (Round(krig_var, 7) >= Round(var_best, 7) And krig_var_old >= var_best) Then var_best = krig_var For f_dim_count_0 = 1 To CDbl(num_dim) RHS_lag_matrix_TEMP(1, f_dim_count_0) = RHS_lag_matrix(1, f_dim_count_0) Next 210 End If 'This temp variable is used to restore the 1st RHS value after the program hits the beginning or end while searching for better solutions temp_rhs_var = RHS_dis_matrix(1, 1) Else If dir_flag = 0 Then dir_flag = 1 ElseIf dir_flag = 1 Then dir_flag = 0 End If design_check_1 = tot_design_check tot_design_check = 0 If CDbl(pd_analysis.lin_cost) = 0 Then bud_remain = bud_remain - cost_tot Else bud_remain = bud_remain - (CDbl(pd_analysis.lin_cost) * cost_tot_new) cost_tot_new = (CDbl(pd_analysis.lin_cost) * cost_tot_new) End If sum_var = sum_var + var_best With lb .AddItem (.ListCount + 1) .Column(1, .ListCount - 1) = Round(var_best, 7) .Column(2, .ListCount - 1) = bud_remain If bud_remain <= 0 And Me.amt_to_paste.Value = "" Then Me.amt_to_paste.Value = .ListCount End If End With For f_dim_count_0 = 1 To CDbl(num_dim) RHS_lag_matrix(1, f_dim_count_0) = RHS_lag_matrix_TEMP(1, f_dim_count_0) Next 'This temp variable is used to restore the RHS value after the program hits the beginning or end while searching for better solutions temp_rhs_var = RHS_dis_matrix(1, 1) krig_var_old = var_best var_best = 0 loc_int = 1 Exit Do End If Loop 'This code is to paste the final FLC into the lists If (CDbl(lb.ListCount) = UBound(problem_space_array)) Then '1st have to determine which FLC is the last For fill_next_can_inp = 1 To (UBound(fp_inp_index_ordered) - 1) final_FLC = final_FLC + 1 If Abs(fp_inp_index_ordered(fill_next_can_inp) - fp_inp_index_ordered(fill_next_can_inp + 1)) > 1 Then 211 For f_dim_count_0 = 1 To CDbl(num_dim) RHS_lag_matrix(1, f_dim_count_0) = problem_space_array(fp_inp_index_ordered(fill_next_can_inp) + 1, f_dim_count_0) Next f_dim_count_0 final_FLC = final_FLC + 1 Exit For End If Next fill_next_can_inp For fill_next_can_inp = 1 To CDbl(lb_1.ListCount) f_euc_cal_0 = 0 For f_dim_count_0 = 1 To CDbl(num_dim) f_euc_cal_0 = f_euc_cal_0 + (problem_space_array(fp_inp_index_ordered(fill_next_can_inp), f_dim_count_0) - RHS_lag_matrix(1, f_dim_count_0)) ^ 2 Next RHS_dis_matrix(fill_next_can_inp, 1) = Sqr(f_euc_cal_0) Next fill_next_can_inp For fill_next_can_inp = 1 To CDbl(lb_1.ListCount) cand_input(fill_next_can_inp) = nugget + c1 * (1 - Exp((-3 * RHS_dis_matrix(fill_next_can_inp, 1)) / a)) Next fill_next_can_inp cand_input(fill_next_can_inp) = 1 krig_var = 0 'Calculate weights ArrInv() = Application.MInverse(cov_matrix) ArrAns() = Application.MMult(ArrInv, Application.Transpose(cand_input)) For krig_count = 1 To lb_1.ListCount + 1 krig_var = krig_var + ArrAns(krig_count, 1) * cand_input(krig_count) var_best = krig_var Next If CDbl(pd_analysis.lin_cost) = 0 Then bud_remain = bud_remain - cost_tot Else bud_remain = bud_remain - (CDbl(pd_analysis.lin_cost) * cost_tot_new) cost_tot_new = (CDbl(pd_analysis.lin_cost) * cost_tot_new) End If With lb .AddItem (.ListCount + 1) .Column(1, .ListCount - 1) = Round(var_best, 7) .Column(2, .ListCount - 1) = bud_remain If bud_remain <= 0 And Me.amt_to_paste.Value = "" Then Me.amt_to_paste.Value = .ListCount End If sum_var = sum_var + var_best End With With lb_1 212 .AddItem (final_FLC) For f_dim_count_0 = 1 To CDbl(num_dim) .Column(f_dim_count_0, .ListCount - 1) = RHS_lag_matrix(1, f_dim_count_0) Next f_dim_count_0 End With End If Loop 'This statements purges the original inp_out array since it is not used in the code anymore. Erase iTemp Erase iTemp_1 '***********End augmented simulated annealing algorithm************* With Me.amt_to_paste For past_data_ind = CDbl(lb_1.ListCount) To 1 Step -1 .AddItem past_data_ind, 0 Next past_data_ind End With For ei_pop = CDbl(2 ^ num_dim) To CDbl(lb.ListCount) lb.Column(3, ei_pop) = Round((CDbl(lb.Column(1, ei_pop)) / sum_var) * 100, 2) Next ei_pop 'Determine end time sngEnd = Timer ' Get end time. 'Determine time elapsed sngElapsed = Format(sngEnd - sngStart, "Fixed") ' Elapsed time. Me.time_elap.Text = sngElapsed Me.iter.Visible = False Me.neighborhood.Visible = True Me.Frame10.Visible = True Me.Frame11.Visible = True Me.Frame12.Visible = True Me.Frame13.Visible = True Me.Frame18.Visible = True Me.CommandButton1.Visible = True End Sub 'Sort code from postman2000 at http://www.ozgrid.com/forum/showthread.php?t=71509 Function SortListBox(oLb As MSForms.ListBox, sCol As Integer, sType As Integer, sDir As Integer) Dim vaItems As Variant Dim i As Long, j As Long Dim c As Integer Dim vTemp As Variant 'Put the items in a variant array vaItems = oLb.List 'Sort the Array Alphabetically(1) If sType = 1 Then For i = LBound(vaItems, 1) To UBound(vaItems, 1) - 1 For j = i + 1 To UBound(vaItems, 1) 213 'Sort Ascending (1) If sDir = 1 Then If vaItems(i, sCol) > vaItems(j, sCol) Then For c = 0 To oLb.ColumnCount - 1 'Allows sorting of multi-column ListBoxes vTemp = vaItems(i, c) vaItems(i, c) = vaItems(j, c) vaItems(j, c) = vTemp Next c End If 'Sort Descending (2) ElseIf sDir = 2 Then If vaItems(i, sCol) < vaItems(j, sCol) Then For c = 0 To oLb.ColumnCount - 1 'Allows sorting of multi-column ListBoxes vTemp = vaItems(i, c) vaItems(i, c) = vaItems(j, c) vaItems(j, c) = vTemp Next c End If End If Next j Next i 'Sort the Array Numerically(2) '(Substitute CInt with another conversion type (CLng, CDec, etc.) depending on type of numbers in the column) ElseIf sType = 2 Then For i = LBound(vaItems, 1) To UBound(vaItems, 1) - 1 For j = i + 1 To UBound(vaItems, 1) 'Sort Ascending (1) If sDir = 1 Then If CInt(vaItems(i, sCol)) > CInt(vaItems(j, sCol)) Then For c = 0 To oLb.ColumnCount - 1 'Allows sorting of multi-column ListBoxes vTemp = vaItems(i, c) vaItems(i, c) = vaItems(j, c) vaItems(j, c) = vTemp Next c End If 'Sort Descending (2) ElseIf sDir = 2 Then If CInt(vaItems(i, sCol)) < CInt(vaItems(j, sCol)) Then For c = 0 To oLb.ColumnCount - 1 'Allows sorting of multi-column ListBoxes vTemp = vaItems(i, c) vaItems(i, c) = vaItems(j, c) vaItems(j, c) = vTemp Next c End If End If 214 Next j Next i End If 'Set the list to the array oLb.List = vaItems End Function 'Post data to sheet Private Sub CommandButton1_Click() If Me.sam_location.ListCount = 0 Then err_msg_1 = MsgBox("You must run the test planning section above prior to pasting data into the worksheet.", vbOKOnly, "Error Handler") Exit Sub End If If pd_analysis.amt_to_paste.Value <> "" Then If CDbl(pd_analysis.amt_to_paste.Value) < 1 Then err_msg_1 = MsgBox("The sample size to paste must be an integer greater than 1.", vbOKOnly, "Error Handler") Exit Sub End If End If 'Calculate the amount to paste Dim past_counter As Integer 'Dim sum_var_1 As Double If Me.var_neg = True Then For ei_pop = 1 To (CDbl(Me.sam_selection.ListCount) - 1) sum_var_1 = sum_var_1 + CDbl(Me.sam_selection.Column(3, ei_pop + 2 ^ CDbl(Me.num_dim) - 1)) past_counter = past_counter + 1 If sum_var_1 > CDbl(Me.ComboBox1.Value) Then Me.amt_to_paste.Value = CDbl(Me.amt_to_paste.Value) + 2 ^ CDbl(Me.num_dim) Exit For End If Me.amt_to_paste.Value = past_counter Next ei_pop End If Dim past_data As Integer Worksheets("Test_Planning").Range("J1").Value = "n=" Worksheets("Test_Planning").Range("K1").Value = CDbl(pd_analysis.amt_to_paste.Value) Worksheets("Test_Planning").Range("J" & 3 + CDbl(Me.amt_to_paste.Value)).Value = "Pilot Design FLCs" Worksheets("Test_Planning").Range("J2").Value = "Sample Size" Worksheets("Test_Planning").Range("K2").Value = "Kriging Variance" Worksheets("Test_Planning").Range("L2").Value = "Balance($)" Worksheets("Test_Planning").Range("M2").Value = "EI" For remov_data = CDbl(Me.sam_selection.ListCount - 1) To (CDbl(Me.amt_to_paste.Value) + 1) Step -1 Me.sam_location.RemoveItem (Me.sam_location.ListCount - 1) Next remov_data 215 'Sort by the 1st column in the ListBox Numerically in Ascending Order Dim lb_1 As MSForms.ListBox Set lb_1 = Me.sam_location Evaluate SortListBox(lb_1, 0, 2, 1) Dim amount_pasted As Integer amount_pasted = CDbl(Me.amt_to_paste.Value) With Worksheets("Test_Planning") For past_data = 1 To amount_pasted With Worksheets("Test_Planning").Cells(2 + past_data, 10).Interior .Pattern = xlSolid .PatternColorIndex = xlAutomatic .ThemeColor = xlThemeColorDark2 .TintAndShade = -9.99786370433668E-02 .PatternTintAndShade = 0 End With With Worksheets("Test_Planning").Cells(2 + past_data, 11).Interior .Pattern = xlSolid .PatternColorIndex = xlAutomatic .ThemeColor = xlThemeColorDark2 .TintAndShade = -9.99786370433668E-02 .PatternTintAndShade = 0 End With With Worksheets("Test_Planning").Cells(2 + past_data, 12).Interior .Pattern = xlSolid .PatternColorIndex = xlAutomatic .ThemeColor = xlThemeColorDark2 .TintAndShade = -9.99786370433668E-02 .PatternTintAndShade = 0 End With With Worksheets("Test_Planning").Cells(2 + past_data, 13).Interior .Pattern = xlSolid .PatternColorIndex = xlAutomatic .ThemeColor = xlThemeColorDark2 .TintAndShade = -9.99786370433668E-02 .PatternTintAndShade = 0 End With 'First list box items .Range("J" & 2 + past_data).Value = Me.sam_selection.Column(0, past_data - 1) .Range("K" & 2 + past_data).Value = Me.sam_selection.Column(1, past_data - 1) .Range("L" & 2 + past_data).Value = Me.sam_selection.Column(2, past_data - 1) .Range("M" & 2 + past_data).Value = Me.sam_selection.Column(3, past_data - 1) For f_dim_count_0 = 1 To CDbl(num_dim) 216 With Worksheets("Test_Planning").Cells(5 + amount_pasted + past_data - 1, 9 + f_dim_count_0).Interior .Pattern = xlSolid .PatternColorIndex = xlAutomatic .ThemeColor = xlThemeColorAccent6 .TintAndShade = 0.799981688894314 .PatternTintAndShade = 0 End With .Range((Chr(73 + f_dim_count_0)) & 5 + amount_pasted + past_data - 1).Value = CDbl(Me.sam_location.Column(f_dim_count_0, past_data - 1)) Next f_dim_count_0 Next past_data For f_dim_count_0 = 1 To CDbl(num_dim) .Range((Chr(73 + f_dim_count_0)) & 4 + amount_pasted).Value = "X" & f_dim_count_0 Next f_dim_count_0 .Range((Chr(73 + f_dim_count_0)) & 4 + amount_pasted).Value = "Y" End With Columns("J:J").EntireColumn.AutoFit Columns("K:K").EntireColumn.AutoFit Columns("L:L").EntireColumn.AutoFit Columns("M:M").EntireColumn.AutoFit Unload pd_analysis End Sub Appendix A.3: Main DFK Kriging Form Option Explicit Private Sub CheckBox1_Click() If SemiForm.Label17.Enabled = False Then SemiForm.Label17.Enabled = True SemiForm.ComboBox1.Enabled = True Else SemiForm.Label17.Enabled = False SemiForm.ComboBox1.Enabled = False End If End Sub Private Sub exp_var_confirm_Click() If SemiForm.exp_var_select <> "Best Estimate (Recommended)" Then SemiForm.nug.Enabled = False SemiForm.sil.Enabled = True SemiForm.ran.Enabled = True Else SemiForm.nug.Enabled = False SemiForm.sil.Enabled = False SemiForm.ran.Enabled = False End If 217 End Sub '*******Help information*********************************** Private Sub Image3_Click() Krig_Help.TextBox1.Text = "This is where the user selects which inputs to use. The selection is to be made by columns. If the selected inputs are a single column " & _ "then it is considered a single dimensional problem. If multiple columns are selected then it is considered a multidimensional problem. Each row is considered " & _ "a seperate input value." Krig_Help.Show End Sub Private Sub Image4_Click() Krig_Help.TextBox1.Text = "This is the lag or also known as binning as it takes the calculated differences and puts them into predetermined bins. This speeds up " & _ "calculation time. The lag aides in properly calculating the experimental semivariogram. It generally will reduce the number of experimental semivariogram points therefore causing the potential for using a fitted model that yields " & _ "a less accurate prediction value. Selecting (Default) will allow the software to try to properly determine the lag/bin size. It is recommended unless a certain lag is required or even preferred." Krig_Help.Show End Sub Private Sub Image5_Click() Krig_Help.TextBox1.Text = "This is the neighborhood. The purpose here is to set bounds for data where the data may not have a strong correlation due to " & _ "the large distances apart. For example, it is hard to say that a sample of soil data taken in the United States is correlated with a soil sample " & _ "taken in Turkey. So, this option allows you to rule out those types of values." Krig_Help.Show End Sub Private Sub Image6_Click() Krig_Help.TextBox1.Text = "This selection is the output. It should directly correspond to the input values, have an equal number of " & _ "values as the inputs, and is limited to a single column." Krig_Help.Show End Sub Private Sub Image7_Click() Krig_Help.TextBox1.Text = "Here is where the user can select specific fitted models to use. If a specific model is selected then " & _ "the user must specify a nugget, sill, and range value all of which must be greater than or equal to zero. It is recommended to let " & _ "the software determine the best fitted model. It accomplishes this through two methods. One method regressors the variogram and then performs " & _ 218 "GRG or EA. The second method perfroms multisampling of various posssible values and selects " & _ "whichever model results in the lowest MSE between the experimental semivariogram and the fitted models. NOTE: If EA is required the software may require " & _ "more CPU time to process the results." Krig_Help.Show End Sub Private Sub Image8_Click() Krig_Help.TextBox1.Text = "Here is where the user can select whichever Kriging type best suits the need of the problem at hand. " & _ "The assumptions should give some further insight into the proper selection although Ordinary Kriging is the generally the most common." Krig_Help.Show End Sub Private Sub Image9_Click() Krig_Help.TextBox1.Text = "Here is where the user can input the nugget, sill, and range if a specific fitted model has been chosen. " & _ "The nugget value is the y-intercept value when looking at the fitted models on a graph. If this value is non zero (discontinous) " & _ "then noise may be present or potential measurement errors may exist (Chiles, 1999). The sill is the limited value of the variogram. " & _ "It states the distance where Z(h) and Z(x+h) start to become uncorrelated (Chiles, 1999). The range is the x-axis distance value that " & _ "corresponds to the sill value." Krig_Help.Show End Sub Private Sub Image10_Click() Krig_Help.TextBox1.Text = "This is essentially the point at which to be predicted. A single column selection corresponds with a single dimension " & _ "and a multicoulmn selection corresponds to a multi-dimensionsal point. If the resolution is entered then this value is the starting value prediction value of the series." Krig_Help.Show End Sub Private Sub Image11_Click() Krig_Help.TextBox1.Text = "This is the resolution in which to perform multiple Kriging predictions at one time. " & _ "The predictions will begin at beginning FLC and end once the prediction value is outside " & _ "the upper bound of the original input data set. The increment counter will be set to the value of the first dimension of the first FLC." Krig_Help.Show End Sub '*******End Help information********************************** Private Sub krigtype_AfterUpdate() 'Populate the Kriging Assumptions in the GUI If SemiForm.KrigType = "Ordinary Kriging" Then 219 SemiForm.KrigAssum.Text = "-Data is spatially correlated in Euclydian space i.e. the closer the distance between them the more correlated." & Chr(13) & "-The number of values per input variable has a large range." & Chr(13) & "-The predictions have the exact same value as the observed observation output." 'ElseIf SemiForm.KrigType = "Universal Kriging" Then ' SemiForm.KrigAssum.Text = "Test." End If End Sub Private Sub CommandButton1_Click() Dim OriginalSampleRange, OutSample As String Dim err_msg_1 'Get user input data 'Get exact location of the cell as a string which can be used in a Range Object with all usual properties 'Err Handling If SemiForm.RefEditOriginalSampleRange.Value = "" Then err_msg_1 = MsgBox("You did not select any inputs. The program will exit now.", vbOKOnly, "Error Handler") Exit Sub End If If SemiForm.Refoutdata.Value = "" Then err_msg_1 = MsgBox("You did not select any inputs. The program will exit now.", vbOKOnly, "Error Handler") Exit Sub End If 'End Err Handling OriginalSampleRange = Range(SemiForm.RefEditOriginalSampleRange.Value).Address(external:=True) ' Turn off screen updating Application.ScreenUpdating = True 'Call the main module to do all the calculations calc_semiv_model_value (OriginalSampleRange) Unload Me End Sub Private Sub CommandButton2_Click() ' Put away the form Unload SemiForm Exit Sub End Sub Appendix A.4: DFK Help Form Private Sub CommandButton1_Click() Unload Me End Sub 220 Appendix A.5: Introduction Form Private Sub CommandButton1_Click() ' Put away the form Unload Me 'Load form pd_analyze_start End Sub Private Sub CommandButton2_Click() ' Put away the form Unload Me 'Load form ShowSemiForm End Sub Private Sub CommandButton3_Click() ' Put away the form Unload Kriging_Intro Exit Sub End Sub Private Sub CommandButton5_Click() Krig_Help.TextBox1.Text = "There are several forms of Kriging, all with the intent to estimate a continuous, spatial attribute at an unsampled site. Kriging is a form of generalized linear regression for the formulation of an optimal estimator in a minimum mean square error sense. Simple Kriging provides a gateway into more detailed methods of Kriging. Simple Kriging is limited due to its simplicity and embedded assumptions. Ordinary Kriging, the method used in this research, is the most widely used Kriging method and is based off of many of the principles founded in simple Kriging. The acronym B.L.U.E is associated with ordinary Kriging. The acronym stands for best linear unbiased estimator. Ordinary Kriging is linear since it estimates weighted linear combinations of data. It is unbiased since it tries to have the mean residual equal to 0." _ & "Finally, ordinary Kriging is considered best since it tries to minimize the variance of the errors. Practically speaking, the goal of ordinary Kriging is unattainable as the mean error and variance are always unknown. This implies that it cannot guarantee the mean error is equal to 0 or that the variance is minimized. The best attempt is to build a model of the data that is available and work with the average error and the error variance. In ordinary Kriging, a probability model is used such that the bias and error variance can both be calculated. By choosing weights for nearby samples this ensures that the average error for the model is exactly 0 and that the error variance is minimized." Krig_Help.Show End Sub Appendix A.6: Results Form Private Sub CommandButton1_Click() 221 Unload Results_Form End Sub Appendix A.7: Workbook Code to Add Ribbon Bar Menu ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' ' START ThisWorkbook Code Module ' Created By Chip Pearson, chip@cpearson.com ' Sample code for Creating An Add-In at http://www.cpearson.com/Excel/CreateAddIn.aspx ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' '''' ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' '''' Option Explicit Private Const C_TAG = "Kriging" ' C_TAG should be a string unique to this add-in. Private Const C_TOOLS_MENU_ID As Long = 30007& Private Sub Workbook_Open() ''''''''''''''''''''''''''''''''''''''''''''''' ' Workbook_Open ' Create a submenu on the Tools menu. The ' submenu has two controls on it. ''''''''''''''''''''''''''''''''''''''''''''''' Dim ToolsMenu As Office.CommandBarControl Dim ToolsMenuItem As Office.CommandBarControl Dim ToolsMenuControl As Office.CommandBarControl ''''''''''''''''''''''''''''''''''''''''''''''' ' First delete any of our controls that ' may not have been properly deleted previously. ''''''''''''''''''''''''''''''''''''''''''''''' DeleteControls '''''''''''''''''''''''''''''''''''''''''''''' ' Get a reference to the Tools menu. '''''''''''''''''''''''''''''''''''''''''''''' Set ToolsMenu = Application.CommandBars.FindControl(ID:=C_TOOLS_MENU_ID) If ToolsMenu Is Nothing Then MsgBox "Unable to access Tools menu.", vbOKOnly Exit Sub End If '''''''''''''''''''''''''''''''''''''''''''''' ' Create a item on the Tools menu. '''''''''''''''''''''''''''''''''''''''''''''' 222 Set ToolsMenuItem = ToolsMenu.Controls.Add(Type:=msoControlPopup, temporary:=True) If ToolsMenuItem Is Nothing Then MsgBox "Unable to add item to the Tools menu.", vbOKOnly Exit Sub End If With ToolsMenuItem .Caption = "&Kriging" .BeginGroup = True .Tag = C_TAG '.OnAction = "'" & ThisWorkbook.Name & "'!Kriging_start" End With '''''''''''''''''''''''''''''''''''''''''''''' ' Create the first control on the new item ' in the Tools menu. '''''''''''''''''''''''''''''''''''''''''''''' Set ToolsMenuControl = ToolsMenuItem.Controls.Add(Type:=msoControlButton, temporary:=True) If ToolsMenuControl Is Nothing Then MsgBox "Unable to add item to Tools menu item.", vbOKOnly Exit Sub End If With ToolsMenuControl '''''''''''''''''''''''''''''''''''' ' Set the display caption and the ' procedure to run when clicked. '''''''''''''''''''''''''''''''''''' .Caption = "Kriging Start" .OnAction = "'" & ThisWorkbook.Name & "'!Kriging_start" .Tag = C_TAG End With 'With ToolsMenuControl '''''''''''''''''''''''''''''''''''' ' Set the display caption and the ' procedure to run when clicked. '''''''''''''''''''''''''''''''''''' ' .Caption = "Get FLC" ' .OnAction = "'" & ThisWorkbook.Name & "'!get_flc" ' .Tag = C_TAG 'End With '''''''''''''''''''''''''''''''''''''''''''''' ' Create the first control on the new item ' in the Tools menu. '''''''''''''''''''''''''''''''''''''''''''''' 223 Set ToolsMenuControl = ToolsMenuItem.Controls.Add(Type:=msoControlButton, temporary:=True) If ToolsMenuControl Is Nothing Then MsgBox "Unable to add item to Tools menu item.", vbOKOnly Exit Sub End If With ToolsMenuControl '''''''''''''''''''''''''''''''''''' ' Set the display caption and the ' procedure to run when clicked. '''''''''''''''''''''''''''''''''''' .Caption = "Test Planning" .OnAction = "'" & ThisWorkbook.Name & "'!pd_analyze_start" .Tag = C_TAG End With Set ToolsMenuControl = ToolsMenuItem.Controls.Add(Type:=msoControlButton, temporary:=True) If ToolsMenuControl Is Nothing Then MsgBox "Unable to add item to Tools menu item.", vbOKOnly Exit Sub End If With ToolsMenuControl '''''''''''''''''''''''''''''''''''' ' Set the display caption and the ' procedure to run when clicked. '''''''''''''''''''''''''''''''''''' .Caption = "Sequential Kriging" .OnAction = "'" & ThisWorkbook.Name & "'!ShowSemiForm" .Tag = C_TAG End With End Sub Private Sub Workbook_BeforeClose(Cancel As Boolean) '''''''''''''''''''''''''''''''''''''''''''''''''''' ' Workbook_BeforeClose ' Before closing the add-in, clean up our controls. '''''''''''''''''''''''''''''''''''''''''''''''''''' DeleteControls End Sub Private Sub DeleteControls() '''''''''''''''''''''''''''''''''''' ' Delete controls whose Tag is ' equal to C_TAG. 224 '''''''''''''''''''''''''''''''''''' Dim Ctrl As Office.CommandBarControl On Error Resume Next Set Ctrl = Application.CommandBars.FindControl(Tag:=C_TAG) Do Until Ctrl Is Nothing Ctrl.Delete Set Ctrl = Application.CommandBars.FindControl(Tag:=C_TAG) Loop End Sub ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' ' END ThisWorkbook Code Module ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' 225 Appendix B: Remaining Bootstrapped Data for Original DFK Process This appendix presents the remaining bootstrapped data from Chapter 4. The bootstrap replicates 1 and 25 are displayed in Chapter 4. These replicates were omitted in the body of the paper as the two displayed replicates are enough for demonstration of the mathematical method involved in performing the bootstrapped calculations. Bootstrap Replicate 2 ? 0.067 0.097 0.161 0.091 0.091 0.097 0.091 0.091 0.125 0.161 0.107 0.257 0.281 0.292 0.203 0.659 0.203 0.659 0.290 0.203 0.292 0.334 0.322 0.805 0.805 0.351 0.193 0.805 0.193 0.132 0.264 0.351 0.422 0.216 0.412 0.216 0.268 0.392 0.392 0.392 0.403 0.412 0.403 0.350 0.188 0.976 0.363 0.421 0.421 0.421 0.219 1.100 0.421 0.421 0.495 Bootstrap Replicate 3 ? 0.125 0.125 0.097 0.091 0.091 0.091 0.091 0.125 0.125 0.091 0.105 0.292 0.257 0.281 0.257 0.292 0.292 0.203 0.203 0.290 0.257 0.262 0.193 0.264 0.322 0.322 0.351 0.193 0.805 0.264 0.264 0.805 0.378 0.412 0.268 0.216 0.392 0.268 0.268 0.412 0.268 0.268 0.216 0.299 0.363 0.188 0.976 0.188 0.976 0.363 0.188 0.188 0.219 0.976 0.462 Bootstrap Replicate 4 ? 0.067 0.091 0.125 0.067 0.097 0.091 0.161 0.125 0.125 0.067 0.101 0.257 0.281 0.659 0.281 0.292 0.659 0.281 0.257 0.257 0.203 0.343 0.805 0.193 0.264 0.351 0.193 0.132 0.805 0.351 0.805 0.193 0.409 0.392 0.392 0.268 0.553 0.216 0.553 0.392 0.392 0.403 0.268 0.383 0.421 0.219 0.219 0.421 0.219 1.100 1.100 0.363 0.219 1.100 0.538 Bootstrap Replicate 5 ? 0.125 0.161 0.125 0.161 0.161 0.097 0.097 0.091 0.067 0.097 0.118 0.257 0.203 0.659 0.257 0.203 0.292 0.292 0.290 0.290 0.257 0.300 0.193 0.805 0.193 0.351 0.351 0.322 0.351 0.805 0.193 0.322 0.389 0.412 0.412 0.392 0.216 0.553 0.403 0.553 0.216 0.412 0.392 0.396 0.219 0.363 0.421 0.976 1.100 0.421 0.188 0.976 0.976 0.976 0.661 226 Bootstrap Replicate 6 ? 0.125 0.091 0.091 0.067 0.067 0.125 0.091 0.091 0.067 0.125 0.094 0.281 0.203 0.257 0.203 0.290 0.281 0.290 0.203 0.292 0.203 0.250 0.193 0.322 0.322 0.805 0.132 0.193 0.322 0.322 0.351 0.193 0.316 0.403 0.553 0.268 0.392 0.392 0.392 0.268 0.553 0.412 0.392 0.402 0.363 0.363 1.100 0.421 1.100 0.363 0.976 0.363 0.219 0.219 0.549 Bootstrap Replicate 7 ? 0.067 0.125 0.067 0.067 0.091 0.091 0.091 0.091 0.091 0.161 0.094 0.659 0.292 0.257 0.203 0.292 0.203 0.290 0.292 0.281 0.292 0.306 0.193 0.351 0.322 0.322 0.805 0.193 0.264 0.193 0.132 0.351 0.313 0.392 0.553 0.412 0.216 0.412 0.268 0.392 0.412 0.216 0.553 0.382 1.100 0.188 1.100 0.363 0.976 0.421 0.421 0.363 0.421 0.421 0.577 Bootstrap Replicate 8 ? 0.161 0.161 0.097 0.067 0.091 0.091 0.067 0.091 0.161 0.125 0.111 0.290 0.659 0.281 0.257 0.203 0.203 0.292 0.281 0.659 0.292 0.342 0.322 0.132 0.351 0.132 0.193 0.351 0.351 0.264 0.132 0.805 0.303 0.268 0.403 0.216 0.412 0.392 0.268 0.216 0.268 0.412 0.392 0.325 0.219 1.100 0.363 1.100 1.100 0.219 0.188 0.976 0.421 1.100 0.679 Bootstrap Replicate 9 ? 0.125 0.091 0.091 0.161 0.161 0.125 0.067 0.091 0.091 0.091 0.109 0.203 0.290 0.203 0.257 0.257 0.290 0.292 0.257 0.257 0.257 0.256 0.351 0.322 0.322 0.351 0.322 0.264 0.132 0.132 0.132 0.264 0.259 0.553 0.268 0.403 0.553 0.392 0.403 0.403 0.553 0.268 0.403 0.420 0.188 0.188 0.188 0.363 1.100 0.421 0.976 1.100 0.421 0.363 0.531 Bootstrap Replicate 10 ? 0.161 0.097 0.067 0.067 0.091 0.161 0.125 0.091 0.091 0.097 0.105 0.281 0.281 0.203 0.290 0.281 0.281 0.292 0.281 0.257 0.257 0.270 0.351 0.351 0.351 0.805 0.264 0.132 0.322 0.264 0.805 0.264 0.391 0.403 0.412 0.392 0.412 0.412 0.392 0.403 0.268 0.553 0.412 0.406 0.219 0.976 0.188 0.188 1.100 0.219 0.421 0.363 0.188 1.100 0.496 227 Bootstrap Replicate 11 ? 0.125 0.161 0.067 0.067 0.097 0.067 0.097 0.161 0.125 0.097 0.106 0.659 0.290 0.290 0.257 0.292 0.281 0.281 0.290 0.290 0.290 0.322 0.805 0.351 0.264 0.193 0.132 0.132 0.351 0.264 0.322 0.351 0.317 0.412 0.403 0.268 0.392 0.412 0.403 0.216 0.268 0.216 0.216 0.320 0.219 0.976 0.188 1.100 0.976 0.188 1.100 0.421 0.188 0.219 0.557 Bootstrap Replicate 12 ? 0.067 0.097 0.091 0.091 0.091 0.097 0.091 0.125 0.125 0.091 0.097 0.290 0.290 0.659 0.290 0.257 0.292 0.203 0.290 0.281 0.257 0.311 0.351 0.132 0.132 0.351 0.805 0.193 0.322 0.264 0.351 0.322 0.322 0.392 0.403 0.403 0.392 0.553 0.412 0.403 0.216 0.553 0.412 0.414 0.976 0.976 0.188 0.363 0.188 0.976 0.188 1.100 0.188 0.363 0.550 Bootstrap Replicate 13 ? 0.097 0.097 0.161 0.067 0.067 0.091 0.091 0.097 0.125 0.091 0.098 0.659 0.659 0.281 0.292 0.290 0.659 0.281 0.290 0.292 0.281 0.399 0.193 0.132 0.132 0.351 0.351 0.193 0.132 0.351 0.193 0.193 0.222 0.553 0.412 0.403 0.268 0.553 0.216 0.216 0.553 0.403 0.268 0.384 1.100 0.421 0.363 0.976 0.976 1.100 0.363 0.363 0.363 0.976 0.700 Bootstrap Replicate 14 ? 0.067 0.091 0.097 0.091 0.125 0.091 0.125 0.097 0.091 0.091 0.097 0.281 0.281 0.257 0.659 0.257 0.281 0.290 0.281 0.659 0.659 0.391 0.805 0.193 0.132 0.351 0.322 0.264 0.351 0.322 0.264 0.805 0.381 0.268 0.412 0.412 0.392 0.216 0.268 0.392 0.268 0.216 0.268 0.311 1.100 0.363 0.188 0.219 0.363 0.219 0.188 0.188 0.976 0.363 0.417 Bootstrap Replicate 15 ? 0.067 0.067 0.091 0.097 0.097 0.091 0.091 0.161 0.091 0.091 0.094 0.659 0.290 0.290 0.292 0.203 0.292 0.203 0.203 0.203 0.203 0.284 0.264 0.193 0.193 0.193 0.132 0.351 0.132 0.351 0.132 0.351 0.229 0.392 0.403 0.553 0.403 0.553 0.553 0.412 0.403 0.553 0.216 0.444 0.219 0.976 0.219 0.976 0.976 0.421 0.976 1.100 0.363 1.100 0.733 228 Bootstrap Replicate 16 ? 0.125 0.097 0.067 0.125 0.125 0.091 0.125 0.091 0.161 0.125 0.113 0.203 0.281 0.292 0.659 0.292 0.290 0.290 0.292 0.203 0.290 0.309 0.805 0.264 0.264 0.351 0.193 0.351 0.132 0.264 0.132 0.805 0.356 0.412 0.412 0.392 0.268 0.403 0.216 0.553 0.553 0.412 0.403 0.402 0.188 0.976 0.421 0.219 0.976 0.421 0.219 0.976 1.100 0.363 0.586 Bootstrap Replicate 17 ? 0.097 0.097 0.091 0.161 0.067 0.091 0.125 0.161 0.091 0.161 0.114 0.290 0.203 0.292 0.203 0.257 0.257 0.257 0.257 0.257 0.257 0.253 0.193 0.264 0.132 0.322 0.132 0.193 0.264 0.264 0.351 0.132 0.225 0.268 0.553 0.268 0.412 0.216 0.412 0.412 0.412 0.392 0.553 0.390 0.363 0.363 0.976 0.188 0.363 0.421 0.188 0.421 0.976 0.219 0.448 Bootstrap Replicate 18 ? 0.091 0.067 0.125 0.091 0.067 0.097 0.067 0.091 0.067 0.091 0.085 0.292 0.203 0.659 0.257 0.257 0.292 0.203 0.281 0.659 0.257 0.336 0.132 0.351 0.351 0.132 0.805 0.351 0.351 0.351 0.805 0.805 0.444 0.392 0.392 0.268 0.392 0.216 0.268 0.268 0.412 0.553 0.403 0.356 0.188 0.219 1.100 0.219 0.188 0.188 1.100 0.976 0.363 0.219 0.476 Bootstrap Replicate 19 ? 0.097 0.067 0.067 0.091 0.097 0.091 0.097 0.091 0.125 0.161 0.098 0.203 0.257 0.659 0.257 0.290 0.290 0.659 0.659 0.292 0.292 0.386 0.351 0.322 0.322 0.322 0.351 0.132 0.351 0.193 0.805 0.805 0.396 0.412 0.392 0.392 0.392 0.553 0.216 0.553 0.216 0.392 0.412 0.393 0.976 0.363 0.976 0.188 0.188 0.219 0.421 0.421 0.219 0.976 0.495 Bootstrap Replicate 20 ? 0.161 0.097 0.091 0.091 0.067 0.097 0.097 0.125 0.067 0.097 0.099 0.257 0.292 0.281 0.292 0.281 0.203 0.203 0.257 0.203 0.292 0.256 0.351 0.805 0.193 0.132 0.132 0.193 0.805 0.322 0.351 0.264 0.355 0.412 0.216 0.403 0.412 0.412 0.216 0.268 0.412 0.412 0.392 0.355 0.976 0.188 0.421 0.219 0.219 0.976 1.100 0.976 0.976 0.976 0.703 229 Bootstrap Replicate 21 ? 0.161 0.091 0.097 0.091 0.097 0.125 0.161 0.097 0.091 0.067 0.108 0.203 0.281 0.659 0.290 0.257 0.292 0.257 0.257 0.257 0.257 0.301 0.264 0.322 0.193 0.264 0.351 0.193 0.322 0.351 0.193 0.264 0.272 0.216 0.403 0.553 0.403 0.412 0.392 0.553 0.268 0.553 0.412 0.416 0.363 0.188 0.421 0.363 0.363 0.976 1.100 0.421 1.100 0.188 0.548 Bootstrap Replicate 22 ? 0.067 0.091 0.125 0.091 0.067 0.067 0.091 0.091 0.091 0.125 0.091 0.257 0.203 0.659 0.203 0.659 0.290 0.257 0.257 0.290 0.203 0.328 0.193 0.322 0.351 0.351 0.193 0.351 0.351 0.351 0.351 0.322 0.314 0.403 0.392 0.412 0.553 0.403 0.216 0.392 0.216 0.412 0.403 0.380 0.976 0.219 0.188 0.421 0.421 0.976 0.363 0.363 0.219 0.421 0.457 Bootstrap Replicate 23 ? 0.125 0.097 0.067 0.091 0.091 0.097 0.067 0.125 0.067 0.161 0.099 0.203 0.281 0.290 0.281 0.292 0.290 0.292 0.292 0.257 0.659 0.314 0.264 0.193 0.264 0.351 0.351 0.193 0.322 0.351 0.132 0.132 0.255 0.403 0.403 0.403 0.216 0.268 0.553 0.392 0.403 0.392 0.392 0.382 0.976 0.421 0.363 0.188 0.188 0.188 0.976 0.363 0.188 0.188 0.404 Bootstrap Replicate 24 ? 0.161 0.067 0.125 0.091 0.097 0.091 0.091 0.091 0.091 0.091 0.100 0.203 0.203 0.257 0.659 0.290 0.257 0.257 0.292 0.203 0.292 0.291 0.264 0.193 0.132 0.805 0.264 0.351 0.351 0.193 0.193 0.351 0.310 0.553 0.268 0.268 0.268 0.412 0.392 0.216 0.392 0.412 0.412 0.359 0.219 0.976 1.100 0.363 0.188 0.188 1.100 0.421 0.421 1.100 0.608 230 Appendix C: Varying Sill and Nugget Variogram Tables and Plots This appendix further demonstrates the relationships in fitted variogram functions. The appendix is broken up into two sections. The first section demonstrates how the variogram functions vary when the sill varies up to the maximum lag. The results are visually presented through four plots at the end of the first section. The second section demonstrates the same as the first section with the exception that the nugget is varied while the range and sill remain constant. Appendix C.1: Varying Sill Variogram Tables and Plots Linear 1 9 0.1111 0.2222 0.3333 0.4444 0.5556 0.6667 0.7778 0.8889 1.0000 2 9 0.2222 0.4444 0.6667 0.8889 1.1111 1.3333 1.5556 1.7778 2.0000 3 9 0.3333 0.6667 1.0000 1.3333 1.6667 2.0000 2.3333 2.6667 3.0000 4 9 0.4444 0.8889 1.3333 1.7778 2.2222 2.6667 3.1111 3.5556 4.0000 5 9 0.5556 1.1111 1.6667 2.2222 2.7778 3.3333 3.8889 4.4444 5.0000 6 9 0.6667 1.3333 2.0000 2.6667 3.3333 4.0000 4.6667 5.3333 6.0000 7 9 0.7778 1.5556 2.3333 3.1111 3.8889 4.6667 5.4444 6.2222 7.0000 8 9 0.8889 1.7778 2.6667 3.5556 4.4444 5.3333 6.2222 7.1111 8.0000 9 9 1.0000 2.0000 3.0000 4.0000 5.0000 6.0000 7.0000 8.0000 9.0000 231 Spherical 1 9 0.1660 0.3278 0.4815 0.6228 0.7476 0.8519 0.9314 0.9822 0.1660 2 9 0.3320 0.6557 0.9630 1.2455 1.4952 1.7037 1.8628 1.9643 0.3320 3 9 0.4979 0.9835 1.4444 1.8683 2.2428 2.5556 2.7942 2.9465 0.4979 4 9 0.6639 1.3114 1.9259 2.4911 2.9904 3.4074 3.7257 3.9287 0.6639 5 9 0.8299 1.6392 2.4074 3.1139 3.7380 4.2593 4.6571 4.9108 0.8299 6 9 0.9959 1.9671 2.8889 3.7366 4.4856 5.1111 5.5885 5.8930 0.9959 7 9 1.1619 2.2949 3.3704 4.3594 5.2332 5.9630 6.5199 6.8752 1.1619 8 9 1.3278 2.6228 3.8519 4.9822 5.9808 6.8148 7.4513 7.8573 1.3278 9 9 1.4938 2.9506 4.3333 5.6049 6.7284 7.6667 8.3827 8.8395 1.4938 Exponential 1 9 0.2835 0.4866 0.6321 0.7364 0.8111 0.8647 0.9030 0.9305 0.9502 2 9 0.5669 0.9732 1.2642 1.4728 1.6222 1.7293 1.8061 1.8610 1.9004 3 9 0.8504 1.4597 1.8964 2.2092 2.4334 2.5940 2.7091 2.7915 2.8506 4 9 1.1339 1.9463 2.5285 2.9456 3.2445 3.4587 3.6121 3.7221 3.8009 5 9 1.4173 2.4329 3.1606 3.6820 4.0556 4.3233 4.5151 4.6526 4.7511 6 9 1.7008 2.9195 3.7927 4.4184 4.8667 5.1880 5.4182 5.5831 5.7013 7 9 1.9843 3.4061 4.4248 5.1548 5.6779 6.0527 6.3212 6.5136 6.6515 8 9 2.2677 3.8927 5.0570 5.8912 6.4890 6.9173 7.2242 7.4441 7.6017 9 9 2.5512 4.3792 5.6891 6.6276 7.3001 7.7820 8.1273 8.3746 8.5519 Gaussian 1 9 0.950213 0.999994 0.0364 0.1377 0.2835 0.4471 0.6038 0.7364 0.8371 2 9 1.900426 1.999988 0.0727 0.2754 0.5669 0.8942 1.2077 1.4728 1.6743 3 9 2.850639 2.999982 0.1091 0.4131 0.8504 1.3413 1.8115 2.2092 2.5114 4 9 3.800852 3.999975 0.1454 0.5508 1.1339 1.7884 2.4153 2.9456 3.3485 5 9 4.751065 4.999969 0.1818 0.6885 1.4173 2.2355 3.0192 3.6820 4.1857 6 9 5.701278 5.999963 0.2182 0.8262 1.7008 2.6826 3.6230 4.4184 5.0228 7 9 6.651491 6.999957 0.2545 0.9639 1.9843 3.1298 4.2268 5.1548 5.8599 8 9 7.601703 7.999951 0.2909 1.1016 2.2677 3.5769 4.8307 5.8912 6.6971 9 9 8.551916 8.999945 0.3272 1.2393 2.5512 4.0240 5.4345 6.6276 7.5342 232 0.0000 1.0000 2.0000 3.0000 4.0000 5.0000 6.0000 7.0000 8.0000 9.0000 10.0000 1 2 3 4 5 6 7 8 9 Linear Series1 Series2 Series3 Series4 Series5 Series6 Series7 Series8 Series9 0.0000 1.0000 2.0000 3.0000 4.0000 5.0000 6.0000 7.0000 8.0000 9.0000 10.0000 1 2 3 4 5 6 7 8 9 Spherical Series1 Series2 Series3 Series4 Series5 Series6 Series7 Series8 Series9 233 0.0000 1.0000 2.0000 3.0000 4.0000 5.0000 6.0000 7.0000 8.0000 9.0000 1 2 3 4 5 6 7 8 9 Exponential Series1 Series2 Series3 Series4 Series5 Series6 Series7 Series8 Series9 0.0000 1.0000 2.0000 3.0000 4.0000 5.0000 6.0000 7.0000 8.0000 9.0000 1 2 3 4 5 6 7 8 9 Gaussian Series1 Series2 Series3 Series4 Series5 Series6 Series7 Series8 Series9 234 Appendix C.2: Varying Nugget Variogram Tables and Plots Nugget = 1 Spherical 9 1 10.0000 10.0000 10.0000 10.0000 10.0000 10.0000 10.000 10.000 10.000 9 2 7.1875 10.0000 10.0000 10.0000 10.0000 10.0000 10.000 10.000 10.000 9 3 5.3333 8.6667 10.0000 10.0000 10.0000 10.0000 10.000 10.000 10.000 9 4 4.3047 7.1875 9.2266 10.0000 10.0000 10.0000 10.000 10.000 10.000 9 5 3.6640 6.1120 8.1280 9.4960 10.0000 10.0000 10.000 10.000 10.000 9 6 3.2292 5.3333 7.1875 8.6667 9.6458 10.0000 10.000 10.000 10.000 9 7 2.9155 4.7522 6.4315 7.8746 9.0029 9.7376 10.000 10.000 10.000 9 8 2.6787 4.3047 5.8252 7.1875 8.3389 9.2266 9.798 10.000 10.000 9 9 2.4938 3.9506 5.3333 6.6049 7.7284 8.6667 9.383 9.84 10.000 Nugget = 2 Spherical 9 1 11.0000 11.0000 11.0000 11.0000 11.0000 11.0000 11.000 11.000 11.000 9 2 8.1875 11.0000 11.0000 11.0000 11.0000 11.0000 11.000 11.000 11.000 9 3 6.3333 9.6667 11.0000 11.0000 11.0000 11.0000 11.000 11.000 11.000 9 4 5.3047 8.1875 10.2266 11.0000 11.0000 11.0000 11.000 11.000 11.000 9 5 4.6640 7.1120 9.1280 10.4960 11.0000 11.0000 11.000 11.000 11.000 9 6 4.2292 6.3333 8.1875 9.6667 10.6458 11.0000 11.000 11.000 11.000 9 7 3.9155 5.7522 7.4315 8.8746 10.0029 10.7376 11.000 11.000 11.000 9 8 3.6787 5.3047 6.8252 8.1875 9.3389 10.2266 10.798 11.000 11.000 9 9 3.4938 4.9506 6.3333 7.6049 8.7284 9.6667 10.383 10.84 11.000 Nugget = 3 235 Spherical 9 1 12.000 12.000 12.0000 12.0000 12.0000 12.0000 12.0000 12.0000 12.000 9 2 9.188 12.000 12.0000 12.0000 12.0000 12.0000 12.0000 12.0000 12.000 9 3 7.333 10.667 12.0000 12.0000 12.0000 12.0000 12.0000 12.0000 12.000 9 4 6.305 9.1875 11.2266 12.0000 12.0000 12.0000 12.0000 12.0000 12.000 9 5 5.664 8.1120 10.1280 11.4960 12.0000 12.0000 12.0000 12.0000 12.000 9 6 5.229 7.3333 9.1875 10.6667 11.6458 12.0000 12.0000 12.0000 12.000 9 7 4.916 6.7522 8.4315 9.8746 11.0029 11.7376 12.0000 12.0000 12.000 9 8 4.679 6.3047 7.8252 9.1875 10.3389 11.2266 11.7979 12.0000 12.000 9 9 4.494 5.9506 7.3333 8.6049 9.7284 10.6667 11.3827 11.8395 12.000 Nugget = 1 0.0000 2.0000 4.0000 6.0000 8.0000 10.0000 12.0000 1 2 3 4 5 6 7 8 9 Sphere Series1 Series2 Series3 Series4 Series5 Series6 Series7 Series8 Series9 236 Nugget = 2 Nugget = 3 0.0000 2.0000 4.0000 6.0000 8.0000 10.0000 12.0000 1 2 3 4 5 6 7 8 9 Sphere Series1 Series2 Series3 Series4 Series5 Series6 Series7 Series8 Series9 0.0000 2.0000 4.0000 6.0000 8.0000 10.0000 12.0000 14.0000 1 2 3 4 5 6 7 8 9 Sphere Series1 Series2 Series3 Series4 Series5 Series6 Series7 Series8 Series9 237 Appendix D: Supplemental Data from Application Area 2: Sample Size Selection Table 50: Pilot Design Estimated Response Data Predicted Response Candidate FLC Smooth Polynomial Curve Rough Polynomial Curve Noisy Curve D= 1; Tota l FLC = 25; Va ria nc e Re duc tion = 25% 5 0.47150 3.21660 51.74491 23 2.02550 7.38003 52.72585 D= 1; Tota l FLC = 25; Va ria nc e Re duc tion = 33% 4 0.37850 2.83269 51.67268 18 1.63550 6.90621 50.77602 D= 1; Tota l FLC = 25; Va ria nc e Re duc tion = 50% 8 0.76519 4.82349 49.85692 21 1.87841 7.18280 50.63585 D= 1; Tota l FLC = 25; Va ria nc e Re duc tion = 66% 3 0.29500 3.94554 51.22746 24 2.11100 7.45919 52.11113 D= 1; Tota l FLC = 25; Va ria nc e Re duc tion = 75% 6 0.58378 4.37253 49.81804 18 1.64113 6.91673 51.04040 238 D= 1; Tota l FLC = 25; Va ria nc e Re duc tion = 90% 6 0.58150 4.46724 49.81804 17 1.55500 6.69211 51.58242 D= 2; Tota l FLC = 25; Va ria nc e Re duc tion = 25% (1, 2) 0.22933 2.87158 51.46236 (4, 1) 1.47455 5.99412 50.84575 D= 2; Tota l FLC = 25; Va ria nc e Re duc tion = 33% (2, 2) 0.62389 4.19965 51.65355 (4, 3) 1.66150 6.74387 51.88809 D= 2; Tota l FLC = 25; Va ria nc e Re duc tion = 50% (1, 4) 0.39688 5.99906 50.95169 (5, 2) 1.96280 9.21019 51.06570 D= 2; Tota l FLC = 25; Va ria nc e Re duc tion = 66% (2, 1) 0.55963 3.62093 50.67789 (4, 5) 1.74794 6.76693 52.17126 D= 2; Tota l FLC = 25; Va ria nc e Re duc tion = 75% (2, 3) 0.76288 5.15992 50.64519 (3, 4) 1.30025 6.36026 51.17188 D= 2; Tota l FLC = 25; Va ria nc e Re duc tion = 90% (3, 2) 1.12307 5.89098 50.98575 (3, 4) 1.30473 6.38044 51.02156 239 D= 1; Tota l FLC = 12 5; Va ria nc e Re duc tion = 25% 6 0.51700 2.77739 51.37124 93 4.96050 12.07827 49.81512 D= 1; Tota l FLC = 125; Va ria nc e Re duc tion = 33% 12 1.10642 5.24324 51.99270 120 4.77128 15.84691 47.63722 D= 1; Tota l FLC = 125; Va ria nc e Re duc tion = 50% 12 1.10883 5.24395 51.99270 120 4.79362 14.77820 47.67108 D= 1; Tota l FLC = 125; Va ria nc e Re duc tion = 66% 25 2.18600 7.71890 52.38430 90 4.94400 11.92503 50.75688 D= 1; Tota l FLC = 125; Va ria nc e Re duc tion = 75% 21 1.87900 7.12575 50.46867 80 4.79850 11.21554 51.80544 D= 1; Tota l FLC = 125; Va ria nc e Re duc tion = 90% 23 2.03500 7.38626 51.42548 70 4.54950 10.61942 50.93005 D= 3; Tota l FLC = 125; Va ria nc e Re duc tion = 25% (1, 1, 2) 0.21768 2.76975 52.00449 (4, 4, 1) 4.85602 11.75360 49.76547 240 D= 3; Tota l FLC = 125; Va ria nc e Re duc tion = 33% (2, 3, 2) 2.85392 9.40406 52.67096 (5, 1, 2) 5.01229 13.19267 50.58011 D= 3; Tota l FLC = 125; Va ria nc e Re duc tion = 50% (1, 1, 3) 0.44948 1.63525 50.83818 (4, 5, 5) 4.80432 12.72027 51.94059 D= 3; Tota l FLC = 125; Va ria nc e Re duc tion = 66% (2, 1, 2) 2.24137 7.15881 52.95494 (3, 5, 4) 4.64795 10.65394 54.03372 D= 3; Tota l FLC = 125; Va ria nc e Re duc tion = 75% (1, 2, 4) 0.88769 5.71143 50.69707 (3, 4, 5) 4.53024 10.05423 52.91640 D= 3; Tota l FLC = 125; Va ria nc e Re duc tio n= 90% (2, 2, 3) 2.69963 8.12533 51.58729 (4, 4, 2) 4.93471 12.30718 51.02052 Table 51: Pilot Design Estimated Variance Data Variance Candidate FLC Smooth Polynomial Curve Rough Polynomial Curve 241 D= 1; Tota l FLC = 25; Va ria nc e Re duc tion = 25% 5 0.43620 3.62242 23 0.27263 2.26401 D= 1; Tota l FLC = 25; Va ria nc e Re duc tion = 33% 4 0.34674 3.11598 18 0.12842 1.15407 D= 1; Tota l FLC = 25; Va ria nc e Re duc tion = 50% 8 0.12027 0.18242 21 0.09622 0.15382 D= 1; Tota l FLC = 25; Va ria nc e Re duc tion = 66% 3 0.07175 0.14951 24 0.09335 0.15862 D= 1; Tota l FLC = 25; Va ria nc e Re duc tion = 75% 6 0.07001 0.12058 18 0.00579 0.15862 D= 1; Tota l FLC = 25; Va ria nc e Re duc tion = 90% 6 0.06965 0.08869 17 0.06965 0.08790 D= 2; Tota l FLC = 25; Va ria nc e Re duc tion = 25% (1, 2) 0.25801 0.27119 (4, 1) 0.36180 0.38278 242 D= 2; Tota l FLC = 25; Va ria nc e Re duc tion = 33% (2, 2) 0.30082 0.07008 (4, 3) 0.23605 0.05504 D= 2; Tota l FLC = 25; Va ria nc e Re duc tion = 50% (1, 4) 0.01027 0.00445 (5, 2) 0.00932 0.00144 D= 2; Tota l FLC = 25; Va ria nc e Re duc tion = 66% (2, 1) 0.05350 0.29321 (4, 5) 0.05430 0.30462 D= 2; Tota l FLC = 25; Va ria nc e Re duc tion = 75% (2, 3) 0.00503 0.31039 (3, 4) 0.00503 0.31050 D= 2; Tota l FLC = 25; Va ria nc e Re duc tion = 90% (3, 2) 0.16216 0.25450 (3, 4) 0.16216 0.25450 D= 1; Tota l FLC = 125; Va ria nc e Re duc tion = 25% 6 0.64876 6.08711 93 0.14971 1.40472 D= 1; Tota l FLC = 125; Va ria nc e Re duc tion = 33% 12 0.17835 3.14344 120 0.20262 3.57136 243 D= 1; Tota l FLC = 125; Va ria nc e Re duc tion = 50% 12 0.33445 3.14344 120 0.16317 3.57136 D= 1; Tota l FLC = 125; Va ria nc e Re duc tion = 66% 25 0.03214 0.58621 90 0.07347 1.33942 D= 1; Tota l FLC = 125; Va ria nc e Re duc tion = 75% 21 0.07341 0.40646 80 0.11011 0.60969 D= 1; Tota l FLC = 1 25; Va ria nc e Re duc tion = 90% 23 0.07204 0.38693 70 0.07204 0.38693 D= 3; Tota l FLC = 125; Va ria nc e Re duc tion = 25% (1, 1, 2) 0.00000 0.00000 (4, 4, 1) 0.00000 0.00000 D= 3; Tota l FLC = 125; Va ria nc e Re duc tion = 33% (2, 3, 2) 0.06866 1.05015 (5, 1, 2) 0.05556 0.68909 D= 3; Tota l FLC = 125; Va ria nc e Re duc tion = 50% (1, 1, 3) 0.22836 0.77976 (4, 5, 5) 0.22917 0.76307 244 D= 3; Tota l FLC = 125; Va ria nc e Re duc tion = 66% (2, 1, 2) 0.21954 0.42767 (3, 5, 4) 0.20727 0.42269 D= 3; Tota l FLC = 125; Va ria nc e Re duc tion = 75% (1, 2, 4) 0.23951 0.36160 (3, 4, 5) 0.23935 0.37415 D= 3; Tota l FLC = 125; Va ria nc e Re duc tion = 90% (2, 2, 3) 0.17403 0.38575 (4, 4, 2) 0.16888 0.37328 Table 52: Pilot Design Squared Residual Data Values Candidate FLC Smooth Polynomial Curve Rough Polynomial Curve Noisy Curve D= 1; Tota l FLC = 25; Va ria nc e Re duc tion = 25% 5 0.00026 1.13472 2.16849 23 0.00010 0.02355 1.81440 D= 1; Tota l FLC = 25; Va ria nc e Re duc tion = 33% 4 0.00018 1.58424 0.63287 18 0.00001 0.00485 0.04039 245 D= 1; Tota l FLC = 25; Va ria nc e Re duc tion = 50% 8 0.00001 0.00852 0.28235 21 0.00000 0.00042 0.11028 D= 1; Tota l FLC = 25; Va ria nc e Re duc tion = 66% 3 0.00000 0.00139 0.01736 24 0.00000 0.01940 0.00041 D= 1; Tota l FLC = 25; Va ria nc e Re duc tion = 75% 6 0.00000 0.00081 0.01059 18 0.00001 0.00349 0.00402 D= 1; Tota l FLC = 25; Va ria nc e Re duc tion = 90% 6 0.00000 0.01516 0.01059 17 0.00000 0.04309 0.00063 D= 2; Tota l FLC = 25; Va ria nc e Re duc tion = 25% (1, 2) 0.00098 0.87603 0.01332 (4, 1) 0.00001 0.28553 1.80126 D= 2; Tota l FLC = 25; Va ria nc e Re duc tion = 33% (2, 2) 0.00266 0.06740 5.24315 (4, 3) 0.00055 0.05380 0.83010 D= 2; Tota l FLC = 25; Va ria nc e Re duc tion = 50% (1, 4) 0.00002 3.63934 0.00556 (5, 2) 0.00002 4.14586 0.11921 246 D= 2; Tota l FLC = 25; Va ria nc e Re duc tion = 66% (2, 1) 0.00050 0.52297 0.92691 (4, 5) 0.00271 0.09642 3.81951 D= 2; Tota l FLC = 25; Va ria nc e Re duc tion = 75% (2, 3) 0.00003 0.18380 1.74144 (3, 4) 0.00000 0.00288 2.08227 D= 2; Tota l FL C= 25; Va ria nc e Re duc tion = 90% (3, 2) 0.00002 0.10416 0.56401 (3, 4) 0.00001 0.00546 2.53870 D= 1; Tota l FLC = 125; Va ria nc e Re duc tion = 25% 6 0.00423 2.45459 2.74270 93 0.00022 0.00402 0.37023 D= 1; Tota l FLC = 125; Va ria nc e Re duc tion = 33% 12 0.00047 0.94183 0.06551 120 0.00082 2.85139 1.67559 D= 1; Tota l FLC = 125; Va ria nc e Re duc tion = 50% 12 0.00037 0.94047 0.06551 120 0.00004 0.38427 1.76440 D= 1; Tota l FLC = 125; Va ria nc e Re duc tion = 66% 25 0.00000 0.01450 0.17824 90 0.00004 0.01253 2.30840 247 D= 1; Tota l FLC = 125; Va ria nc e Re duc tion = 75% 21 0.00000 0.00133 0.02719 80 0.00000 0.00101 0.01620 D= 1; Tota l FLC = 125; Va ria nc e Re duc tion = 90% 23 0.00000 0.02550 0.00217 70 0.00000 0.00009 0.02622 D= 3; Tota l FLC = 125; Va ria nc e Re duc tion = 25% (1, 1, 2) 0.00039 1.07702 0.18208 (4, 4, 1) 0.01071 0.08062 4.43182 D= 3; Tota l FLC = 125; Va ria nc e Re duc tion = 33% (2, 3, 2) 0.02611 0.66917 0.06609 (5, 1, 2) 0.00020 0.35083 0.39473 D= 3; Tota l FLC = 125; Va ria nc e Re duc tion = 50% (1, 1, 3) 0.02371 5.16667 0.27149 (4, 5, 5) 0.03829 0.05015 1.53263 D= 3; Tota l FLC = 125; Va ria nc e Re duc tion = 66% (2, 1, 2) 0.00886 0.45425 0.17203 (3, 5, 4) 0.00020 0.06160 8.79880 D= 3; Tota l FLC = 125; Va ria nc e Re duc tion = 75% (1, 2, 4) 0.00079 0.85430 1.13720 (3, 4, 5) 0.00039 0.30880 4.61510 248 D= 3; Tota l FLC = 125; Va ria nc e Re duc tion = 90% (2, 2, 3) 0.00312 0.10331 0.62485 (4, 4, 2) 0.00111 0.05932 0.03426 Appendix E: Supplemental Data from Application Area 2: and FLC Selection Table 53: Application Area 2 Variance Reduction Data Pilot Design FLC Selection Va ria nc e Re duc tion 25% 3 1, 13, 25 33% 4 1, 13, 19, 25 50% 7 1, 4, 7, 13, 19, 22, 25 66% 10 1, 2, 4, 7, 10, 13, 16, 19, 22, 25 75% 14 1, 2, 4, 5, 7, 8, 10, 13, 16, 19, 21, 22, 24, 25 90% 20 1-5, 7, 8, 10, 11, 13, 15, 16, 18-25 Pilot Design FLC Selection Va ria nc e R educ tion 25% 7 (1,1); (1,3); (1, 5); (3; 3); (5; 1); (5; 3); (5, 5) 33% 8 (1,1); (1,3); (1, 5); (3; 3); (3, 5); (5, 1); (5, 3); (5, 5) 50% 12 (1,1); (1, 3); (1, 5); (2, 2); (3, 1); (3, 3); (3, 5); (4, 2); (4, 4); (5, 1); (5, 3); (5, 5) 66% 16 (1,1); (1, 2); (1 , 3); (1, 5); (2, 2); (2, 4); (2, 5); (3,1); (3, 3); (3, 5); (4, 1); (4, 2); (4, 4); (5, 1); (5, 3); (5, 5) 75% 18 (1,1); (1, 2); (1 , 3); (1, 4); (1, 5); (2, 2); (2, 4); (2, 5); (3,1); (3, 3); (3, 5); (4, 1); (4, 2); (4, 4); (5, 1); (5, 3); (5, 4); (5, 5) 90% 23 (1,1); (1, 2); (1 , 3); (1, 4); (1, 5); (2, 1); (2, 2); (2, 3); (2, 4); (2, 5); (3,1); (3, 3); (3, 5); (4, 1); (4, 2); (4,3); (4, 4); (4,5); (5, 1); (5, 2); (5, 3); (5, 4); (5, 5) Pilot Design FLC Selection Va ria nc e Re duc tio n 25% 5 1, 32, 63, 94, 125 33% 7 1, 16, 32, 63, 94, 110, 125 50% 14 1, 16, 24, 32, 47, 55, 63, 71, 79, 94, 102, 110, 118, 125 249 66% 28 1, 8, 12, 16, 20, 24, 28, 32, 39, 43, 47, 51, 55, 63, 67, 71, 75, 79, 83, 87, 94, 98, 102, 106, 110, 114, 118, 125 75% 43 1, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 28, 32, 35, 39, 43, 47, 51, 55, 59, 63, 67, 71, 75, 79, 83, 87, 91, 94, 98, 102, 104, 106, 108, 110, 112, 114, 116, 118, 120, 122, 125 90% 84 1-22, 24, 26-28, 30, 32, 33, 35, 37, 39, 41, 43, 45, 47, 49, 51, 53, 55, 57, 59, 61, 63, 65, 67, 69, 71, 73, 75, 77, 79, 81, 83, 85, 87, 89, 91, 93, 94, 96, 98, 100, 102, 104, 106, 108-125 Pilot Design FLC Selection Va ria nc e R educ tion 25% 27 (1, 1,1); (1,1,5); (1,3,1); (1,3,3); (1,3,5); (1,5,1); (1,5,3); (1,5,5); (3,1,1); (3,1,3);(3,1,5); (3,3,1); (3,3,3); (3,3,5); (3,5,3); (5,1,1); (5,1,5); (5, 2,2); (5,2,4); (5,3,1); (5,3,3); (5,3,5); (5,4,2); (5,4,4); (5,5,1); (5,5,3); (5,5,5) 33% 34 (1,1,1); (1,1,2); (1,1,4); (1,1,5); (1,3,1); (1,3,3); (1,3,5); (1,5,1); (1,5,3); (1,5,5); (2,4,4); (2,5,1); (3,1,1); (3,1,3); (3,1,5); (3,3,1); (3, 3,3); (3,3,5); (3,5,3); (3,5, 5); (4,5,1); (5, 1,1); (5,1,3);(5,1,5); (5,2,2); (5,2,4); (5, 3,1); (5,3,3); (5,3,5); (5,4,2); (5,4,4); (5,5,1); (5,5,3); (5,5,5) 50% 53 (1,1,1); (1,1,2); (1,1,4); (1,1,5); (1,3,1); (1,3,3); (1,3,5); ( 1,4,2); (1,5,1); (1,5, 3); (1,5,5); (2,2,1); (2,2,2); (2,2,4); (2,2,5); (2,4,2); (2,4,4); (2,4,5); (2,5,1); (2,5,4); (3,1,1); (3,1,3); (3,1,5); (3,3,1); (3,3,3); (3,3,5); (3,4,1); (3,5,2); (3,5,3); (3,5,5); (4,1,2); (4,1,4); (4,2,1); (4,2,3); (4,2, 5); (4,3,4); (4,4,3); (4,4,5); (4,5,1); (5,1,1); (5,1,3);(5,1,5); (5,2,2); (5,2,4); (5,3,1); (5,3,3); (5,3,5); (5,4,2); (5,4,4); (5,5,1); (5,5,3); (5,5,5) 66% 75 (1, 1, 1); (1, 1, 2); (1, 1, 4); (1, 1, 5); (1, 2, 1); (1, 2, 3); (1, 2, 5); (1, 3, 1); (1, 3, 3); (1, 3, 5); (1, 4, 1); (1, 4, 2); (1, 4, 4); (1, 5, 1); (1, 5, 2); (1, 5, 3); (1, 5, 4); (1, 5, 5); (2, 1, 1); (2, 1, 3); (2, 1, 5); (2, 2, 1); (2, 2, 2); (2, 2, 4); (2, 2, 5); 250 (2, 3, 3); (2, 4, 2); (2, 4, 4); (2, 4, 5); (2, 5, 1); (2, 5, 4); (2, 5, 5); (3, 1, 1); (3, 1, 3); (3, 1, 5); (3, 2, 2); (3, 2, 4); (3, 3, 1); (3, 3, 3); (3, 3, 5); (3, 4, 1); (3, 4, 4); (3, 5, 2); (3, 5, 3); (3, 5, 5); (4, 1, 1); (4, 1, 2); (4, 1, 4); (4, 2, 1); (4, 2, 3); (4, 2, 5); (4, 3, 2); (4, 3, 4); (4, 4, 1); (4, 4, 3); (4, 4, 5); (4, 5, 1); (4, 5, 2); (4, 5, 4); (4, 5, 5); (5, 1, 1); (5, 1, 3); (5, 1, 4); (5, 1, 5); (5, 2, 1); (5, 2, 2); (5, 2, 4); (5, 3, 1); (5, 3, 3); (5, 3, 5); (5, 4, 2); (5, 4, 4); (5, 5, 1); (5, 5, 3); (5, 5, 5) 75% 87 (1, 1, 1); (1, 1, 2); (1, 1, 3); (1, 1, 4); (1, 1, 5); (1, 2, 1); (1, 2, 3); (1, 2, 5); (1, 3, 1); (1, 3, 2); (1, 3, 3); (1, 3, 4); (1, 3, 5); (1, 4, 1); (1, 4, 2); (1, 4, 4); (1, 4, 5); (1, 5, 1); (1, 5, 2); (1, 5, 3); (1, 5, 4); (1, 5, 5); (2, 1, 1); (2, 1, 2); (2, 1, 3); (2, 1, 4); (2, 1, 5); (2, 2, 1); (2, 2, 2); (2, 2, 4); (2, 2, 5); (2, 3, 1); (2, 3, 3); (2, 4, 2); (2, 4, 4); (2, 4, 5); (2, 5, 1); (2, 5, 3); (2, 5, 4); (2, 5, 5); (3, 1, 1); (3, 1, 3); (3, 1, 5); (3, 2, 2); (3, 2, 4); (3, 3, 1); (3, 3, 3); (3, 3, 5); (3, 4, 1); (3, 4, 2); (3, 4, 4); (3, 5, 2); (3, 5, 3); (3, 5, 5); (4, 1, 1); (4, 1, 2); (4, 1, 4); (4, 1, 5); (4, 2, 1); (4, 2, 3); (4, 2, 5); (4, 3, 2); (4, 3, 4); (4, 4, 1); (4, 4, 3); (4, 4, 5); (4, 5, 1); (4, 5, 2); (4, 5, 4); (4, 5, 5); (5, 1, 1); (5, 1, 3); (5, 1, 4); (5, 1, 5); (5, 2, 1); (5, 2, 2); (5, 2, 4); (5, 3, 1); (5, 3, 3); (5, 3, 5); (5, 4, 1); (5, 4, 2); (5, 4, 4); (5, 4, 5); (5, 5, 1); (5, 5, 3); (5, 5, 5) 90% 110 (1, 1, 1); (1, 1, 2); (1, 1, 3); (1, 1, 4); (1, 1, 5); (1, 2, 1); (1, 2, 2); (1, 2, 3); (1, 2, 4); (1, 2, 5); (1, 3, 1); (1, 3, 2); (1, 3, 3); (1, 3, 4); (1, 3, 5); (1, 4, 1); (1, 4, 2); (1, 4, 3); (1, 4, 4); (1, 4, 5); (1, 5, 1); (1, 5, 2); (1, 5, 3); (1, 5, 4); (1, 5, 5); (2, 1, 1); (2, 1, 2); (2, 1, 3); (2, 1, 4); (2, 1, 5); (2, 2, 1); (2, 2, 2); (2, 2, 4); (2, 2, 5); (2, 3, 1); (2, 3, 3); (2, 3, 5); (2, 4, 1); (2, 4, 2); (2, 4, 4); (2, 4, 5); (2, 5, 1); (2, 5, 2); (2, 5, 3); (2, 5, 4); (2, 5, 5); (3, 1, 1); (3, 1, 2); (3, 1, 3); (3, 1, 5); (3, 2, 2); (3, 2, 4); (3, 2, 5); (3, 3, 1); (3, 3, 3); (3, 3, 5); (3, 4, 1); (3, 4, 2); (3, 4, 4); (3, 5, 1); (3, 5, 2); (3, 5, 3); (3, 5, 4); (3, 5, 5); (4, 1, 1); 251 (4, 1, 2); (4, 1, 3); (4, 1, 4); (4, 1, 5); (4, 2, 1); (4, 2, 3); (4, 2, 4); (4, 2, 5); (4, 3, 1); (4, 3, 2); (4, 3, 4); (4, 3, 5); (4, 4, 1); (4, 4, 3); (4, 4, 5); (4, 5, 1); (4, 5, 2); (4, 5, 3); (4, 5, 4); (4, 5, 5); (5, 1, 1); (5, 1, 2); (5, 1, 3); (5, 1, 4); (5, 1, 5); (5, 2, 1); (5, 2, 2); (5, 2, 3); (5, 2, 4); (5, 2, 5); (5, 3, 1); (5, 3, 2); (5, 3, 3); (5, 3, 4); (5, 3, 5); (5, 4, 1); (5, 4, 2); (5, 4, 3); (5, 4, 4); (5, 4, 5); (5, 5, 1); (5, 5, 2); (5, 5, 3); (5, 5, 4); (5, 5, 5) Appendix F: MC Response Data Set and Augmented data set Table 54: Monte Carlo Data Sets - System 1 - System 2 1.00 6.798998681 25.96423782 2.00 39.20088643 26.83080437 3.00 120.7742288 27.38741513 4.00 277.3377493 28.43806341 5.00 525.5891761 28.80622686 6.00 900.1173153 28.92755346 7.00 1416.673854 29.28707671 8.00 2118.296116 30.6192136 9.00 2981.370319 31.0832378 10.00 4066.837971 31.55190042 11.00 5467.005852 31.73343556 12.00 7123.517974 32.68922354 13.00 8989.831558 33.3273705 14.00 11223.89717 33.55929165 15.00 13641.91875 33.99557626 16.00 16507.60378 34.5891125 17.00 20076.91167 35.11661943 18.00 23844.52609 35.19720066 19.00 27508.46435 35.29033292 20.00 32193.68279 35.48173608 252 21.00 37574.64921 35.71051947 22.00 43056.76744 35.83052729 23.00 49309.72943 36.22838575 24.00 56233.81793 36.23234205 25.00 62767.05007 36.3186531 26.00 71004.70142 36.40782906 27.00 79277.58911 36.89625045 28.00 88248.21932 36.90168468 29.00 97605.90169 36.93945145 30.00 108343.5368 37.00202468 31.00 120606.1794 37.36379894 32.00 130622.8326 37.40786396 33.00 146249.6938 38.33809514 34.00 157646.417 38.773925 35.00 172027.4515 38.84404704 36.00 187155.3353 38.98124431 37.00 203848.8216 39.11639523 38.00 220638.7253 39.2291271 39.00 240023.5079 39.3741915 40.00 256562.1858 39.62250629 41.00 279219.6379 39.88628249 42.00 295371.1084 40.110814 43.00 322052.3614 40.40346665 44.00 342493.596 40.46390258 45.00 365872.0923 40.63606992 46.00 390821.0363 40.94009127 47.00 421033.9519 40.96887677 48.00 446147.9875 41.14913862 49.00 472860.8039 41.1717623 50.00 503594.6952 41.5549779 51.00 536075.2562 41.58335413 52.00 568951.7473 41.5888111 53.00 593618.1417 41.86775312 54.00 634050.9005 41.87521098 55.00 666695.4716 41.9251195 253 56.00 711200.5516 42.04746928 57.00 739593.3392 42.2525376 58.00 778167.6302 42.30922185 59.00 824730.3776 42.3143605 60.00 869762.6102 42.4238864 61.00 907166.1314 42.43304956 62.00 962608.6404 42.4757617 63.00 1005810.023 42.61212906 64.00 1055453.011 42.65427277 65.00 1097689.124 42.71026354 66.00 1153461.244 42.7491217 67.00 1203959.502 42.7521003 68.00 1250402.16 42.81264991 69.00 1320917.656 43.27262685 70.00 1376327.818 43.38824637 71.00 1444749.612 43.43385753 72.00 1504225.325 43.4585162 73.00 1569478.194 43.5670985 74.00 1640224.533 43.65902568 75.00 1673881.4 44.15820184 76.00 1759440.408 44.24688895 77.00 1824768.804 44.30995103 78.00 1914540.966 44.49249799 79.00 1976080.431 44.51563326 80.00 2056605.75 44.76381163 81.00 2130335.715 44.76906396 82.00 2224269.352 45.09871031 83.00 2289766.683 45.27221917 84.00 2378400.539 45.27221917 85.00 2462327.28 45.30898549 86.00 2541476.247 45.32262791 87.00 2605987.964 45.37378699 88.00 2765311.609 45.41460056 89.00 2825989.644 45.42139903 90.00 2922527.544 45.50618668 254 91.00 3028085.649 45.59993284 92.00 3109106.208 45.63530082 93.00 3212835.277 45.6857323 94.00 3346107.221 45.80464873 95.00 3466549.855 45.87721504 96.00 3522833.372 45.90136213 97.00 3647467.093 45.99519924 98.00 3742708.899 46.12251713 99.00 3880245.295 46.43732735 100.00 4003763.519 46.48456424 101.00 4119662.276 46.54065732 102.00 4242915.493 46.58609795 103.00 4407326.277 46.68001465 104.00 4520080.139 46.74057563 105.00 4651376.968 46.89428023 106.00 4776308.696 46.96404984 107.00 4912306.465 46.98086413 108.00 5060561.904 47.22738266 109.00 5190668.86 47.47142283 110.00 5300763.436 48.01841568 111.00 5482268.832 48.1423116 112.00 5665537.935 48.15087222 113.00 5805244.484 48.35825975 114.00 5954992.728 48.53866939 115.00 6148459.14 48.58040383 116.00 6219370.684 48.6167154 117.00 6397475.503 48.76176844 118.00 6567145.906 48.80493533 119.00 6679552.676 48.96042482 120.00 6969852.076 49.10722863 121.00 7059941.632 49.4799623 122.00 7271409.705 49.5511871 123.00 7458622.957 49.57875616 124.00 7613585.012 49.592535 125.00 7745872.508 49.82671852 255 126.00 8042189.485 50.37837253 127.00 8198976.434 50.41435442 128.00 8393665.715 50.4886374 129.00 8657512.299 50.70244823 130.00 8800848.915 50.70474471 131.00 8988174.761 51.23513928 132.00 9217052.355 51.23976633 133.00 9339650.97 51.25903625 134.00 9684586.608 51.291437 135.00 9842126.553 51.58980811 136.00 10049038.1 51.72476575 137.00 10310965.89 51.82501481 138.00 10467165.03 52.02606998 139.00 10765988.16 52.03075388 140.00 10994127.65 52.0393486 141.00 11246532.15 52.05028527 142.00 11500265.03 52.07998028 143.00 11617519.92 52.21541541 144.00 12089006.5 52.35990001 145.00 12249891.22 52.44415332 146.00 12420107.42 52.62109552 147.00 12787795.15 52.70987357 148.00 13017123.82 52.73289515 149.00 13279966.88 52.87053581 150.00 13519452.39 52.92558298 151.00 13871955.95 52.98232408 152.00 14099208.19 53.00151441 153.00 14427453.39 53.04717105 154.00 14485317.04 53.1434638 155.00 15073203.74 53.19012088 156.00 15281094.27 53.53415999 157.00 15537569.04 53.762284 158.00 15798595.82 53.85934982 159.00 16011202.29 54.29245119 160.00 16390851.71 54.35123866 256 161.00 16715548.23 54.37564722 162.00 17023585.36 54.4608214 163.00 17285074.71 54.57860097 164.00 17764456.9 54.8280981 165.00 18074003.11 54.90647381 166.00 18180119.36 54.94014785 167.00 18613399.29 55.09282927 168.00 19098113.69 55.11026883 169.00 19368427.41 55.19675041 170.00 19712850.13 55.24935331 171.00 20110615.24 55.34089395 172.00 20319137.24 55.36472271 173.00 20685912.37 55.49415518 174.00 21102992.06 55.55207862 175.00 21522826.85 55.92365268 176.00 21606361.31 56.0379989 177.00 22138684.19 56.17442311 178.00 22710102.59 56.28774615 179.00 22946791.5 56.56328893 180.00 23066222.93 56.69383553 181.00 23757838.06 56.76386662 182.00 24136169.5 56.76964191 183.00 24604449.8 56.83327244 184.00 24992233.25 56.89135504 185.00 25283281.09 56.91563855 186.00 25860058.3 56.96038569 187.00 26122857.56 56.99357088 188.00 26724560.61 57.45335456 189.00 27145031.53 57.50089839 190.00 27551035.7 57.55474048 191.00 28099493 57.67022357 192.00 28461413.43 57.69077815 193.00 28855907.03 57.95359938 194.00 29479647.59 58.01671831 195.00 29646733 58.02516524 257 196.00 30173953.46 58.08336154 197.00 30710199.32 58.4450221 198.00 31220954.29 58.62894467 199.00 31523039.93 58.9909463 200.00 32110147.85 59.06106834 201.00 32614588.17 59.09572009 202.00 32787102.07 59.17002581 203.00 33483035.05 59.32841431 204.00 33780394.21 59.52645678 205.00 34613439.46 59.54332791 206.00 35154488.91 59.6062422 207.00 35237131.41 59.61838396 208.00 36109420.77 59.74796421 209.00 36450987.26 59.89916771 210.00 36972996.84 59.91167326 211.00 37603789.94 60.01976671 212.00 38181870.4 60.18156581 213.00 38473145 60.30177827 214.00 39479950.94 60.30566636 215.00 39886857.19 60.42612894 216.00 40123172.36 60.56412202 217.00 40927040.07 60.67435278 218.00 41485954.18 61.16022759 219.00 41856353.68 61.28514668 220.00 42989979.06 61.31413683 221.00 43182665.8 61.36797891 222.00 44243616.84 61.4045406 223.00 44586029.87 62.08609319 224.00 44653404.57 62.14343683 225.00 45930403.97 62.53320079 226.00 46422631.58 62.87785381 227.00 46909818.08 63.25506673 228.00 47571552.49 63.52696017 229.00 48024505.81 63.54223969 230.00 48781255.67 64.70289135 258 231.00 48762747.78 65.15031727 232.00 49808021.92 65.58373697 233.00 50710921.49 66.58727342 234.00 51473388.14 66.58727342 235.00 52420685.72 67.1321517 236.00 52769229.83 67.62377906 237.00 53204848.49 67.86756911 238.00 54203237.71 68.21408659 239.00 54364575.81 68.28693712 240.00 54878193.13 68.31140253 241.00 55742779.47 68.62717909 242.00 56787118.21 68.74113877 243.00 57590041.03 69.28656275 Table 55: Augmented Data Sets to Include Interpolated Data - System 1 - System 2 1.00 6.8 32.90095391 2.00 3755.6 33.04298327 3.00 7504.4 33.18501263 4.00 11253.2 33.32704199 5.00 15002 33.46907134 6.00 18750.8 33.6111007 7.00 22499.6 33.75313006 8.00 26248.4 33.89515942 9.00 29997.2 34.03718878 10.00 33746 34.17921814 11.00 37494.8 34.32124749 12.00 41243.6 34.46327685 13.00 44992.4 34.60530621 14.00 48741.2 34.74733557 15.00 52490 34.88936493 16.00 56238.8 35.03139429 17.00 59987.6 35.17342364 259 18.00 63736.4 35.315453 19.00 67485.2 35.45748236 20.00 71234 35.59951172 21.00 74982.8 35.74154108 22.00 78731.6 35.88357044 23.00 82480.4 36.02559979 24.00 86229.2 36.16762915 25.00 89978 36.30965851 26.00 93726.8 36.45168787 27.00 97475.6 36.59371723 28.00 101224.4 36.73574659 29.00 104973.2 36.87777594 30.00 108722 37.0198053 31.00 200859.0114 37.05959115 32.00 292996.0229 37.09937699 33.00 385133.0343 37.13916284 34.00 477270.0457 37.17894869 35.00 569407.0572 37.21873453 36.00 661544.0686 37.25852038 37.00 753681.08 37.29830622 38.00 774624.7345 37.33809207 39.00 795568.389 37.37787791 40.00 816512.0435 37.41766376 41.00 837455.698 37.45744961 42.00 858399.3525 37.49723545 43.00 879343.007 37.5370213 44.00 900286.6615 37.57680714 45.00 921230.316 37.61659299 46.00 942173.9705 37.65637883 47.00 936800.1154 37.69616468 48.00 931426.2604 37.73595053 49.00 926052.4054 37.77573637 50.00 920678.5503 37.81552222 51.00 915304.6953 37.85530806 52.00 909930.8403 37.89509391 260 53.00 904556.9852 37.93487975 54.00 899183.1302 37.9746656 55.00 893809.2752 38.01445145 56.00 888435.4201 38.05423729 57.00 883061.5651 38.09402314 58.00 877687.7101 38.13380898 59.00 872313.855 38.17359483 60.00 866940 38.21338068 61.00 935462 38.3199295 62.00 1003984 38.42647832 63.00 1072506 38.53302714 64.00 1141028 38.63957596 65.00 1209550 38.74612478 66.00 1278072 38.8526736 67.00 1346594 38.95922242 68.00 1415116 39.06577124 69.00 1483638 39.17232006 70.00 1552160 39.27886888 71.00 1620682 39.3854177 72.00 1689204 39.49196652 73.00 1757726 39.59851534 74.00 1826248 39.70506416 75.00 1894770 39.81161298 76.00 1963292 39.9181618 77.00 2031814 40.02471062 78.00 2100336 40.13125944 79.00 2168858 40.23780826 80.00 2237380 40.34435708 81.00 2305902 40.4509059 82.00 2374424 40.55745472 83.00 2442946 40.66400354 84.00 2511468 40.77055236 85.00 2579990 40.87710118 86.00 2648512 40.98365 87.00 2717034 41.09019882 261 88.00 2785556 41.19674764 89.00 2854078 41.30329646 90.00 2922600 41.40984528 91.00 3023242.4 42.92003849 92.00 3123884.8 44.43023171 93.00 3224527.2 45.94042492 94.00 3335309.8 46.15677098 95.00 3446092.4 46.37311703 96.00 3556875 46.58946308 97.00 3667657.6 46.80580913 98.00 3778440.2 47.02215518 99.00 3889222.8 47.23850124 100.00 3728610.099 48.08721517 101.00 3990981.65 48.93592909 102.00 4253353.2 49.78464302 103.00 4383329.1 50.36512517 104.00 4513305 50.94560733 105.00 4643280.9 51.52608948 106.00 4773256.8 52.10657163 107.00 4912115.467 52.85703739 108.00 5050974.133 53.60750316 109.00 5189832.8 54.35796892 110.00 5336543.6 54.36021234 111.00 5483254.4 54.36245576 112.00 5629965.2 54.36469918 113.00 5784744.8 54.84377172 114.00 5939524.4 55.32284427 115.00 6094304 55.80191681 116.00 6257368.4 56.13476535 117.00 6420432.8 56.46761388 118.00 6583497.2 56.80046242 119.00 6755062.4 57.97103894 120.00 6926627.6 59.14161546 121.00 7098192.8 60.31219199 122.00 7275541.2 63.14901965 262 123.00 7469392.6 63.17506758 124.00 7663244 63.20111551 125.00 7857095.4 63.22716344 126.00 8050946.8 63.25321136 127.00 8244798.2 63.27925929 128.00 8438649.6 63.30530722 129.00 8632501 63.33135515 130.00 8826352.4 63.35740308 131.00 9020203.8 63.38345101 132.00 9214055.2 63.40949893 133.00 9437943.822 63.41578961 134.00 9661832.444 63.42208028 135.00 9885721.067 63.42837095 136.00 10109609.69 63.43466162 137.00 10333498.31 63.44095229 138.00 10557386.93 63.44724296 139.00 10781275.56 63.45353364 140.00 11005164.18 63.45982431 141.00 11229052.8 63.46611498 142.00 11645785.68 63.57124674 143.00 12062518.56 63.67637849 144.00 12479251.43 63.78151025 145.00 12789926.69 63.88664201 146.00 13100601.95 63.99177377 147.00 13411277.21 64.09690552 148.00 13721952.47 64.20203728 149.00 14032627.73 64.30716904 150.00 14343302.99 64.4123008 151.00 14653978.24 64.51743255 152.00 14964653.5 64.62256431 153.00 15275328.76 64.72769607 154.00 15586004.02 64.83282783 155.00 15896679.28 64.93795958 156.00 16207354.54 65.04309134 157.00 16518029.79 65.1482231 263 158.00 16828705.05 65.25335485 159.00 17139380.31 65.35848661 160.00 17450055.57 65.46361837 161.00 17760730.83 65.56875013 162.00 18071406.09 65.67388188 163.00 18382081.35 65.77901364 164.00 18692756.6 65.8841454 165.00 19003431.86 65.98927716 166.00 19314107.12 66.09440891 167.00 19624782.38 66.19954067 168.00 19935457.64 66.30467243 169.00 20246132.9 66.40980419 170.00 20556808.16 66.51493594 171.00 20867483.41 66.6200677 172.00 21178158.67 66.72519946 173.00 21488833.93 66.83033122 174.00 21755315.08 66.93546297 175.00 22021796.24 67.04059473 176.00 22288277.39 67.14572649 177.00 22554758.54 67.25085825 178.00 22821239.69 67.35599 179.00 23087720.85 67.46112176 180.00 23354202 67.56625352 181.00 23763069.87 67.58604008 182.00 24171937.73 67.60582664 183.00 24580805.6 67.62561321 184.00 24989673.47 67.64539977 185.00 25398541.33 67.66518633 186.00 25807409.2 67.68497289 187.00 26216277.07 67.70475946 188.00 26625144.93 67.72454602 189.00 27034012.8 67.74433258 190.00 27495515.69 67.95451031 191.00 27957018.57 68.16468804 192.00 28418521.46 68.37486577 264 193.00 28880024.34 68.5850435 194.00 29341527.23 68.79522123 195.00 29803030.11 69.00539896 196.00 30264533 69.21557669 197.00 30726035.89 69.42575441 198.00 31187538.77 69.63593214 199.00 31649041.66 69.84610987 200.00 32110544.54 70.0562876 201.00 32572047.43 70.26646533 202.00 33033550.31 70.47664306 203.00 33495053.2 70.68682079 204.00 34095268.58 70.82481856 205.00 34695483.95 70.96281633 206.00 35295699.33 71.1008141 207.00 35895914.7 71.23881187 208.00 36496130.08 71.37680964 209.00 37096345.46 71.51480741 210.00 37696560.83 71.65280518 211.00 38296776.21 71.79080295 212.00 38896991.59 71.92880072 213.00 39497206.96 72.06679849 214.00 40097422.34 72.20479626 215.00 40697637.71 72.23727991 216.00 41297853.09 72.26976356 217.00 41898068.47 72.3022472 218.00 42498283.84 72.33473085 219.00 43098499.22 72.3672145 220.00 43698714.6 72.39969815 221.00 44298929.97 72.4321818 222.00 44899145.35 72.46466545 223.00 45499360.72 72.49714909 224.00 46099576.1 72.52963274 225.00 46699791.48 72.56211639 226.00 47300006.85 72.59460004 227.00 47900222.23 72.62708369 265 228.00 48500437.61 72.65956733 229.00 49100652.98 72.69205098 230.00 49700868.36 72.72453463 231.00 50301083.73 72.75701828 232.00 50901299.11 72.78950193 233.00 51501514.49 72.82198558 234.00 52085669.66 72.85446922 235.00 52669824.84 72.88695287 236.00 53253980.02 72.91943652 237.00 53833065.09 72.95192017 238.00 54412150.17 72.98440382 239.00 54991235.24 73.01688747 240.00 55570320.32 73.04937111 241.00 56149405.4 73.08185476 242.00 56728490.47 73.11433841 243.00 57443273.2 73.14682206 Appendix G: Additional Initial Sample Size and FLC Selection Data Table 56: Initial Sample Size and FLC Selection Data Compilation Pilot Design FLC Selection Smooth Expected Response Rough Expected Response 10 4 6 (1); (5); (8); (10) (1); (3); (5); (7); (8); (10) 25 7 10 (1); (4); (7); (13); (19); (22); (25) (1); (2); (4); (7); (10); (13); (16); (19); (22); (25) 50 9 15 (1); (7); (13); (19); (25); (32); (38); (44); (50) (1); (4); (7); (10); (13); (16); (19); (25); (29); (32); (38); (41); (44); (47); (50) 75 11 21 (1); (10); (19); (28); (33); (38); (43); (48); (57); (66); (75) (1); (5); (7); (10); (14); (16); (19); (23); (28); (33); (38); (43); (48); (53); (57); (60); (62); (66); (69); (71); (75) 266 100 13 25 (1); (7); (13); (25); (37); (43); (50); (57); (63); (75); (82); (88); (100) (1); (4); (7); (10); (13); (16); (19); (25); (31); (37); (43); (46); (50); (54); (57); (63); (69); (75); (79); (82); (88); (91); (94); (97); (100) 125 14 28 (1); (16); (24); (32); (47); (55); (63); (71); (79); (94); (102); (110); (118); (125) (1); (8); (12); (16); (20); (24); (28); (32); (39); (43); (47); (51); (55); (63); (67); (71); (75); (79); (83); (87); (94); (98); (102); (106); (110); (114); (118); (125) Pilot Design FLC Selection Smooth Expected Response Rough Expected Response 9 6 7 (1, 1); (1, 2); (1, 3); (2, 2); (3, 1); (3, 3) (1, 1); (1, 2); (1, 3); (2, 2); (3, 1); (3, 2); (3, 3) 25 12 16 (1, 1); (1, 3); (1, 5); (2, 2); (3, 1); (3, 3); (3, 5); (4, 2); (4, 4); (5, 1); (5, 3); (5, 5) (1, 1); (1, 2); (1, 3); (1, 5); (2, 2); (2, 4); (2, 5); (3, 1); (3, 3); (3, 5); (4, 1); (4, 2); (4, 4); (5, 1); (5, 3); (5, 5) 49 20 28 (1, 1); (1, 4); (1, 7); (2, 2); (2, 6); (3, 3); (3, 5); (3, 7); (4, 1); (4, 4); (4, 7); (5, 1); (5, 3); (5, 5); (6, 2); (6, 6); (7, 1); (7, 4); (7, 5); (7, 7) (1, 1); (1, 3); (1, 4); (1, 5); (1, 7); (2, 2); (2, 4); (2, 6); (3, 1); (3, 3); (3, 5); (3, 7); (4, 1); (4, 4); (4, 6); (4, 7); (5, 1); (5, 3); (5, 5); (5, 7); (6, 2); (6, 4); (6, 6); (7, 1); (7, 3); (7, 4); (7, 5); (7, 7) 81 28 43 (1, 1); (1, 3); (1, 5); (1, 7); (1, 9); (2, 2); (3, 1); (3, 3); (3, 5); (3, 7); (3, 9); (5, 1); (5, 3); (5, 5); (5, 7); (5, 9); (7, 1); (7, 3); (7, 5); (7, 7); (7, 9); (8, 2); (8, 8); (9, 1); (9, 3); (9, 5); (9, 7); (9, 9) (1, 1); (1, 2); (1, 3); (1, 5); (1, 7); (1, 9); (2, 2); (2, 4); (2, 6); (2, 8); (3, 1); (3, 3); (3, 5); (3, 7); (3, 9); (4, 2); (4, 4); (4, 6); (4, 8); (5, 1); (5, 3); (5, 5); (5, 7); (5, 9); (6, 2); (6, 4); (6, 6); (6, 8); (7, 1); (7, 3); (7, 5); (7, 7); (7, 9); (8, 1); (8, 2); (8, 4); (8, 6); (8, 8); (9, 1); (9, 3); (9, 5); (9, 7); (9, 9) 267 100 34 53 (1, 1); (1, 3); (1, 6); (1, 8); (1, 10); (2, 1); (2, 5); (2, 7); (3, 3); (3, 8); (3, 10); (4, 1); (4, 5); (4, 7); (4, 9); (5, 2); (5, 4); (5, 10); (6, 3); (6, 7); (6, 9); (7, 1); (7, 5); (7, 8); (8, 2); (8, 7); (8, 10); (9, 4); (9, 9); (10, 1); (10, 3); (10, 6); (10, 8); (10, 10) (1, 1); (1, 3); (1, 6); (1, 8); (1, 10); (2, 1); (2, 5); (2, 7); (3, 3); (3, 8); (3, 10); (4, 1); (4, 4); (4, 5); (4, 7); (4, 9); (5, 2); (5, 4); (5, 6); (5, 8); (5, 10); (6, 1); (6, 2); (6, 3); (6, 5); (6, 7); (6, 9); (7, 1); (7, 3); (7, 5); (7, 6); (7, 8); (7, 10); (8, 2); (8, 3); (8, 4); (8, 5); (8, 7); (8, 8); (8, 10); (9, 1); (9, 4); (9, 6); (9, 9); (9, 10); (10, 1); (10, 2); (10, 3); (10, 5); (10, 6); (10, 7); (10, 8); (10, 10) 121 41 62 (1, 1); (1, 4); (1, 6); (1, 9); (1, 11); (3, 3); (3, 9); (3, 11); (4, 1); (4, 4); (4, 6); (4, 11); (5, 3); (5, 5); (5, 8); (5, 10); (6, 1); (6, 4); (6, 6); (6, 11); (7, 3); (7, 5); (7, 7); (7, 9); (8, 1); (8, 4); (8, 8); (8, 11); (9, 3); (9, 6); (9, 10); (10, 2); (10, 5); (10, 8); (10, 11); (11, 1); (11, 4); (11, 6); (11, 7); (11, 9); (11, 11) (1, 1); (1, 3); (1, 4); (1, 6); (1, 8); (1, 9); (1, 11); (2, 2); (2, 4); (2, 7); (2, 8); (2, 10); (3, 1); (3, 3); (3, 5); (3, 6); (3, 8); (3, 9); (3, 11); (4, 1); (4, 2); (4, 4); (4, 6); (4, 9); (4, 11); (5, 3); (5, 5); (5, 7); (5, 8); (5, 10); (6, 1); (6, 4); (6, 6); (6, 9); (6, 11); (7, 2); (7, 3); (7, 5); (7, 7); (7, 9); (7, 10); (8, 1); (8, 4); (8, 6); (8, 8); (8, 11); (9, 1); (9, 3); (9, 6); (9, 9); (9, 10); (10, 2); (10, 5); (10, 8); (10, 11); (11, 1); (11, 3); (11, 4); (11, 6); (11, 7); (11, 9); (11, 11) Pilot Design FLC Selection Smooth Expected Response Rough Expected Response 27 17 20 (1, 1, 1); (1, 1, 2); (1, 1, 3); (1, 2, 2); (1, 3, 1); (1, 3, 3); (2, 1, 2); (2, 2, 1); (2, 2, 2); (2, 2, 3); (2, 3, 2); (3, 1, 1); (3, 1, 3); (3, 2, 2); (3, 2, 3); (3, 3, 1); (3, 3, 3) (1, 1, 1); (1, 1, 2); (1, 1, 3); (1, 2, 2); (1, 3, 1); (1, 3, 3); (2, 1, 2); (2, 2, 1); (2, 2, 2); (2, 2, 3); (2, 3, 1); (2, 3, 2); (2, 3, 3); (3, 1, 1); (3, 1, 3); (3, 2, 1); (3, 2, 2); (3, 2, 3); (3, 3, 1); (3, 3, 3) 268 64 32 41 (1, 1, 1); (1, 1, 3); (1, 1, 4); (1, 2, 2); (1, 3, 1); (1, 3, 3); (1, 4, 1); (1, 4, 2); (1, 4, 4); (2, 1, 1); (2, 1, 3); (2, 2, 1); (2, 2, 4); (2, 3, 2); (2, 4, 1); (2, 4, 4); (3, 1, 2); (3, 1, 4); (3, 2, 3); (3, 3, 1); (3, 3, 4); (3, 4, 2); (4, 1, 1); (4, 1, 3); (4, 1, 4); (4, 2, 1); (4, 2, 2); (4, 2, 4); (4, 3, 3); (4, 4, 1); (4, 4, 3); (4, 4, 4) (1, 1, 1); (1, 1, 2); (1, 1, 3); (1, 1, 4); (1, 2, 2); (1, 2, 4); (1, 3, 1); (1, 3, 3); (1, 3, 4); (1, 4, 1); (1, 4, 2); (1, 4, 4); (2, 1, 1); (2, 1, 3); (2, 1, 4); (2, 2, 1); (2, 2, 4); (2, 3, 2); (2, 4, 1); (2, 4, 3); (2, 4, 4); (3, 1, 1); (3, 1, 2); (3, 1, 4); (3, 2, 3); (3, 3, 1); (3, 3, 4); (3, 4, 1); (3, 4, 2); (3, 4, 4); (4, 1, 1); (4, 1, 3); (4, 1, 4); (4, 2, 1); (4, 2, 2); (4, 2, 4); (4, 3, 2); (4, 3, 3); (4, 4, 1); (4, 4, 3); (4, 4, 4) 125 53 75 (1, 1, 1); (1, 1, 2); (1, 1, 4); (1, 1, 5); (1, 2, 3); (1, 3, 1); (1, 3, 3); (1, 3, 5); (1, 4, 2); (1, 5, 1); (1, 5, 3); (1, 5, 5); (2, 2, 1); (2, 2, 2); (2, 2, 4); (2, 2, 5); (2, 4, 2); (2, 4, 4); (2, 4, 5); (2, 5, 1); (2, 5, 4); (3, 1, 1); (3, 1, 3); (3, 1, 5); (3, 3, 1); (3, 3, 3); (3, 3, 5); (3, 4, 1); (3, 5, 2); (3, 5, 3); (3, 5, 5); (4, 1, 2); (4, 1, 4); (4, 2, 1); (4, 2, 3); (4, 2, 5); (4, 3, 4); (4, 4, 3); (4, 4, 5); (4, 5, 1); (5, 1, 1); (5, 1, 3); (5, 1, 5); (5, 2, 2); (5, 2, 4); (5, 3, 1); (5, 3, 3); (5, 3, 5); (5, 4, 2); (5, 4, 4); (5, 5, 1); (5, 5, 3); (5, 5, 5) (1, 1, 1); (1, 1, 2); (1, 1, 4); (1, 1, 5); (1, 2, 1); (1, 2, 3); (1, 2, 5); (1, 3, 1); (1, 3, 3); (1, 3, 5); (1, 4, 1); (1, 4, 2); (1, 4, 4); (1, 5, 1); (1, 5, 2); (1, 5, 3); (1, 5, 4); (1, 5, 5); (2, 1, 1); (2, 1, 3); (2, 1, 5); (2, 2, 1); (2, 2, 2); (2, 2, 4); (2, 2, 5); (2, 3, 3); (2, 4, 2); (2, 4, 4); (2, 4, 5); (2, 5, 1); (2, 5, 4); (2, 5, 5); (3, 1, 1); (3, 1, 3); (3, 1, 5); (3, 2, 2); (3, 2, 4); (3, 3, 1); (3, 3, 3); (3, 3, 5); (3, 4, 1); (3, 4, 4); (3, 5, 2); (3, 5, 3); (3, 5, 5); (4, 1, 1); (4, 1, 2); (4, 1, 4); (4, 2, 1); (4, 2, 3); (4, 2, 5); (4, 3, 2); (4, 3, 4); (4, 4, 1); (4, 4, 3); (4, 4, 5); (4, 5, 1); (4, 5, 2); (4, 5, 4); (4, 5, 5); (5, 1, 1); (5, 1, 3); (5, 1, 4); (5, 1, 5); (5, 2, 1); (5, 2, 2); (5, 2, 4); (5, 3, 1); (5, 3, 3); (5, 3, 5); (5, 4, 2); (5, 4, 4); (5, 5, 1); (5, 5, 3); (5, 5, 5) 269 216 91 197 (1, 1, 1); (1, 1, 3); (1, 1, 5); (1, 1, 6); (1, 2, 3); (1, 2, 5); (1, 3, 1); (1, 3, 2); (1, 3, 4); (1, 3, 6); (1, 4, 1); (1, 4, 3); (1, 4, 5); (1, 4, 6); (1, 5, 2); (1, 5, 4); (1, 6, 1); (1, 6, 3); (1, 6, 5); (1, 6, 6); (2, 1, 2); (2, 1, 4); (2, 1, 5); (2, 2, 1); (2, 2, 2); (2, 2, 6); (2, 3, 1); (2, 3, 3); (2, 3, 5); (2, 4, 2); (2, 4, 5); (2, 5, 1); (2, 5, 4); (2, 5, 6); (2, 6, 2); (2, 6, 5); (3, 1, 1); (3, 1, 3); (3, 1, 4); (3, 1, 6); (3, 2, 3); (3, 2, 5); (3, 3, 2); (3, 3, 4); (3, 3, 6); (3, 4, 1); (3, 4, 3); (3, 4, 6); (3, 5, 2); (3, 5, 5); (3, 6, 1); (3, 6, 3); (3, 6, 4); (3, 6, 6); (4, 1, 2); (4, 1, 4); (4, 1, 6); (4, 2, 1); (4, 2, 3); (4, 2, 5); (4, 3, 1); (4, 3, 4); (4, 3, 6); (4, 4, 2); (4, 4, 4); (4, 4, 6); (4, 5, 1); (4, 5, 3); (4, 5, 5); (4, 5, 6); (4, 6, 1); (4, 6, 2); (4, 6, 4); (4, 6, 6); (5, 1, 1); (5, 1, 2); (5, 1, 3); (5, 1, 4); (5, 1, 5); (5, 1, 6); (5, 2, 1); (5, 2, 2); (5, 2, 4); (5, 2, 5); (6, 1, 1); (6, 1, 6); (6, 3, 3); (6, 4, 1); (6, 6, 1); (6, 6, 4); (6, 6, 6) 270 (1, 1, 1); (1, 1, 2); (1, 1, 3); (1, 1, 4); (1, 1, 5); (1, 1, 6); (1, 2, 1); (1, 2, 2); (1, 2, 3); (1, 2, 4); (1, 2, 5); (1, 2, 6); (1, 3, 1); (1, 3, 2); (1, 3, 3); (1, 3, 4); (1, 3, 5); (1, 3, 6); (1, 4, 1); (1, 4, 2); (1, 4, 3); (1, 4, 4); (1, 4, 5); (1, 4, 6); (1, 5, 1); (1, 5, 2); (1, 5, 3); (1, 5, 4); (1, 5, 5); (1, 5, 6); (1, 6, 1); (1, 6, 2); (1, 6, 3); (1, 6, 4); (1, 6, 5); (1, 6, 6); (2, 1, 1); (2, 1, 2); (2, 1, 3); (2, 1, 4); (2, 1, 5); (2, 1, 6); (2, 2, 1); (2, 2, 2); (2, 2, 4); (2, 2, 6); (2, 3, 1); (2, 3, 2); (2, 3, 3); (2, 3, 5); (2, 3, 6); (2, 4, 1); (2, 4, 2); (2, 4, 4); (2, 4, 5); (2, 4, 6); (2, 5, 1); (2, 5, 2); (2, 5, 3); (2, 5, 4); (2, 5, 5); (2, 5, 6); (2, 6, 1); (2, 6, 2); (2, 6, 3); (2, 6, 4); (2, 6, 5); (2, 6, 6); (3, 1, 1); (3, 1, 2); (3, 1, 3); (3, 1, 4); (3, 1, 5); (3, 1, 6); (3, 2, 1); (3, 2, 3); (3, 2, 4); (3, 2, 5); (3, 2, 6); (3, 3, 1); (3, 3, 2); (3, 3, 4); (3, 3, 6); (3, 4, 1); (3, 4, 3); (3, 4, 5); (3, 4, 6); (3, 5, 1); (3, 5, 2); (3, 5, 4); (3, 5, 5); (3, 5, 6); (3, 6, 1); (3, 6, 2); (3, 6, 3); (3, 6, 4); (3, 6, 5); (3, 6, 6); (4, 1, 1); (4, 1, 2); (4, 1, 3); (4, 1, 4); (4, 1, 5); (4, 1, 6); (4, 2, 1); (4, 2, 2); (4, 2, 3); (4, 2, 5); (4, 2, 6); (4, 3, 1); (4, 3, 3); (4, 3, 4); (4, 3, 5); (4, 3, 6); (4, 4, 1); (4, 4, 2); (4, 4, 4); (4, 4, 6); (4, 5, 1); (4, 5, 2); (4, 5, 3); (4, 5, 5); (4, 5, 6); (4, 6, 1); (4, 6, 2); (4, 6, 3); (4, 6, 4); (4, 6, 5); (4, 6, 6); (5, 1, 1); (5, 1, 2); (5, 1, 3); (5, 1, 4); (5, 1, 5); (5, 1, 6); (5, 2, 1); (5, 2, 2); (5, 2, 3); (5, 2, 4); (5, 2, 5); (5, 2, 6); (5, 3, 1); (5, 3, 2); (5, 3, 4); (5, 3, 5); (5, 3, 6); (5, 4, 1); (5, 4, 3); (5, 4, 5); (5, 4, 6); (5, 5, 1); (5, 5, 2); (5, 5, 4); (5, 5, 5); (5, 5, 6); (5, 6, 1); (5, 6, 2); (5, 6, 3); (5, 6, 4); (5, 6, 5); (5, 6, 6); (6, 1, 1); (6, 1, 2); (6, 1, 3); (6, 1, 4); (6, 1, 5); (6, 1, 6); (6, 2, 1); (6, 2, 2); (6, 2, 3); (6, 2, 4); (6, 2, 5); (6, 2, 6); (6, 3, 1); (6, 3, 2); (6, 3, 3); (6, 3, 4); (6, 3, 5); (6, 3, 6); (6, 4, 1); (6, 4, 2); (6, 4, 3); (6, 4, 4); (6, 4, 5); (6, 4, 6); (6, 5, 1); (6, 5, 2); (6, 5, 3); (6, 5, 4); (6, 5, 5); (6, 5, 6); (6, 6, 1); (6, 6, 2); (6, 6, 3); (6, 6, 4); (6, 6, 5); (6, 6, 6) Pilot Design FLC Selection Smooth Expected Response Rough Expected Response 271 81 46 57 (1, 1, 1, 1); (1, 1, 1, 2); (1, 1, 1, 3); (1, 1, 2, 2); (1, 1, 3, 1); (1, 1, 3, 3); (1, 2, 1, 2); (1, 2, 2, 1); (1, 2, 2, 3); (1, 2, 3, 1); (1, 2, 3, 2); (1, 3, 1, 1); (1, 3, 1, 3); (1, 3, 2, 2); (1, 3, 3, 1); (1, 3, 3, 3); (2, 1, 1, 2); (2, 1, 2, 1); (2, 1, 2, 3); (2, 1, 3, 2); (2, 1, 3, 3); (2, 2, 1, 1); (2, 2, 1, 3); (2, 2, 2, 2); (2, 2, 3, 1); (2, 2, 3, 3); (2, 3, 1, 2); (2, 3, 2, 1); (2, 3, 2, 3); (2, 3, 3, 2); (3, 1, 1, 1); (3, 1, 1, 3); (3, 1, 2, 2); (3, 1, 3, 1); (3, 1, 3, 3); (3, 2, 1, 1); (3, 2, 1, 2); (3, 2, 2, 1); (3, 2, 2, 3); (3, 2, 3, 2); (3, 3, 1, 1); (3, 3, 1, 3); (3, 3, 2, 2); (3, 3, 2, 3); (3, 3, 3, 1); (3, 3, 3, 3) (1, 1, 1, 1); (1, 1, 1, 2); (1, 1, 1, 3); (1, 1, 2, 1); (1, 1, 2, 2); (1, 1, 3, 1); (1, 1, 3, 3); (1, 2, 1, 2); (1, 2, 2, 1); (1, 2, 2, 3); (1, 2, 3, 1); (1, 2, 3, 2); (1, 3, 1, 1); (1, 3, 1, 2); (1, 3, 1, 3); (1, 3, 2, 2); (1, 3, 2, 3); (1, 3, 3, 1); (1, 3, 3, 2); (1, 3, 3, 3); (2, 1, 1, 2); (2, 1, 1, 3); (2, 1, 2, 1); (2, 1, 2, 3); (2, 1, 3, 2); (2, 1, 3, 3); (2, 2, 1, 1); (2, 2, 1, 3); (2, 2, 2, 2); (2, 2, 3, 1); (2, 2, 3, 3); (2, 3, 1, 1); (2, 3, 1, 2); (2, 3, 2, 1); (2, 3, 2, 3); (2, 3, 3, 2); (3, 1, 1, 1); (3, 1, 1, 3); (3, 1, 2, 1); (3, 1, 2, 2); (3, 1, 3, 1); (3, 1, 3, 2); (3, 1, 3, 3); (3, 2, 1, 1); (3, 2, 1, 2); (3, 2, 1, 3); (3, 2, 2, 1); (3, 2, 2, 3); (3, 2, 3, 1); (3, 2, 3, 2); (3, 3, 1, 1); (3, 3, 1, 3); (3, 3, 2, 2); (3, 3, 2, 3); (3, 3, 3, 1); (3, 3, 3, 2); (3, 3, 3, 3) 272 256 117 158 (1, 1, 1, 1); (1, 1, 1, 2); (1, 1, 1, 3); (1, 1, 1, 4); (1, 1, 2, 1); (1, 1, 2, 3); (1, 1, 3, 2); (1, 1, 3, 3); (1, 1, 3, 4); (1, 1, 4, 1); (1, 1, 4, 3); (1, 1, 4, 4); (1, 2, 1, 1); (1, 2, 1, 4); (1, 2, 2, 2); (1, 2, 2, 4); (1, 2, 3, 1); (1, 2, 3, 3); (1, 2, 4, 1); (1, 2, 4, 2); (1, 2, 4, 4); (1, 3, 1, 2); (1, 3, 1, 3); (1, 3, 2, 1); (1, 3, 3, 1); (1, 3, 3, 4); (1, 3, 4, 2); (1, 4, 1, 1); (1, 4, 1, 3); (1, 4, 1, 4); (1, 4, 2, 2); (1, 4, 2, 4); (1, 4, 3, 1); (1, 4, 3, 3); (1, 4, 4, 1); (1, 4, 4, 3); (1, 4, 4, 4); (2, 1, 1, 1); (2, 1, 1, 4); (2, 1, 2, 2); (2, 1, 3, 1); (2, 1, 3, 4); (2, 1, 4, 2); (2, 2, 1, 1); (2, 2, 1, 3); (2, 2, 2, 4); (2, 2, 4, 3); (2, 2, 4, 4); (2, 3, 1, 4); (2, 3, 2, 1); (2, 3, 2, 3); (2, 3, 4, 1); (2, 3, 4, 3); (2, 4, 1, 1); (2, 4, 1, 3); (2, 4, 3, 1); (2, 4, 3, 2); (2, 4, 3, 4); (2, 4, 4, 4); (3, 1, 1, 2); (3, 1, 2, 1); (3, 1, 2, 3); (3, 1, 3, 4); (3, 1, 4, 1); (3, 1, 4, 3); (3, 1, 4, 4); (3, 2, 1, 4); (3, 2, 2, 1); (3, 2, 3, 2); (3, 2, 4, 1); (3, 3, 1, 1); (3, 3, 1, 3); (3, 3, 2, 2); (3, 3, 2, 4); (3, 3, 3, 3); (3, 3, 4, 2); (3, 3, 4, 4); (3, 4, 1, 2); (3, 4, 1, 4); (3, 4, 2, 1); (3, 4, 2, 3); (3, 4, 2, 4); (3, 4, 4, 1); (3, 4, 4, 2); (3, 4, 4, 3); (4, 1, 1, 1); (4, 1, 1, 3); (4, 1, 1, 4); (4, 1, 2, 1); (4, 1, 2, 4); (4, 1, 3, 2); (4, 1, 3, 3); (4, 1, 4, 1); (4, 1, 4, 2); (4, 1, 4, 4); (4, 2, 1, 2); (4, 2, 2, 3); (4, 2, 3, 1); (4, 2, 3, 4); (4, 2, 4, 2); (4, 2, 4, 4); (4, 3, 1, 1); (4, 3, 1, 4); (4, 3, 2, 3); (4, 3, 3, 1); (4, 3, 4, 1); (4, 3, 4, 3); (4, 4, 1, 1); (4, 4, 1, 3); (4, 4, 1, 4); (4, 4, 2, 2); (4, 4, 3, 1); (4, 4, 3, 3); (4, 4, 3, 4); (4, 4, 4, 1); (4, 4, 4, 2); (4, 4, 4, 4) 273 (1, 1, 1, 1); (1, 1, 1, 2); (1, 1, 1, 3); (1, 1, 1, 4); (1, 1, 2, 1); (1, 1, 2, 3); (1, 1, 3, 2); (1, 1, 3, 3); (1, 1, 3, 4); (1, 1, 4, 1); (1, 1, 4, 3); (1, 1, 4, 4); (1, 2, 1, 1); (1, 2, 1, 4); (1, 2, 2, 2); (1, 2, 2, 4); (1, 2, 3, 1); (1, 2, 3, 3); (1, 2, 4, 1); (1, 2, 4, 2); (1, 2, 4, 4); (1, 3, 1, 1); (1, 3, 1, 2); (1, 3, 1, 3); (1, 3, 1, 4); (1, 3, 2, 1); (1, 3, 2, 3); (1, 3, 3, 1); (1, 3, 3, 2); (1, 3, 3, 4); (1, 3, 4, 2); (1, 3, 4, 4); (1, 4, 1, 1); (1, 4, 1, 2); (1, 4, 1, 3); (1, 4, 1, 4); (1, 4, 2, 1); (1, 4, 2, 2); (1, 4, 2, 4); (1, 4, 3, 1); (1, 4, 3, 3); (1, 4, 4, 1); (1, 4, 4, 2); (1, 4, 4, 3); (1, 4, 4, 4); (2, 1, 1, 1); (2, 1, 1, 3); (2, 1, 1, 4); (2, 1, 2, 2); (2, 1, 2, 4); (2, 1, 3, 1); (2, 1, 3, 3); (2, 1, 3, 4); (2, 1, 4, 1); (2, 1, 4, 2); (2, 1, 4, 4); (2, 2, 1, 1); (2, 2, 1, 3); (2, 2, 2, 4); (2, 2, 3, 1); (2, 2, 3, 2); (2, 2, 4, 3); (2, 2, 4, 4); (2, 3, 1, 2); (2, 3, 1, 4); (2, 3, 2, 1); (2, 3, 2, 3); (2, 3, 3, 4); (2, 3, 4, 1); (2, 3, 4, 3); (2, 4, 1, 1); (2, 4, 1, 3); (2, 4, 1, 4); (2, 4, 2, 2); (2, 4, 3, 1); (2, 4, 3, 2); (2, 4, 3, 4); (2, 4, 4, 2); (2, 4, 4, 4); (3, 1, 1, 1); (3, 1, 1, 2); (3, 1, 1, 4); (3, 1, 2, 1); (3, 1, 2, 3); (3, 1, 3, 2); (3, 1, 3, 4); (3, 1, 4, 1); (3, 1, 4, 3); (3, 1, 4, 4); (3, 2, 1, 2); (3, 2, 1, 4); (3, 2, 2, 1); (3, 2, 2, 3); (3, 2, 3, 2); (3, 2, 3, 4); (3, 2, 4, 1); (3, 2, 4, 3); (3, 3, 1, 1); (3, 3, 1, 3); (3, 3, 2, 2); (3, 3, 2, 4); (3, 3, 3, 1); (3, 3, 3, 3); (3, 3, 4, 2); (3, 3, 4, 4); (3, 4, 1, 2); (3, 4, 1, 4); (3, 4, 2, 1); (3, 4, 2, 3); (3, 4, 2, 4); (3, 4, 3, 2); (3, 4, 4, 1); (3, 4, 4, 2); (3, 4, 4, 3); (3, 4, 4, 4); (4, 1, 1, 1); (4, 1, 1, 3); (4, 1, 1, 4); (4, 1, 2, 1); (4, 1, 2, 2); (4, 1, 2, 4); (4, 1, 3, 1); (4, 1, 3, 2); (4, 1, 3, 3); (4, 1, 4, 1); (4, 1, 4, 2); (4, 1, 4, 3); (4, 1, 4, 4); (4, 2, 1, 1); (4, 2, 1, 2); (4, 2, 1, 3); (4, 2, 2, 3); (4, 2, 2, 4); (4, 2, 3, 1); (4, 2, 3, 4); (4, 2, 4, 2); (4, 2, 4, 4); (4, 3, 1, 1); (4, 3, 1, 2); (4, 3, 1, 4); (4, 3, 2, 1); (4, 3, 2, 3); (4, 3, 3, 1); (4, 3, 3, 2); (4, 3, 3, 4); (4, 3, 4, 1); (4, 3, 4, 3); (4, 4, 1, 1); (4, 4, 1, 3); (4, 4, 1, 4); (4, 4, 2, 2); (4, 4, 2, 4); (4, 4, 3, 1); (4, 4, 3, 3); (4, 4, 3, 4); (4, 4, 4, 1); (4, 4, 4, 2); (4, 4, 4, 4)