Design Evaluation - Graphs - FDS Graph¶
Fraction of Design Space (FDS) graph is used to calculate the volume of the design space having a prediction variance less than or equal to a specified value.
The ratio of this volume to the total volume of the design space is the fraction of design space.
The goal is to produce a single plot showing the cumulative fraction of the design space on the x-axis (from zero to one) versus the prediction variance on the y-axis.
Stat-Ease, Inc. recommends an FDS score of at least 0.8 or 80% for exploration and optimization, and 100% for stability and robustness testing such as demonstrating the design space for quality by design (QbD) work.
The FDS Graph tool provides options for evaluating the FDS as a function of four error types:
Mean - Calculates FDS based on the half-width of the confidence interval. Use the Mean error type if the goal of the experiment is to find the optimized factor settings for specific response goals.
Pred - Calculates FDS based on the half-width of the prediction interval. The Pred error type is rarely used but can be used when more precision predicting the average of future samples is desired.
Diff - Calculates FDS based on the size of the difference between pairs of observations. Use the Diff error type as a substitute for power on response surface and mixture designs.
Tolerance - Calculates FDS based on the half-width of the tolerance interval. Use the Tolerance setting if you need to prove individual outcomes will meet specifications.
One or two-sided intervals can be specified on the pull down. Specify the type and direction to match the goals for the most critical response. The default two-sided interval is good for initial estimates of the FDS score.
There are three parameters: “delta”, “sigma”, and “alpha” for each error type and a fourth “Proportion” for the Tolerance error type.
“delta” specifies the maximum acceptable half-width (margin of error) of the respective interval for the Mean, Pred, and Tolerance error types. The best way to determine the delta is to answer the question, “plus or minus how much is an acceptable estimate?”
When the Diff error type is used, delta is the minimum desired change in the response The best way to determine this delta is to answer the question, “how much of a change in the response is important?”
In either case a larger delta produces a higher FDS score.
“sigma” is an estimate for the standard deviation that will appear on the ANOVA. It can be obtained from previous work with this system, work from a similar system, or outright guessing. If the unexplained nuisance variation can be minimized during the experiment then a smaller sigma can be entered to improve the FDS.
“alpha” is the significance level used throughout statistical analysis. Our default is 0.05 or 5%. It is the acceptable risk of a Type I error. Larger alpha increases FDS. Two-sided intervals use alpha/2 whereas one-sided interval use alpha to compute the t critical value.
“Proportion” is only used for Tolerance error type. It is the proportion of the individual outcomes required to be contained by the tolerance interval.
The FDS score can be increased by building a larger design, increasing the delta, reducing the sigma, increasing the alpha and/or decreasing the Proportion.
References
Alan Agresti. Categorical Data Analysis. Wiley, 2013.
Brian Beachkofski and Ramana Grandhi. Improved Distributed Hypercube Sampling, pages 1–7. Number 1274. American Institute of Aeronautics and Astronautics, 2002.
George E.P. Box and Norman R. Draper. Empirical Model-Building and Response Surfaces. Wiley, 1987.
George E.P. Box and K.B. Wilson. On the experimental attainment of optimum conditions. Journal of the Royal Statistical Society: Series B, 13(1):1–45, 1951.
Brusco, Cradit, and Steinly. An exact algorithm for hierarchically well-formulated subsets in second-order polynomial regression. Technometrics, 51(3):306–315, 2009.
Christensen, Pearson, and Johnson. Case-deletion diagnostics for mixed models. Technometrics, 34(1):38–45, 1992.
William S. Cleveland. Robust locally weighted regression and smoothing scatterplots. Journal of the American Statistical Association, 74(368):829–836, 1979.
J. Cornell. Experiments with Mixtures. Wiley, 3rd edition, 2002.
D. Cuthbert. Use of half-normal plots in interpreting factorial two-level experiments. Technometrics, 1(4):311, 1959.
DeGryze, Langhans, and Vandebroek. Using the correct intervals for prediction: a tutorial on tolerance intervals of ordinary least-squares regression. Chemometrics and Intelligent Laboratory Systems, 87(2):147–154, 2007.
N. Draper and D. Lin. Small response-surface designs. Technometrics, 32(2):187–194, 1990.
D. J. Finney. On the distribution of a variate whose logarithm is normally distributed. Journal of the Royal Statistical Society Supplement, 7:155–161, 1941.
Hahn and Meeker. Statistical Intervals, A Guide for Practitioners. 1991.
David W. Hosmer and Stanley Lemeshow. Applied Logistic Regression. Wiley, New York, second edition, 2000.
Rob J. Hyndman and Yanan Fan. Quantiles in statistical packages. The American Statistician, 50(4):361–365, 1996.
Kenward and Roger. Small sample inference for fixed effects from restricted maximum likelihood. Biometrics, 53(3):983–997, 1997.
J. S. Long. Regression Models for categorical and limited dependent variables. Sage Publications, Thousand Oaks, CA, 1997.
D. McFadden. Conditional logit analysis of qualitative choice behavior. In P. Zarembka, editor, Frontiers in Econometrics, chapter Four, pages 104–142. Academic Press, New York, 1974.
M.D. McKay, R.J. Beckman, and W.J. Conover. A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics, 21(2):239–245, 1979.
Raymond H. Meyers and Douglas C. Montgomery. Response Surface Methodology: Process and Product in Optimization Using Designed Experiments. Wiley, 1995.
D. Miller. Reducing transformation bias in curve fitting. The American Statistician, 38(2):124–126, 1984.
Douglas C. Montgomery, Elizabeth A. Peck, and . Geoffrey Vining. Introduction to Linear Regression Analysis. Wiley, 5th edition, 2012.
Douglas C. Montgomery, Elizabeth A. Peck, and G. Geoffrey Vining. Introduction to Linear Regression Analysis. Wiley, 3rd edition, 2001.
Raymond H. Myers. Classical and Modern Regression with Applications. Duxbury Press, 1986.
John A. Nelder. The selection of terms in response-surface models; how strong is the weak-heredity principle? The American Statistician, 1988.
G.W. Oehlert. A note on the delta method. American Statistician, 46:27–29, 1992.
Gary W. Oehlert. A first course in design and analysis of experiments. Gare W. Oehlert, 2010. ISBN 0-7167-3510-5.
J. Peixoto. A property of well-formulated polynomial regression models. The American Statistician, 44(1):26–30, 1990.
G. Piepel, J. Szychowski, and J. Loeppky. Augmenting scheffe linear mixture models with squared and/or crossproduct terms. Journal of Quality Technology, 2002.
Peter Z. G Qian, Huaiqing Wu, and C. F. Jeff Wu. Gaussian process models for computer experiments with qualitative and quantitative factors. Technometrics, 50(3):383–396, 2008. URL: https://doi.org/10.1198/004017008000000262, arXiv:https://doi.org/10.1198/004017008000000262, doi:10.1198/004017008000000262.
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
Salvador B. Rodriguez. Hotel room computational fluid dynamics to investigate airborne pathogen dispersal patterns. 1 2021. URL: https://www.osti.gov/biblio/1761600, doi:10.2172/1761600.
K. G. Roquemore. Hybrid designs for quadratic response surfaces. Technometrics, 18(4):419–423, 1976.
Jerome Sacks, William J. Welch, Toby J. Mitchell, and Henry P. Wynn. Design and Analysis of Computer Experiments. Statistical Science, 4(4):409 – 423, 1989. URL: https://doi.org/10.1214/ss/1177012413, doi:10.1214/ss/1177012413.
Thomas Santner, Brian Williams, and William Notz. The Design and Analysis Computer Experiments. Springer, 01 2003. ISBN 978-1-4419-2992-1. doi:10.1007/978-1-4757-3799-8.
T.J. Smith and C.M. McKenna. A comparison of logistic regression pseudo R^2 indices. Multiple Linear Regression Viewpoints, 39(2):17–26, 2013.
Michael Stein. Large sample properties of simulations using latin hypercube sampling. Technometrics, 29(2):143–151, 1987.
Tue Tjur. Coefficients of determination in logistic regression models–a new proposal: the coefficient of discrimination. The American Statistician, 63(4):366–372, November 2009.
Geoff Vining. Technical advice: residual plots to check assumptions. Quality Engineering, 23(1):105–110, 2011.
Weisberg and Stanford. Applied Linear Regression. John Wiley & Sons, Inc., 1985.
A Zahran, CM Anderson-Cook, and RH Myers. Fraction of design space to assess prediction capability of response surface designs. Journal of Quality Technology, 35(4):377–386, 2003.