Issue: Volume 10, Number 6 (Circulation: Over 5500 worldwide)
Date: June 2010
From: Mark J. Anderson, Stat-Ease, Inc., Statistics Made Easy® Blog

Dear Experimenter,

Here's another set of frequently asked questions (FAQs) about doing design of experiments (DOE), plus alerts to timely information and free software updates. If you missed the previous DOE FAQ Alert, see below.

==> Tip: Get immediate answers to questions about DOE via the Search feature on the main menu of the Stat-Ease® web site. This not only pores over previous alerts, but also the wealth of technical publications posted throughout the site.

Feel free to forward this newsletter to your colleagues. They can subscribe by going to http://www.statease.com/doealertreg.html. If this newsletter prompts you to ask your own questions about DOE, please address them via mail to: [email protected].

Also, Stat-Ease offers an interactive web site — the Support Forum for Experiment Design at http://forum.statease.com. Anyone (after gaining approval for registration) can post questions and answers to the Forum, which is open for all to see (with moderation). Furthermore, the Forum provides program help for Design-Ease® and Design-Expert® software. Check it out and search for answers. If you come up empty, do not be shy: Ask your question! Also, this being a forum, we encourage you to weigh in!

For an assortment of appetizers to get this Alert off to a good start, follow this link, http://www.StatsMadeEasy.net* (-> new web site!), and see a number of new blogs (listed below, beginning with the most recent one):

—PB&J please, but hold the jelly (and margarine) and put it on toast — a mixture design combined with a categorical factor
—Stat-Ease Corporation celebrates 25 years in business
—Two-level factorial experimentation might make music for my ears

Also see the new comments on post #520 (3/28/10): "Misuse of statistics calls into question the credibility of science."

*Need a feed or e-mail updates from StatsMadeEasy? Go to http://feeds.feedburner.com/StatsMadeEasy. It's easy!

"Your StatsMadeEasy blogs brighten up a dreary workday!"
—Applied Statistician, Florida

Topics in the body text of this DOE FAQ Alert are headlined below (the "Expert" ones, if any, delve into statistical details).

1. FAQ: Is it necessary to introduce replicate runs for factorial designs?
2. Info Alert: Practical considerations for the implementation of design of experiments in quality by design (QBD)
3. Book Giveaway: Winners of 1 copy each of "DOE Simplified, 2nd Ed" (signed) and "Response Surface Methodology, 3rd Ed"
4. Webinar Alert (1st): DOEMade Easy and More Powerful via Design-Expert Software,Part 2 — Response Surface Methods (RSM) for Process Optimization
5. Events Alert: Where and when to attend talks by experts from Stat-Ease and see its software exhibited
6. Workshop Alert: Unique opportunity to learn factorial DOE and RSM in back-to-back workshops — a powerful one-two punch

P.S. Quote for the month: Soundness of observations on "obvious things" versus the "fancy" of the brain. (Wisdom from the first Curator of Experiments.)

(Page down to the end of this ezine to enjoy the actual quote.)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

1. FAQ: Is it necessary to introduce replicate runs for factorial designs?

-----Original Question-----
From: Fabrication Engineer
"I have two questions:
—First: I would like to know how calculate the effect of each factor with no replicates for a general factorial design on 4 factors with 4 levels each. My Design-Expert software somehow determines the significance of each factor without the presence of any replicates in the design of experiment. How can this be? How does it calculates the variance? Should I introduce replicates?
—Second: What is the 'lack of fit'? It appears when I add replicates. If there is not any replicate, this parameter disappears from the ANOVA (analysis of variance)."

Answer (from Stat-Ease Consultant Wayne Adams):
"Effects in a factorial design can be estimated without any true replicates because there is hidden replication. The average outcome of a given treatment (level) can always be estimated and the difference between treatment means (or grand mean in the case of categoric factors) can be calculated. This produces an estimated effect and will eventually lead to an estimated coefficient if included in the fitted model.

If all the effects are estimated and included in the model then you are correct, there is not a valid test for significance. What if some of the treatment means were only slightly different? Would this small difference be attributed to a true factor effect or considered random noise in the system? We have a choice when fitting the model. We can choose to say the variation in the treatment means is mostly coming from random noise, or mostly coming from a true effect. If it is determined to be mostly noise (small effects) we pool all the noise effects together (pooled error). This pool of errors is compared to the larger effects. If the larger effects are enough larger (F-ratio), then they "test" significant.

Replicates show us variation that we KNOW belongs in the error pool. Nothing changed in the factors, yet the response measurements are somewhat different. This is called "pure error" in Design-Expert. The pure error is literally added to the pool of errors when the effects are tested. Because we know that some of the error comes from replicates we can also test the pooled error from the "too small" effects noted above against the "pure error". This is also a ratio of variances (F-ratio) and generates a p-value. A significant lack-of-fit indicates the pool of errors is larger than the pure error, which is usually interpreted to indicate the wrong model has been fitted.

Take care with the interpretation of lack-of-fit tests. If a process is extremely stable, the replicates will have very little variation. The tiny pure error this produces will create a significant lack-of-fit test. In these cases take a look at the adjusted and predicted R-squareds. If they are high and within 0.2 of each other then the model is representing the data and true response surface well. If there are no replicates, then there is no "pure error" estimate, therefore no lack-of-fit test.

Do you always need to replicate? No. Do replicates improve the analysis? Yes. Whether or not replicates are needed in a particular experiment depends on many things. For example, if the initial design falls short on power then we advise you add runs. However, if the initial design is a fraction of all combinations, then building a larger fraction is better than replicating the existing one, at least so far as power is concerned. Adding a number of replicated runs (we advise at least 4 — otherwise do not bother to do any) is useful for enabling the lack-of-fit test."

Comment:
This answer by Wayne covers many bases. One that I'd like to reinforce is that when replicates are added to a design, be very wary of their agreement with each other being too good to be true, that is, not really reflecting the true error overall. I've seen cases where every replicate agreed exactly! Obviously someone simply copied over the responses rather than actually replicating the run. Other common mistakes are to simply re-sample or re-test the materials made during a given run, which underestimates the error by not going through a compleat process re-set. Finally, even replicates done fair and square, re-done from start to finish, might produce results that are too far off for the experimenter to accept, so they get discarded and another run is done that agrees better. That is not fair! — Mark

PS. Version 8 of Stat-Ease software offers a wonderful new tool for general factorials such as this 4x4x4x4 —the ability to plot effects on a half-normal plot. This makes it really easy to sort out at a glance which, if any, effects stand out from the rest (the vital few). Simply click these into your provisional model (subject to verification via ANOVA and residual diagnostics) and leave the remainder of effects, those near zero that are lined up (the trivial many, normally distributed) for the error pool. This is really handy for all factorials, but especially ones that are unreplicated. For details on Design-Expert V8 and a link to the free trial download (at the bottom of the long list of new features), go to http://www.statease.com/dx8descr.html.

(Learn more about factorials by attending the two-day computer-intensive workshop "Experiment Design Made Easy." See http://www.statease.com/clas_edme.html for a description of this class and link from this page to the course outline and schedule. Then, if you like, enroll online.)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

PLEASE DO NOT SEND ME REQUESTS TO SUBSCRIBE OR UNSUBSCRIBE — FOLLOW THE INSTRUCTIONS AT THE END OF THIS MESSAGE.

Sincerely,

Mark

Mark J. Anderson, PE, CQE
Principal, Stat-Ease, Inc. (http://www.statease.com)
2021 East Hennepin Avenue, Suite 480
Minneapolis, Minnesota 55413 USA

PS. Quote for the month—Soundness of observations on "obvious things" versus the "fancy" of the brain.

"The science of nature has been already too long made only a work of the brain and the fancy: It is now high time that it should return to the plainness and soundness of observations on materials and obvious things."

—Robert Hooke, first Curator of Experiments for the Royal Society of England, founded 1662. (Source: "The Discoverers" by Daniel J. Boorstin, Random House, New York, 1983.)

Trademarks: Stat-Ease, Design-Ease, Design-Expert and Statistics Made Easy are registered trademarks of Stat-Ease, Inc.

Acknowledgements to contributors:
—Students of Stat-Ease training and users of Stat-Ease software
—Stat-Ease consultants Pat Whitcomb, Shari Kraber, Wayne Adams and Brooks Henderson (see http://www.statease.com/consult.html for resumes)
—Statistical advisor to Stat-Ease: Dr. Gary Oehlert (http://www.statease.com/garyoehl.html)
—Stat-Ease programmers led by Neal Vaughn and Tryg Helseth (http://www.statease.com/pgmstaff.html)
—Heidi Hansel Wolfe, Stat-Ease marketing director, Karen Dulski, and all the remaining staff that provide such supreme support!

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Interested in previous FAQ DOE Alert e-mail newsletters?
To view a past issue, choose it below.

#1 Mar 01, #2 Apr 01, #3 May 01, #4 Jun 01, #5 Jul 01 , #6 Aug 01, #7 Sep 01, #8 Oct 01, #9 Nov 01, #10 Dec 01, #2-1 Jan 02, #2-2 Feb 02, #2-3 Mar 02, #2-4 Apr 02, #2-5 May 02, #2-6 Jun 02, #2-7 Jul 02, #2-8 Aug 02, #2-9 Sep 02, #2-10 Oct 02, #2-11 Nov 02, #2-12 Dec 02, #3-1 Jan 03, #3-2 Feb 03, #3-3 Mar 03, #3-4 Apr 03, #3-5 May 03, #3-6 Jun 03, #3-7 Jul 03, #3-8 Aug 03, #3-9 Sep 03 #3-10 Oct 03, #3-11 Nov 03, #3-12 Dec 03, #4-1 Jan 04, #4-2 Feb 04, #4-3 Mar 04, #4-4 Apr 04, #4-5 May 04, #4-6 Jun 04, #4-7 Jul 04, #4-8 Aug 04, #4-9 Sep 04, #4-10 Oct 04, #4-11 Nov 04, #4-12 Dec 04, #5-1 Jan 05, #5-2 Feb 05, #5-3 Mar 05, #5-4 Apr 05, #5-5 May 05, #5-6 Jun 05, #5-7 Jul 05, #5-8 Aug 05, #5-9 Sep 05, #5-10 Oct 05, #5-11 Nov 05, #5-12 Dec 05, #6-01 Jan 06, #6-02 Feb 06, #6-03 Mar 06, #6-4 Apr 06, #6-5 May 06, #6-6 Jun 06, #6-7 Jul 06, #6-8 Aug 06, #6-9 Sep 06, #6-10 Oct 06, #6-11 Nov 06, #6-12 Dec 06, #7-1 Jan 07, #7-2 Feb 07, #7-3 Mar 07, #7-4 Apr 07, #7-5 May 07, #7-6 Jun 07, #7-7 Jul 07, #7-8 Aug 07, #7-9 Sep 07, #7-10 Oct 07, #7-11 Nov 07, #7-12 Dec 07, #8-1 Jan 08, #8-2 Feb 08, #8-3 Mar 08, #8-4 Apr 08, #8-5 May 08, #8-6 June 08, #8-7 July 08, #8-8 Aug 08, #8-9 Sep 08, #8-10 Oct 08, #8-11 Nov 08, #8-12 Dec 08, #9-01 Jan 09, #9-02 Feb 09, #9-03 Mar 09, #9-04 Apr 09, #9-05 May 09, #9-06 June 09, #9-07 July 09, #9-08 Aug 09, #9-09 Sep 09, #9-10 Oct 09, #9-11 Nov 09, #9-12 Dec 09, #10-1 Jan 10, #10-2 Feb 10, #10-3 Mar 10, #10-4 April 10, #10-5 May 10, #10-6 June 10 (see above)

Click here to add your name to the DOE FAQ Alert newsletter list server.

Statistics Made Easy®

DOE FAQ Alert ©2010 Stat-Ease, Inc.
All rights reserved.*

* Feel free to forward this newsletter to your colleagues. They can subscribe at http://www.statease.com/doealertreg.html.