Fractional Factorial with Foldover

Introduction

At the outset of your experimental program you may be tempted to design one comprehensive experiment that includes all known factors - to get the BIG Picture in one shot. This assumes that you can identify all the important factors and their optimal levels. A more efficient, and less risky, approach consists of a sequence of smaller experiments. You can then assess results after each experiment and use what you learn for designing the next experiment. Factors may be dropped or added in mid-stream, and levels evolved to their optimal range.

Highly fractionated experiments make good building blocks for sequential experiments. Many people use Plackett-Burman designs for this purpose, but we prefer the standard two-level approach or the minimum-run (“Min Run”) options offered by Stat-Ease^® software. Regardless of your approach, you may be confounded in the interpretation of effects from these low-resolution designs. Of particular concern, main effects may be aliased with plausible two-factor interactions. If this occurs, you might be able to eliminate the confounding by running further experiments using a foldover design. This technique adds further fractions to the original design matrix.

We will discuss foldover designs that offer the ability to:

• free main effects from two-factor interactions, or

• de-alias a main effect and all of its two-factor interactions from other main effects and two-factor interactions.

Stat-Ease’s “Foldover” feature automatically generates the additional design points needed for either type of foldover.

Saturated Design Example

Lance Legstrong has one week to fine-tune his bicycle before the early-bird Spring meet. He decides to test seven factors in only eight runs around the quarter-mile track. We show Lance’s design below, which can be set up from Stat-Ease via its factorial tab with the default design builder for a “2-Level Factorial”. It is “saturated” with factors, that is, no more can be added for the given number of runs. (Inspiration for this case study came from Statistics for Experimenters, 2nd Edition (Wiley, 2005) by Box, Hunter and Hunter (“BHH”). In Section 6.5, BHH reports a very similar study with a few factors and results differing from those shown below.)

“When your television set misbehaves, you may discover that a kick in the right place often fixes the problem, at least temporarily. However, for a long-term solution the reason for the fault must be discovered. In general, problems can be fixed or they may be solved. Experimental design catalyzes both fixing and solving.” – Box, Hunter and Hunter.

As you will see in a moment via an evaluation, this is a resolution III design in which all main effects are confounded with two-factor interactions. (To alert users about these low-resolution designs, they are red-flagged in the design builder within Stat-Ease.)

Lance would prefer to run a resolution V or higher experiment, which would give clear estimates of main effects and their two-factor interactions. Unfortunately, at the very least this would require 30 runs via the “min-run Res V” design (also known as “MR5”) invented by Stat-Ease statisticians. But 30 runs are too many for the short time remaining before competition. Lance briefly considers choosing an intermediate resolution IV design (color-coded a cautionary yellow in Stat-Ease), but seeing that it requires 16 runs, he decides to perform only the 8 runs needed to achieve the lesser resolution III.

To save time, load the experimental results by clicking Help, Tutorial Data -> Biker. After looking over the data, check out the alias structure of the design by clicking the Evaluation node on the left. Note that Stat-Ease recognizes that this meager design can only model response(s) to the order of main effects. The program is set to produce results showing aliasing up to two-factor interactions (2FI) – ignoring terms of 3FI or more.

Now press the Results button at the top of the data window, and then the Aliases tab to see the impact of only running a Res III design in this case.

This design is a 1/16th fraction, so every effect will be aliased with 15 other effects, most of which are ignored by default to avoid unnecessary screen clutter. The output indicates that each main effect will be confounded with three two-factor interactions.

Don’t bother studying the remaining design evaluation report. Click the Time ¼ mile node in the analysis branch at the left and press the Start Analysis button to bring up the Effects window. Notice that there are three effects that stand off to the right of the line of trivial effects near zero. Starting from the right, click those three largest effects (or rope them off as pictured below).

Then on the Pareto plot, right-click the biggest bar (E) to show its aliases as below.

Factors B (Tires), E (Gear), and G (Generator) are clearly significant. Click ANOVA to test this statistically. Before you can go there, you are reminded via the warning shown below that this design is aliased.

You know about this from the design evaluation, so click No to continue. (Option: click Yes to go back and see the list. Then click the ANOVA button.)

Lance gets a great fit to his data. Based on the negative coefficient for factor B, it seems that hard tires decrease his time around the track. On the other hand, the positive coefficients for factors E and G indicate that high gear gives him better speed and the generator should be left off. The whole approach looks very scientific and definitive. In fact, after adjusting his bicycle according to the results of the experiment, Lance goes out and wins his race!

However, Lance’s friend (and personal coach) Sheryl Songbird spots a fallacy in the conclusions. She points out that all the main effects in this design are confounded with two-factor interactions. Maybe one of those confounded interactions is actually what’s important. “You must run at least one more experiment to clear this up,” says Sheryl. “That sounds like a great idea,” says Lance tiredly, realizing that he will never sleep easy knowing that some doubt remains as to the ideal settings for his bike.

Complete Foldover Design

To untangle the main effects from the interactions in his initial resolution III design, Lance can run a “foldover,” which requires a reversal in all of the signs on the original eight runs. Combining both blocks of runs produces a resolution IV design – all main effects will be free and clear of two-factor interactions (2FI’s). However, all the 2FI’s remain confounded with each other.

To create the new runs for the foldover design, click the Design node and then select Design Tools, Augment Design, Augment.

This selection brings up the following screen.

The default suggestion by Stat-Ease for Foldover is good, so click OK to continue. The program now lists which factors to be folded over. By default, Stat-Ease selects all the factors. This represents a complete foldover. Factors can be selected or cleared with a right click on the factor, or by double-clicking on the factor.

In this case, it is best that all factors be folded over, so click OK to continue. The program displays the following warning (really just a suggestion).

Click Yes and you will see another warning.

Click Yes. The warning is a reminder that if you choose the response with missing data for evaluation, only rows with data will be evaluated. To evaluate all the data, leave the Response on “Design Only”, and press Results and Factorial Effects Aliases to see the augmented design evaluation – much better!

Complete foldover of a resolution III design results in a resolution IV design. The proof for this case is that the main effects are now aliased only with three-factor interactions – ignored by default because they are so unlikely. See Montgomery’s Design and Analysis of Experiments textbook for the details on why this happens. By the way, the power also increases due to the doubling of experimental runs. If you’d like to assess this, go back to the Model screen and change the Order to Main effects before re-computing the Results. Then press the Terms (Power) button.

Now return to the main Design node and select View, Display Columns, Sort, Std Order, Ascending. (Tip: A quicker route to sorting any column is via a double-click on the header.) Notice how each run in block two is the reverse of block one. For example, whereas the seat (factor A) went up, down… in block 1, it goes down, up… in block 2 – everything goes opposite.

Lance’s results for the additional runs appear in the following table.

To continue following along with Lance, within your Response 1 column, enter his above times for the foldover block (Block 2) of eight runs, as shown below. First be sure you have changed the design matrix to standard order. Input factors must match up with the responses, otherwise the analysis is nonsense.

Click the Time ¼ mile node and press the Start Analysis button to display the Effects plot. Starting from the right, click the three largest effects.

It now appears that what looked to be effect G really must be AF. Be careful though – recall from the design evaluation that AF is aliased with BE and CD in this folded over design. (To see this, you can go back and look at the screen shot from before or press Evaluation on your screen and look at the Report. Or, even better, just right click on AF on the Half-Normal Plot (as pictured above)). Stat-Ease picks AF only because it is the lowest in alphabetical order of all the aliased terms.

From their subject matter knowledge, Lance and Sheryl know that AF, the interaction of the seat position with the wheel covers, cannot occur; and CD, the interaction of handle bar position with the brand of helmet, should not exist. They are sure the only feasible interaction is BE, the interaction of tire pressure and gears. However, proceed with the ANOVA on the model with the two-factor interaction at the default of AF.

The software gives the following warning about model hierarchy.

Because you selected interaction AF, the parent factors A and F must be included in the model to preserve hierarchy. This makes the model mathematically complete. Click Yes to correct the model for hierarchy.

Notice that these two new model terms, A and F, fall in the trivial-many near-zero population of effects. Therefore you may find that one or both are not very significant statistically. Nevertheless, it is important to carry them along.

Press forward to ANOVA again. You are now warned that “This design contains aliased terms. Do you want to switch to the View, Alias list…?” Click No to continue. (You will return later to this Alias List, so don’t worry.)

The ANOVA looks good. (Factor A is not significant but it’s in the model to preserve hierarchy.)

Click Diagnostics. You will see nothing abnormal here. Then click Model Graphs to view the “AF” interaction plot. Because neither “A” nor “F” as main effects changes the response very much, the interaction forms an X, as shown below.

Keep in mind that Lance and Sheryl know this interaction of the seat position with the wheel covers really cannot occur, so what’s shown on the graph is not applicable.

Investigation of Aliased Interactions

Lance decides to investigate the other, more plausible, interactions aliased with AF. You can follow along on your computer to see how this works. Lance clicks back onto the Effects tab, then selects the Numeric tab and maximizes this window.

He locates and right clicks term AF, below left, then selects BE, resulting in the next screenshot shown.

Note that this action is reversible, that is, AF can be substituted back in for BE. But for now, just to do something different, Lance selects the Pareto tab. The effects of A and F remain from the prior model in hierarchical support of the AF interaction.

Now with interaction BE as the substitute, these two main effects, clearly insignificant, are no longer needed, so click the bars labeled “F” (ranked 4th) and “A” (6th) to turn off these vestigial model terms. Notice how this simplifies into a family of effects: B, E and BE. In other words, this model achieves hierarchy with all terms being significant on an individual basis. As a general rule, a simpler (or as statisticians say, “more parsimonious”) model like this should be favored over one that requires more terms. This is especially true for models with parent terms that are not significant on their own, such as the one you created earlier with the orphan AF.

Lance now displays the BE interaction. Take a look at it yourself by clicking the Model Graphs button. (Click No if advised to view aliased interactions.)

Lance feels that it really makes much better sense this way.

Notice that Stat-Ease displays a red point on the graph, which in fractional factorials, only appears under certain conditions. Click it to identify these conditions.

At the left of the graph you now see that at this point the tire pressure (B) equals 40 psi and the gear (E) is set high. Above that you see the run number identified (yours may differ due to randomization) and the actual time of 83 seconds. The other factors, not displayed on the graph, would normally default to their average levels, but because they are categorical, the software arbitrarily chooses their low levels (red slide bars to the left). This setup matches the last row of the foldover in standard order. (Note: due to randomization, your run number may differ from that shown at the left of the graph.)

If you prefer to display the results as an average of the low and high levels, click the list arrow on the floating Factors Tool and make Average your selection, as shown below (already done for factors A:Seat, C:Handle Bars, D:Helmet Brand, F:Wheelcovers, and G:Generator).

Notice that the point on the graph now disappears because the average seat height was not one of the levels actually tested. To return the interaction plot back to original settings, press Default on the Factors Tool. Then play with the other settings. See if you can find any other conditions at which an actual run was performed. Remember that even with the foldover, you’ve only run 16 out of the possible 128 (27) combinations, so you won’t see very many points, if any, on the graphs. For example, as shown below, one of the other actual points (the green circle) can be found with the factors set on the tool as shown below (high generator).

Just to cover all bases, Lance decides to look at the second alias for AF – the CD effect, even though it just doesn’t make sense that C (handlebars) and D (helmet brand) would interact. Lance sees that the interaction plots look quite different. What a surprise! If you can spare the time, check this out by going back to the Effects button and selecting Numeric from the selection tool. Then replace BE with CD via the right-click menu. (You must then select the factors C and D in the model to make it hierarchical.) Based on his knowledge of biking, Lance makes a “leap of faith” decision: He assumes that BE is the real effect, not CD.

Now would be a good time to save your work by doing a File, Save As and typing in a new name, such as “Biker2”.

Inspired by these exciting results, Lance begins work on a letter advising the bicycle supplier to change how they initially set them up for racers. However, Lance’s friend and personal coach, Sheryl, reminds him that due to the aliasing of two-factor interactions, they still have not proven their theory about the combined impact of tire pressure and gear setting. “You must run a third experiment to confirm the BE interaction effect,” admonishes Sheryl. “I suppose I must,” replies Lance resignedly.

Single Factor Semi-Foldover

Sheryl declares, “By making just eight more runs you can prove your assumption.” “Wonderful,” sighs Lance as he rolls his eyes, “Please show me how.” “Well,” replies Sheryl, “If you run the same points as in the first two experimental blocks, but reverse the pattern only for the B factor, then B and all of its two-factor interactions will be free and clear of any other two-factor interactions.” “But that requires 16 more runs,” Lance complains. “Hold on,” says Sheryl, “I saw an article posted by Stat-Ease on ‘How To Save Runs, Yet Reveal Breakthrough Interactions, By Doing Only A Semifoldover On Medium-Resolution Screening Designs’ at statease.com/pubs/semifold.pdf . It details how you can do only half the foldover and still accomplish the objective for de-aliasing a 2FI.” Thus, Lance allows himself to be cajoled into another eight times around the track.

You can create the extra runs by returning to the Design node and from there selecting Design Tools, Augment Design, Augment. The program now advises you do the Semifold.

Press OK to bring up the dialog box for specifying which runs will be added in what now will become a third block of runs for Lance to ride. In the field for Choose the factor to fold on, select B. Lance and Sheryl puzzle over the next question, Choose the factor and its level to retain, but they quickly settle on D+ – the Windy brand of helmet, figuring that this factor definitely does not affect the response (and Sheryl likes the look of Lance wearing this brand of helmet!).

Give this the OK, click Yes as advised to go to design evaluation, and if the warning about missing data comes up, press Yes again. You should now see the starting screen for Evaluation – press Results to see the new Aliases. Scroll down to view the 2FI’s and notice that BE is now aliased only with three-factor interactions.

Notice that some effects, for example AB, are partially aliased with other effects. This is an offshoot of the semifold, causing the overall design to become non-orthogonal; that is, effects no longer can be independently estimated. You will see a repercussion of this when analyzing the data.

Click the main Design branch to see the augmented runs. Then double-click the Std column header to get it back in standard order.

Lance’s results can be seen in the following table.

Enter the times shown above into your blank response fields. Again, be certain everything matches properly. Then go to the Time ¼ Mile node on the left and press the Start Analysis button. The following warning comes up that the design is not orthogonal (as explained a bit earlier).

Press OK. Then, rope the three largest effects as pictured below. You will find B, E, and interaction BE to be significant.

The effects shift – you may not notice the change when all are selected at once.

When you press ANOVA, the “This design contains aliased terms” warning again appears. Click Yes to see the alias list, which now looks much better than before: The model (“M”) terms are aliased only with high-order interactions that are very unlikely to have any impact on the response.

Click ANOVA again and review the results. Then move on to Diagnostics. Check out all the graphs. Then click Model Graphs. The interaction plot of BE looks the same as before. Check it out. Just to put a different perspective on things, on the Factors Tool, right click E: Gear and select it for the X1 axis. You now get a plot of EB, rather than BE. In other words, the axes are flipped. Note that the lines are now dotted to signify that gear is a categorical factor (high or low). On the BE plot the lines were solid because B (tire pressure) is a numerical factor, which can be adjusted to any level between the low and high extremes.

Conclusion

“Thank goodness for designed experiments!” exclaims Lance. “I know now that if I am in low gear then it doesn’t matter much what my tire pressure is, but if I am in high gear then I could improve my time with higher tire pressure.”

“Or,” cautions Sheryl, “You could say that at low tire pressure it doesn’t really matter what gear you are in, but at high pressure you had better be in high gear!”

“Whatever,” moans Lance.

“Another thing I noticed,” prods Sheryl, “Your average time for each set of eight runs increased a lot. It was 70 seconds in the first experiment, 80 in the second set and almost 85 seconds in the last set of runs. You must have been getting tired. It’s a good thing we could extract the block effect or we might have been misled in our conclusions.”

“It doesn’t get any better than DOE,” agrees Lance.

In this instance, by careful augmentation of the saturated design through the foldover techniques as suggested by Stat-Ease (defaults provided via augmentation tool), Lance quickly found the significant factors among the seven he started with, and he pinned down how they interacted. He needed only 24 runs in total – broken down into three blocks. A more conservative approach would be to start with a Resolution V design, where all main effects and two-factor interactions would be free from other main effects and two-factor interactions. But this would have required 30 experimental runs at the least with the MR5 design (offered only by Stat-Ease!), or far more (64) with the standard, classical two-level fractional factorial design (27-1). And lest you forget, Lance Legstrong may not have won the early-bird Spring meet without the results from his first eight-run experiment.

In this case, Lance discovered a winning combination with the saturated Resolution III design. Thus, he developed a fix for the problem via design of experiments. However, without the prompting of Sheryl Songbird, his friend and personal coach, Lance may not have done the necessary follow up experiments via foldover to determine exactly what caused the improvement. She helped him use DOE to solve the problem.