One needs to understand how to obtain fair and representative samples. It clearly isn’t enough to just pick the first 60 loaves that come off the assembly line during the early morning hours because this wouldn’t give every loaf the same chance of being in the sample.
When carrying out experiments to learn about the effects of certain controlled factors (that is, factors that can be changed on purpose), one must be keenly aware that results can be affected by “nuisance“ factors that are not controlled during the course of the experiment.
Quite often, one doesn’t even know what these nuisance factors are or how they enter into the experiment. For illustration, consider an experiment that attempts to compare the impact of two training methods by exposing different groups of employees to the two methods and studying whether the type of training affects certain test scores. One must find a “fair” way to partition the work force into two comparable groups. It certainly would be bad to have responsible workers in one training method and the rest in the other; such an arrangement could bias the arrangement because responsible workers might score higher on any test.
Randomization, where the participants in method 1 (and method 2) are selected at random, provides a fair way. Such randomization is essential, as quite often we don’t even know which employee characteristics could have an impact on the comparison.
Variation in experimental results is a fact of life. It arises because of the many uncontrolled and uncontrollable factors factors that influence the process. By replicating (repeating) the experiment, one guards against interpreting too much from the result of a single run.
Learning is an iterative process. Each tme we look at the results of an experiment we obtain further information which allows us to focus our investigation even better. Because of this fact, one should always carry out investigations in the form of a sequence of small experiments. Experiments coat money and normally one operates within a certain fixed budget. If all resources are used up in one giant experiment, then there is no opportunity to put the resulting knowledge to use and to ask more further, more focused, questions. Hence it is a bad idea to exhaust one’s budget with a single massive experiment. It is much better to to conduct a sequence of small experiments. The strategy of changing only one factor at a time is also a very poor approach.
Get the Rare Faith Newsletter
Let me show you how to use the kind of faith that causes things to happen in finances, marriage, and parenting.
You’ll receive a weekly Newsletter, Podcast, and monthly Digest with fresh articles, special offers, and more! Serving over 50,000 subscribers since 2002, but don’t worry – It’s easy to cancel at any time with just one click.