I thought I'd take a moment to discuss an interesting (to me anyway) way I am using Statistics101.
So called, "Shotgun" approaches to research are common in my field. A shotgun approach is a theory generating method in which an investigator examines lots of possible relationships among large sets of variables. For instance, on might be interested in knowing if a measure of extraversion predicts any number of 64 behaviors from an observed social interaction. So one might correlate an extraversion score with scores on each of these 64 behaviors, resulting in 64 total correlations. However, if one is using hypothesis testing and a .05 alpha level, one would expect a certain number of those results to be "statistically" significant by chance alone.
This is a cause for concern among reviewers of this type of research. They complain that the results are nothing more than capitilizing on chance findings.
So I've used permutation tests and statistics101 to develop a way to refute this complaint.
Imagine the 64 correlations between a measure of extraversion and these 64 behaviors. Say 22 were found to be statistically significant at the .05 alpha level. The question of interst is, "How likely is it to find 22 statistically significant by chance alone?"
To answer this question, I do the following steps: 1) Shuffle the extraversion scores such that each sampling unit (participant) has an equal probability of being assigned any extraversion score by random chance. 2) Compute the 64 correlations between behavior and extraversion. 3) Record the number of statistically significant correlations if "extraversion scores were randomly assigned." 4) Repeat 1-3 perhaps 10,000 times to create a sampling distribution of "significant results by chance alone." 5) Compare the original observed number of statistically significant correlates with this sampling distribution to determine the probability of my observed result by chance alone.
The results have been extremely promising and I am currently working on extending this procedure in several different ways. First, I have already developed a way to extend this to situations in which one has two large sets of variables rather than one variable (e.g. extraversion) and one large set (behavior). Using Coordinated sorts, the same basic procedure outlined above is used. Additionally, I have worked with using a Bootstrap sampling technique rather than a permutation (shuffling) technique. The shuffling technique uses all the original data, just randomly assigned to participants without replacement. The bootstrap technique does the same, only it uses replacement such that participant 1's extraversion score could get assigned to 0 or more than 1 participant on any given trial.
So that is my current application of statistics101 and it is looking quite promising.
Sherman