How Analysis of Variance Works


Intuitive explanations of statistical concepts for novices #1


Lots of people use Analysis of Variance (Anova) without really understanding how it works, so I thought I'd have a go at explaining the basics in an intuitive fashion.



Consider three experiments, A, B and C, each of which compares the impact of an intervention on an outcome measure. The three experiments each have 20 people in a control group and 20 in an intervention group.

Figure 1 shows the individual scores on an outcome measure for the two groups as blobs, and the mean score for each group as a dotted black line.






Figure 1: Simulated data from 3 intervention studies





In terms of average scores of control and intervention groups, the three groups look very similar, with the intervention group about .4 to .5 points higher than the control group.

But we can't interpret this difference without having an idea of how variable scores are in the two groups.



For experiment A, there is considerable variation within each group, that swamps the average difference between the groups. In contrast, for experiment C, the scores within each group are tightly packed. Group B is somewhere in between.



If you enter these data into a one-way Anova, with group as a between-subjects factor, you get out a F-ratio, which can then be evaluated in terms of a p-value which gives the probability of obtaining such an extreme result if there is really no impact of the intervention. As you will see, the F-ratios are very different for A, B, and C, even though the group mean differences are the same. And in terms of the conventional .05 level of significance, the result from experiment A is not significant, experiment C is significant at the .001 level, and experiment B shows a trend (p = .051).



So how is the F-ratio computed? It just involves computing a number that reflects the ratio between the variance of the means of the groups, and the average variance within each group.

When we just have two groups, as here, the first value just reflects how far away the two group means are from the overall mean. This is the Between Groups term, which is just the Variance of the two means multiplied by the number in each group (20). That will be similar for A, B and C, because the means for the two groups are similar and the numbers in each group are the same.



But the Within Groups term will differ substantially for A, B, and C, because it is computed as the average variance for the two groups.

The F-ratio is obtained by just dividing the between groups term by the within groups term. If the within groups term is big, F is small, and vice versa.



The R script used to generate Figure 1 can be found here: https://github.com/oscci/intervention/blob/master/Rftest.R



PS. 20/11/2017. Thanks to Jan Vanhove for providing code to show means rather than medians in Fig 1. 

Comments

Post a Comment

Popular posts from this blog

Open code: not just data and publications

Image
I had a nice exchange on Twitter this week. Nick Fay had found a tweet I had posted over a year ago, asking for advice on an arcane aspect of statistical analysis: I'd had some replies, but they hadn’t really helped. In the end, I’d worked out that there was an error in the stats bible written by Rand Wilcox, which was leading me astray. Once I’d overcome that, I managed to get the analysis to work. It was clear Nick was now having the same problem and going round in exactly the same circles I had experienced. My initial thought was that I could probably dig out the analysis and reconstruct what I’d done, but my heart sank at the prospect. However, then I had a cheerful thought. I had deposited the analysis scripts for my project on the Open Science Framework, here . I checked, and the script was pretty well annotated, and as a bonus you got a script showing you how to make a nice swarm plot. This experience comes hard on the heels of another interaction, this time around a paper ...
Read more

Data from the phonics screen: a worryingly abnormal distribution

Image
The new phonics screening test for children has been highly controversial.   I’ve been surprised at the amount of hostility engendered by the idea of testing children’s knowledge of how letters and sounds go together. There’s plenty of evidence that this is a foundational skill for reading, and poor ability to do phonics is a good predictor of later reading problems. So while I can see there are aspects of the implementation of the phonics screen that could be improved,   I don’t buy arguments that it will ‘confuse’ children, or prevent them reading for meaning. I discovered today that some early data on the phonics screen had recently been published by the Department for Education , and my inner nerd was immediately stimulated to visit the website and download the tables.   What I found was both surprising and disturbing. Most of the results are presented in terms of proportions of children ‘passing’ the screen, i.e. scoring 32 or more. There are tables showing how ...
Read more

Has the Society for Neuroscience lost its way?

Image
The tl;dr version: The Society for Neuroscience (SfN) makes humongous amounts of money from its journal and meetings, but spends very little on helping its members, while treating overseas researchers with indifference bordering on disdain. I first became concerned about the Society for Neuroscience back in 2010 when I submitted a paper to the Journal of Neuroscience. The instructions to authors explained that there was a submission fee (at the time about $50). Although I 'd never come across such a practice before, I reckoned it was not a large sum, and so went ahead. The Instructions for Authors explained that there was a veto on citation of unpublished work. I wanted to cite a paper of mine that had been ‘accepted in principle’ but needed minor changes, and I explained this in my cover letter. Nevertheless, the paper was desk-rejected because of this violation. A week later, after the other paper was accepted, I updated the manuscript and resubmitted it, but was told that I had ...
Read more