di: Hattie's comparison of effect sizes

[Bonnie Grossen]

Kerry,
I’m late on this, but my attention just came to Hattie’s Visible Learning where he compares 252 “influences” in education and ranks them by Effect Size. Here’s the link:
https://visible-learning.org/hattie-ranking-influences-effect-sizes-learning-achievement/

My DI friends in Australia are chagrined that DI came out relatively low in the stack. I did a little searching and researching and came up with this analysis. I would like to know what you think of it, and if you have anything to add, or see any problems with my thinking.

I was able to read the first two chapters of Hattie's book, "Visible Learning: A Synthesis of over 800 Meta-Analyses Relating to Achievement" (2008 or 2011 or 2017, not sure). He makes it clear that he is not using Effect Size to mean the size of the difference between a treatment and a control group. In fact, very few comparison studies are included in his analyses: “The wars as to what counts as evidence for causation are raging as never before. Some have argued that the only legitimate support for causal claims can come from randomized control trials (RCTs, trials in which subjects are allocated to an experimental or a control group according to a strictly random procedure.). There are few such studies among the many outlined in this book."
I would agree that causal conclusions, e.g., DI causes higher achievement, can only be made from studies comparing the effects of two treatments: the one being studied and a reasonable alternative instructional model. 
Hattie describes 3 types of Effect Sizes that he has calculated in this massive meta-analysis. He then compares the ESs without regard for type. I find that very misleading rather than informative. Here are the three:
The first two:
“Statistically, an effect size can be calculated in two major ways:
	Effect size = [Mean treatment – Mean control] / SD
Or 
	Effect size = [Mean end of treatment – Mean beginning of treatment] / SD”
So he uses a traditional ES calculation, comparing the difference between two treatments AND he calculates effect size as a difference in the pre and posttest scores, with no comparison group: 
An Effect size for growth without a comparison group is an entirely different metric. Effect sizes of differences in the performance of comparison groups are likely to be so much smaller than an ES calculated on the growth from pre- to posttest. Both groups could grow enormously and have only a small difference. In addition, did he control for time. Certainly, instruction over a year is going to show more growth than a two-week intervention. (Many DI studies are only two-weeks long because that is all it took to get a significant difference.) Using ES for growth to say something contributed to that growth is completely unacceptable in the scientific method. Growth happens with time. You can't measure the additive value that a teaching strategy might have if you do just pre- and posttest. It’s basic logic.
And you certainly can’t mix Effect Sizes for group comparisons with simply growth over time (from pre to post) if you want to show the relative power of one teaching model over another. 
Here’s the real digression from scientific thinking. Hattie includes a 3rd way to calculate ES from correlation studies: 

I don't see the math involved in calculating the ES from a correlation. The example of the different heights of women and men is comparable to the common design of studies of metacognition: the better readers reported using metacognitive strategies more than the poor readers did. This is ridiculous.
Even though Hattie agrees that causal claims cannot be made from correlation studies, he still includes mostly correlational studies in his meta-analysis:
“Throughout this book, many correlates will be presented, as most meta-analyses seek such correlates of enhanced student achievement. A major aim is to weave a story from these data that has some convincing power and some coherence, although there is no claim to make these ‘beyond reasonable doubt’.”
If Hattie wants to use three different types of Effect Sizes, he at least has to put them into three different categories, three different lists. It is completely inappropriate and very unscientific to compare all these ESs with each other, calling them all “Cohen’s D”. 
Hattie received a lot of recognition for his earlier analyses of Effect Size, comparing the results of many comparison studies. I suspect he came under pressure from folks like Allington, Marzano, Goodman, those guys that are highly invested in educational nonsense. I know that when I managed to write something that people read and respected, I got a lot of hate mail. Hattie has to be a big boy and not succumb to their constant antagonism. He has to do what’s right.
So tell me what you think, Kerry?  I only got the first two chapters of his book online. Chapter 3 might provide something important that I am missing. But I don’t see 3 buckets of ES analysis, to match his 3 definitions of Effect Size.
Did I miss something? If you agree with me, I’m going to write to him and tell him what I think. I need confirmation because I can’t believe he could be so stupid. Up until now, I thought he was such a clear thinker and admired him. 
Thanks for all the informative postings you do for us on this website.
Warm regards,
Bonnie
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists-prod.uoregon.edu/pipermail/di/attachments/20191021/ee9bb586/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 5C443912350A438CAE3E4A5FA39BC0F3.png
Type: image/png
Size: 178642 bytes
Desc: not available
URL: <http://lists-prod.uoregon.edu/pipermail/di/attachments/20191021/ee9bb586/attachment-0001.png>