Some thoughts on the replication issue in Psychology

So new book contract for Dai Jones, Phil Tyson and me and the idea this summer is to make some inroads into my chapters. I am going to use the blog to work through some of the issues in my chapters with the idea that it will help the writing process.

One of the things I am writing about is methodological issues.

This article in the Guardian set me to thinking.

The first thought is the state of crisis the discipline of psychology is in.

For generations psychologists have been doing experimental work with too few participants, leading to a chronic lack of power. This has been made worse by researchers throwing as many variables as possible into studies, so that data collection is more 'efficient' and a tendency to write up studies as if some of the variables never existed.

Allied to a lack of pre-registration of studies and the well known positive result bias in publication then we get to a situation where I am unsure of any findings in psychology, beyond a handful of well  verified findings. While I might not be a experimentalist I still want psychology to be able to act in that mode, albeit with an acknowledgement that findings are probably time and culture specific until we have positive evidence to the contrary.

Adding to this a culture that is hostile to replication studies then I am not sure what, if anything, we should be teaching undergraduate students.

Some of the recent historical research on famous findings like the Milgram obedience study suggests that a tendency to "clean up" findings and present results in an overly positive way has been standard practice for over 60 years.

The second thought is what a bunch of hypocrites we are.

The article draws parallels with the natural sciences in making a call for more replication. Yet psychologists have been quite happily attacking parapsychologists over just this issue (along with the issue of methodological transparency). Some of that fire and methodological exactitude should be turned on the rest of the discipline.

The third thought is it should be of no surprise if even well done, with sufficient participants and replicated in their own time, historical results are not replicated.

People change, and humanity changes. In so far as one can believe any psychology research, the research on stereotype threat suggests that cultural expectations can affect performance. The looping effect, as Ian Hacking calls it, is one possible mechanism where past research on psychology can change the nature of the object of study (people). None of which is an argument against doing careful well replicated research in the here and now, it is useful to know about the psychology of people, even if that knowledge has a shelf life.

The final thought is we need to be much more explicit about linking the models of human action and thought psychologists use in the laboratory to real world. If we establish beyond reasonable doubt that a given independent variable can have a measurable effect in the laboratory we need to be able to say what that effect looks like in the wider world.