Psychology, methods and science, why worry?

Over the last couple of months I have been toying with ideas around Psychology, methods and science.

I suspect that what I want to say will be a bit bigger than one blog post, so I intend to do a series of posts.

This installment is about why I think we have a problem.

As someone with an interest in conceptual and historical issues I tend to range a bit more freely over the psychological literature than specialists in a particular area. Some of the problems which are acknowledged in one aspect of psychology seem to be mirrored in other areas of the discipline. If the problems are even more widespread than that it seems to implicate the whole discipline.

We do not routinely test for similarity instead of difference

One of the outstanding issues in Psychology is something I first became aware of with the Psychology of Sex and Gender.

While most of us, most of the time, act and talk as if there are psychological differences between men and women the scientific evidence for those differences is at best contested, at worst none existent.

What tends to happen is that an area of possible, psychological, difference is opened up by a statistically significant difference being published. Over time a bunch of other people do similar (although not identical) work. After some more time a meta analysis is carried out on the findings, and the results of the meta analysis suggest that the original difference is small and inconsistent. This may lead us to believe that the genders are more (psychologically) similar than different.

However, while I strongly believe that psychological differences that we can detect using our current methods are small and inconsistent, that is not the same thing as saying the men and women are psychologically similar. We simply have not been testing for similarity, and lack of evidence for difference is not the same as evidence in favour of similarity.

Psychology studies are routinely under powered

In the paper Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49, 997-1003. Cohen argues that there are a number of problems with null hypothesis significance testing. One of the problems highlighted is that psychology studies are routinely under powered.

This suggested teaching exercise might help you grasp what this means, and may act as a useful teaching resource if you ever have to teach statistics.

One trouble with having a routine of under powered studies (studies with too few participants for a given effect size) is that we get studies published followed by a number of failures to replicate. After some time someone will do a meta analysis and suggest that the original finding is smaller and more inconsistent than originally thought. All this strikes me as a huge waste of resources.

We do not routinely try to replicate results

While the recent controversies around Bem may have put this into focus in psychology it is very difficult to get a straightforward replication published. Psychologists work around this by doing "conceptual replications" replicating the idea, but not the study. However, this is surely missing the point. When researchers do meta analyses of psychology studies they try to include unpublished studies (normally Ph.D. dissertations) that did attempt replication. Unfortunately we do not know about the missing studies because of the next problem.

We do not routinely register studies before they are run

We simply don't know how big the 'file drawer' of studies is, people occasionally try to make a guess but without routinely registering studies before they run we cannot know so fairly important information about whether an apparently interesting finding has been extensively tested.   

We do not know the relationship between our studies and the world outside of the laboratory

In order to carry out science like investigations of human psychology it is necessary to simplify phenomena so that we can test for some of the things that might affect human psychology, while concentrating on a sub-set of things that might affect human psychology. What psychologists mean when they talk about 'controlling unwanted variables'.

One of the questions is can we do that in a meaningful way. I suspect we can but all too often we do not try to. There is a famous talk given by the physicist Richard Feynman on Cargo Cult Science.

All too often we don't know what our laboratory studies, with humans, actually mean because we haven't done the basic work (Feynman also makes some points about replication, which means the problem of lack of replications stretches back at least to the mid twentieth century).

Most psychometric tests have not been tested for predictive validity

Which is ultimately the same problem as above, but for personality tests.

We routinely use poor sampling methods

Psychologists routinely use undergraduate students, self selecting samples, and samples drawn purely from clinical populations. That they/we then go on to make universal claims about human psychology from these samples is just bizarre.

Next episode being careful with 'brain scans'