Pre-registration and Psychology publications

A couple of months ago the Guardian published this article on pre-registration of (mainly) psychology studies. This was followed by this article arguing against pre-registration.

This second article made me despair.

So I am going to look at it point by point after a bit of an introduction.

While there are a range of methods used in psychology the discipline has taken to a possible over reliance on null hypothesis significance testing (NHST).

There are conditions where NHST is great, especially when effect size estimations are also used, clinical trials, and in psychology when you are testing for an effect where you have a good idea of what size of effect matters outwith the laboratory, and you have a good model of that process to test in the laboratory. If NHST is the right way to understand data then there are a set of rigorous procedures that need to be followed else the results are likely to be inaccurate.

Now despite the fact that every psychology degree includes extensive research methods training it does appear that psychologists generally don't really understand what NHST is for, possibly because we use some fairly inaccurate short hand to try to introduce the concept to students.

We tend to say things like, probability that the effect is non-random, or that the effect is real.

However the NHST is not telling us the probability, it is telling us if an effect in a particular study will apply to the population that the sample was drawn from.

Obviously not all research is suited for NHST, I do qualitative research and historical research, neither of which use NHST, but also if you do not have a representative sample then NHST are not the thing you should be using. There are also arguments for using Bayesian statistics more generally, but that is not the argument I am making here. All quotes hereafter are from the Times Higher article.

Limiting more speculative aspects of data interpretation risks making papers more one-dimensional in perspective.

One of the issues is that these 'speculative' aspects of data interpretation tend to be done with NHST after looking at the data, that is not speculative it is wrong. Where exploratory data analysis is done it should clearly be labelled as such, and reporting significance levels shouldn't happen.

..commitment to publish with the journal concerned would curtail researchers’ freedom to choose the most appropriate forum for their work after they have considered the results.

I am not sure what this means, the meaning I can put on it is that if you find something unexpected, by doing statistical tests ad hoc you may want to publish these somewhere. Well you can but as you have broken the fundamental rules of NHST then that is fraudulent unless you clearly mark it as exploratory data analysis. The correct thing to do in this case is to design a study to test the potential phenomena, and pre-register that study.

With no results to go on, reviewers would be more likely than ever to rely on reputation, which would count against junior scientists.

 Most journals claim their review process is blind. Good to see proof that it actually isn't. I have long suspected this, even without names one tends to know what colleagues are working on in the same field. It might be important to work out how we can do genuine blind peer reviewing else our discipline is a laughing stock.

In addition, the requirement to refine studies and their interpretation prior to data collection would prevent us from learning from our mistakes along the way.

This tends to lead to data peaking and changing stimulus materials, instructions and even measures as one goes along, all of which invalidates NHST. The correct thing to do is of course to learn the lessons and run a fresh study.

Moreover, in my fields (cognitive neuroscience and psychology), a significant proportion of studies would simply be impossible to run on a pre-registration model because many are not designed simply to test hypotheses.

Well yes, so in those studies NHST results are not being erroneously reported, no problem with that at all and that is a good point. Psychology needs to look carefully at what it is doing and what science actually might be. However it does on

...many of the participant populations introduce significant sources of complexity and noise

 And these participant populations are not ones that can be used to make statements about the general population, so why exactly would NHST be used here.

So in summary, lots of reasons not to use NHST, however, we want to use them badly and fear pre-registration might stop that being done.