Sunday 21 January 2018

Will the new reporting guidelines for quantitative studies end the replication crisis

The narrative around the new reporting guidelines is very technocratic but the subtext is all about the much reported replication crisis. Back when I used to teach critical reading of psychology journal articles my approach was if something that should be in the report is missing that is because the authors are covering up a mistake. Hopefully the new guidelines will mean that authors will have to report the dodgy things they are doing, or they will be outright fraudulent. This can only be a good thing.

So lets get into some detail.

Preregistration
 An encouragement to preregister all psychology studies, but an acknowledgement it is only really happening in studies with a clinical application. This really does have to become routine across quantitative psychology.

Sampling

"Describe procedures for selecting participants, including...
• Percentage of sample approached that actually participated
• Whether self-selection into the study occurred (either by individuals or by units, such as schools or clinics)

• Settings and locations where data were collected as well as dates of data collection."

The idea of recording the number of potential participants approached is good, will give some information about possible volunteer bias. Self selection is another way that the characteristics of the participants can be problematic.
The requirement for date and location seems to be an attempt to deal with fraudulent data collection, follow this link for a recent example. 

Measuring instruments


"Define all primary and secondary measures and covariates, including measures collected but not included in this report."
After several failures to replicate 'power posing' findings one of the original authors made the claim that there were a whole bunch of measures that were not reported. That was always poor practice but these reporting conventions make sure that things are reported accurately, or the author(s) deliberately mislead.
Masking



"Report whether participants, those administering the experimental manipulations, and those assessing the outcomes were aware of condition assignments.
If masking took place, provide statement regarding how it was accomplished and if and how the success of masking was evaluated."

Everyone knows it is best that the person talking to the participant doesn't know the hypothesis and conditions, and if there is any chance of human error in scoring it is better that the scorer doesn't as well. But that is difficult to achieve and makes doing psychology experiments more expensive.
  
Analytical strategy

This works best if these are registered in advance.

Describe the analytic strategy for inferential statistics and protection against experiment-wise error for
Primary hypotheses
Secondary hypotheses
Exploratory hypotheses 


One thing that I suspect happens too often is people pretending exploratory hypotheses were the hypothesis they were always looking for. 

Participant Flow
Careful accounting of why any data was dropped, and keeping all the original data so others can look at it, both are necessary.

There is a lot more detail, but these are the things that seem to be important additions. It will be 2-3 years before the impact of this is felt, hopefully it is a step in the right direction. 
 

No comments:

Post a Comment