Some thoughts on Education, Genetics and Heritability and IQ

Some of you may have seen this story in the Guardian. I have been avoiding commenting on it until after the comments section was closed, because I hope to do some discourse analysis on both the newspaper report and the comments section, to try to find out how people are using the terms intelligence and heritability in everyday(ish) language.

The Guardian also usefully made the source material available. What I want to do in this blog entry is discuss the substantive points Dominic Cummings makes with regard to education, genetics, heritability and IQ.

IQ as a measurement of Cognitive Ability

On page 68 of  Some Thoughts on Education and Political Priorities Cummings suggests that if you ask the question of people what proportion of kids with an IQ of X could master integration, and then goes on to say that research on children with different cognitive ability should be done.

IQ, of course, isn't a measure of cognitive ability. If one wanted to do this research then measuring actual cognitive ability (in mathematics) would be the thing to do, not measuring IQ. One reason for this is that IQ tests are made up of several different abilities, another is that the nature of standardisation of the IQ tests obscures the underlying cognitive abilities.

It may be that the people that Cummings has asked this question are just being polite by saying that no empirical research has been done, rather than pointing out to him the stupidity of the question.

IQ not the only thing that affects academic performance

Cummings cites Lewis Terman on the idea that IQ is not the only thing that affect academic performance. This is of course true, but given that IQ was developed to predict educational attainment I tend to see this as a proof of failure of concept rather than anything else.

The extremely limited capacity of short term working memory (p72)

In talking about the way that basic research is being ignored in education Cummings talks about the extremely limited capacity of short term working memory being a reason why information may not be retained from a lecture. I often use short term memory as a great example of the difference between academic and lay understandings of psychology concepts. Short term memory refers to the few seconds after being exposed to, say a list of words, and the capacity of short term memory is normally seen as 7 plus or minus two things.

Working memory is a different topic it is a theoretical framework that refers to structures and processes used for temporarily storing and manipulating information.

So short term memory doesn't include memory of what happened over the last hour.
And short term working memory appears to be a conflation of two separate things.

On page 72 Cummings begins his confused ramblings about genetics.

The heritability of educational outcomes

Cummings cites with approval some work by Plomin on using heritability statistics on things such as phonics tests (70% heritability), reading and maths tests (at ages 7,9 and 12) as showing 60-70% heritability and GCSE as showing approximately 60% heritability.

Cummings is careful to talk about heritability as being about the amount of variance as due to genetic factors, but, like Plomin, seems to have a willfully ingenuous view on what, if anything, heritability statistics might reveal.

Like that other greatly misused statistic regression there appears to a tendency to conflate the assumptions of the test with what the test can tell us.

With regression one of the assumptions of a regression model is that the predictor variables cause the dependent variable; however it is often used as if the regression equation reveals whether or not a predictor variable (in a battery of tests) is causal.

With heritability there are a whole bunch of assumptions that need to be met before the statistic can be used.

The heritability statistic was created by Ronald Fisher who is also responsible for creating the ANOVA statistic so beloved in social sciences. A lot of Fisher's work related to problem particularly in crop breeding, where it could be know with a high degree of certainty the precise environment crops were exposed to, and a good idea of what we would now call the genetic background of the crops.

While there is no need to know which gene (or genes) are responsible for any trait that may be open to heritability analysis if it is assumed there is more than one gene they need to act upon each other in an additive way, rather than interacting with each other.

The other assumption is that 'the environment' acts in an additive way rather than interacting with the genetic component.

These assumptions seem far fetched when it comes to IQ tests. They are stretched beyond any credible breaking point when it comes to testing educational outcomes.

I have no doubt that genetics and environment play an important role in educational outcomes, but I am sceptical that we can reduce the complexities of their interactions to a single number. Even if we can then heritability is not that number for any trait where there is interaction between genes or interaction between genes and the environment.

Why study Conceptual and Historical Issues in Psychology

I expect that this will be covered either in the editorial or one of the earlier articles.

The position is the one that Dai Jones has developed writing in our books, and that I teach to in the current year three module.

Even if Psychology were a natural science there would still be good reasons for studying historical and conceptual issues.

• Interest - historical gossip, who did what to whom and how it affects the academic discipline.
• Understanding - Why did theory (x) being and why did it drop out of fashion.
• Learning past lessons - There are lots of examples of poor psychology, how can we avoid that in future,
• Critical understanding - Trying to understand psychology in its social context.

Different reasons for studying history need different types of historical writing.

However if psychology is not a natural science, because of the reflexivity issue (what psychologists say about human psychology directly and indirectly changes human psychology) then it becomes more important to study historical and conceptual issues.

There are also a bunch of reasons to do with current psychology and a need to understand how things have reached this position.

• In 2013 the Clinical Division of the BPS published a position paper highly critical of DSM V. (http://dcp.bps.org.uk/dcp/the_dcp/news/dcp-position-statement-on-classification.cfm)
• In the UK professional psychology is overseen by the Health Professions Council
• What are the relationships between psychology and health professions, how might these impact upon psychology?

There is an historical precedent, with the way that psychotherapists lobbied to become part of the medical profession in the 1910s, this had several impacts on the discipline and on the practice of psychotherapy in the USA.

• Upon formation of the Coalition the current government set up the 'Nudge Unit' using insights from psychology to change people's behaviour without recourse to legislation.
• Seligman the founder of positive psychology has met the PM. Positive psychology techniques have been used by the Nudge Unit.
• What networks does psychology as a discipline tie into, how might this affect the discipline?

Psychology has long tried to have influence with government, and some of things psychologists do seem to tie into this, but also governments (mis)use psychological findings a process which needs to be critically interogated.

• The American Psychological Association excuses psychologists working with the CIA and US military from normal ethical processes. "Enhanced interrogations" normally have a psychologist present.
• How have ethical codes developed and why are they important?
• What is the relationship between psychology and the military? 

Psychology has long had relationships with the military, from the Great War forwards. The military often have protected budgets and can be a good source of research funding.

• Cohen (1992) suggests Psychology studies are chronically underpowered.
• Straightforward replications are hardly ever published.
• Fraud in published psychology papers threatens our attempts to be taken seriously (http://www.guardian.co.uk/science/2012/sep/13/scientific-research-fraud-bad-practice)
• The credibility of psychology has also been undermined by a recent US supreme court ruling. (http://www.psychologytoday.com/blog/media-spotlight/201304/can-video-games-cause-violence
• Why are we so bad at quantitative methods when part of the way that, some, define psychology as a science is by the methods being used. 

I argue that part of the reason for this is the mechanical way that methods in psychology are understood, arguably a lingering affect of methodological behaviourism.

Historical understanding helps to open up these questions, and in addition helps to tie psychology back into its philosophical background.

Emebedding Conceptual and Historical issues in the Psychology curriculum

Opportunity for a possible paper, just about to send an abstract off for this, after chance to review and revise it for a special issue of of History and Philosophy of Psychology, coming out of some workshops on teaching CHiP Dai and I have contributed to over the last two years.

As it seems to help my process of writing I intend to blog about the paper as I write it.

Here is the first draft of the abstract.

For over a decade the CHiPs coverage at the University of Gloucestershire was isolated in three stand alone modules. The optional year two module recruited poorly, losing out to traditional applied modules, and although the compulsory year three module always had enthusiastic support from a minority of students, module evaluations revealed a group of students who felt that the coverage came too late in the degree. Spurred by University wide changes following the new fees regime in England the whole degree was redesigned for the 2012 academic year. A commitment to teach CHiPs across the whole of the first year provision was established. Integrating conceptual and historical issues into first term research methods teaching, in particular, has led to the opportunity to consider how psychological knowledge production works, rather than teaching research methods as a set of procedures to follow. In addition combining this with material on critical thinking has enabled us to move beyond seeing critical thinking as merely the application of scientific methods to everyday life and towards helping our students be critical about the ways that psychology operates in society. Initial student evaluation of this material has been much more positive than for the stand alone modules.

Any comments welcome. The abstract needs to be submitted by 30 September.

Pre-registration and Psychology publications

A couple of months ago the Guardian published this article on pre-registration of (mainly) psychology studies. This was followed by this article arguing against pre-registration.

This second article made me despair.

So I am going to look at it point by point after a bit of an introduction.

While there are a range of methods used in psychology the discipline has taken to a possible over reliance on null hypothesis significance testing (NHST).

There are conditions where NHST is great, especially when effect size estimations are also used, clinical trials, and in psychology when you are testing for an effect where you have a good idea of what size of effect matters outwith the laboratory, and you have a good model of that process to test in the laboratory. If NHST is the right way to understand data then there are a set of rigorous procedures that need to be followed else the results are likely to be inaccurate.

Now despite the fact that every psychology degree includes extensive research methods training it does appear that psychologists generally don't really understand what NHST is for, possibly because we use some fairly inaccurate short hand to try to introduce the concept to students.

We tend to say things like, probability that the effect is non-random, or that the effect is real.

However the NHST is not telling us the probability, it is telling us if an effect in a particular study will apply to the population that the sample was drawn from.

Obviously not all research is suited for NHST, I do qualitative research and historical research, neither of which use NHST, but also if you do not have a representative sample then NHST are not the thing you should be using. There are also arguments for using Bayesian statistics more generally, but that is not the argument I am making here. All quotes hereafter are from the Times Higher article.

Limiting more speculative aspects of data interpretation risks making papers more one-dimensional in perspective.

One of the issues is that these 'speculative' aspects of data interpretation tend to be done with NHST after looking at the data, that is not speculative it is wrong. Where exploratory data analysis is done it should clearly be labelled as such, and reporting significance levels shouldn't happen.

..commitment to publish with the journal concerned would curtail researchers’ freedom to choose the most appropriate forum for their work after they have considered the results.

I am not sure what this means, the meaning I can put on it is that if you find something unexpected, by doing statistical tests ad hoc you may want to publish these somewhere. Well you can but as you have broken the fundamental rules of NHST then that is fraudulent unless you clearly mark it as exploratory data analysis. The correct thing to do in this case is to design a study to test the potential phenomena, and pre-register that study.

With no results to go on, reviewers would be more likely than ever to rely on reputation, which would count against junior scientists.

 Most journals claim their review process is blind. Good to see proof that it actually isn't. I have long suspected this, even without names one tends to know what colleagues are working on in the same field. It might be important to work out how we can do genuine blind peer reviewing else our discipline is a laughing stock.

In addition, the requirement to refine studies and their interpretation prior to data collection would prevent us from learning from our mistakes along the way.

This tends to lead to data peaking and changing stimulus materials, instructions and even measures as one goes along, all of which invalidates NHST. The correct thing to do is of course to learn the lessons and run a fresh study.

Moreover, in my fields (cognitive neuroscience and psychology), a significant proportion of studies would simply be impossible to run on a pre-registration model because many are not designed simply to test hypotheses.

Well yes, so in those studies NHST results are not being erroneously reported, no problem with that at all and that is a good point. Psychology needs to look carefully at what it is doing and what science actually might be. However it does on

...many of the participant populations introduce significant sources of complexity and noise

 And these participant populations are not ones that can be used to make statements about the general population, so why exactly would NHST be used here.

So in summary, lots of reasons not to use NHST, however, we want to use them badly and fear pre-registration might stop that being done.

Why Psychology cannot be a science

At some point in the future when the science of psychology is well established ideas like this will be taught to undergraduates in the discipline so they can have a good laugh at foolish people who died before they were born, a bit like we do at the moment with Freud, Watson, Eysenck and so forth.

According to the standard, textbook, histories of the discipline psychology overcame the objections of Kant in establishing psychology as a discipline. Kant argued that an empirical science of psychology was impossible for two reasons. The first that psychological phenomena were not open to quantification, and so the only quantity that psychologists would be able to measure was time (either the time psychological phenomena persisted, or the time gaps between psychological phenomena). The second that psychological phenomena were subjective and would not be open to objective measurement.

Now there was a lot more going on than this, and there is a nice overview here with suggestions for further reading. I think one could plausibly argue that psychology has not been successful in meeting the objections of Kant, and one can see the impact of trying to overcome the objections in the way the discipline has changed across time. Those are not the arguments I am going to make.

The argument I am going to make is based on the work of Kurt Danziger, especially the sort of work he does here, and which is also associated with the work of Ian Hacking.

Danziger makes two points, one of which is common to all disciplines, the other of which is only common to a subset of disciplines.

The first point is, that areas of knowledge begin with a vocabulary that comes from the language of the host society, and which contains assumptions, some of which may remain unexamined for some time because they seem natural. In all disciplines the vocabulary of the area of knowledge changes across time, a specialist vocabulary comes into existence, and this specialist vocabulary may feed back to the language of the host societies. In natural sciences, to some extent, as evidence is gathered the nature of concepts and their associated vocabularies may change because of that process. However, the changing vocabulary does not affect the objects being studied; although they may change our understanding of those objects and regardless of language as knowledge increases peoples may gain technologies that can affect the objects of study.

In disciplines like psychology (other examples would include aspects of psychiatry, sociology and economics) things are different, because the description of the object of study can change the object of study.

Danziger in his 1997 book Naming the Mind makes the case for how this has happened for the concepts of intelligence, motivation and personality, attitudes, behaviour and learning; and variables. His more recent book, Marking the Mind makes a similar case for memory.

So what does this mean. I am going to use my usual teaching examples.

Natural Science

Across time our understanding of dinosaurs has changed. During the nineteenth century our understanding of dinosaurs, based on fossil evidence, was often of great lumbering creatures which inevitably became extinct. In the early twenty first century our understanding of dinosaurs is of a complex variety of animals, each superbly suited to their own ecological niche, some of which became extinct through extreme events, some of which through selection pressures evolved into different species, like, for example, chickens.

Neither of these understandings of dinosaurs affected how dinosaurs understood themselves.

Now of course the objection to this example is that dinosaurs (although not their descendants) are no longer with us, so a second example, I normally use Ether theory or Germ theory depending on how I feel.

Across time our understanding of infection and diseases has changed.

Prior to the nineteenth century the predominant theory of disease transmission was miasma theory:

The miasmatic position was that diseases were the product of environmental factors such as contaminated water, foul air, and poor hygienic conditions. Such infection was not passed between individuals but would affect individuals within the locale that gave rise to such vapors. It was identifiable by its foul smell. (wiki)

While there was earlier work leading up to it, in the nineteenth century germ theory developed, with evidence suggesting that for some infectious diseases microorganisms are the cause.

As germ theory developed and came to be accepted, how we reacted to the possibility of infection through germs changed our behaviour, and we developed technologies to ameliorate the possibility of infections.

However diseases did not change because our understanding of them changed, from being contained in Miasmas to being contained in microorganisms.


Disciplines like Psychology

Bystander inaction. People are less likely to act to help someone if they see themselves as one of many witnessing whatever it is that needs help.

There are two general exceptions to this, people who believe they have the professional skills to help, and people who have been thoroughly taught about the bystander inaction effect.

The study of psychology alters the psychology of people. Psychological research alters how people think about themselves and affects their behaviour.

It is not just psychology where this happens, there are other disciplines with similar looping effects. Belief that a certain level of government debt is harmful has led to austerity, which has because it is such a ridiculous mistake, led to government debt growing. Economic theory affects economies.

For a more fun example watch Hysteria.

Does it matter?

Yes and no. There is no point in trying to 'prove' psychology is a science by doing research that gets more and more obtuse.

There is a point in trying to make our measurement tools and theories the best we can so we can best explain what is going on now, without worrying about the universality of findings.

Letting go of a narrow obsession with a limited notion of what psychology should be like because it is a science is I think a good thing. And ironically it might lead to us being better at collecting data, testing hypotheses and building theories.

Brains!

 

Journal of Serendipitous and Unexpected Results, 2010. 1(1):1-5

Unless you are a dualist who believes that consciousness exists apart from the body it will come as no shock that brains are a necessary part of human (and salmon) psychology.

However, there are some issues around what we ought to, and ought not to, say on the basis of techniques such as fMRI (Functional magnetic resonance imaging)

The first point is this is a graph, not a direct image of the brain.

The technique, as explained in the Wiki article, is to compare blood flow across time.

In the case of the graph above the dead salmon was exposed to pictures of human individuals in social situations with a specified emotional valence, either socially inclusive or socially exclusive. The salmon was asked to determine which emotion the individual  in the photo must have been experiencing.

The brain activation graph shows the difference in activation, as detected by fMRI when the dead salmon was exposed to the two types of picture.

Except of course the salmon was dead, the researchers bought it at a supermarket, and unless we have thoroughly misunderstood the difference between dead and alive there was no blood flow.

Which leads to the second point, the need for adjustment for multiple comparisons.

The anomalous graph is caused because there is no (or inadequate) adjustment made to the null hypothesis statistical tests used to 'detect' the difference given the (large) number of multiple comparisons being made.

This issue is so well documented at this point that any researcher who publishes research based on fMRI data who has failed to do appropriate adjustments for multiple comparisons ought to be seen as fraudulent. Although that may not have been as apparent in the early years of this century, and there is a technical argument about the correct way of doing these adjustments for multiple comparison.

So there we have the basics, however seductive it is to think we are looking at something akin to an X-ray of brain activation we are actually looking at a graph. Beware of any research which does not robustly adjust for multiple comparisons.

Just because it appears in brain imaging it doesn't mean it is just a biological phenomena

In the next post I will address some of the issues around the philosophy of what I am discussing, and thereafter will post something more concrete about the relationships between brains, bodies and the environment for human psychology. For right now hold on to the idea (unless you are a substance dualist) that anything that happens in the mind also happens in the brain.

So if I have learnt to fear dentistry, which I have, then that fear of dentists is materially represented in my body and brain. That is not the same as saying that my fear of dentistry is caused by my biology.

This blog post by Neurocritic illustrates the problems of inadequate reporting of research to make it sound like once a psychological issue can be 'brain scanned' it is biological and open to straightforward intervention with drug therapies. However, it is not only the reporting that is at fault here, it also appears to be a default position of at least some psychology and neuroscience authors that the 'brain scan' (graph of difference in blood flow) shows a biological reality for psychological phenomena.

Beware of the dichotomy that biological is real, not biological is false (or constructed).

This is a particular issue in sex and gender research, where if a graph of difference in blood flow can be   shown for men and women then that particular difference is seen as biological in origin. Despite the possibility that the differences between men and women, for a particular difference, can be at least as plausibly be explained by social process.

Just showing a graph of differences in blood flow in the brains of two categories of people does not equal an explanation of the differences between those categories of people.

Psychology, methods and science, why worry?

Over the last couple of months I have been toying with ideas around Psychology, methods and science.

I suspect that what I want to say will be a bit bigger than one blog post, so I intend to do a series of posts.

This installment is about why I think we have a problem.

As someone with an interest in conceptual and historical issues I tend to range a bit more freely over the psychological literature than specialists in a particular area. Some of the problems which are acknowledged in one aspect of psychology seem to be mirrored in other areas of the discipline. If the problems are even more widespread than that it seems to implicate the whole discipline.

We do not routinely test for similarity instead of difference

One of the outstanding issues in Psychology is something I first became aware of with the Psychology of Sex and Gender.

While most of us, most of the time, act and talk as if there are psychological differences between men and women the scientific evidence for those differences is at best contested, at worst none existent.

What tends to happen is that an area of possible, psychological, difference is opened up by a statistically significant difference being published. Over time a bunch of other people do similar (although not identical) work. After some more time a meta analysis is carried out on the findings, and the results of the meta analysis suggest that the original difference is small and inconsistent. This may lead us to believe that the genders are more (psychologically) similar than different.

However, while I strongly believe that psychological differences that we can detect using our current methods are small and inconsistent, that is not the same thing as saying the men and women are psychologically similar. We simply have not been testing for similarity, and lack of evidence for difference is not the same as evidence in favour of similarity.

Psychology studies are routinely under powered

In the paper Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49, 997-1003. Cohen argues that there are a number of problems with null hypothesis significance testing. One of the problems highlighted is that psychology studies are routinely under powered.

This suggested teaching exercise might help you grasp what this means, and may act as a useful teaching resource if you ever have to teach statistics.

One trouble with having a routine of under powered studies (studies with too few participants for a given effect size) is that we get studies published followed by a number of failures to replicate. After some time someone will do a meta analysis and suggest that the original finding is smaller and more inconsistent than originally thought. All this strikes me as a huge waste of resources.

We do not routinely try to replicate results

While the recent controversies around Bem may have put this into focus in psychology it is very difficult to get a straightforward replication published. Psychologists work around this by doing "conceptual replications" replicating the idea, but not the study. However, this is surely missing the point. When researchers do meta analyses of psychology studies they try to include unpublished studies (normally Ph.D. dissertations) that did attempt replication. Unfortunately we do not know about the missing studies because of the next problem.

We do not routinely register studies before they are run

We simply don't know how big the 'file drawer' of studies is, people occasionally try to make a guess but without routinely registering studies before they run we cannot know so fairly important information about whether an apparently interesting finding has been extensively tested.   

We do not know the relationship between our studies and the world outside of the laboratory

In order to carry out science like investigations of human psychology it is necessary to simplify phenomena so that we can test for some of the things that might affect human psychology, while concentrating on a sub-set of things that might affect human psychology. What psychologists mean when they talk about 'controlling unwanted variables'.

One of the questions is can we do that in a meaningful way. I suspect we can but all too often we do not try to. There is a famous talk given by the physicist Richard Feynman on Cargo Cult Science.

All too often we don't know what our laboratory studies, with humans, actually mean because we haven't done the basic work (Feynman also makes some points about replication, which means the problem of lack of replications stretches back at least to the mid twentieth century).

Most psychometric tests have not been tested for predictive validity

Which is ultimately the same problem as above, but for personality tests.

We routinely use poor sampling methods

Psychologists routinely use undergraduate students, self selecting samples, and samples drawn purely from clinical populations. That they/we then go on to make universal claims about human psychology from these samples is just bizarre.

Next episode being careful with 'brain scans'