Is the government’s PREVENT ‘science’ really peer-reviewed?

By Asim Qureshi Last month, CAGE published a report critiquing the flawed “science” underpinning the British government’s PREVENT strategy. Among the many criticisms in the report, some attention was given to the nature of the peer review process. At the time of writing, we did not envisage the importance of this single issue, particularly in light of the more substantive points we were making. In conjunction with the launch of the report, the Guardian published a news article detailing how 140 academics had written an open letter to the government asking for the flawed science to be made available to academics and psychologists, in order for it to be scrutinised. In that very article, the Home Office responded to the report by stating that the study, used as a basis for PREVENT , had been through a “peer review” process. It is important to mention that had we known that the response from the Home Office would have been to clarify such a small issue in our report, instead of the graver issues contained within it, we would have made greater effort to address this point. As a matter of transparency, it is important that we address this issue since it has created further questions that the Home Office must now answer. CAGE’s report found the scientific basis of PREVENT was flawed. This led to government officials and PREVENT officers making a number of remarks to newspapers and on social media on two aspects of the study. Read more: A serious blow for PREVENT: reactions to the CAGE report Upon reading their remarks, it became clear – as mentioned by a host of academics and psychologists – that either the government did not understand the process by which the PREVENT strategy went through a “scientific” check, or its spokespeople were being intentionally disingenuous in their explanations. For clarity, we will first summarise the actual pieces of work the government claimed were peer reviewed. The research that the Home Office described was an ERG22+ study, which was conducted by two psychologists, Monica Lloyd and Christopher Dean, during their time at the National Offender Management Service (NOMS) in 2010. The original classified study as described by the Home Office was: HM Government – National Offender Management Service, Extremism Risk Guidelines: ERG 22+ Structured Professional Guidelines for Assessing Risk of Extremist Offending, London, Ministry of Justice Publications In 2015 a summary of the study was published as a journal article in a niche psychology journal under the American Psychological Association (APA): Lloyd M and Dean C, The Development of Structured Guidelines for Assessing Risk in Extremist Offenders, Journal of Threat Assessment and Management Finally, during the summer of 2016, an article was published by Monica Lloyd in a magazine of the British Psychological Society (BPS): Lloyd M, Structured guidelines for assessing risk in extremist offenders, Journal of Assessment and Development Matters, vol.8 No.2 In a quote to the Guardian in response to our report, the Home Office claimed that based on the publications of their study in the “Journal of Threat Assessment and Management” and “Assessment and Development Matters”, their study was not only carried out to “meticulous academic guidelines”, but it was also “peer reviewed” beforehand and then published in “ two publicly available academic journals”. However, the Home Office did not provide any evidence to substantiate their claims. In fact, when CAGE contacted the BPS recently to determine on what grounds the article was published in one of their “academic journals” (Assessment and Development Matters), they stated that this so-called “academic journal”, was in fact only a magazine distributed to psychology practitioners – there was no peer review process involved at all. Moving on to the point of the study having been through a credible peer review process, it is important to point out that for any study to have been put through any such process, it would need to have been extensively critiqued to assess the quality of the article by experts in their field. The only way this could take place is for the experts to have been given the original study including the original dataset from which the ERG22+ tool was developed, as well as the methodology involving how the data was interpreted. Based on the admissions provided in the journal article itself, the authors of the study admit that it did not go through any such process. Furthermore, the academic community has asked for proof of this process which the Home Office said took place without having provided any evidence to substantiate their claim. This proof has so far not been forthcoming. For that reason, the journal article written by Monica Lloyd in 2016 for the BPS should be discounted as a source of credible evidence as to the scientific rigour of the study, as it is a four page summary of the article that was produced in the American Psychology Association journal. Key among the quotes in Lloyd’s article for the BPS in 2016 was her recognition that, “Reliability and validity studies are in progress.” [1] Any claim that a peer review process of the science of the ERG22+ took place or was possible in the BPS article, would not stand the test of any academic journal. In fact, the BPS should issue a statement clarifying whether or not that is the case. If they confirm that their publishing process is equated to an acceptable peer review of the entire science, then it will suggest that the BPS is providing fraudulent academic cover for the government’s original study. As mentioned earlier, Home Office responded to criticisms of the government’s study by claiming that the ERG22+ had been verified by two peer-reviewed studies: “The guidance that is used was based on a peer-reviewed study, carried out to meticulous academic guidelines and published in two publicly available academic journals.” On the other hand, PREVENT officials, normally active on social media, largely remained silent with the exception of senior PREVENT officials Abu Ahmed and William Baldet. While the initial statements claimed that the peer review was conducted by the journals themselves, perhaps a more accurate version was presented by the Security Minister Ben Wallace: “The guidance that is used was based on a peer-reviewed study, carried out to meticulous academic guidelines and published in two publicly available academic journals.”  The statement by the Security Minister appears to indicate that the study was peer reviewed, and then due to that process of peer review, acceptably published in two journals. The BPS journal, we have already explained, could not possibly provide a credible peer review of the science, as it is a three page summary that provides none of the data – to claim it would have been possible for two blind reviewers who had never had access to the data, to carry out a sufficient review of the ‘science’ is less than credible. Watch: Seminar with Professor David Miller and Asim Qurehsi on the ERG22+ In a similar vein, the longer journal piece in the American Psychological Association Journal of Threat Assessment and Management, presents a similar problem. While it is a somewhat longer discussion of the original study, the possibility for an adequate blind review to have taken place would simply have not been possible without the reviewers having been given access to the original ‘study’ – and this is not indicated anywhere in the journal article or the editorial introduction to the journal. Where the real point of contention lies, is between the view of Monica Lloyd, the author of the original study, and the government. According to Lloyd, the ‘study’ was, “not an academic piece of work”, but rather was an internal report for practitioners. This is key, as the author is claiming that the ‘study’ the government has relied on is in no way an actual ‘scientific study’. How could it then be possible: 1. For that internal report to be blind peer reviewed to an acceptable standard to consider it legitimate ‘science’? 2. For the government to place the ‘study’ on a statutory footing? These questions sit at the very heart of the government’s claims of validity – again, something the authors stress has not been attained. For Lloyd, both in 2015: “The current lack of demonstrated reliability and validity remains the main limitation of the ERG at this time.” [2] And again in 2016, “Reliability and validity studies are in progress…” [3] These two statements from the author of the ‘study’ itself, present the clearest indication that whatever ‘science’ the government has relied upon to form the 22+ factors of vulnerability factors to indicate ‘radicalisation’, it is based on nothing more than a non-peer reviewed internal report. Furthermore, the peer review process is the backbone of science. For the government to toy with this term in order to manipulate it for its own ends – as seems to be the indication here – threatens the integrity of scientific study. This should be challenged vigorously by practitioners and civil society members alike. As such, continuing with such a policy as the basis for referring individuals to security services and making court decisions that have a lasting impact on individuals and communities, is a matter of grave concern, not least of all considering the unethical way in which it has been implemented. [1] Lloyd M, Structured guidelines for assessing risk in extremist offenders, Journal of Assessment and Development Matters, vol.8 No.2, p.18 [2] Lloyd M and Dean C, The Development of Structured Guidelines for Assessing Risk in Extremist Offenders, Journal of Threat Assessment and Management, 2015, p.50 [3] Lloyd M, Structured guidelines for assessing risk in extremist offenders, Journal of Assessment and Development Matters, vol.8 No.2, p.18 (CC image courtesy of Maina Kiai on Flickr)