Fact Check on ‘Latest evidence casts further doubt on the effectiveness of headspace’ (Kisely & Looi, 2022)

13/09/2022

Authors:
Professor Debra Rickwood, Chief Scientific Advisor
Nic Telford, Head of Evaluation and Monitoring

September 2022

Background

The Medical Journal of Australia has published a perspectives paper by Kisely and Looi, entitled ‘Latest evidence casts further doubt on the effectiveness of headspace’ (Kisely & Looi, 2022).

We contend that the paper shows evidence of bias, containing seven misleading or misrepresentative statements, four irrelevant and inappropriate comparisons, and four errors— a total of 15 very evident demonstrations of bias in this brief 1500-word paper. Table 1 lists these misleading claims and each instance and counters them with the facts. These are presented in the order that they appear in the publication.

Table 1. Fact Check

Erroneous/misleading claim	Fact
1. Misrepresentation of information “Of the $2.3 billion allocated in the 2020–21 federal budget on new mental health expenditure, $765.8 million was directed to headspace, as well as related services for older age groups in Head to Health centres.2”	The amount reported ($765.8M) adds together separate funding for headspace ($278.6M) and Head to Health ($487.2M), which are separate services and government initiatives. Head to Health is not a youth mental health initiative. The headspace funding comprises only 8.25% of the $2.3B.
2. Error of omission “This expansion has occurred despite limited evidence of effectiveness, with many studies using either process measures or uncontrolled satisfaction surveys.”	No citations are provided for this assertion, which fails to acknowledge the three independent quantitative evaluations commissioned by the Australian Government (Hilferty et al., 2015; Muir et al., 2009), a comparison outcome study with better access (Bassilios et al., 2017), two major reports of client outcomes (Rickwood et al., 2022; Rickwood et al., 2015), and a longitudinal follow-up study (Telford & Rickwood, 2019).
3. Misrepresentation of results “A small uncontrolled study from September 2012 to July 2017 (n = 77) did report statistically significant benefits on the K10. However, the proportion of participants in a lower clinical stage of illness at the end of treatment was not statistically significant.8”	This is a very small study of a brief intervention conducted by graduate students in a few headspace centres (Schley et al., 2019). Kisely and Looi dig deeply into the results of this paper to find the one non-significant result, failing to report the main findings which show a medium effect sized change, and that significantly more young people scored in the non-clinical range after the intervention.
4. Misleading claim “These findings are reflected in a more recent study of 1510 young people that found only 35% had good functional outcomes after 24 months in the period 2008–2018.9”	This study is not from a headspace client group (and does not claim to be), as clarified in a letter to the editor and accompanying the paper (Rickwood et al., 2022).
5. Error “The available evidence is further limited by the relatively small sample sizes, the highest of which was 2222 participants from 2008–09, less than 0.5% of the 700 000 young people seen by headspace since 2006.3”	The statement uses a numerator from a one-year period when there were only 10 headspace centres initially opened (2008-09) and a denominator from a 16-year period (2006-2021) to derive the very small (and non-sensical) percentage reported. This statement is also incorrect as there are outcome data publicly reported for 15,496 (Rickwood et al., 2015) and 58,233 (Rickwood et al., 2022) young people. These studies are of census samples of all young people attending headspace for mental health care in their specified time frames.
6. Misrepresentation of quote “For instance, in 2014, the National Mental Health Commission noted that the creation of headspace mental health centres was conducted without sufficient consultation, leading to “duplication of, and competition with, other community, private and state government services”.12”	The full reference for this quote is, “While headspace has been enthusiastically received by many communities, there also have been clear indications that problems can arise due to a lack of local planning, along with duplication of, and competition with, other community, private and state government services in some regions.” (National Mental Health Commission, 2014 p.82). Further, this is a 9-year-old report; the National Mental Health Commission publishes a report every year (see https://www.mentalhealthcommission.gov.au/monitoring-and-reporting/national-reports)
7. Misrepresentation of results “This is despite the fact that conventional headspace centres may be ill-equipped to manage this population as just under 40% of clients have no recorded diagnosis.14”	This percentage seems to come from the following statement, “Twenty-nine per cent of clients were estimated by clinicians to have full-threshold, remission, or serious and ongoing disorder, yet almost a third of these had no actual clinical diagnosis recorded at presentation, and a further 6.7% were reported as diagnosis not yet assessed” (Rickwood et al., 2018); a misrepresentation of these results.
8. Error “should return to evidence-based care delivered in public and private settings”	No evidence is provided to support the claim that ‘evidence-based’ care was the previous norm. headspace was set up to address the well-acknowledged failings of the previous system for youth mental health.
9. Misrepresentation of evidence “based on existing early episode psychosis services, already linked to headspace and said to have a strong evidence base.13 However, in comparison with treatment as usual, the advantages of these services are restricted to the initial 2 years with limited evidence beyond that.15“	Cites a non-peer reviewed report rather than the 22 peer reviewed studies of early intervention in psychosis that show strong cost effectiveness (McGorry & Mei, 2020).
10. Irrelevant comparison “Most information on effectiveness concerns inpatient settings and data from the Australian Institute of Health and Welfare and the Private Psychiatric Hospitals Data Reporting and Analysis Service suggest high effect sizes that easily exceed those of headspace.17”	Data on inpatient services are not comparable with headspace, which is community-based primary care and focused on mild-moderate presentations within an early intervention context –very different to an inpatient population. The ‘high effect sizes’ noted here come from analysis of all adult inpatients in private psychiatric hospitals (Looi et al., 2022), using measures that cannot be sensibly compared to headspace clients in any possible way.
11. Misleading inference “At a minimum, headspace services should be situated and funded within state/territory publicly provided mental health services with established systems for clinical governance and subject to a similar level of accountability”	This inference that headspace centres do not have established systems for clinical governance is incorrect. As part of the headspace trade mark licencing process, all headspace centres must demonstrate sound clinical governance, which is the responsibility of the lead agency and commissioning PHN. Clinical governance is checked regularly through the headspace Model Integrity Framework and the headspace Trade Mark Licence Deed process.
12. Irrelevant comparator “In particular, the routine use of standardised clinician and patient reported outcome measures, such as the Health of the Nation Outcome Scales,18 would enable comparisons to existing public and private sector services”	The HoNOS is an inappropriate measure for the headspace client group (Jacobs, 2009). The HoNOS is a clinician-rated tool developed by the United Kingdom Royal College of Psychiatrist’s Research Unit to measure change in the health and social functioning of adults experiencing severe mental illness in secondary care mental health services (James et al., 2018). headspace includes the K10+ as a mandated outcome measure for PHN-funded mental health services, as well as the SOFAS, and also developed a purpose-built routine outcome measure (MyLifeTracker) for its centre services (Kwan et al., 2018; Rickwood et al., 2022) after undertaking a systematic review of outcome measures for youth mental health and finding few suitable measures (Kwan & Rickwood, 2015).
13. Irrelevant comparator and error of omission “In addition to the integration of existing centres into mainstream mental health services, alternatives to headspace include the diversion of proposed future funding to public sector mental health services where there is better evidence for improved outcomes, or to private psychological and psychiatric services through the Better Access program.20”	No evidence or citations are provided to support claims made in relation to the effectiveness of community-based public mental health services for young people. The reference given here is to the Australian Government’s website, which lists its initiatives. Furthermore, a comparative study of young people accessing headspace and Better Access showed comparable outcomes (Bassilios et al., 2017) is not cited.
14. Irrelevant comparator and error of omission “In 2019–2020, baseline psychiatric symptoms for new outpatient referrals as measured by the Health of the Nation Outcome Scales were 11.1 (standard deviation, 6.2) for 15–24-year- olds compared with 7.4 (standard deviation, 5.8) at discharge to no further care.21 This approximates an effect size of 0.62, which represents a change of medium magnitude”	Again, the HoNOS is not an appropriate outcome measure for the headspace client service group; they do not have severe mental illness. Further, it is expected that those with more severe scores on outcome measures demonstrate greater change. For example, a meta-analysis of depression treatment reported posttreatment effect sizes were much greater for high-severity patients (d = 0.63) than for low-severity patients (d = 0.22) (Driessen et al., 2010).
15. Error of omission “It is time for a national discussion about youth mental health in the context of declining mental health outcomes in young people, rather than doing the same thing repeatedly and expecting different results.”	There is robust ongoing national discussion of youth mental health in Australia, and has been for some time. Australia is acknowledged as a global leader in youth mental health research and treatment development (see https://www.internationalaffairs.org.au/australianoutlook/australia-must-take-the-lead-in-global-youth-mental-health/). headspace, itself, provides a platform for national discussion and this is a major part of its remit. Australia also has many more major initiatives all focused on understanding and improving youth mental health and providing platforms for discussion (eg, ReachOut, Orygen Youth Health, Youth Beyondblue, BlackDog). The International Association of Youth Mental Health was set up and is auspiced from Australia to facilitate discussion around youth mental health, internationally. Its 6th biennial conference commences September 29th this year.

References

Bassilios, B., Telford, N., Rickwood, D., Spittal, M.J., & Pirkis, J. (2017). Complementary primary mental health programs for young people in Australia: Access to Allied Psychological Services (ATAPS) and headspace. International Journal of Mental Health Systems, 11(1), 19. https://doi.org/10.1186/s13033-017-0125-7

Driessen, E., Cuijpers, P., Hollon, S. D., & Dekker, J. J. (2010). Does pretreatment severity moderate the efficacy of psychological treatment of adult outpatient depression? A meta-analysis. Journal of Consulting and Clinical Psychology, 78(5), 668. https://doi.org/10.1037/a0020570

Hilferty, F., Cassells, R., Muir, K., Duncan, A., Christensen, D., Mitrou, F., Gao, G., Mavisakalyan, A., Hafekost, K., Tarverdi, Y., Nguyen, H., Wingrove, C., & Katz, I. (2015). Is headspace making a difference to young people’s lives? Final Report of the independent evaluation of the headspace program. https://headspace.org.au/assets/Uploads/Evaluation-of-headspace-program.pdf

Jacobs, R. (2009). Investigating patient outcome measures in mental health. CHE Research Paper 48. https://www.york.ac.uk/che/pdf/rp48.pdf

James, M., Painter, J., Buckingham, B., & Stewart, M. W. (2018). A review and update of the Health of the Nation Outcome Scales (HoNOS). BJPsych Bull, 42(2), 63-68. https://doi.org/10.1192/bjb.2017.17

Kisely, S., & Looi, J.C.L. (2022). Latest evidence casts further doubt on the effectiveness of headspace. Medical Journal of Australia. https://doi.org/10.5694/mja2.51700

Kwan, B., & Rickwood, D. J. (2015). A systematic review of mental health outcome measures for young people aged 12 to 25 years. BMC Psychiatry, 15(1). https://doi.org/10.1186/s12888-015-0664-x

Kwan, B., Rickwood, D. J., & Telford, N. R. (2018). Development and validation of MyLifeTracker: A routine outcome measure for youth mental health. Psychology Research and Behavior Management, 11. https://doi.org/10.2147/PRBM.S152342

Looi, J. C., Bastiampillai, T., Pring, W., Kisely, S.R., & Allison, S. (2022). Private psychiatric hospital care in Australia: a descriptive analysis of casemix and outcomes. Australasian Psychiatry, 30, 174-178. https://doi.org/https://doi.org/10.1177/1039856220924323

McGorry, P. D., & Mei, C. (2020). Why do psychiatrists doubt the value of early intervention? The power of illusion. Australasian Psychiatry, 28(3), 331-334. https://doi.org/10.1177/1039856220924323

Muir, K., Powell, A., Patulny, R., Flaxman, S., McDermott, S., Oprea, I., Gendera, S., Vespignani, J., Sitek, T., Abello, D., & Katz, I. (2009). Independent evaluation of headspace: the National Youth Mental health Foundation. Social Policy Research Centre, University of New South Wales. https://headspace.org.au/assets/Uploads/Corporate/Publications-and-research/final-independent-evaluation-of-headspace-report.pdf

National Mental Health Commission. (2014). Contributing lives, thriving communities: Report of the National Review of Mental Health Programmes and Services. Vol 1. https://www.mentalhealthcommission.gov.au/monitoring-and-reporting/national-reports

Rickwood, D., McEachran, J., Saw, A., Telford, N., Trethowan, J., & McGorry, P. (2022). Sixteen years of innovation in youth mental healthcare in Australia: Outcomes for young people attending headspace centre services. medRxiv. https://doi.org/10.1101/2022.08.24.22279102

Rickwood, D., Paraskakis, M., Quin, D., Hobbs, N., Ryall, V., Trethowan, J., & McGorry, P. (2018). Australia’s innovation in youth mental health care – the headspace centre model. Early Intervention in Psychiatry, 13, 159-166. https://doi.org/https://doi.org/10.1111/eip.12740

Rickwood, D., Trethowan, J., & Carruthers, A. (2022). Letter to the Editor: Social and occupational outcomes for young people who access early intervention services. Medical Journal of Australia. https://doi.org/https://doi.org/10.5694/mja2.51426

Rickwood, D. J., Telford, N. R., Mazzer, K., Parker, A. G., Tanti, C. P., & McGorry, P. D. (2015). Changes in psychological distress and psychosocial functioning for young people accessing headspace centres for mental health problems. Medical Journal of Australia, 202(10), 573-542. https://doi.org/10.5694/mja14.01696

Schley, C., Pace, N., Mann, R., McKenzie, C., McRoberts, A., & Parker, A. (2019). The headspace Brief Interventions Clinic: Increasing timely access to effective treatments for young people with early signs of mental health problems. Early Intervention in Psychiatry, 13, 1073-1082. https://doi.org/10.1111/eip.12729

Telford, N., & Rickwood, D. (2019). headspace centre young person follow up study. https://headspace.org.au/assets/headspace-centre-young-person-follow-up-study-Sept-2019.PDF