Recovery Colleges Characterisation and Testing in England (RECOLLECT): rationale and protocol | BMC Psychiatry

Design

The RECOLLECT programme comprises six work packages structured to relate to the RECOLLECT multilevel change model. Work Package 1 is a national survey of managers of Recovery Colleges in England, supplemented by qualitative interviews with a subsample of survey respondents. Work Package 2 is a prospective pre-post study of Recovery College fidelity and service user student outcomes and service use. Work Package 3 is a prospective controlled study comparing outcomes and service use among newly registered Recovery College service user students with a matched service user control group. Work Package 4 is a retrospective controlled study comparing routine clinical data for service user students with a matched control group. Work package 5 is a series of organisational case studies. Work Package 6 will mobilise the resulting knowledge from Work Packages 1 to 5. The inter-relationships and objectives addressed in Work Package 1 to 5 are shown in Fig. 3.

Recovery college sample

The inclusion criteria for Recovery Colleges are:

1. 1.

   A focus on supporting personal recovery in mental health or substance use;

2. 2.

   An aspiration to use co-production, defined as individuals with lived experience working with staff or subject experts to design and deliver all aspects of the Recovery College;

3. 3.

   An aspiration to use adult learning approaches, in which students and trainers collaborate and learn from each other by sharing experiences, knowledge, and skills;

4. 4.

   Location in England.

These criteria were agreed by a Recovery College stakeholder group, consisting of academics, LEAP members involved with Recovery Colleges, Recovery College Managers, and members of ImROC, which supports recovery college development as part of its partnership work on recovery and wellbeing practice. The rationale for the criteria was based on: the common purpose of Recovery Colleges to support personal recovery, the foundational importance of co-production and an adult learning approach identified through analysis of the fidelity measure data [33] and the geographical scope of the programme. The organisation did not need to be called a ‘Recovery College’ to be included, since some use other names such as discovery centre, empowerment college or recovery academy.

Project management

The RECOLLECT programme is jointly led by King’s College London and University of Nottingham, and researchers will be employed at both sites. The applicant team comprises 14 members with expertise in all aspects of the programme. The Programme Steering Committee (PSC) comprises six independent members to ensure governance and independent oversight. The International Advisory Board (IAB) will maximise the scientific quality of the programme, fair and impartial evaluation of Recovery Colleges, and contribute to dissemination. The 11 IAB members have multidisciplinary and cross-cultural expertise in all programme aspects. The Lived Experience Advisory Panel (LEAP) comprises 10 individuals who either have lived experience of mental health problems or are carers, some but not all of whom have used Recovery Colleges. To ensure that expertise by experience informs all aspects of the programme, the LEAP will provide input into choice of measures, support design decision-making, inform documentation for participants, interview participants, be involved in interpretation of data, and assist with publicising findings. Lived experience is also represented on the Programme Steering Committee, International Advisory Board and in the research team.

Work packages 1 to 6 are now described.

Work package 1: National survey and qualitative interviews

Work package 1 will address Objectives 1 (Characterise) and 2 (Costs).

Design and rationale

Work Package 1 consisted of a national survey of all Recovery College managers [36] and qualitative interviews with a subsample of managers. In addition to addressing Objectives 1 and 2, results will be used to design the sampling frame for Work Packages 2 and 5, inform the economic evaluations in Work packages 3 and 4, and describe the impact of the COVID-19 pandemic on Recovery Colleges. In relation to this latter aim, the COVID-19 pandemic began after the RECOLLECT Programme was funded. When the programme began in December 2020, the long-term impact of the pandemic on Recovery Colleges was not clear, but the short-term impact was high, as many Recovery Colleges temporarily closed and all moved from primarily or exclusively face-to-face provision to entirely online provision during periods of national lock-down. The design of Work Package 1 was therefore supplemented with semi-structured qualitative interviews with Recovery College managers, to understand the impact of COVID-19 as a contextual influence on Recovery Colleges, including its impact on fidelity.

Sample

The survey sample is managers of Recovery Colleges in England, per the above criteria. The semi-structured interviews will be conducted with a sub-group of survey participants. It is anticipated that saturation [37] will be achieved through no more than 40 interviews.

Measures

The RECOLLECT Fidelity Measure is a 12-item Recovery College manager-rated assessment [33]. It comprises seven non-modifiable components (Valuing equality; Learning; Tailored to the student; Co-production of the Recovery College; Social connectedness; Community focus; and Commitment to recovery) and five modifiable components (Available to all; Location; Distinctiveness of course content; Strengths-based; and Progressive). Each non-modifiable component is scored on a three-point scale from 0 (low fidelity) to 2. Each modifiable component is rated as either Type 1 or Type 2 (e.g. eligibility restricted to those affiliated with the host organisation, vs available to all), and as each modifiable component is independent, Recovery Colleges can score Type 1 on some components and Type 2 on others. No summary score is produced, since the relationship between fidelity and outcome is not empirically established.

Procedures

To maximise coverage, Recovery Colleges were identified using multiple approaches: web searches containing key words related to Recovery Colleges and geographical area, advertising of the survey through social media and Recovery College networks, involvement of study partner organisations such as ImROC, and snowball sampling. All Recovery College managers were then invited to participate by email and provided with an information sheet. After giving informed consent and responding to screening questions to ensure their organisation meets the inclusion criteria, managers of those screening as eligible were asked to complete the online survey.

The survey covered characteristics of the Recovery College structure, courses, students, fidelity, and costs. Attention was paid to capturing both current and pre-pandemic data where relevant, and free-text elaboration of responses was allowed. The survey was implemented online on Qualtrics, with an equivalent Microsoft Excel version as an alternative where participants cannot access Qualtrics due to organisational firewalls. The survey was piloted with a small convenience sample of Recovery College managers and refined.

The semi-structured qualitative interview topic guide covers history, organisational context, COVID-related changes and future plans. Interviews were conducted by RECOLLECT researchers, LEAP members who will receive qualitative interviewer training and supervision, and RECOLLECT research team members with expertise in qualitative methodologies and organisational sociology. Recovery College managers who participate in the national survey were asked to participate in a (maximum) one-hour semi-structured interview either online or by telephone. After the consent process, the interview was conducted, with notes kept by the interviewer on their impressions and key discussion points. Interviews will be audio recorded, transcribed and anonymised.

Analysis

Recovery College characteristics were investigated using descriptive analysis exploring RECOLLECT Fidelity Measure scores and numerical characteristics such as student numbers. Hierarchical cluster analysis was used to create a national Recovery College typology. Content analysis of any free text survey responses and collaborative data analysis [34] using framework analysis [38, 39] of interview transcripts will be employed to explore organisational characteristics and pandemic impact. To assess costs, the expenses associated with Recovery Colleges will be identified and quantified, including salaries of those employed by the Recovery College and by other organisations who release employees as sessional trainers, building and IT costs, curriculum development and delivery, and time spent by stakeholders such as service users, informal carers and mental health staff who attend as students [26]. Costs will then be apportioned to specific sectors as appropriate with input and feedback from stakeholders.

Work package 2: Prospective pre-post study exploring Recovery College impact on student mental health outcomes

Work package 2 will address Objectives 3 (Student outcomes), 4 (Fidelity, mechanisms and outcomes) and 9 (change model).

Design and rationale

Work Package 2 will consist of a prospective pre-post study at 25 Recovery Colleges selected from the survey sample to provide the full range of Fidelity Measure item scores. The change model and input from LEAP informed the development of items to assess the mechanisms of action and the choice of outcome measures, i.e. the primary outcome of quality of life and secondary outcome measures covering intrapersonal changes and changes to the students’ life including changes in service use. In addition, we will use the generic health status measure, EQ-5D-5L [40] to allow comparison of the samples in Work Package 2 and Work Package 3, and hence assess generalisability of the intervention group in Work Package 3.

Sample

To address Objective 3 we have conservatively assumed a small but clinically significant within-person effect size of 0.2 [41]. With standard deviation (SD) of 5 based on the mean difference of total score of the primary outcome measure, the Manchester Short Assessment (MANSA) [42] between baseline and follow up at 12 months with 95% power and 5% significance, assuming an intra-class correlation coefficient (ICC) of 0.02 [43, 44], an individual autocorrelation of MANSA score over time of 0.5 [45] and within-cluster standard deviation (SD) of 5, we require a sample size of 500. With an estimate of 20% attrition rate by 12 months, we will recruit 625 participants, for an analysable sample of 500. We will recruit 625 newly registering students (25 per college) who currently use secondary mental health services. Exclusion criteria will be insufficient knowledge of English to complete the outcome measures.

Measures

The primary outcome is quality of life as measured by the MANSA [46]. This comprises 12 items rated using a seven-point Likert scale, with higher scores indicating higher quality of life. The MANSA has been shown to have good psychometric properties, including internal consistency and validity [42, 47]. Secondary outcomes comprise: empowerment in relation to mental health service use, measured by the Mental Health Confidence Scale [48]; personal recovery measured by Brief INSPIRE-O [49]; social inclusion measured by the Social Inclusion Scale/Measure (Social inclusion) [50]; resilience measured by the Brief Resilience Scale [51]; hope as measured by the Herth Hope Index [52]; Social network size using the Lubben Social Network Scale 6 [53]; and mental wellbeing, using the Warwick-Edinburgh Mental Wellbeing Scale (Short version) [54].

Student service use will be measured using a Service Use Questionnaire designed for the project and will capture both NHS and non-NHS service use. The SUQ covers contacts with health and social care, use of leisure facilities, employment, income and out of pocket expenses. We will include the number and types of RC courses attended. To provide a societal perspective, it will be tailored to cover work status and absence, sources of income, domestic activities, informal care, medication use, out-of-pocket expenses, and consumption of non-medical resources. Where consent is given by students, mental health clinical records will be accessed to supplement findings.

The EQ-5D-5L is a self-rated measure of health-related quality of life [40]. The scale measures quality of life on a 5-component scale including mobility, self-care, usual activities, pain/discomfort, and anxiety/depression. Psychometrically, it is less prone to ceiling effects than the EQ-5D-3L across a range of diseases [55, 56]. The National Institute for Health and Care Excellence (NICE) requires use of a 5L to 3L crosswalk algorithm for their reference-case to estimate QALYs [57].

Procedures

All new students will be made aware of the research by Recovery College staff as they register. Those who express interest in participating will be contacted by a member of the research team who will provide the participant information sheet either remotely or in person. Participation will be optional, and students can choose to register with the college and not take part in the RECOLLECT Study. Eligible students who provide informed consent will then complete baseline measures, sociodemographic data and provide follow-up contact information for re-administration of all measures at 4, 8, and 12 months. Clinical and service use data will be collected from electronic medical records given participant consent for access; this consent will be optional for those consenting to participate in measure completion.

Analysis

To address Objective 3 (Student outcomes), we will compare primary outcome (total MANSA score) at baseline, 4, 8, and 12 months follow-up adjusted for baseline total MANSA score and accounting for hierarchical clustering of students at the level of Recovery College. The model will include the MANSA score as the outcome variable, baseline MANSA score and socio-demographic characteristics as explanatory variables at each time point. Specifically, we will fit a three-level random intercept linear regression model, considering observations from baseline, 4, 8 and 12 months at level 1, individuals at level 2 and Recovery Colleges at level 3. Secondary outcomes will be assessed similarly to the primary outcomes, using generalized linear mixed models (logistic, linear and Poisson depending on the distribution of our outcome variable).

The relationship between Fidelity Measure scores at the Recovery College level with primary and secondary outcomes will also be assessed using generalised linear mixed models. Where significant associations emerge, theoretical mechanisms of action will be added to models to examine potential mediating effects.

If indicated, we will refine the RECOLLECT multilevel change model by removing outcomes where there is no evidence of change at any time point.

Work package 3: Prospective controlled study exploring the effectiveness and cost effectiveness of Recovery Colleges

Work package 3 will address Objectives 5 (Effectiveness) and 6 (Cost-effectiveness).

Design and rationale

Work package 3 will employ a prospective controlled study design and include seven of the Recovery Colleges from Work Package 2, with which it will run in parallel. A non-randomised design will be employed due to the impossibility of individual level randomisation; most mental health service users have access to a Recovery College, even if not available in their Trust. The seven Recovery Colleges were selected for being linked to an NHS Trust which has access to a Clinical Record Interactive Search (CRIS) system, as this will be used to identify the control group.

Sample

Intervention group eligibility will be as for Work Package 2; eligible controls will not be attending a Recovery College at baseline, or have previously attended a Recovery College. Work Package 3 will require approximately 48 students at each of the seven Recovery Colleges to form an intervention group of 330 participants; and 660 participants in total with the control group. This sample size is needed for 90% power to detect an effect size of 0.2 on the MANSA [46] using a two-sided test for the difference between two independent means and assuming α = 0.05. To meet this recruitment target, students at the seven Recovery College sites for Work Package 3 will be oversampled (i.e. 48 instead of 25 per site) relative to the others in Work Package 2.

Settings

The seven participating Recovery Colleges linked to an NHS Trust with access to a CRIS system cover areas within five of the nine regions of England defined by the Office for National Statistics: South West, South East, East and North West England and London.

Measures

Work Package 3 will use the same primary and secondary outcome measures as for Work Package 2.

Procedures

Work Package 3 will run alongside Work Package 2. At the seven Recovery College sites involved in Work Package 3, participants will be asked if they would like to take part in Work Package 3 by consenting to data linkage using CRIS. Those that consent will comprise the intervention group for Work Package 3. This is outlined in Fig. 4 below. To identify the control group, propensity score matching [58] will be employed using the CRIS system at each site. This is a statistical matching technique that attempts to create a comparable control group for the Recovery College intervention group by accounting for the covariates, such as demographics and clinical history, that predict outcomes. Reverse anonymisation will be undertaken by CRIS staff and potential control participants will be contacted by NHS Trust staff working with CRIS staff to find out if participants are interested in hearing more about the study. Those that are will be provided with information sheets and will need to give informed consent to take part. The intervention group will complete those measures used in Work Package 2 and listed above at the same time points of baseline, 4, 8 and 12 months. The control group will complete the primary outcome measure and a restricted set of those measures used in Work Package 2 and the Work Package 3 intervention group, to reduce burden. Participants in Work Packages 2 and 3 will be compensated for their time completing assessments.

Analysis

We will compare the total MANSA score for service users who attended a Recovery College (intervention group) with those who did not (control group) at each follow up, controlling for baseline scores and hierarchical clustering of patients in Recovery Colleges. Specifically, we will fit a three-level random intercept linear regression model, considering observations from baseline, 4, 8 and 12 months at level 1, individuals at level 2 and Recovery Colleges at level 3. The model will include the MANSA scores at 4, 8 and 12 months as the outcome variables with intervention group, baseline MANSA score and sociodemographic characteristics, including the matching factors. A time x intervention interaction will be included to allow the effect to differ at each time point. Those secondary outcomes collected in both the intervention and control groups will be assessed similarly to the primary outcomes, using generalized linear mixed model (linear, logistic and Poisson depending on the distribution of the outcome.

Incremental economic analyses will be conducted from both a healthcare and a societal perspective with analyses conducted at 4, 8 and 12-month follow-up, controlling for baseline. Uncertainty will be addressed by generating cost-effectiveness planes from bootstrapped resamples. Cost effectiveness acceptability curves will examine the probability that the intervention is cost-effective for different decision-maker willingness-to-pay thresholds, as well as generating net benefit estimates. Sensitivity analysis will investigate key factors including the impact of costs of different Recovery College delivery models and the use of EQ-5D-5L/EQ-5D-3L cross-walk methods to generate utility [57]. The benefits of Recovery Colleges are likely to continue after the 12-month primary endpoint. Therefore, we will also carry out an economic evaluation informed by modelling to estimate longer-term benefits and NHS/Personal and Social Service and wider costs, using current design quality standards [59]. This requires an expert consensus panel to support model design, made up of Lived Experience Advisory Panel, Recovery College staff and the applicants, and representatives of Trusts and other affiliated organisations.

Work package 4: Retrospective controlled study of the effectiveness of Recovery Colleges

Work Package addresses Objective 7 (clinical data).

Design and rationale

Work Package 4 will employ a retrospective controlled study using routine clinical data and measures, with an 18 month follow up period. In comparison to Work Package 3, this study will benefit from a larger sample size and absence of selection bias in the intervention group, to produce results that will be more generalisable across NHS sites.

Measures

The Health of the Nation Outcomes Scale (HoNOS) [60] is a 12-item measure completed by clinical staff at all NHS mental health Trusts which covers a patient’s health and social functioning. Each item is scored on a five-point Likert scale. It has moderately high construct validity, fair to moderate test–retest reliability and good predictive validity [61].

DIALOG is an 11-item measure exploring health status and quality of life used routinely in many NHS mental health trusts, and based on the MANSA [62]. There are eight items on quality of life and three items on treatment, each scored on a seven-point Likert scale. Internal consistency is 0.71 for the eight quality of life measures and 0.57 for the three treatment items [62].

The Clinical Outcomes in Routine Evaluation – Outcome Measure (CORE-OM) [63] is a 34 item self-report measure covering subjective wellbeing, symptoms, function and risk used in many NHS mental health trusts. Each item is scored on a four-point Likert-scale. Internal consistency and test–retest reliability are good (0.75–0.95) [64].

Procedures

Five of the seven Recovery College sites used in Work Package 3 will be employed for this study (see Fig. 5), based on availability of student data and affiliation with an NHS Trust which has a CRIS. At each site service user student and CRIS data will be linked.

The intervention group will comprise all service user students who first registered at the Recovery College over 18 months prior to the selected audit date (the date of the data extraction from the Recovery College). Based on student numbers and the proportion who are service users we estimate 400 per site (total n = 2,000).

Individual service users in the control group will be matched with the students using a similar propensity scoring approach to that described in Work Package 3, so that as far as possible the only difference between groups will be Recovery College attendance. For the intervention group Recovery College registration will be used as the baseline date, and the last HoNOS and other available outcome scores recorded before Recovery College registration will be used as the baseline data. Eligible controls will also have a HoNOS score at the equivalent timepoint. The follow up HoNOS and other outcome scores to be used will be those entered as an update at the closest time point to the 18 month follow up point. For both groups, data on service use will be extracted for the 18 months after the baseline, including number of psychiatric inpatient days; number of contacts with staff in community mental health teams and specialist services such as psychology and outpatient services; and medications including doses prescribed.

Analysis

We will fit a two-level random intercept linear regression model, considering observations from individuals at level 1 and Recovery Colleges at level 2. We will compare HONOS scores of intervention group individuals at 18 months with 18-month HoNOS for matched control individuals adjusting for baseline HONOS and sociodemographic characteristics. The model will include 18 months HoNOS score as the outcome variable, and the baseline HoNOS score and sociodemographic characteristics as explanatory variables.

The relationship between Recovery College use, and mental health service use will be examined using both resource consumption parameters (categorical data) and derived individual-level costs (continuous data). Individual level costs will be calculated by combining service use data with appropriate unit costs, using the methods outlined in Work Package 1. Costs will be compared for the intervention group compared with matched controls using a bootstrapped regression model, adjusting for HoNOS and other outcome measures as available.

Work package 5: Organisational case studies of Recovery Colleges

Work Package 5 addresses Objectives 8 (Organisational) and 9 (Change model).

Design and rationale

Work Package 5 will consist of organisational case studies. It will include six Recovery Colleges from Work Package 2, to be purposefully sampled to capture variation in Fidelity Measure score, length of time in operation, size, and funding arrangements. Within the constraints of maximum variation sampling and the need to manage burden on each Recovery College, we will prioritise Recovery Colleges which were also sites for Work Packages 3 and 4; this will enhance finalisation of the multilevel change model given the evidence from these sites on effectiveness and cost effectiveness.

Procedures

Qualitative data collection using in-depth interviews and focus groups will be undertaken with stakeholders: Recovery College Trust managers; peer and non-peer trainers; and commissioners, and past and present students, including but not limited to service user students. For interviews of professionals, snowball sampling will be used, starting with the Recovery College manager. Students will be recruited via Recovery College staff, as well as posters, emails and flyers. Topic guides tailored to the participant will explore: how contextual and organisational factors influence fidelity; the mechanisms proposed in the RECOLLECT change model leading to outcomes and how fidelity influences these mechanisms. Content will include: 1) manager and other staff views on what influences fidelity items; 2) student experiences of attending, using probes based on Fidelity Measure items; 3) observations by those in contact with Trust services (service users, carers and staff) of any changes they have observed in the Trust since the Recovery College opened which they think may be influenced by the presence of the college, and 4) stakeholder views about the impact of the Recovery College on the local community.

Analysis

Collaborative data analysis involving LEAP members again using Framework analysis [38, 39] will be employed to facilitate analysis within and between groups of participants, using NVivo [65]. Context will be explored by learning from experiences of managing organisational interfaces and commissioning arrangements. Cross-case comparison will elaborate the key contextual and organisational factors influencing the delivery of service and explaining variation in access, fidelity and outcome. Analysis will focus on the relationships posited in the multilevel change model.

Work package 6: Knowledge mobilisation

Design and rationale

RECOLLECT will produce: 1) An evidence-based understanding of what a Recovery College is, captured by the RECOLLECT Fidelity Measure; 2) A contextualised understanding of how Recovery Colleges work, expressed through the RECOLLECT multilevel change model; and 3) the first rigorous evidence about the effectiveness and cost effectiveness of Recovery Colleges.

The following key stakeholders will be informed of the findings: Recovery College students, mental health service staff, policy makers, commissioners, researchers, research participants, and the general public. Dissemination will include tailored messaging through various networks (professional, service user and carer, research networks including in low- and middle-income countries, ImROC, the Recovery College Network), international Recovery College community of practice online including through social media and the study website (researchintorecovery.com/recollect), and through a knowledge mobilisation event at the end of the programme. Outputs will include papers in high impact academic and practitioner journals, posters, conferences, reports and briefings.