1Champion KE, et al. BMJ Open 2018;8:e020433. doi:10.1136/bmjopen-2017-020433

Open access

Cluster randomised controlled trial of an online intervention to prevent ecstasy and new psychoactive substance use among adolescents: final results and implications for implementation

Katrina E Champion, Nicola Clare Newton, Lexine Stapinski, Maree Teesson

To cite: Champion KE, Newton NC, Stapinski L, et al. Cluster randomised controlled trial of an online intervention to prevent ecstasy and new psychoactive substance use among adolescents: final results and implications for implementation. BMJ Open 2018;8:e020433. doi:10.1136/bmjopen-2017-020433

► Prepublication history and additional material for this paper are available online. To view these files, please visit the journal online (http:// dx. doi. org/ 10. 1136/ bmjopen- 2017- 020433).

Received 2 November 2017Revised 20 August 2018Accepted 10 October 2018

NHMRC Centre of Research Excellence in Mental Health and Substance Use, National Drug and Alcohol Research Centre, UNSW Sydney, Sydney, New South Wales, Australia

Correspondence toDr Katrina E Champion; k. [email protected] unsw. edu. au

Research

© Author(s) (or their employer(s)) 2018. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

AbstrACt Objectives To evaluate the effectiveness of the online Climate Schools: Ecstasy and Emerging Drugs module over 2 years, and examine the impact of intervention dose on outcomes.Design Cluster randomised controlled trial.setting Secondary schools in Australia.Participants 1126 students (aged 14.9 years) from 11 schools.Intervention Five schools were randomly allocated to the four-lesson internet-based Climate Schools: Ecstasy and Emerging Drugs module. This universal intervention uses cartoon storylines to deliver harm-minimisation information about ecstasy and new psychoactive substances (NPS). It was delivered during health education classes over 4 weeks. Six schools were randomised to the control group (health education as usual). Participants were not blinded to intervention allocation.Outcomes measures Students completed self-report surveys at baseline, post-test, 6, 12 and 24 months post-baseline. Intentions to use ecstasy and NPS (including synthetic cannabis and synthetic stimulants), knowledge about ecstasy and NPS and lifetime use of ecstasy and NPS were assessed. This paper reports the results at 24 months post-baseline.Analysis Mixed effects regressions were conducted to analyse intervention effects from baseline to 24 months. Post hoc analyses using Inverse Probability of Treatment Weighting compared controls with students who: i) completed all four lessons (‘full dose’) and ii) partially completed the intervention (≤three lessons, ‘incomplete dose’).results Primary analyses found that controls were significantly more likely to intend on using synthetic cannabis compared with intervention group students (OR=3.56, p=0.01). Results from the weighted analyses indicated that controls reported significantly lower knowledge about ecstasy (p=0.001) and NPS (p=0.04) compared with the full-dose group. No significant differences were observed between the incomplete dose and control groups.Conclusions The online intervention was effective in modifying students’ intentions to use synthetic cannabis up to 24 months; however, this study highlights the importance of delivering prevention programmes in full to maximise student outcomes.

trial registration number ACTRN12613000708752.

Ecstasy use among young Australians is relatively low, with 3.2% of adolescents aged 14–19 years reporting past year use in 2016.1 However, for those adolescents who do use ecstasy, the potential for harm is considerable, with adverse effects ranging from confusion and dehydration to impaired brain functioning and mental health prob-lems.2 3 In addition to concerns about estab-lished illicit drugs such as ecstasy, there are new and emerging issues in the global land-scape of drug use that also need to be consid-ered. One such challenge is the growth in the number and availability of new psychoactive substances (NPS) and the severity of conse-quences associated with their use.4

NPS are substances that are specifically designed to imitate the effects of established illicit drugs and to circumvent existing drug laws. Although NPS are typically sold online or in ‘headshops’ as legal alternatives to

strengths and limitations of this study

► A key strength of this study is its robust and longi-tudinal design. A cluster randomised controlled trial was conducted in 11 Australian schools (n=1126 students) over a 2-year period.

► Sophisticated statistical analyses were employed. Multilevel regressions were estimated to analyse differential group change in outcomes from baseline to the 24-month follow-up.

► This study used novel online intervention comple-tion data to examine the effect of dosage on student outcomes.

► Attrition at the 24-month follow-up was higher than anticipated, despite comprehensive efforts to assess students who were absent from school on the day of a survey.

2 Champion KE, et al. BMJ Open 2018;8:e020433. doi:10.1136/bmjopen-2017-020433

Open access

traditional illicit drugs,5 laws about NPS are complex and frequently changing and many countries have introduced legislation to ban them. According to the European Monitoring Centre for Drugs and Drug Addiction,6 NPS continue to pose a significant public health problem, with 66 new NPS detected in 2016. Two of the most common types of NPS are synthetic cannabinoids (synthetic cannabis), which are designed to mimic the effects of cannabis and are sold under brand names such as Spice and Kronic, and synthetic cathinones (synthetic stimulants, ‘bath salts’), stimulant-like substances intended to imitate the psychoactive effects of ecstasy and amphetamine.7 8 Approximately 1% of Australians aged 14–19 years have used synthetic cannabis in their lifetime and the preva-lence is <1% for stimulant-type NPS use.1 Although these rates are not high, the severity of adverse consequences associated with their use is concerning. Very little is known about the long-term effects of synthetic canna-binoid use; however, short-term effects include extreme agitation, anxiety and psychosis9 10 and serious physical side effects, including seizures, cardiac arrests and even death have been documented.11 Similarly, commonly reported side effects of synthetic stimulant use include aggression, heart palpitations and hallucinations.12–14 NPS use is also associated with other adverse conse-quences including alcohol, tobacco and cannabis use, truancy and risky sexual behaviour.15–17 Furthermore, misleading marketing, including use of colourful pack-aging and inaccurate terms such as ‘legal highs’, is likely to influence young people’s perceptions of the risks asso-ciated with NPS use, leading them to incorrectly believe that NPS are low-risk and safe to use.18 Therefore, it is clear that NPS use and related harms among youth need to be prevented. In response to this, the universal Climate Schools: Ecstasy and Emerging Drugs module was developed as the first online school-based prevention programme to simultaneously target ecstasy and NPS use among adolescents.19

The Climate Schools: Ecstasy and Emerging Drugs module is a four-lesson online universal programme delivered to year 10 students (aged 15 years) during health educa-tion classes. It is based on the effective Climate Schools prevention framework, which has previously been shown to reduce alcohol consumption, binge drinking and cannabis use, increase knowledge and modify inten-tions.20–24 Underpinned by social influence theory and a harm-minimisation approach, the Climate Schools: Ecstasy and Emerging Drugs module uses interactive online cartoon storylines to provide information about ecstasy and NPS use, challenge students’ perceptions of peer drug use and build resistance skills. Full details of the intervention content has been published elsewhere.19 The short-term efficacy of the module has been demonstrated previ-ously25 via a cluster randomised controlled trial (RCT) with Australian students (n=11 schools, 1126 students, mean age=14.9 years). Results indicated that the interven-tion was effective in reducing intentions to use any NPS and synthetic cannabis (up to 12 months postbaseline)

and increasing knowledge about ecstasy (at post-test) and NPS (up to 6 months postbaseline). In the context of universal prevention for illicit substances, an indication of the efficacy of a programme is often reliant on measures of intentions and knowledge.26 This is especially true for low-prevalence illicit substances such as ecstasy and NPS. Informed by principles of social influence theory,27 28 a key component of the Climate Schools programme is the provision of information about the short-term negative consequences of ecstasy and NPS use. The assumption is that by providing students with accurate, immediately relevant and evidence-based information, they will be equipped with the knowledge to make informed decisions when they are exposed to drug use.29 Knowledge about substance use is a precursor to attitude and beliefs, which in turn are predictive of intentions to use substances.30 Furthermore, intentions to use alcohol,31 tobacco32 and illicit substances have been shown to predict subsequent initiation of use.33 Therefore, an examination of the long-term effectiveness of the Climate Schools intervention in increasing knowledge and modifying future intentions to use ecstasy and NPS is important. Long-term outcomes of the Climate Schools: Ecstasy and Emerging Drugs module have not yet been examined, nor has the impact of programme adherence on outcomes, an area of growing importance in the literature.

The importance of high-quality implementation on programme outcomes is now well recognised in the prevention science field.34–37 In school settings, achieving high-quality implementation is often challenging given the many obstacles that can impede programme delivery such as a lack of resources in terms of teachers, time and money,38 as well as the fact that teachers often make adaptations to programme content that under-mine efficacy.39 40 Although several different elements of implementation have been proposed, in the context of school-based prevention key factors to consider are adher-ence to the intervention by teachers and/or facilitators, quality of delivery, dosage of the intervention and student engagement.36 39 The internet-based nature of the Climate Schools programmes affords several advantages in terms of the degree of fidelity that can potentially be achieved. For example, the core content is preprogrammed online and completion is self-directed by students, thus delivery is not dependent on teacher training or skills, ensuring faithful delivery of the core programme components.41 42 The portable nature of the internet also means that students can complete missed lessons at home, or in a follow-up lesson, increasing their potential to receive the interven-tion in full.43 Related to this is the ability of online interven-tions, such as Climate Schools, to track and collect objective dosage data in real-time rather than relying on teacher self-report.44 Although previous studies have examined the impact of implementation of school-based substance use prevention programmes on student outcomes,36 45 the influence of implementation quality of online drug educa-tion prevention programmes has not yet been explored. Taking this all into account, the aims of the current paper

3Champion KE, et al. BMJ Open 2018;8:e020433. doi:10.1136/bmjopen-2017-020433

Open access

are twofold: to evaluate the effectiveness of the Climate Schools: Ecstasy and Emerging Drugs module over a 2-year period, and examine the impact of intervention dose (number of lessons received by students) on outcomes. We hypothesised that compared with the control group, the Climate Schools intervention group would:i. Report significantly greater knowledge about ecstasy

and NPS at 24 months;,ii. Be significantly less likely to intend on using ecstasy

and NPS at 24 months;iii. Be significantly less like to have ever used ecstasy and

NPS at 24 months.

MethODsDesign and procedureA cluster RCT was conducted in 11 secondary schools in Australia between 2014 and 2016. Schools were randomly allocated to the Climate Schools intervention or to a control group (health education as usual). Cluster randomisation was employed to avoid contam-ination of the control group by the intervention group through student communication. Blocked randomisa-tion was conducted by an independent researcher using the RALLOC function in Stata. Participants were not blinded to intervention allocation. Ninety indepen-dent schools (not run by the government or Catholic Church) in Sydney were contacted in 2013 and invited to participate. Schools that had previously collaborated with the researchers or that had expressed an interest in participating in future research were approached initially. School principals were sent a letter outlining the aims of the study and seeking their permission to conduct research with their students. Twelve schools agreed to take part and were randomised; however, one school withdrew from the study prior to baseline, as most students did not speak English as their first language and could not complete the intervention. Consent forms were sent home to parents of year 10 students (aged 15–16 years) at participating schools. Students with parental consent (88%) completed a confidential online self-report survey during class time at baseline, postintervention and 6, 12 and 24 months postbaseline. The final sample consisted of 1126 students (n=490 control, n=636 intervention) at baseline. Full details of the study protocol have been reported previ-ously46 and details of participant recruitment, randomi-sation and retention are provided in figure 1.

Patient and public involvementExtensive consultation was conducted with students, teachers and health professionals to inform the devel-opment of the Climate Schools: Ecstasy and Emerging Drugs module. Feedback was sought about the relevance and acceptability of the intervention, including language and aesthetics. Full details of the co-development process have been published previously.19 47 A summary of find-ings from the present study have been disseminated to

teachers and students at participating schools, and results have been presented at scientific meetings and published in peer-reviewed journals.25

sample size calculationsTo account for cluster randomisation, sample size calcu-lations were based on sample size requirements devel-oped by Heo and Leon48 to detect intervention by time interactions in longitudinal cluster RCTs. We estimated that 10 schools (5 in each group) were needed to detect a standardised between-group mean difference of 0.20 (equivalent to an OR of 1.44), with 80% power (p=0.05) in primary and secondary outcomes at the end of the trial (see protocol for full details).49

Intervention and control groupsThe Climate Schools: Ecstasy and Emerging Drugs moduleSchools allocated to the intervention group were asked to implement the Climate Schools: Ecstasy and Emerging Drugs module during health education classes in 2014. The programme consists of four 40 min lessons to be deliv-ered 1 week apart, over a 4-week period. Teachers were instructed to complete all four lessons of the programme, and as the intervention is preprogrammed online, lessons were designed to be delivered in order (ie, starting with lesson 1, then lessons 2, 3 and 4) and in full. The inter-vention aims to impart evidence-based information about ecstasy and NPS, correct overestimates of peer ecstasy and NPS use, teach refusal skills and address misconceptions. Full details of the lesson content have been described in detail elsewhere.19 25 46 Each lesson consists of a 20 min online cartoon component completed individually by students in a classroom setting, under teacher supervi-sion. The remaining 20 min consist of optional activities, including online worksheets and teacher-led discus-sions and role plays. Teachers were provided with four preplanned activities per lesson and were able to choose which activities to implement to best meet the needs of their class. The core component of the intervention is the online cartoon lessons, therefore it was not essential that teachers completed the same activities or same number of activities. Participants accessed the programme via the study website using secure login details. The researchers met with teachers at the start of the study to assist in the navigation of the website, reiterate the study aims and to distribute hard-copy teacher manuals. The teacher manuals contained implementation guidelines, links to the syllabus, lesson summaries (for teachers and students) and materials for implementation of the optional activ-ities (eg, suggested discussion points, worksheets and homework tasks). All the activities required minimal preparation by teachers and no ongoing training was required. This information was also available to teachers via the study website.

The control groupControl schools implemented health education classes as usual (which covers drug education) to their year 10

4 Champion KE, et al. BMJ Open 2018;8:e020433. doi:10.1136/bmjopen-2017-020433

Open access

students in 2014. Teachers at control schools were invited to provide information about the amount and format of any drug education they delivered to their students (see online supplementary file 1).

MeasuresDemographic data collected at the individual level included gender, age, country of birth and self-reported grades, which were scored categorically using the following

Figure 1 Consolidated Standards of Reporting Trials flow diagram of participant recruitment, allocation and assessment. 

5Champion KE, et al. BMJ Open 2018;8:e020433. doi:10.1136/bmjopen-2017-020433

Open access

response options: ‘90%–100%’, ‘80%–89%’, ‘70%–79%’, ‘60%–69%’, ‘59% or below’. The primary outcomes of interest are self-reported intentions to use ecstasy and any NPS, and secondary outcomes are intentions to use synthetic cannabis and synthetic stimulants, knowledge about ecstasy and NPS and lifetime use of ecstasy and NPS (including synthetic cannabis and synthetic stimu-lants). These findings have been reported previously at 12 months postbaseline,25 with the current paper focusing on 24-month outcomes.

Intentions to use ecstasy and any NPS, synthetic cannabis and synthetic stimulantsFour items, adapted from a previous prevention trial,50 were used to assess intentions. Students rated how likely they were to use ecstasy/any NPS/synthetic cannabis/synthetic stimulants in the next 6 months and responses were coded on a dichotomous scale ‘very unlikely, unlikely, unsure’=0 or ‘likely, very likely’=1.

Knowledge about ecstasy and NPSEcstasy knowledge was measured using a 15-item scale developed to reflect the intended content of the Climate Schools intervention. Items assessed knowledge about the prevalence of ecstasy use, risks and harm-minimis-ation information. For each statement, students were required to answer ‘true’, ‘false’ or ‘do not know’. Items were summed to provide a total continuous score, with higher scores indicating greater knowledge. A 5-item scale was used to assess NPS knowledge, including knowl-edge about associated harms and prevalence of use. Total continuous scores ranged from 0 to 5, with higher scores indicating greater NPS knowledge. The ecstasy knowl-edge scale demonstrated acceptable reliability (α=0.72), and the NPS scale had questionable reliability (α=0.67).

Ecstasy and NPS useLifetime ecstasy use was measured using a single dichoto-mous item (“Have you ever used ecstasy?” yes/no), adapted from the Australian National Drug Strategy Household Survey.51 Three equivalent items were used to assess use of ‘any NPS’, ‘synthetic cannabis (marijuana), for example, Spice, Kronic, K2’ and ‘synthetic stimulants’ (‘bath salts’).

Intervention implementationIn the present study, implementation fidelity was primarily assessed by the dose of the intervention received. Teachers at intervention schools were asked to complete a logbook indicating which lessons (eg, “Did your students complete the online Climate Schools cartoon component for lesson 1?”) and activities (“Which activities did your students complete?’) they completed with their class. Teachers were also asked to rate the level of engagement among students for each lesson (‘very disengaged’=0 to ‘very engaged’=5) and to complete a programme evaluation (fidelity and eval-uation summary available online). The use of objec-tive evidence to assess dosage, in addition to teacher self-report, is advocated in the literature.44 Thus, lesson completion data for the online cartoon component were

collected at the student and school level, and extracted via the study website. For each lesson, data were coded on a binary scale (0=‘did not complete online lesson’, 1=‘lesson completed in full’). To ensure the reliability of the online dosage data, teachers at participating schools were contacted via email and asked to retrospectively describe details of the intervention implementation. One teacher indicated that students at their school had completed the intervention as a group, rather than individually, due to limited computer resources, thus all students at that school were coded as completing all four lessons. A single binary measure was generated to represent overall dose, with completion of either one, two or three lessons representing an ‘incomplete dose’ and completion of all four lessons, a ‘full dose’.

statistical analysesThe present data were analysed in two phases. First, multilevel regressions were conducted to estimate the overall effectiveness of the Climate Schools intervention for primary and secondary outcomes (intentions, knowledge and lifetime use). Second, in light of the low adherence to the intervention (59% received a full dose), post hoc inverse probability of treatment weighting analyses were conducted to examine the effect of dose on outcomes. Full details of the analyses are described below.

Primary analysesTo analyse differential group change in outcomes at the 24-month follow-up, multilevel regressions including fixed and random effects were estimated in Stata V.14. Specification of a random intercept in these models take into account variation at the individual level, and random effects at the school level were estimated to account for clustering or potential lack of independence in data from students within the same school.52 Preliminary models were estimated to determine the most appropriate cova-riance structure for the linear regressions (knowledge), and the optimal number of integration points for each logistic regression (intentions and lifetime use).52 An unstructured covariance structure was selected and 15 integration points were used. A three-level regression model was estimated for each outcome, with repeated observations (level 1) nested within students (level 2), and students clustered within schools (level 3). As per our previous analyses of the 12-month data,25 for each outcome, gender and baseline academic grades were included in the model as covariates. Rates of synthetic stimulant use were too low for multilevel analyses to be conducted for this variable. Time at each assessment occa-sion was modelled using a categorical variable (coded 0–5, ie, baseline, post-test, 6, 12 and 24 months), and the intervention effect was assessed with a trial group×time interaction. These models use baseline measurements as the reference point to estimate participant-specific starting points and change over time, and allows us to examine change over time from baseline to 24 months. This approach is ideal for analysis of longitudinal and

6 Champion KE, et al. BMJ Open 2018;8:e020433. doi:10.1136/bmjopen-2017-020433

Open access

clustered data.53 Multilevel mixed effects models are recognised as an emerging, rigorous method of analysing trial data and has several advantages over conventional approaches, such as better treatment of missing data, accommodation of variance at the individual level, and flexible modelling of time effects and the within-subject covariance structure.52–54

Missing dataMissing data were handled using maximum likelihood (ML) estimation, based on the intention-to-treat prin-ciple, which includes all participants in the sample (n=1126). This approach is widely accepted and has been employed in previous studies applying multilevel models to substance use.55 ML is a highly efficient way of using all available information to estimate parameters rather than deleting cases with missing data.52 56 It is a superior method for handling missing data as all avail-able data are used to compensate for missing data for a particular student.57 ML is valid under the assumption that data are missing completely at random or missing at random (MAR; meaning missingness does not depend on unobserved response, given the observed covariates and outcomes in the model).54 In the present study, the assumption of MAR is plausible given the variables in the analysis models, including outcome measurements at

other time-points, which are used to estimate unobserved outcomes.

Examining the impact of intervention dose on outcomesIn view of the observed intervention adherence (59% of students received a full dose), post hoc inverse prob-ability of treatment weighting analyses were conducted to examine the effect of dose on intervention outcomes. Inverse probability weightings are based on propensity scores, defined as the probability of treatment assignment (or in the present study, of full exposure to the Climate Schools intervention), given participants’ observed base-line characteristics.58 Intervention exposure, or dose, is likely to be associated with participant characteristics (eg, truant or high-risk students may be more likely to receive a lower intervention dose), and thus propensity scores provide a way of balancing out these differences in baseline characteristics. In order to attain unbiased esti-mates of intervention effects at incomplete and full dose, analyses are adjusted using inverse probability weights to balance the distribution on measured baseline covariates between adherent and non-adherent participants.59 That is, comparisons are made between the incomplete dose group and the control group, and the full dose group and the control group, with the three weighted groups similar in all respects at baseline except for intervention

Table 1 Descriptive statistics for outcome variables over time and by group

Outcome GroupBaseline n=1126

Post-test n=729 6 months, n=874 12 months, n=717 24 months, n=684

Ecstasy intentions, % (n) CLIMATE 2.20 (14) 3.79 (10) 3.30 (15) 1.55 (6) 5.63 (22)

CONTROL 0.82 (4) 2.80 (12) 3.51 (14) 2.15 (7) 3.48 (10)

NPS intentions, % (n) CLIMATE 1.73 (11) 3.79 (10) 1.98 (9) 0.52 (2) 3.33 (13)

CONTROL 0.8 (4) 3.27 (14) 3.01 (12) 1.84 (6) 2.79 (8)

Synthetic cannabis intentions, % (n)

CLIMATE 6.92 (44) 7.20 (19) 6.39 (29) 6.74 (26) 7.42 (29)

CONTROL 4.49 (22) 6.78 (29) 8.27 (33) 7.06 (23) 8.71 (25)

Synthetic stimulants intentions, % (n)

CLIMATE 2.20 (14) 2.65 (7) 1.54 (7) 0.52 (2) 3.10 (12)

CONTROL 0.82 (4) 2.34 (10) 2.26 (9) 1.23 (4) 1.05 (3)

Ecstasy knowledge (M, SD)

CLIMATE 10.59 (2.53) 11.57 (3.61) 11.65 (2.77) 11.90 (2.36) 11.20 (3.48)

CONTROL 9.18 (2.87) 9.57 (3.31) 10.11 (3.25) 10.40 (3.13) 9.99 (3.57)

NPS knowledge (M, SD) CLIMATE 3.33 (1.32) 3.85 (1.49) 3.93 (1.24) 3.99 (1.13) 3.71 (1.44)

CONTROL 2.71 (1.48) 2.78 (1.48) 3.12 (1.51) 3.28 (1.45) 3.07 (1.50)

Ever used ecstasy, % (n) CLIMATE 2.50 (16) 4.10 (11) 3.07 (14) 2.14 (7) 7.36 (29)

CONTROL 1.23 (6) 1.63 (7) 1.25 (5) 2.07 (8) 2.79 (8)

Ever used NPS, % (n) CLIMATE 2.98 (19) 3.02 (8) 3.08 (14) 1.30 (5) 4.58 (18)

CONTROL 2.86 (14) 2.80 (12) 3.52 (14) 1.23 (4) 2.44 (7)

Ever used synthetic cannabis, % (n)

CLIMATE 2.35 (15) 2.63 (7) 1.57 (7) 1.30 (5) 4.31 (17)

CONTROL 2.45 (12) 2.33 (10) 2.05 (8) 0.93 (3) 1.38 (4)

Ever used synthetic stimulants, % (n)

CLIMATE 0.63 (4) 1.51 (4) 0.88 (4) 0.00 (0) 3.00 (12)

CONTROL 0.0 (0) 0.70 (3) 0.50 (2) 0.00 (0) 1.02 (3)

Total knowledge scores ranged from 0 to 5, with higher scores indicating greater knowledge.NPS, new psychoactive substances.

7Champion KE, et al. BMJ Open 2018;8:e020433. doi:10.1136/bmjopen-2017-020433

Open access

allocation. In the present analysis, the propensity score was estimated by regressing intervention dose (0=control group, 1=incomplete dose, 2=full dose) on a set of 27 baseline covariates (categorical and continuous) using multinominal regression. In line with previous research,60 covariates included in the model were: i) vari-ables found to predict lesson completion at the bivariate level, ii) risk factors associated with substance use in the literature (eg, mental health outcomes), iii) baseline measures of all outcome variables and iv) socioeconomic variables (eg, gender). To increase the explained varia-tion and improve the propensity score model, two-way interaction and quadratic terms were also included where these improved prediction.61 All included variables were measured at baseline and therefore were not influenced by exposure to the intervention (a full list of covariates are summarised in online supplementary file 2). The final multinominal logit model used to derive propen-sity scores was successful at modelling full dose of the intervention (likelihood ratio χ2=672.9 (df=58; p<0.001); R2=29.0%). Balance checks were conducted on baseline covariates comparing weighted means and proportions across the three weighted groups (control, incomplete, full dose) and results indicated that baseline equivalence was achieved on all covariates.

Inverse probability of treatment weightsWeights were generated and summary statistics were examined to determine whether stabilising weights were required. The use of stabilising weights is recommended when extreme weights are present due to inaccurate or unstable weights for students with a very low probability of receiving the intervention.59 62 Results indicated that there were extreme weights present (M=3.05, SD=6.99; range=1.00–158.53), thus stabilised weights were gener-ated. As the stabilised weights also varied in range

(M=1.00, SD=1.76, range=0.23–36.46), a series of sensi-tivity analyses using truncated weights were conducted to examine the influence of the extreme weights. To create truncated weights, the cut point of the 1st and 99th percentiles was calculated and any values outside the cut points were replaced with those values. Mixed effects regressions using the truncated weights were conducted and compared with models using the stabilised weights. No material differences were observed between the two models for any outcome, thus results from analyses using the full range of weights are reported. Consistent with the primary analyses, mixed effects regressions were esti-mated to examine change over time in outcomes across the three weighted groups. These models were based on a sample of 1088 participants, as 38 cases were excluded due to missing covariate data, and included a variable

Table 2 Results from multilevel regression models at the 24-month follow-up

Outcome OR 95% CI P values

Ecstasy intentions 2.00 0.42 to 9.17 0.39

NPS intentions 2.26 0.48 to 10.75 0.31

Synthetic cannabis intentions

3.56 1.33 to 9.54 0.01

Synthetic stimulant intentions

1.78 0.50 to 6.34 0.37

Ever used ecstasy 1.08 0.19 to 6.15 0.93

Ever used NPS 0.57 0.14 to 2.30 0.43

Ever used synthetic cannabis

0.17 0.03 to 0.97 0.05

B 95% CI P values

Ecstasy knowledge 0.07 −0.38 to 0.52 0.75NPS knowledge −0.02 −0.23 to 0.19 0.85

Reference group=Climate Schools intervention.NPS,  new psychoactive substances.

Table 3 Results from multilevel regression models using weighted data (n=1088)

Outcome OR 95% CI P values

Climate full dose vs control

Ecstasy intentions 1.44 0.17 to 12.19 0.74

NPS intentions 0.56 0.05 to 5.81 0.63

Synthetic cannabis intentions

1.01 0.29 to 3.50 0.98

Synthetic stimulant intentions

0.90 0.13 to 6.44 0.91

Ever used ecstasy 1.28 0.15 to 10.90 0.82

Ever used NPS 0.33 0.04 to 3.02 0.33

Ever used synthetic cannabis

0.98 0.15 to 6.66 0.99

Β 95% CI P values

NPS knowledge 0.54 0.00 to 1.07 0.04Ecstasy knowledge 1.14 0.35 to 1.92 0.001

Outcome OR 95% CI P values

Climate incomplete dose vs control

Ecstasy intentions 1.35 0.24 to 7.58 0.73

NPS intentions 2.23 0.52 to 9.62 0.28

Synthetic cannabis intentions

2.86 0.67 to 12.25 0.16

Synthetic stimulant intentions

0.88 0.18 to 4.27 0.88

Ever used ecstasy 1.48 0.21 to 10.50 0.70

Ever used NPS 0.46 0.8 to 2.67 0.39

Ever used synthetic cannabis

2.82 0.49 to 16.20 0.25

Β 95% CI P values

NPS knowledge −0.11 −0.37 to 0.60 0.65Ecstasy knowledge −0.01 −0.64 to 0.62 0.98

Reference group=control group.NPS,   new psychoactive substances.

8 Champion KE, et al. BMJ Open 2018;8:e020433. doi:10.1136/bmjopen-2017-020433

Open access

representing dose, time×group interaction and the stabi-lised weight.

resultssample characteristicsA total of 1126 students completed the baseline survey (Mage=14.92 years, SD=0.43; 57% male). Almost all students (96%) provided follow-up data on at least one occasion, and 79% provided data at two or more follow-ups. The descriptive statistics for outcomes over time are reported in table 1 and participant retention is presented in figure 1. At baseline, there was a signifi-cantly greater proportion of females in the control group (78%) compared with the intervention group (16%; χ2 (1, n=1126)=438.29, p<0.001). Students in the interven-tion group also reported significantly greater ecstasy knowledge (F(1, 1124)=76.44, p<0.001) and NPS (F(1, 1123)=56.79, p<0.001) compared with controls.

Intervention effectsTable 2 presents the results from the mixed effects regres-sions of the 24-month follow-up data. Results indicate that a significantly greater proportion of students in the control group reported being likely to use synthetic cannabis in the future, compared with the intervention group (OR=3.56, 95% CI 1.32 to 9.54, p=0.01). There were no significant group differences for any other outcome variable; however, students in the control group were trending towards being less likely to use synthetic cannabis compared with the inter-vention group (p=0.05), when dose was not considered.

Impact of intervention dose on outcomesAnalysis of online dosage data indicated that 59% of students (n=378) completed the Climate Schools module in full. Analyses examining the baseline equivalence among the unweighted groups indicated that the full-dose group had significantly greater knowledge about ecstasy (p<0.005) and NPS (p<0.005) compared with controls at baseline. Although the incomplete dose group had significantly greater knowledge of harms associated with ecstasy (p<0.001) and NPS (p<0.001) than controls at baseline, they were significantly more likely to have used ecstasy (p=0.008), any NPS (p=0.03) and synthetic stimu-lants (p=0.04) and to intend on using ecstasy (p=0.003), synthetic cannabis (p=0.03) and synthetic stimulants (p=0.04) in the future (the complete baseline and 24-month data for the unweighted groups are available in online supplementary file 3). Mixed effects regressions using the weighted data revealed that at 24 months, control students reported significantly less knowledge about both ecstasy (p=0.001) and NPS compared with students who received a full dose of the intervention (p=0.04). There were no significant differences between the incomplete dose and controls on any outcome (table 3).

DIsCussIOnThis study evaluated the effectiveness of the first online school-based prevention programme for ecstasy and

NPS over a 2-year period. Results indicated that at the 24-month follow-up, students in the intervention group were significantly less likely to intend on using synthetic cannabis in the next 6 months, compared with controls. This finding indicates that there was a sustained effect over time for synthetic cannabis intentions, with anal-ysis of the short-term data previously demonstrating a significant group difference up to 12 months. However, beneficial short-term effects previously observed in rela-tion to NPS intentions and increased knowledge about ecstasy and NPS were not sustained at 24 months, and there was no effect in terms of reducing lifetime use of ecstasy or NPS. The ability of the intervention to modify students’ future intentions to use synthetic cannabis up to 24 months after the intervention is notable, as future intentions to use illicit substances, such as cannabis, have been shown to predict initiation of use.33 This finding is especially important in light of the severe side effects asso-ciated with synthetic cannabis use, including renal and pulmonary injuries and cardiac arrests,11 63 as well as the fact that synthetic cannabis use has been associated with poor mental health outcomes, risky sexual behaviour, binge drinking, illicit drug use and injury or violence.15–17 Future research should specifically examine the relation-ship between NPS attitudes, intentions and use.

This study makes an important contribution to the literature by using novel online lesson completion data to examine the effect of dosage on student outcomes. Overall, we found that lesson completion was poor, with just over half of students (59%) completing the Climate Schools: Ecstasy and Emerging Drugs module in full. Results from the weighted analyses indicated that at the 24-month follow-up, control students reported signifi-cantly lower knowledge about ecstasy and NPS, compared with students who completed all four lessons of the programme. Students who received an incomplete dose of the intervention did not report significantly greater knowledge than controls. This informs interpretation of the primary analyses, in which effects for ecstasy and NPS knowledge were only evident at post-test and 12 months, respectively, and not sustained at 24 months. The present findings emphasise the importance of delivering the Climate Schools: Ecstasy and Emerging Drugs module in full to maximise student outcomes.

Despite the potential of internet-based interventions to improve implementation fidelity, some students did not receive a full dose of the intervention in the present study. Although we cannot be certain of the reasons for poor adherence, it is likely attributable to the timing of intervention delivery and competing demands faced by teachers. Unlike previous Climate Schools modules which are typically delivered in years 8 or 9, this module was deliv-ered in year 10 (aged 15 years) to align with the later age of initiation for ecstasy and NPS use. The low adherence to the intervention and modest follow-up rates achieved in this study highlight the difficulties of conducting research among year 10 students. This is consistent with previous research that has shown that school-based drug

9Champion KE, et al. BMJ Open 2018;8:e020433. doi:10.1136/bmjopen-2017-020433

Open access

education often exists in competition with academic agendas,64 including exams, excursions and a lack of time in the curriculum. Indeed, teachers in this study indi-cated that it was hard to implement the surveys and/or intervention due to examinations, excursions and a lack of lesson time. Implications for programme developers include considering ways of improving student engage-ment with the intervention to increase lesson completion and thus improve programme outcomes,45 65 such as the use of smartphone applications and novel technology, email/SMS reminders to encourage students to complete missed lessons and automatic notifications for teachers about student lesson completion. In terms of real-world implementation, teachers should consider their existing timetables and lesson plans to determine when the full four-lesson Climate Schools programme can be successfully implemented among students.

Examination of the baseline descriptive data indicated several differences between students who did, versus did not, complete the programme (see online supplementary file 3). Students who did not complete the programme in full (incomplete dose) tended to be ‘riskier’ at baseline than students in the control group, reporting greater use of ecstasy and NPS and greater intentions to use these substances. This pattern of results suggests that students in the incomplete dose group were the students at greatest risk of substance use and harms at the beginning of the study, which could impact both their propensity to complete the intervention, as well as their substance use behaviour at 24 months. This suggests that this subgroup of students may require more intensive or targeted inter-vention, which addresses underlying risk factors such as personality traits or emotional/behavioural difficulties,66 rather than a universal approach. Targeted, or ‘selec-tive’ prevention approaches are aimed at individuals who are at high risk of developing substance use prob-lems or who are showing very early signs or symptoms.67 There were no significant differences between control students and those who partially completed the interven-tion, suggesting that the intervention needs to be imple-mented in full to achieve beneficial student outcomes. This may be particularly important when implementing prevention programmes for low prevalence, but high-risk substances such as ecstasy and NPS, where students need to receive comprehensive harm-minimisation informa-tion and skills to fully understand the harms and learn the prevention messages.

In addition, the intervention effect found for NPS intentions at the 12-month follow-up was not sustained at 24 months. This suggests that additional booster lessons may be needed in the year following the intervention to reinforce key messages among students who received the intervention in full, and to provide an opportunity for students who only received an incomplete dose to acquire new knowledge and skills. It is also possible that greater effects would be found if schools implemented the Climate Schools: Ecstasy and Emerging Drugs module in combination with the other existing modules in the Climate School suite,

rather than as a stand-alone intervention. If feasible for schools, students could receive the effective 12-lesson Climate Schools: Alcohol and Cannabis module in years 8 or 9 (aged 13–14 years), and the four-lesson Ecstasy and Emerging Drugs module in years 9 or 10 (aged 15–16 years), providing them with comprehensive harm-minimisation knowledge about licit and illicit drugs, as well as rein-forcing refusal and resistance skills.

Although the prevalence of ecstasy use appeared to increase from baseline (2.5%) to 24 months (7.3%) among the intervention group, importantly, there were no significant group differences. These rates are consistent with patterns of ecstasy use observed among Australian students of comparable ages in the most recent Austra-lian Secondary Students’ Alcohol and Drug (ASSAD) survey, a large survey of just under 25 000 adolescents. The ASSAD reported rates of ecstasy use to be approx-imately 3% among those aged 15 years, and increasing to 7% among those aged 17 years.68 It should also be noted that although there were no significant differences between the intervention and control groups in terms of use of any NPS or synthetic cannabis, the intervention group appeared to be trending towards being more likely to have ever used synthetic cannabis group than controls. An inspection of the unweighted data by dose suggests that it is the incomplete dose group driving this pattern of results.

The present findings should be interpreted in light of several limitations. First, our measure of implemen-tation fidelity focused on only one dimension, interven-tion dose,36 and we did not assess teacher-related factors, such as quality of delivery and adherence to programme content. However, since the Climate Schools programme is completed individually by students via the internet, these teacher-related factors are potentially less important for this study. Additionally, the propensity model to predict likelihood of a full dose was constructed primarily through student-level variables, such as mental health outcomes and truancy rates, and did not include school-specific or teacher-specific factors. It should be noted that it is possible that other factors may have influenced the likelihood that a student would complete the Climate Schools programme in full, such as teachers’ attitudes and willingness to prior-itise intervention delivery among timetabling issues and competing demands in the school environment.37 None-theless, the use of online dosage data extracted from the study website, rather than relying solely on teacher self-re-port, is novel and a strength of this study. Attrition was higher than anticipated, despite comprehensive efforts to follow-up students who were absent from school on the day of a survey. Importantly, missing data were handled in our analyses using ML estimation, a widely accepted approach and highly efficient way of using all available information to estimate parameters rather than deleting cases with missing data.52 56 Finally, the low numbers of students in our sample who had used any NPS, synthetic cannabis and synthetic stimulants meant that cell sizes were small and our results should be interpreted with

10 Champion KE, et al. BMJ Open 2018;8:e020433. doi:10.1136/bmjopen-2017-020433

Open access

some caution. The low cell counts also meant that many outcomes were dichotomised and were analysed via multi-level logistic regressions, which deviates from the planned Poisson analyses in our protocol.49

COnClusIOnsThis study evaluated the effectiveness of the Climate Schools: Ecstasy and Emerging Drugs module at the final 24-month follow-up assessment, and examined the impact of intervention dose on outcomes. The present findings demonstrated a sustained impact of the programme in terms of modifying students’ intentions to use synthetic cannabis up to 24 months; however, no other interven-tion effects were observed. An examination of outcomes by dose revealed that the intervention was effective in increasing knowledge about both ecstasy and NPS up to 24 months for students who received the programme in full, compared with the control group. These findings highlight the importance of ensuring that online inter-ventions, such as the Climate Schools: Ecstasy and Emerging Drugs module, are delivered in full to maximise student outcomes.

Acknowledgements The authors would like to thank the teachers and students at participating schools, Philip Clare for statistical advice and Beth Turner and Joanne Cassar for their assistance with data entry.

Contributors KEC, NCN and MT designed the study. KEC coordinated data collection and intervention implementation in schools. KEC and LS led the statistical analysis. All authors read, edited and approved the final manuscript.

Funding This work was supported by the National Health and Medical Research Council (APP1056432) and the Australian Government Department of Health. NN and LS were supported by Society for Mental Health Research Early Career Fellowships.

Disclaimer The funding bodies had no role in the design of the study and collection, analysis and interpretation of data and in writing the manuscript.

Competing interests MT, NCN and KEC are the developers of the Climate Schools: Ecstasy and Emerging Drugs module. MT and NCN are Directors of Climate Schools Pty Ltd.

Patient consent Obtained.

ethics approval The study was approved by the UNSW Sydney Human Research Ethics Committee (HREC HC13075).

Provenance and peer review Not commissioned; externally peer reviewed.

Data sharing statement The datasets generated and/or analysed during the current study are not publicly available, but are available from the corresponding author on reasonable request.

Open access This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http:// creativecommons. org/ licenses/ by- nc/ 4. 0/.

reFerenCes 1. Australian Institue of Health and Welfare. National Drug Strategy

Household Survey 2017 Key Findings. Canberra: AIHW, 2017. 2. Fisk JE, Murphy PN, Montgomery C, et al. Modelling the adverse

effects associated with ecstasy use. Addiction 2011;106:798–805. 3. Baylen CA, Rosenberg H. A review of the acute subjective effects of

MDMA/ecstasy. Addiction 2006;101:933–47.

4. United Nations Office on Drugs and Crime. World Drug Report. New York: United Nations, 2015.

5. Araújo AM, Valente MJ, Carvalho M, et al. Raising awareness of new psychoactive substances: chemical analysis and in vitro toxicity screening of ‘legal high’ packages containing synthetic cathinones. Arch Toxicol 2015;89:757–71.

6. European Monitoring Centre for Drugs and Drug Addiction. European Drug Report 2017: Trends and Developments. Luxembourg: EMCDDA, 2017.

7. Baumeister D, Tojo LM, Tracy DK. Legal highs: staying on top of the flood of novel psychoactive substances. Ther Adv Psychopharmacol 2015;5:97–132.

8. Zawilska JB, Andrzejczak D. Next generation of novel psychoactive substances on the horizon – A complex problem to face. Drug Alcohol Depend 2015;157:1–17.

9. van Amsterdam J, Brunt T, van den Brink W. The adverse health effects of synthetic cannabinoids with emphasis on psychosis-like effects. J Psychopharmacol 2015;29:254–63.

10. Winstock A, Lynskey M, Borschmann R, et al. Risk of emergency medical treatment following consumption of cannabis or synthetic cannabinoids in a large global sample. J Psychopharmacol 2015;29:698–703.

11. Davis C, Boddington D. Teenage cardiac arrest following abuse of synthetic cannabis. Heart Lung Circ 2015;24:e162–e163.

12. Dargan PI, Albert S, Wood DM. Mephedrone use and associated adverse effects in school and college/university students before the UK legislation change. QJM 2010;103:875–9.

13. Hohmann N, Mikus G, Czock D. Effects and risks associated with novel psychoactive substances: mislabeling and sale as bath salts, spice, and research chemicals. Dtsch Arztebl Int 2014;111:139–U16.

14. Karila L, Megarbane B, Cottencin O, et al. Synthetic cathinones: a new public health problem. Curr Neuropharmacol 2015;13:12–20.

15. Patrick ME, O’Malley PM, Kloska DD, et al. Novel psychoactive substance use by US adolescents: Characteristics associated with use of synthetic cannabinoids and synthetic cathinones. Drug Alcohol Rev 2016;35:586–90.

16. Champion KE, Teesson M, Newton NC. Patterns and correlates of new psychoactive substance use in a sample of Australian high school students. Drug Alcohol Rev 2016;35:338–44.

17. Clayton HB, Lowry R, Ashley C, et al. Health risk behaviors with synthetic cannabinoids versus marijuana. Pediatrics 2017;139:e20162675.

18. Corazza O, Demetrovics Z, van den Brink W, et al. ‘Legal highs’ an inappropriate term for ‘Novel Psychoactive Drugs’ in drug prevention and scientific debate. Int J Drug Policy 2013;24:82–3.

19. Champion KE, Teesson M, Newton NC. Development of a Universal Internet-Based Prevention Program for Ecstasy and New Psychoactive Substances. Open J Prev Med 2015;05:23–30.

20. Newton NC, Andrews G, Teesson M, et al. Delivering prevention for alcohol and cannabis using the Internet: a cluster randomised controlled trial. Prev Med 2009;48:579–84.

21. Newton NC, Teesson M, Vogl LE, et al. Internet-based prevention for alcohol and cannabis use: final results of the Climate Schools course. Addiction 2010;105:749–59.

22. Newton NC, Vogl LE, Teesson M, et al. CLIMATE Schools: alcohol module: cross-validation of a school-based prevention programme for alcohol misuse. Aust N Z J Psychiatry 2009;43–201–7.

23. Champion KE, Newton NC, Stapinski L, et al. A cross-validation trial of an Internet-based prevention program for alcohol and cannabis: Preliminary results from a cluster randomised controlled trial. Aust N Z J Psychiatry 2016;50–64–73.

24. Teesson M, Newton NC, Slade T, et al. Combined universal and selective prevention for adolescent alcohol use: a cluster randomized controlled trial. Psychol Med 2017;47:1761–70.

25. Champion KE, Newton NC, Stapinski LA, et al. Effectiveness of a universal internet-based prevention program for ecstasy and new psychoactive substances: a cluster randomized controlled trial. Addiction 2016;111:1396–405.

26. Faggiano F, Minozzi S, Versino E, et al. Universal school-based prevention for illicit drug use. Cochrane Database Syst Rev 2014;18:CD003020.

27. Evans RI. Smoking in children: developing a social psychological strategy of deterrence. Prev Med 1976;5:122–7.

28. Botvin GJ. Preventing drug abuse in schools: social and competence enhancement approaches targeting individual-level etiologic factors. Addict Behav 2000;25–887–97.

29. Meyer L, Cahill H. Principles for school drug education. Canberra: Australian Government Department of Education, Science and Training, 2004.

30. Ajzen I, Fishbein M. Understanding attitudes and predicting social behaviour, 1980.

11Champion KE, et al. BMJ Open 2018;8:e020433. doi:10.1136/bmjopen-2017-020433

Open access

31. Webb JA, Baer PE, Getz JG, et al. Do fifth graders’ attitudes and intentions toward alcohol use predict seventh-grade use? J Am Acad Child Adolesc Psychiatry 1996;35:1611–7.

32. Maher RA, Rickwood D. The theory of planned behavior, domain specific self-efficacy and adolescent smoking. J Child Adolesc Subst Abuse 1998;6:57–76.

33. Stewart MW, Moreno MA. Changes in attitudes, intentions, and behaviors toward tobacco and marijuana during u.S. Students’ first year of college. Tob Use Insights 2013;6:TUI.S11325–16.

34. Durlak JA, DuPre EP. Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol 2008;41(3-4):327–50.

35. Elliott DS, Mihalic S. Issues in disseminating and replicating effective prevention programs. Prev Sci 2004;5:47–53.

36. Bast LS, Due P, Bendtsen P, et al. High impact of implementation on school-based smoking prevention: the X:IT study-a cluster-randomized smoking prevention trial. Implement Sci 2016;11:125.

37. Durlak JA. Studying program implementation is not easy but it is essential. Prev Sci 2015;16:1123–7.

38. Bumbarger B, Perkins D. After randomised trials: issues related to dissemination of evidence‐based interventions. J Child Serv 2008;3:55–64.

39. Dusenbury L, Brannigan R, Hansen WB, et al. Quality of implementation: developing measures crucial to understanding the diffusion of preventive interventions. Health Educ Res 2005;20:308–13.

40. Ringwalt CL, Ennett S, Johnson R, et al. Factors associated with fidelity to substance use prevention curriculum guides in the nation’s middle schools. Health Educ Behav 2003;30:375–91.

41. Backer TE. Finding the balance: Program fidelity and adaptation in substance abuse prevention: a state of the art review. Rockville, MD: Substance Abuse and Mental Health Services Administration, Center for Substance Abuse Prevention, 2001.

42. Pankratz MM, Jackson-Newsom J, Giles SM, et al. Implementation fidelity in a teacher-led alcohol use prevention curriculum. J Drug Educ 2006;36:317–33.

43. Portnoy DB, Scott-Sheldon LA, Johnson BT, et al. Computer-delivered interventions for health promotion and behavioral risk reduction: a meta-analysis of 75 randomized controlled trials, 1988-2007. Prev Med 2008;47:3–16.

44. Bishop DC, Pankratz MM, Hansen WB, et al. Measuring fidelity and adaptation. Eval Health Prof 2014;37:231–57.

45. Pettigrew J, Graham JW, Miller-Day M, et al. Adherence and delivery: implementation quality and program outcomes for the seventh-grade keepin’ it REAL program. Prev Sci 2015;16:90–9.

46. Champion KE, Teesson M, Newton NC. A cluster randomised controlled trial of the climate schools: Ecstasy and emerging drugs modulein australian secondary schools: Study protocol. BMC Public Health 2013;13:1168.

47. Newton NC, Teesson M, Newton KL. Developing the climate schools: ecstasy module-a universal Internet-based drug prevention program. J Psychoactive Drugs 2012;44:372–80.

48. Heo M, Leon AC. Sample size requirements to detect an intervention by time interaction in longitudinal cluster randomized clinical trials. Stat Med 2009;28:1017–27.

49. Champion KE, Teesson M, Newton NC. A cluster randomised controlled trial of the climate schools: Ecstasy and emerging drugs

module in australian secondary schools: Study protocol. BMC Public Health 2013;13:1168.

50. Vogl LE, Newton NC, Champion KE, et al. A universal harm-minimisation approach to preventing psychostimulant and cannabis use in adolescents: a cluster randomised controlled trial. Subst Abuse Treat Prev Policy 2014;9:24.

51. Australian Institue of Health and Welfare. 2010 National Drug Strategy Household Survey report. Drug statistics series no 25 Cat no PHE 145. Canberra: AIHW, 2011.

52. Rabe-Hesketh S, Skrondal A. Multilevel and longitudinal modeling using Stata. 2nd Edition. Texas: STATA press, 2008.

53. West BT, Welch KB, Galecki AT. Linear mixed models: A practical guide using statistical software. 2nd ed. Boca Raton: FL: CRC Press, 2014.

54. Gueorguieva R, Krystal JH. Move over ANOVA: progress in analyzing repeated-measures data and its reflection in papers published in the Archives of General Psychiatry. Arch Gen Psychiatry 2004;61:310–7.

55. Strøm HK, Adolfsen F, Handegård BH, et al. Preventing alcohol use with a universal school-based intervention: results from an effectiveness study. BMC Public Health 2015;15:337–37.

56. Schafer JL, Graham JW. Missing data: our view of the state of the art. Psychol Methods 2002;7:147–77.

57. Mallinckrodt CH, Sanger TM, Dubé S, et al. Assessing and interpreting treatment effects in longitudinal clinical trials with missing data. Biol Psychiatry 2003;53:754–60.

58. Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika 1983;70:41–55.

59. Austin PC. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behav Res 2011;46:399–424.

60. Eisner M, Nagin D, Ribeaud D, et al. Effects of a universal parenting program for highly adherent parents: a propensity score matching approach. Prev Sci 2012;13:252–66.

61. Jones AS, D’Agostino RB, Gondolf EW, et al. Assessing the effect of batterer program completion on reassault using propensity scores. J Interpers Violence 2004;19:1002–20.

62. Robins JM, Hernán MA, Brumback B. Marginal structural models and causal inference in epidemiology. Epidemiology 2000;11:550–60.

63. Gurney SM, Scott KS, Kacinko SL, et al. Pharmacology, toxicology, and adverse effects of synthetic cannabinoid drugs. Forensic Sci Rev 2014;26:53–78.

64. Van Hout MC, Foley M, McCormack A, et al. Teachers’ perspectives on their role in school-based alcohol and cannabis prevention. Int J Health Promot Educ 2012;50:328–41.

65. Low S, Van Ryzin MJ, Brown EC, et al. Engagement matters: lessons from assessing classroom implementation of steps to respect: a bullying prevention program over a one-year period. Prev Sci 2014;15:165–76.

66. Griffin KW, Botvin GJ. Evidence-based interventions for preventing substance use disorders in adolescents. Child Adolesc Psychiatr Clin N Am 2010;19:505–26.

67. Mrazek P, Haggerty R. Reducing Risks for Mental Disorders:: Frontiers for Preventive Intervention Research. Washington, DC: National Academy Press, 1994.

68. White V, Williams T. Australian secondary school students’use of tobacco, alcohol, and over-thecounter and illicit substances in 2014. Victoria: Cancer Council Victoria, 2016.

© 2018 Author(s) (or their employer(s)) 2018. Re-use permitted under CCBY-NC. No commercial re-use. See rights and permissions. Published by

BMJ. This is an open access article distributed in accordance with theCreative Commons Attribution Non Commercial (CC BY-NC 4.0) license,

which permits others to distribute, remix, adapt, build upon this worknon-commercially, and license their derivative works on different terms,

provided the original work is properly cited, appropriate credit is given, anychanges made indicated, and the use is non-commercial. See:

http://creativecommons.org/licenses/by-nc/4.0/. Notwithstanding theProQuest Terms and Conditions, you may use this content in accordance

with the terms of the License.