This study sought to address this gap in data collection through the inclusion of opentext and quantitative items in the posttest instrument to gather data on student perceptions of the RAYS program and their overall experiences. Items included in the survey assessed resource awareness, self-reflection of AOD use, and the likelihood of recommending the RAYS program to peers. Overall, student resource awareness increased for substance use treatment and support services. This puts RAYS on track to meet the program objective of increasing access to and awareness of SUD treatment services . Another quantifiable objective was to provide SUD treatment services to at least 100 students by April 2024. Based on case data, of the 63 students who enrolled in RAYS during this evaluation period, 40 were referred and participated in at least one substance use counseling session . Therefore, at this state, the program is just under 50% of the target goal. It is important to note that not all students who go through RAYS are diverted from drug-related disciplinary incidents, thus may not require SUD treatment services.Prior to the launch of the RAYS program, Nevada County sites reported higher overall and drug-related suspensions in relation to selected comparable sites in a neighboring county. Overall suspension counts were notably higher in Nevada County schools, reporting over double the number of suspensions in relation to comparable sites during the 2017-18 and 2018-19 school years.
Nonetheless, a sharp decline in the 2021-22 academic year, 4×4 grow table after the launch of RAYS, implies some effect of the program on the disciplinary landscape in Nevada County sites. This notable reduction in the number disciplinary incidents after implementation of an RJP program has been reported in previous studies. Hashim and colleagues noted that at the 1-year mark after implementation of a restorative justice program, there was a significant reduction in the number of suspensions. This same trend is seen in the initial decline in the number of disciplinary incidents in Nevada County sites implementing RAYS. Furthermore, Hollands et al and Gregory et al conducted comparative analyses of schools implementing RJP programs and found that at schools with no restorative programs in place, there were higher suspension rates over time. Similarly, when comparing the Nevada County sites to the four schools in the neighboring county, there was a drastic difference in the number of overall and drug-related suspension counts, with Nevada County schools reporting lower numbers. To our knowledge, data on drug-related suspensions has not been examined in previous restorative program evaluations. Therefore, this study provides insight into the potential impacts of an RJP program with substance use components in addressing drug-related disciplinary incidents in particular. As reported, Nevada County sites had slightly higher drug-related suspensions in the 2018-19 academic year, which corresponds with the uptick in the number of relative incidents at the state level .
Prior to the launch of RAYS, drug-related suspensions were higher at Nevada County sites in relation to comparable sites; however, similar to the number in overall suspensions, there was a sharp decrease in the academic year following the launch of RAYS in August 2021. In contrast, at the comparable school sites the number of drug-related suspensions nearly doubled during the 2021-22 academic year in relation to 2018- 19. Although these shifts may not be directly related to the implementation of RAYS, they may be indicators of larger program impacts on the discipline landscape at the school level. As is evident by pretest and posttest data, the majority of students who exited RAYS reported decreased use behaviors or frequency of use. As such, these students may no longer be using, or if they are, they are choosing not to use at school where they are most likely to be caught. At the systemic level, administrators at the Nevada sites may be increasing the number of referrals they are making to RAYS from drug-related incidents. If so, this may imply an increased awareness of the RAYS program amongst school district and site staff which is crucial for program sustainability and continuous support. This increased awareness may be supported by the high number of school staff and administrator exposures to information presentations conducted by the RAYS program coordinators . Past RJP evaluation studies have cited the negative impacts of punitive measures, particularly the counter intuitive effects they have in increasing the number of disciplinary incidents and repeat offenses.
It is evident from Nevada’s suspension data that in the absence of the RJP program, there was a higher number of overall and drug-related suspensions compared to when after RAYS was launched. Furthermore, authors have noted the positive effects RJP programs in improving academic achievement among participants.26 Although this study did not collect student academic data, it may be interesting to examine the potential impacts of RAYS in increasing academic success as students who are diverted to the program tend to stay in school rather than being sent home. Studies have also found that RJPs decrease the suspension gap between racial/ethnic minorities and non-Hispanic White students. Such gaps were not noted within the Nevada County sites implementing RAYS as the majority of enrollees self-identified as non-Hispanic White, representative of the student body in Nevada County . One of the objectives set by NCSOS was a reduction in drug-related suspensions by 20% by April 2024 . To calculate drug-related suspension rates, the total number of drug related suspensions was divided by the cumulative enrollment for the 2018-19 and 2021-22 academic years . Calculations revealed that in 2018-19, 6.03% of all students at the four sites were suspended for a drug-related reason, whereas in 2021-22, 2.17% of students were suspended. Based on these raw calculations, there was an approximate 64.06% reduction in drug-related suspensions among the entire population at the four sites. This supports the conclusion that RAYS is on track to meet the 20% reduction goal by April 2024 if this decreasing trend continues. This decrease in drug-related suspension rates follows similar patterns with overall suspensions seen in other studies examining the effects of RJPs on disciplinary rates over time. It is crucial to continue tracking drug-related suspensions to assess potential time-dependent effects of the RAYS program across the implementation period.The attainment of program-level goals and objectives pertaining to the timeline for program implementation, case load, and activities is discussed here. To our knowledge, few evaluation studies have examined implementation fidelity of RJP programs, only focusing on program effectiveness on student behavioral outcomes and student experiences. Nonetheless, some researchers have highlighted the critical role that implementation fidelity assessments play in evaluating program effectiveness. Based on case and activity-level data, RAYS is on track to meet process and outcome measure goals and objectives as outlined in the original project proposal and LEP . Behavioral measures related to AOD use are in line with program goals and objectives of reducing youth marijuana and other drug use. Implementation of program components is also on track to meet calendar goals with respect to timeline for implementing activities and expanding access to services for students across the four sites. Program completion rates also align with the goals and objectives NCSOS outlined in their project proposal and LEP. Another one of the program objectives was to assess changes in the recidivism rate over time . Based on data from the current evaluation period, cannabis drying system a total of 2 students reenrolled in RAYS while 48 students exited the program. Following the recidivism rate formula in Table C, the recidivism rate for May 2021 through January 2023 was 5.13%. To ensure that RAYS will meet the 25% reduction objective, it will be crucial to monitor this rate through the remainder of the implementation period.Despite having a set evaluation plan with outlined process and outcome measures with methodologies for data collection and reporting, certain aspects of the evaluation are falling short in providing the full scope of evaluating overall program effectiveness. It is important to highlight gaps and limitations in order to optimize the present evaluation strategy, while also informing future evaluations of RJP programs with substance use components. Furthermore, the limitations discussed here will help to improve current methods for data collection and reporting for the larger evaluation report to be submitted to state funders in October 2024. One of the limitations for this study was the small sample size and low response rate attained for pretest and posttest data collected from students who exited RAYS. During this evaluation period, only 21 out of 48 students who exited the program submitted posttest responses, equating to an approximate response rate of 43.75%.
This omits exit data from over half of the sample which would be crucial to provide a more accurate assessment of individual level behavioral variables captured via pretest and posttest data. Discussions with program coordinators revealed logistical obstacles with administering post-surveys to students uponexiting RAYS, mainly with re-contacting students and scheduling a time to administer the post survey, which is done in-person. Regardless, the UCSD research team is currently working with coordinators to identify these logistical gaps and revising current protocols to increase the pretest and posttest response rate. Current protocol adaptations have included providing “quick response” codes and posttest links directly to students and having them take the survey remotely. This strategy would ideally increase accessibility of the survey and would eliminate the need for an in-person administration. Nonetheless, study strengths lie in the pretest and posttest instrument design. Scales and measures were adapted from the existing 2019-20 CSTS and 2021-22 MYHB instruments which were developed and implemented by Dr. Shu-Hong Zhu’s research team in measuring population-level substance use behaviors among adolescents in California public schools. Furthermore, working directly with NCSOS coordinators, the UCSD research team was further able to adapt the language in the survey instruments and optimize flow to meet accessibility needs of the RAYS students. As such, established behavioral scales were incorporated into the surveys, while also taking a tailored approach to meet program needs which ensured surveys were measuring outcomes of interest for the RAYS program and the NCSOS team. The use of aggregate discipline data from the CDE’s public data repository may also be cited as a limitation as it may not inform individual-level impacts of the program in mitigating re-offenses among students in Nevada County. Furthermore, discipline data reported to the CDE may not be representative of all suspensions or disciplinary incidents occurring at Nevada sites. Thus, more comprehensive datasets are needed to examine site-level changes in disciplinary incidents over time and to assess whether RAYS is in fact having an impact in reducing these counts. Despite these limitations, publicly available datasets from the CDE still reveal stark differences in the number of overall and drug-related suspensions between RAYS and non-RAYS schools. These findings highlight that there is indeed some effect of the program in mitigating the number of suspensions post-launch. Therefore, it is crucial to continue examining these changes over time while working with NCSOS coordinators to gain access to site-level discipline data versus using publicly available data which may not be providing the full scope of the impacts on the disciplinary landscape in Nevada County. Furthermore, the study being intermediate in nature may not allow for a comprehensive evaluation of the program for the entire implementation period . Current findings are only reporting data from a snapshot of the entire implementation period, thus excluding over one years’ worth of pretest and posttest data, discipline data, case management data, and activity-level data. A complete evaluation, which will be conducted May 2024 upon completion of the grant period, will inform the entire implementation period answering more evaluation questions and assessing program status in meeting all program goals and objectives outlined in the proposal and LEP. Nonetheless, these intermediate findings highlight the potentially positive impacts that RAYS is already having with regards to adolescent AOD use rates, resource awareness, and the number of disciplinary incidents. This is the first study, to our knowledge, to report evaluative findings for an RJP program with substance use intervention education, counseling, and treatment components. Findings from this report provide a first-hand look into the potential impacts of an RJP program for students diverted from drug-related disciplinary incidents in addressing AOD use behaviors and harm perceptions, self-responsibility, and resource awareness. Furthermore, this is one of the few studies evaluating an RJP program being implemented in a school system from a rural setting with a prevalent marijuana cultivation industry.