After 6 months, 5 sites, 8,930 miles, 6 reports, and 46,500 words, on the 31st of July we (myself, Gary Zajac, and Courtney Meyer of the Justice Center for Research) finally submitted Penn State’s evaluability assessment of Circles of Support and Accountability to the U.S. National Institute of Justice.
Below is the executive summary from the project’s main cross-site report. The final reports can be obtained from the authors via email (firstname.lastname@example.org). It was a truly fascinating project to work on. I have worked for a COSA provider, conducted research on the program, and going into this project considered myself to be pretty knowledgeable about the program – however, I was still exposed to many unique new perspectives that I hadn’t even considered with every site visit and each conversation with those dedicated people who make the program happen.
In textbook cover-my-backside style I also include the NIJ’s disclaimer: “This project was supported by Award No. 2012-IJ-CX-0008, awarded by the National Institute of Justice, Office of Justice Programs, U.S. Department of Justice. The opinions, findings, and conclusions or recommendations expresses in this publication are those of the author(s) and do not necessarily reflect those of the Department of Justice.” You also should not quote/cite the executive summary below without obtaining the express permission of the report authors.
According to the U.S. National Reentry Resource Center (NRRC) at least 95% of state prisoners are released back to their communities after a period of incarceration. Both criminal justice agencies and the general public are often particularly conscious of the issue of sex offenders returning to the community because of the potentially negative biological and psychiatric outcomes for victims (e.g., Andersen, Tomada, Vincow, Valente, Polcari, & Teicher, 2008; Chen et al., 2010). Due to these negative outcomes, criminal justice responses to sex offender reentry have typically involved tightening supervision for sex offenders. Conversely, the base rate of recidivism for sex offenders is lower than is often expected at around 12.4%. There is also a growing interest in using restorative justice approaches with this population, which redirect society’s punitive response to crime with the aim of increasing public safety through reconciliatory action between offenders, victims, and the community (Sullivan & Tifft, 2005)
Circles of Support and Accountability (COSA) is a restorative justice-based community reentry program for high-risk sex offenders with little or no pro-social community support. COSA originated in 1994 in response to the release of Charlie Taylor, a high-profile, high-risk, repeat child sex offender in Hamilton, Ontario, Canada. A ‘Circle of Support’ was arranged – a select group from the church congregation maintaining daily contact with Taylor (Hannem & Petrunik, 2004). Taylor did not reoffend and the program was extended in Canada, and similar programs grew in, among other places, the United Kingdom, the Netherlands, and the U.S. There have been no rigorous large-scale outcome evaluations of COSA conducted to date. Some small-scale outcome evaluations have been published that vary in quality. A weighted average of the three significant estimated reductions suggest that COSA may be responsible for a reduction of 77% in sexual recidivism. However, because of the varying quality of these studies in terms of retroactive and imperfect matching of samples, the integrity of statistical analyses, and the lack of statistically significant results, it could be argued that this figure should be considered only an estimate of effectiveness. Therefore, at this time there is not enough evidence to confidently state that COSA is proven to be effective in reducing sexual recidivism.
This report outlines an evaluability assessment of COSA across five sites with the goal of assessing the readiness of COSA provision in the U.S. for rigorous evaluation. Evaluability assessments examine the demand for information that might come from a large-scale evaluation and seek to match supply with demand by proposing designs that are feasible, relevant and useful. The assessment aimed to clarify program intent, explore program reality, examine program data capacity, analyze program fidelity, and propose potential evaluation designs for future evaluation.
An ‘intended model’ was developed that sought to illustrate the espoused theory of COSA. A logic model was developed to define the three key problems that COSA seeks to address: (1) the increased frequency of recidivism for high-risk sex offenders; (2) the lack of formal supervision for offenders who have completed their sentences in full; and (3) the lack of social capital and community support for returning sex offenders. A model of COSA program operations, adapted from a model developed by Correctional Services Canada (CSC, 2002; 2003), was also developed that outlined stakeholders and operations. The stakeholders form four broad categories: COSA project staff, service users, formal criminal justice organizations, and community service providers. COSA operations involved five phases: (1) establishing the COSA team and program; (2a) Core Member enrolment; (2b) volunteer enrolment; (3) forging the Circle; (4) ongoing support; (5) dissolution of the Circle.
COSA program reality was established via site visits to five locations delivering, or intending to deliver, COSA programs in the U.S.: Fresno, CA; Denver, CO; Durham, NC; Lancaster, PA; and Burlington, VT. During these site visits in-person interviews were conducted with key program personnel, other stakeholders, and any documented material related to COSA policies and procedures was collected. Data was collected and analyzed using a fidelity item measurement tool that examines 41 items across 10 fidelity categories, including management, model, operations, outcomes, staff, Core Members and volunteers and a data item tool that examined the availability of 23 key data variables.
In summary, all of the sites have implemented versions of the CSC model, adapted to suit their needs. Only COSA Fresno appeared to be running the program in the absence of formal parole or probation supervision in the community. Management structures and financial and operational security differed between sites. Fidelity scores at the sites were (in descending order): Vermont COSA – 86%; COSA Fresno – 58%; COSA Lancaster – 52%; Colorado COSA – 27%; and COSA Durham – 24%. The site reports suggest that VT-COSA alone could be considered to have high program fidelity, with COSA Fresno and COSA Lancaster demonstrating adequate fidelity, and Colorado COSA and COSA Durham demonstrating low fidelity (due mainly to their lack of capacity).
It is concluded that there are five potential obstacles that need to be addressed in order to conduct a successful experimental evaluation of COSA. Firstly, a myopic focus on recidivism may not adequately measure the success of COSA as in some circumstances the detection of a new offense by the Circle may be a marker of program success. Secondly, significant differences in program implementation, namely grass-roots versus institutional models and fully-completed versus supervised Core Members, could result in key differences within the population from which samples might be drawn. Thirdly, there are concerns regarding the systematic selection of highly-motivated offenders and the apparent flexibility in the application of selection criteria. Fourthly, the low capacity at sites, and thus the small populations from which to draw numbers of COSA-eligible participants, combined with the low rates of recidivism expected for both COSA Core Members and controls, may make the detection of any observable effects of COSA more difficult. Finally, in many instances key data, particularly for the Core Member, were not solicited, collected, or reported by the COSA programs. The site reports also noted that both the quality of the relationships between the program and their criminal justice partners and the importance of program stability would need to be addressed for successful evaluation.
It is concluded that there is no methodological or ethical reason why a randomized control trial of COSA provision in the U.S. could not be conducted. The obstacles to an RCT are all such that they can be addressed with a combination of realistic tightening of program implementation, rigorous experimental control, and an increase in real-world resources. It was concluded that there are no major benefit to the use of non-experimental studies over a randomized control trial for the evaluation of COSA. Consequently, three action recommendations for future evaluative activity are presented: (1) conduct an experimental evaluation of the Vermont COSA program alone; (2) conduct an experimental evaluation that combines the Vermont COSA and COSA Fresno programs; or (3) allow the fledgling sites to develop and conduct a multi-site evaluation of COSA in the future.
=== End of executive summary ===
(To reiterate: Do not quote/cite the above executive summary without obtaining the express permission of the report authors.)
We are aware that the findings of the assessment may not to be everyone’s liking – but we sought to provide an objective outlook on a program that is much-vaunted, but sometimes subject to assumption and under-critiqued. I like to think the outcomes of the evaluability assessment were positive and encouraging, and the sheer will and motivation of the vast majority of individuals we met at the five sites was humbling at times. Essentially, our only real criticism is that even the most determined belief in a program and its effectiveness is not a substitute for evidence of its effectiveness. What we hope is that the COSA community take the opportunity to assess how the program and its aims have evolved since the 1990s, how the problem and contexts for which it was developed have changed, and how service provision can be developed moving forward to establish and ensure best practice.