Quality of Service Implementation Assessment

Department of Public Safety - Division of Juvenile Justice & Delinquency Prevention

Juvenile Community Programs

Standardized Program Evaluation Protocol (SPEP)

Quality of Service Implementation Assessment

Juvenile Community Programs - Provider Information

Instructions: In each of five measures of quality program implementation, sections A-E that follow, please select one description that best describes the program's current operating condition. Enter the selection's corresponding numeric rating (0, 1, 2, 3, or 4) at the bottom of each section. (The aggregate score will automatically total at the bottom of page 6.)

Sponsoring /Lead Agency:

Program Name/Component Name:

JCPC Area Consultant/
Contract Service Administrator:

Program Manager/Director:

Newly Funded Community Programs Service Provider
Existing Funded Community Programs Provider

Multi-County: Are services provided in more than one county by the same agency?

Yes
No

County:

Date of Completion:

SPEP (Select ONE below):

Component has SPEP service
No SPEP service

SPEP Primary Service (Select ONE of the SPEP Services below):
--Select Program Type--
asdf

Purpose of the Quality of Service (QoS) Implementation Assessment:

To identify and measure the degree of quality of service delivery in JCPC/Contractual funded programs. To assess quality of implementation relative to Department policies regarding service delivery, guided by principles shown to positively affect outcomes. To support continuous improvement to help programs more closely align their service to evidence-based methods shown to reduce recidivism.

Category A: Defined Protocol for Program Services and Delivery
Rating	Description
4	In addition to agency policies as required by JCPC/Contractual policy, program has a written and organized protocol manual that describes the manner of service delivery and includes the suggested interventions, number of sessions, content and flow. Evidence may include: specific methods for providing services curriculum workbook/instructor manual lesson plan(s) script Individual Service Plans are consistently present and show evidence of involvement of youth and family in planning and includes client-specific concerns to be addressed, recommended frequency/duration of contact that follow the manual/protocol.
3	In addition to agency policies as required by JCPC/Contractual policy, program has some written program operational guidelines and policies, but is not organized into protocol manual. These operational guidelines and policies may not necessarily specify a particular flow or the number of sessions; however, they include: a general outline of what suggested interventions are to be delivered lesson plan(s) content of sessions or curriculum to be used or provides the method of service delivery Individual Service Plans are consistently present and show evidence of youth OR parent involvement in planning, included suggested length of stay and/or frequency of contact, and consistently includes a description of the services to be provided, though less individualized.
2	Program does not have a written protocol manual that defines the service delivery but does have required written program policies as per JCPC/Contractual policy. Individual Service Plans are consistently present and show evidence of youth OR parent involvement in planning, includes suggested length of stay and/or frequency of contact, and consistently includes a description of the services to be provided, though less individualized.
1	Program does not have a written protocol manual that defines the service delivery but does have required written program policies as per JCPC/Contractual policy. Individual Service Plans are consistently present; however, there is little specificity in service plan content, and the description of services provided may be generalized. Individual Service Plans do not consistently show evidence of youth or parent involvement.
0	Program does not have written protocols for service delivery, or required program policies per JCPC/Contractual policy. Individual Service Plans are not consistently present.
Rating for Category A =
Comments:

Category B: Staff Training
Rating	Description
4	Documented evidence of a systematic training plan which has specified minimum annual training hours, the plan includes: An individualized annual training plan for each position with training goals and objectives related to service provided, and delivery of interventions to youth and families. Evidence of a regular and consistent training schedule, with training provided as scheduled, training provided to staff meets JCPC/Contractual policy minimum requirements, and all direct service staff meet/exceed agency’s minimum annual training hours. Training sessions in program service delivery, as well as clinical supervision, case staffing and/or consultation sessions: address identified training goals and objectives; and are consistently documented and maintained in an individualized training record. Direct service staff are highly experienced and highly qualified. Direct service staff possess licensing/degrees/credentials/certifications required by the program service type and/or the specified model. Trainers (or clinical supervisor/consultants) are highly experienced and highly qualified.
3	Documented evidence of an annual schedule of staff training, and has specified minimum annual training hours, with A majority of staff meeting/exceeding agency’s minimum annual training hours. Agency has documentation of direct service staff participating in training related to program service and delivery of interventions. Documented training meets JCPC/Contractual policy minimum standards. Direct service staff possess licensing/degrees/credentials/certifications required by the program service type and/or the specified model. Trainers (or clinical supervisor/consultants) are experienced and qualified.
2	Documented training meets JCPC/Contractual policy minimum standards. There is no clear or established training plan (schedule, minimum annual hours, or established curriculum).
1	Documented training does not meet JCPC/Contractual policy minimum standards. There is some documented evidence of staff participation in occasional training.
0	Documented training does not meet JCPC/Contractual policy minimum standards. No documented evidence of training provided to staff.
Rating for Category B =
Comments:

Category C: Staff Retention (Prior 12 Months)

Total Staff Positions includes all direct service staff, volunteers, or contractors AND program management staff, volunteers, or contractors (i.e.: Program Manager, anyone responsible for program oversight, consultation, or supervision).

Programs with six or more staff, volunteer, or contractor positions

Programs with six or more staff, volunteer, or contractor positions
Retention %	Vacancy Period	Points	Retention %	Vacancy Period	Points
81-100%	< 90 days	4			4
71-80%	< 90 days	3	81-100%	> 90 days	3
61-70%	< 90 days	2	71-80%	> 90 days	2
51-60%	< 90 days	1	61-70%	> 90 days	1
≤ 50%	< 90 days	0	≤ 60%	> 90 days	0

Programs with five or fewer staff, volunteer or contractor positions

Agency has a written continuity plan for vacancies, and it was implemented during any and all vacancies.

Programs with five or fewer staff, volunteer or contractor positions
Vacancy Period	Points
≤ 30 days	4
31-60 days	3
61-90 days	2
91-120 days	1
> 120 days, OR has no plan, OR failed to implement plan	0

Programs with five or fewer staff, volunteer or contractor positions
Rating for Category C =
Comments:

Category D: Agency's Own Program Evaluation, Monitoring, and Corrective Action
Rating	Description
4	Agency has clearly defined, written effectiveness measures, with specified methods and procedures to collect and examine data to measure and improve Program performance. Program has an established process in which the program monitors its service delivery against the protocol manual. The monitoring process is done on at least a quarterly basis, or in accordance with parameters of the model, with documented results in writing. Deviations from the protocol manual are addressed with corrective actions which are specified in writing, tracked, and documented accordingly. NCALLIES data is accurate and completed within the required JCPC/Contractual parameters.
3	Agency has general goals and desired outcomes for the program beyond the required measurable objectives, but lacks specific definition of program effectiveness. The program has an established process to collect and evaluate information and data to measure progress towards goals and outcomes. Program has an established process in which the delivery of program services is monitored by the agency at least annually to examine how closely actual implementation of the services matches the agency’s model/protocol. Deviations from the model/protocol are addressed with corrective actions which are specified in writing, monitored, and documented accordingly. NCALLIES data is accurate and completed within the required JCPC/Contractual parameters.
2	Program has a minimal process to demonstrate program effectiveness through data collection at least annually. Data collection efforts for only the required JCPC measurable objectives. Although there is no established process to track, document and address unmet measurable objectives or program fidelity concerns, there is some evidence of corrective actions taken to address inadequacies. NCALLIES data is entered but shows data quality errors and/or data is not completed within required JCPC/Contractual parameters.
1	Program has minimal process to demonstrate program effectiveness through data collection or data collection efforts are restricted to data for the required JCPC/Contractual measurable objectives. There is no evidence that unmet measurable objectives generate any corrective actions. There is no established process to track, document and address program fidelity concerns. NCALLIES data is entered but shows data quality errors and data is not completed within required parameters.
0	Program lacks evidence to demonstrate any systematic effort to measure program fidelity or effectiveness. No evidence of addressing unmet measurable objectives. Data has not been entered in NCALLIES within required JCPC/Contractual parameters.
Rating for Category D =
Comments:

Category E: Staff Evaluation
Rating	Description
4	A written overall plan of staff evaluation and development is present in order to formally and specifically review, appraise and improve work performance. Staff members are evaluated on a specified schedule of at least annually, according to JCPC/Contractual policy. Staff evaluation includes measurements of the staff member's performance in implementing the DPS funded program model/protocol; adherence to required policies and procedures for JCPC/Contractual funded programs, and quality of client/family engagement. Any deviations/violations of the program policies/model/protocol are documented in a written staff development plan. Areas of improvement are identified to include knowledge, skills, abilities necessary for enhancing program service delivery, including client/family engagement. Plan(s) are implemented to improve the staff member’s capacity (knowledge, skills, and abilities).
3	Staff members are evaluated annually, according to JCPC/Contractual policy. The staff evaluations include assessing staff member's performance in implementing the DPS funded program model/protocol; adherence to required policies and procedures for JCPC funded programs, and quality of client/family engagement. There are written staff development plans. The plans do minimally address ongoing follow-up of any violation, deviation of program policies, and/or the model protocol.
2	Staff members are evaluated annually, according to JCPC/Contractual policy, but the evaluation elements are general and broad. Evaluation is present, but lacks specifics to the particular JCPC/Contractual program and does not address adherence to program or JCPC/Contractual policies and requirements, and/or fidelity of the model or protocol. There are no formal, written staff development plans.
1	Evaluations are documented but are performed on less than an annual basis. The evaluation elements are general and broad and do not address program or JCPC/Contractual policy and requirements, and/or fidelity of the model or protocol. There are no formal, written staff development plans.
0	Program lacks documented evaluations of program staff.
Rating for Category E =
Comments:
Total for Categories A - E =	0

Glossary
Term	Definition
Consistently Present	Present in no less than 66% of a random sampling of files.
Policy	A guiding principle used to set direction in an organization; a course of action to guide and influence decisions. A policy designates a process (procedure).
Procedure	A series of actions that are done in a certain way or order to be followed as a consistent approach; a particular course of action or process intended to achieve a result.
Protocol	A formal set of written procedures to follow; prescribed methods to be used in delivering program services.