Effectiveness & Maintenance – Organizational Implementation and Effectiveness

This indicator measures implementation and effectiveness of PSE changes. Implementation is defined as the aggregate number of sites or organizations in each type of setting within the eat, live, work, learn, shop, and play domains that report a multi-component and multi-level intervention. Effectiveness is defined as the aggregate number of sites or organizations with improved physical activity environment assessment scores.

Implementation pertains to whether the intervention was delivered with fidelity or as intended and whether the essential elements known to be important to the achievement of positive outcomes were actually and consistently implemented. To be effective, organizational policy changes and environmental supports should be made as part of multi-component and multi-level interventions to sustain the new changes or standards over time. This indicator is “long-term” in that it follows the medium-term adoption indicator; but, in addition, the implementation features are intended to enhance the likelihood of impact and sustainability that derives when PSE change is part of a multi-component/multi-level intervention.

Effectiveness pertains to the achievement of the intended outcomes. Implementation and effectiveness are closely linked because the quality of implementation will directly affect the outcomes achieved. At the environmental level, effectiveness is defined as, and is measured by, improvements in the physical activity environment and/or organizational changes, policies, rules, marketing, and access that make healthy choices easier. Increased environmental assessment scores provide objective, systematic evidence of documented environment-level improvements.

LT6a.	Total number of sites or organizations that implemented a multi-component and multi-level intervention with one or more changes in MT6 (site or organizational adoption of PSE changes and promotion) and one or more of the following additional components: Evidence-based education Marketing Parent/community involvement Staff training on continuous program and policy implementation
LT6b.	Total number of components per site or organization, and types of components implemented during the period assessed
LT6c.	Number of sites or organizations that made at least one PSE change (MT6) and show improved physical activity environment assessment scores using a reliable and, if possible, valid environmental assessment tool

Implementation (LT6a–b):

LT6a. Number of sites or organizations that report a multi-component and multi-level intervention with one or more changes in MT6 (PSE changes). Multi-component intervention refers to sites making at least one PSE change (MT6) implemented together with evidence-based education, marketing, parent/community involvement, and/or staff training on continuous program and policy implementation.

LT6b. Total number of components per site or organization, and types of components:

• Evidence-based education
• Marketing
• Parent/community involvement
• Staff training on continuous program and policy implementation

These four components were identified through practitioner input during development of the Western Region SNAP-Ed Evaluation Framework.

Report implementation in two ways:

1. With the ideal being implementation of one MT6 change plus all four additional components, record the number of sites or organizations implementing one MT6 change and one, two, three, or all four components. Also note how many sites did not implement a multi-component intervention (i.e., only made change in MT6 but did not implement any additional components).
2. Record the number of sites or organizations implementing each type of specific additional component—evidence-based education, marketing, parent/community involvement, and staff training.

Effectiveness (LT6c):

Number of sites or organizations with improved physical activity assessment scores using a reliable and, if possible, valid environmental assessment tool. Follow-up assessment scores should be compared to baseline scores to determine whether, and what, improvement has been made.

Trained community members, employees, or participants should conduct assessments using a consistent process to ensure that results across jurisdictions and over time are reliable and comparable.

The timing for the follow-up assessment(s) depends on the program’s implementation schedule and should be conducted after changes have been implemented.

Programs may elect to assess and report the implementation and effectiveness indicators in a sample of sites or organizations. If so, a sampling approach should be used that ensures the subset of sites selected are representative of the type of SNAP-Ed setting where one or more PSE changes are being made (MT6).

Alternatively, programs may elect to assess and report the implementation and effectiveness indicators for all sites or organizations where one or more PSE changes are being made (MT6). In this case, programs might want to calculate either the “Coverage of all potential/eligible SNAP-Ed sites” or “Coverage of actual SNAP-Ed sites” similar to the denominators described for indicators MT5 and MT6.

See Appendix C for description of sampling strategies.

N/A

Implementation: Implementation can be measured through ongoing activity tracking and process monitoring systems. Measuring implementation requires a system for documenting changes by organizational site or system. Good and clear definitions of the implementation components are required for consistent and comparable tracking and reporting of multi-component and multi-level interventions.

• While it would be desirable to have examples of good process-monitoring tools for tracking the implementation of multi-component and multi-level interventions, few, if any, are available. Process-monitoring tools are typically internal documents that are rarely disseminated beyond associated programs.

Effectiveness: See the listing of reliable tools for needs assessments and environmental scans in ST5 Readiness appropriate to the eat, learn, live, play, shop, and work domains (e.g., Nutrition and Physical Activity Self-Assessment for Child Care [NAP SACC], School Health Index, School Physical Activity Policy Assessment [S-PAPA], Physical Activity Resource Assessment [PARA], Walkability Checklist – Safe Routes to School).

• While many good and reliable environmental assessment scans are available, few have been validated.
• For some settings or MT6 PSE changes, such as physical activity facilities, no specific assessment tools are available.

Domain
Environmental
Evidence-based education
Marketing activities
Policy
RE-AIM model
Settings
Sites
Supports
Systems

N/A