Framework Component
Effectiveness & Maintenance - Organizational Implementation and EffectivenessIndicator Description
This indicator measures implementation and effectiveness of PSE changes. Implementation is defined as the aggregate number of sites or organizations in each type of setting within the eat, learn, live, play, shop, and work domains that report a multi-component and multi-level intervention. Effectiveness is defined as the aggregate number of sites or organizations with improved food environment assessment scores.
Background and Context
Implementation pertains to whether the intervention was delivered with fidelity or as intended and whether the essential elements known to be important to the achievement of positive outcomes were actually and consistently implemented. To be effective, organizational policy changes and environmental supports should be made as part of multi-component and multi-level interventions to sustain the new changes or standards over time. This indicator is "long-term" in that it follows the medium-term adoption indicator; but, in addition, the implementation features are intended to enhance the likelihood of impact and sustainability that derives when PSE change is part of a multi-component/multi-level intervention.
Effectiveness pertains to the achievement of the intended outcomes. Implementation and effectiveness are closely linked since the quality of implementation will directly affect the outcomes achieved. At the environmental level, effectiveness is defined as, and is measured by, improvements in the food environment and/or organizational changes, policies, rules, marketing, and access that make healthy choices easier. Increased environmental assessment scores provide objective, systematic evidence of documented environment-level improvements.
Outcome Measures
| LT5a. | Total number of sites or organizations that implemented a multi-component and multi-level intervention with one or more changes in MT5 (site or organizational adoption of PSE changes and promotion) and one or more of the following additional components:
|
| LT5b. | Total number of components per site or organization, and types of components implemented during the period assessed |
| LT5c. | Number of sites or organizations that made at least one PSE change (MT5) and show improved food environment assessment scores using a reliable and, if possible, valid environmental assessment tool |
What to Measure
Implementation (LT5a-b):
LT5a. Number of sites or organizations that report a multi-component and multi-level intervention with one or more changes in MT5 (PSE changes). Multi-component intervention refers to sites making at least one PSE change (MT5) implemented together with evidence-based education, marketing, parent/community involvement, and/or staff training on continuous program and policy implementation.
LT5b. Total number of components per site or organization, and types of components:
- Evidence-based education
- Marketing
- Parent/community involvement
- Staff training on continuous program and policy implementation
These four components were identified through practitioner input during development of the Western Region SNAP-Ed Evaluation Framework.
Report implementation in two ways:
- With the ideal being implementation of one MT5 change plus all four additional components, record the number of sites or organizations implementing one MT5 change and one, two, three, or all four components. Also note how many sites did not implement a multi-component intervention (i.e., only made change in MT5 but did not implement any additional components).
- Record the number of sites or organizations implementing each type of specific additional component-evidence-based education, marketing, parent/community involvement, and staff training.
Effectiveness (LT5c):
Number of sites or organizations with improved food environment assessment scores using a reliable and, if possible, valid environmental assessment tool. Follow-up assessment scores should be compared to baseline scores to determine whether, and what, improvement has been made. Report date and actual score for each administration of the assessment tool.
Trained community members, employees, or participants should conduct assessments using a consistent process to ensure that results across jurisdictions and over time are reliable and comparable.
The timing for the follow-up assessment(s) depends on the program's implementation schedule and should be conducted after changes have been implemented.
Programs may elect to assess and report the implementation and effectiveness indicators in a sample of sites or organizations. If so, a sampling approach should be used that ensures the subset of sites selected are representative of the type of SNAP-Ed setting where one or more PSE changes is being made (MT5).
Alternatively, programs may elect to assess and report the implementation and effectiveness indicators for all sites or organizations where one or more PSE changes is being made (MT5). In this case, programs might want to calculate either the "Coverage of all potential/eligible SNAP-Ed sites" or "Coverage of actual SNAP-Ed sites" similar to the denominators described for indicators MT5 and MT6.
See Appendix C for description of sampling strategies.
Surveys and Data Collection Tools
Implementation: Implementation can be measured through ongoing activity tracking and process monitoring systems. Measuring implementation requires a system for documenting changes by organizational site or system. Good and clear definitions of the implementation components are required for consistent and comparable tracking and reporting of multi-component and multi-level interventions.- While it would be desirable to have examples of good process-monitoring tools for tracking the implementation of multi-component and multi-level interventions, few, if any, are available. Process-monitoring tools are typically internal documents that are rarely disseminated beyond associated programs.
- While many good and reliable environmental assessment scans are available, few have been validated.
- For some settings or MT5 PSE changes, such as edible gardens, no specific assessment tools are available.
Key Glossary Terms
Domain, Environmental, Evidence-based education, Marketing activities, Policy, RE-AIM model, Settings, Sites, Supports, Systems