Framework Component
Effectiveness & Maintenance Refers to the extent to which a learner continues to perform a behavior after a portion or all of the intervention responsible for the initial change in behavior has been removed. - Organizational Implementation Pertains to whether the intervention was delivered with fidelity or as intended and whether the essential elements known to be important to the achievement of positive outcomes were actually and consistently implemented. and EffectivenessIndicator Description
This indicator measures implementation and effectiveness of PSE changes. Implementation Pertains to whether the intervention was delivered with fidelity or as intended and whether the essential elements known to be important to the achievement of positive outcomes were actually and consistently implemented. is defined as the aggregate number of sites or organizations in each type of setting within the eat, live, work, learn, shop, and play domains that report a multi-component and multi-level intervention. Effectiveness is defined as the aggregate number of sites or organizations with improved physical activity environment assessment scores.Background and Context
Implementation Pertains to whether the intervention was delivered with fidelity or as intended and whether the essential elements known to be important to the achievement of positive outcomes were actually and consistently implemented. pertains to whether the intervention was delivered with fidelity or as intended and whether the essential elements known to be important to the achievement of positive outcomes were actually and consistently implemented. To be effective, organizational policy changes and environmental supports should be made as part of multi-component and multi-level interventions to sustain the new changes or standards over time. This indicator is "long-term" in that it follows the medium-term adoption indicator; but, in addition, the implementation features are intended to enhance the likelihood of impact and sustainability that derives when PSE change is part of a multi-component/multi-level intervention.
Effectiveness pertains to the achievement of the intended outcomes. Implementation Pertains to whether the intervention was delivered with fidelity or as intended and whether the essential elements known to be important to the achievement of positive outcomes were actually and consistently implemented. and effectiveness are closely linked because the quality of implementation will directly affect the outcomes achieved. At the environmental level, effectiveness is defined as, and is measured by, improvements in the physical activity environment and/or organizational changes, policies, rules, marketing, and access that make healthy choices easier. Increased environmental assessment scores provide objective, systematic evidence of documented environment-level improvements.
Outcome The desired benefit, improvement, or achievement of a specific program or goal. Measures
| LT6a. | Total number of sites or organizations that implemented a multi-component and multi-level intervention with one or more changes in MT6 (site or organizational adoption of PSE changes and promotion) and one or more of the following additional components:
|
| LT6b. | Total number of components per site or organization, and types of components implemented during the period assessed |
| LT6c. | Number of sites or organizations that made at least one PSE change (MT6) and show improved physical activity environment assessment scores using a reliable and, if possible, valid environmental assessment tool |
What to Measure
Implementation Pertains to whether the intervention was delivered with fidelity or as intended and whether the essential elements known to be important to the achievement of positive outcomes were actually and consistently implemented. (LT6a-b):
LT6a. Number of sites or organizations that report a multi-component and multi-level intervention with one or more changes in MT6 (PSE changes). Multi-component intervention refers to sites making at least one PSE change (MT6) implemented together with evidence-based education, marketing, parent/community involvement, and/or staff training on continuous program and policy implementation.
LT6b. Total number of components per site or organization, and types of components:
- Evidence-based education The integration of the best research evidence with the best available practice-based evidence into education interventions.
- Marketing
- Parent/community involvement
- Staff training on continuous program and policy implementation
These four components were identified through practitioner input during development of the Western Region SNAP-Ed Evaluation Framework.
Report implementation in two ways:
- With the ideal being implementation of one MT6 change plus all four additional components, record the number of sites or organizations implementing one MT6 change and one, two, three, or all four components. Also note how many sites did not implement a multi-component intervention (i.e., only made change in MT6 but did not implement any additional components).
- Record the number of sites or organizations implementing each type of specific additional component-evidence-based education, marketing, parent/community involvement, and staff training.
Effectiveness (LT6c): Number of sites or organizations with improved physical activity assessment scores using a reliable and, if possible, valid environmental assessment tool. Follow-up A minimum of 6 months post-intervention. assessment scores should be compared to baseline scores to determine whether, and what, improvement has been made.
Trained community members, employees, or participants should conduct assessments using a consistent process to ensure that results across jurisdictions and over time are reliable and comparable.
The timing for the follow-up assessment(s) depends on the program's implementation schedule and should be conducted after changes have been implemented.
Programs may elect to assess and report the implementation and effectiveness indicators in a sample of sites or organizations. If so, a sampling approach should be used that ensures the subset of sites selected are representative of the type of SNAP-Ed setting where one or more PSE changes are being made (MT6).
Alternatively, programs may elect to assess and report the implementation and effectiveness indicators for all sites or organizations where one or more PSE changes are being made (MT6). In this case, programs might want to calculate either the "Coverage of all potential/eligible SNAP-Ed sites" or "Coverage of actual SNAP-Ed sites" similar to the denominators described for indicators MT5 and MT6.
See Appendix C for description of sampling strategies.
Population
N/ASurveys and Data Collection Tools
Implementation Pertains to whether the intervention was delivered with fidelity or as intended and whether the essential elements known to be important to the achievement of positive outcomes were actually and consistently implemented. : Implementation Pertains to whether the intervention was delivered with fidelity or as intended and whether the essential elements known to be important to the achievement of positive outcomes were actually and consistently implemented. can be measured through ongoing activity tracking and process monitoring systems. Measuring implementation requires a system for documenting changes by organizational site or system. Good and clear definitions of the implementation components are required for consistent and comparable tracking and reporting of multi-component and multi-level interventions.
- While it would be desirable to have examples of good process-monitoring tools for tracking the implementation of multi-component and multi-level interventions, few, if any, are available. Process-monitoring tools are typically internal documents that are rarely disseminated beyond associated programs.
Effectiveness: See the listing of reliable tools for needs assessments and environmental scans in ST5 Readiness The state of being fully prepared to make a behavior change. appropriate to the eat, learn, live, play, shop, and work domains (e.g., Nutrition and Physical Activity Self-Assessment for Child Care [NAP SACC], School Health Index, School Physical Activity Policy A written statement of an organizational position, decision, or course of action. Assessment [S-PAPA], Physical Activity Resource Assessment [PARA], Walkability Checklist - Safe Routes to School).
- While many good and reliable environmental assessment scans are available, few have been validated.
- For some settings or MT6 PSE changes, such as physical activity facilities, no specific assessment tools are available.
Key Glossary Terms
Domain Categories of settings where people eat, learn, live, play, shop, and work. , Environmental Includes the built or physical environments, which are visible or observable and may include economic, social, normative, or message environments. , Evidence-based education The integration of the best research evidence with the best available practice-based evidence into education interventions. , Marketing activities Marketing Activities, by type, include advertising, in-language, public relations, promotion, and personal sales. , Policy A written statement of an organizational position, decision, or course of action. , RE-AIM model A model to help structure evaluation to answer practical questions about program implementation and sustainability (see https://re-aim.org/). , Settings Types of sites, for example schools, work sites, food stores, and parks. , Sites The physical locations or places where SNAP-Ed activities occur. , Supports Changes in written policies, organizational systems, and the observable (physical or "built") or communications environments that make healthy choices easier and more desirable. , Systems A group of related parts that move or work together within a whole organization or a network of organizations.