Effectiveness & Maintenance – Organizational Implementation and Effectiveness

This indicator focuses on unanticipated or unexpected benefits occurring during the period assessed that accrued incidental to Adoption, Implementation, and/or Maintenance of SNAP-Ed programming conducted by Implementing Agencies. It reports the number, type, and sectors in which the benefits occurred. The benefits may take many different forms, be associated with SNAP-Ed activities conducted in environmental settings or in a broader, multi-sector context at the local, state, territorial, or tribal levels, and take form in the public, nonprofit or business sectors. The benefits will be serendipitous, resulting from new priorities, indirect relationships, or word-of-mouth information that occurred with little direct involvement or intentional planning by SNAP-Ed staff.

Unexpected benefits grow out of efforts in which SNAP-Ed was a key player. Distinguishing questions to determine attribution might be the following: “Was this unexpected benefit a result of SNAP-Ed leadership, initiative, or results?” and “How likely is it that this unexpected benefit would have occurred without SNAP-Ed?” Very sophisticated and longer-term evaluations are needed to establish causation by and attribution to any single effort, such as SNAP-Ed. This indicator is intended to capture distal and indirect results of SNAP-Ed activity. Some unexpected benefits will be activities that only SNAP-Ed partners may conduct, such as advocating for new public policies, or that fund activities or programs that go beyond what SNAP-Ed may do, such as to non-SNAP-Ed eligible population segments or higher-income geographic areas.

Characterizing unexpected benefits will encourage cooperation and collaboration with communities and partners. This leverages the impact of SNAP-Ed, increasing all partners’ effectiveness and efficiency, and builds momentum for broader diffusion of positive programs. The goal is to continually move toward new, positive norms in the public, nonprofit, and business sectors; in environmental settings, organizations, and systems that reach large numbers of people; and in larger geographic areas.

LT11, Unexpected Benefits, differs from LT9, Leveraged Resources, in that the benefits are unexpected or serendipitous rather than planned or intentional, and the outcome measures are primarily descriptive rather than quantitative. Although this indicator is located in the Environmental Settings chapter and includes unexpected benefits in any of the Environmental Settings in LT5–LT8, unexpected benefits may occur as outgrowths of work with partners and champions in Sectors of Influence (LT12–LT19). Both are reported in LT11.

This indicator measures unexpected benefits that occur in any setting, sector, or jurisdiction. Where feasible, the number of people expected to experience the unexpected benefit (reach) may be cited. Outcomes include, but are not limited to:

LT11a.	Government: Number and topics of unexpected new statutes, regulations, and changes in administrative interpretations of governmental policy; new grants or permanent budget allocations; Executive Orders, local ordinances, regional policies; initiatives started by governors, mayors, city councils, county commissions, school boards, regional governments; new or redirected staff assignments toward SNAP-Ed related goals; legal reinterpretations or decisions
LT11b.	Nonprofit: Number and topics of unexpected new organizational initiatives; resource allocations such as “dedicated” personnel, budget, or grant-making; new organizations established to carry forward SNAP-Ed-related initiatives (coalitions or collaboratives establishing themselves as nonprofit organizations, like food policy councils or standing committees within civic organizations)
LT11c.	Business: Number and topics of unexpected new company initiatives; new resource allocations such as “dedicated” personnel or budget; peer leadership within a domain or setting that results in new practices or resource allocation by a chain of stores, fitness centers, or home supply centers
LT11d.	Other multi-sector attributions: Number and types of other unexpected benefits that go beyond those specified in other indicators (MT7–13 and LT12–19), such as higher rates of school attendance, reduced walking/biking injuries, measures of community livability, food insecurity, economic stimulus, or unemployment

Keeping track of unexpected benefits need not be labor intensive or complicated. For example:

• Measurement of unexpected benefits may be quantitative, as with the dollar value of a new grant, or qualitative such as a case study showing how SNAP-Ed activities contributed to a new policy, initiative, or business decision. In some cases, the unexpected benefit may have occurred after SNAP-Ed funding ended because the experience prepared partners to take advantage of new opportunities for which they otherwise might not have been able to qualify.
• Where scoring tools have already been completed, for example at baseline and subsequent to implementation, they can be repeated after SNAP-Ed has stepped back and any change in scores explored using key informant interviews to understand how unexpected benefits evolved from SNAP-Ed.
• Relevant instruments dealing with sustainability may be available in various repositories (NCCOR, RWJF centers, various policy and social science institutes). The “diffusion of innovations” literature and collective impact metrics may provide other ways to approach this question. Other resources can be found:
  • https://www.collectiveimpactforum.org/resources
  • https://www.tamarackcommunity.ca/read 
• Survey instruments or interviews should use metrics that are linked to SNAP-Ed objectives and interventions, thus making an association with SNAP-Ed reasonable.
• In some cases, often in casual communications, third parties independently cite the SNAP-Ed interventions or give testimonials and anecdotes that attribute unexpected benefits to SNAP-Ed and can be used to explore the link with SNAP-Ed as a contributor.

The types and scale of unexpected benefits accrued by site, organization, system, and jurisdiction, and attributed by partners in whole or in part to SNAP-Ed can be expressed through:

1. New, unexpected resources dedicated to maintaining SNAP-Ed initiatives, either supporting former SNAP-Ed functions or addressing newly justified needs, specify (grants, donations, budget allocations, personnel reassignments, community benefit commitments)
2. New, unexpected policies and initiatives with “dedicated” resources (budget, personnel)
3.  Educational outcomes, specify (average daily attendance, academic performance, graduation rates) – see LT15
4. Social outcomes, specify (employment, career advancement, encounters with justice system)
5. Social capital in communities, specify (newly established leadership bodies to address problems identified partially or wholly through SNAP-Ed, community organizations, advocacy bodies
6. Health-related statistics in a population segment, community, county, region, or statewide that reasonably could be seen as co-occurring with significant, comprehensive, and long-term SNAP-Ed intervention activity – see LT17 and Population Results

N/A

One approach to measuring and reporting on LT11: Unexpected Benefits is to utilize Ripple Effects Mapping.

Ripple Effects Mapping (REM)(1–4) is a participatory evaluation method to identify intended and unintended impacts of complex community-based interventions and strategies. REM involves facilitating a collaborative discussion and mapping exercise with a range of stakeholders, including key staff and partners, people who are intended to be beneficiaries of the project, and community members who may be positioned to speak about the project’s impact to a broader audience.

As SNAP-Ed considers participatory methods and equity in evaluation, REM offers several strengths. REM has the potential to uplift and center participant and community members’ voices in the evaluation process and can illuminate which benefits may be most important and impactful to them. REM can also help Implementing Agencies understand connections and mechanisms between benefits. Lastly, REM produces a visual map of a project’s successes, which can be used to disseminate information back and/or give ownership of these data to community members and other key stakeholders.

Michigan Fitness Foundation and UNC Chapel Hill (UNC-CH) have both utilized REM to evaluate SNAP-Ed programming. Michigan Fitness Foundation used REM to understand the holistic effect of SNAP-Ed programs in communities with 37 SNAP-Ed funded organizations, and UNC-CH used REM to understand the unexpected benefits of a community gardens initiative with six SNAP-Ed funded organizations. They provide case studies, sample materials, and lessons learned to consider as you explore this method for programs.

• Evaluation Reports
  • MFF FY 2019 Summary 
  • MFF REM Aggregate Report
• Templates
  • UNC-CH handout with CCF and ripples
  • UNC-CH REM protocol
  • UNC-CH facilitator guide
  • MFF agenda
  • MFF protocol
• Example Outputs
  • UNC-CH Coharie CG REM Final Map
  • UNC-CH Entity Enterprise CG REM Final Map
  • UNC-CH Fairview CG REM Final Map
  • UNC-CH Garland CG REM Final Map
  • UNC-CH Little Bud Thorbs CG REM Final Map

For reporting purposes, consider classifying unexpected benefits using the SNAP-Ed domains (eat, learn, live, play, shop, and work) in the priority intervention settings and sites at local, regional, or statewide levels. This would permit SNAP-Ed programs and partners to more effectively communicate impact with different stakeholder segments.

Additional Information can be found in the followings publications:

• Chazdon S, Emery M, Hansen D, Higgins L, Sero R. A Field Guide to Ripple Effects Mapping. Minneapolis, MN: University of Minnesota Libraries Publisher; 2017.
• Chazdon S. Ripple Effect Mapping: A “Radiant” Way to Capture Program Impacts. Journal of Extension. 2012 Oct;50(5).
• Kollock DH. Ripple Effects Mapping: Ripple Effects Mapping for Evaluation. Washington State University Extension; 2011.
• Olfert M, Hagedorn RL, White JA, Baker BA, Colby SE, Franzen-Castle L, et al. An Impact Mapping Method to Generate Robust Qualitative Evaluation of Community-Based Research Programs for Youth and Adults. MPS. 2018 Jul 17;1(3):25.