Catalyst Evaluation: Baby steps in measuring health equity and systems change

By ShaCoria Winston

No matter your role in community-based programming, it’s likely that you have given some thought to evaluation. How do I show impact? What are the outcomes that best tell the story? How do I involve stakeholders in the process and actually incorporate their feedback? Or you may be wondering, how can I ensure that my data collection tool is measuring all the appropriate indicators? If this is you, please know that you are not alone. The field of evaluation is evolving as we learn new ways to collect data, focus our outcomes and respond to directives.

Evaluation of the Catalyst for Healthy Eating and Active Living program has greatly evolved since 2012. We have come a long way from painstakingly reporting and compiling data in Microsoft Word. As the Catalyst expanded, we transitioned to using an online survey platform for data collection. In 2015, the Catalyst team was trained in Results Based Accountability which shifted our evaluation plan and the way we created action plans for our work in the community. We clarified our evaluation measures, collected data more consistently across counties, and began to add measures related to health equity.

Catalyst team members strategizing at team retreats.

In 2017, we created new logic models that embed health equity approaches into our work and more clearly strive toward systems changes for long-term outcomes. (See our previous blog, health equity lessons, to read about the process for creating the logic models.)

A logic model is a simplified picture or roadmap of a program. It shows logical relationships among resources invested, activities and changes that results. Simply put, it is a way of thinking! We utilized logic models to assist in evaluation efforts in the following ways:

1. Embedding health equity

With guidance from the CDC’s Practitioner’s Guide for Advancing Health Equity along with insight from our work experience  incorporating equity into program strategies, we selected approaches that would drive towards eliminating disparities and achieving health equity. We were able to identify approaches for each strategy by gathering feedback from Catalyst Coordinators working in community. We ensured that the approaches were both replicable and measurable in the communities that we serve. For example, when planning for community gardens, a measurable health equity approach would be to partner with trusted organizations to identify residents to serve as community liaisons.

2. Measuring activities and outputs

We considered activities that would contribute best to the long-term goals of our program. Activities within the logic models were organized to include community supports that drove towards our main functions. The categories included activities that contributed towards a) plans/policies, b) environments and/or collaboration and c) programming/promotion. As a part of the evaluation of each strategy, we hoped to identify common activities and trace their connection to outcomes.

3. Systems change

In each logic model, we identified a clear connection between our healthy eating and active activities and three types of long-term outcomes; a) sustainable policy and environmental changes for healthy eating and active living, b) changes in healthy eating and active living-related behavior and status, and c) equitable food, recreation and active transportation systems. We did this by including achievable activities and outcomes that fed into each other. Without a clear connection between inputs, activities, outputs and outcomes, it is difficult to understand what leads to long-term change.

Catalyst team members strategizing at team retreats.

While the Catalyst program is ending, we learned some important lessons in evaluation. If we were moving forward, we would have built a real-time database to constantly capture data for constant monitoring and feedback.

Evaluation is an iterative process. It will evolve as your program evolves. Community-based programs are dynamic – so it’s important to involve the community when developing your program’s evaluation plan. Be open to learning throughout the process knowing it’s okay if you don’t get everything right at first. You’ll be happy with the results you are able to achieve from an intentional and engaged process!

Stay tuned for our next blog Catalyst Always: Celebrations, transitions and thanks  

 ————

The Catalyst for Healthy Eating and Active Living is a program of the Foundation for Health Leadership & Innovation in partnership with the NC Division of Public Health’s Community and Clinical Connections for Prevention and Health Branch with funding from the Kate B. Reynolds Charitable Trust.

To see our Five Year Summary and Catalyst Video, please visit our webpage.

Contact Jamie Cousins, Catalyst Program Director, at Jamie.cousins@foundationhli.org / (919) 707-5241 for info or with questions