The recent tragic incident at the Pulse nightclub in Orlando and the many similar incidents before that continue to bring to the surface issues about mental health (feelings of hopelessness, sense of isolation, and mismanaged anger), broken systems (ability of individuals with mental health issues to purchase assault weapons so easily), sense of community (individuals’ need to belong to a group of people with shared experiences), and intersectionality (people who struggle with multiple social identities, but societal and group norms prescribe which singular identity they should have). The incident was triggered by hatred toward a group of people with different sexual orientations and gender identities; there has been speculation that the perpetrator himself may have been struggling with multiple social identities. The incident unleashed more fear and prejudice against an entire community of people with a different faith. The incident also fueled national debate about gun control, causing some to sharpen their focus on stopping the violence and others to become more adamant about the ability to purchase weapons to protect themselves and their loved ones. The entire situation reflects a complex web of biases, misconceptions, systemic issues, and the human need to belong, that combined, can create inequitable outcomes for certain communities.
Inequitable. Inequity. Equity. Equality. Disparity. Unequal. These terms have become increasingly common in communities and national discussions about fighting poverty, lowering the rates of obesity and diabetes, keeping young men of color in school and out of jail, and assuring that everyone has healthcare coverage. Inequity refers to lack of fairness and justice, while equity means that everyone has fair opportunity to attain their fullest potential. The absence of disparities—health, economic, education, and any other condition—should not be mistaken for equity. Elimination of such disparities is merely one step toward achieving equity. But do we really understand what these terms mean? More important, do we really understand the capacities and circumstances needed to achieve equity, what is it we are trying to achieve concretely, and how do we measure our progress toward the “it”?
Evaluating Equity-Promoting Efforts
As evaluators of equity-promoting efforts, we frequently come across people responsible for implementing these efforts, and they are often overwhelmed by the idea of measuring their outcomes. Indeed, it is not a simple feat, and there is no magical answer. Current advancements in data visualization techniques and Geographic Information Systems (GIS) technology mask the thoughtfulness and perseverance and sometimes even infuriation it takes to evaluate equity-promoting initiatives and how we are making progress toward equity. These tools can support the evaluation approach, but they are not the answer. Instead, the best possible answer lies in: 1) clarifying what equity means for the effort, 2) asking the right questions (i.e., challenging assumptions that reflect the status quo or that well-intentioned initiatives will lead to the desired equity outcomes), 3) using the basic tools of evaluation, 4) managing expectations, and 5) communicating effectively. All five principles must be combined to form the best approach to evaluating equity-promoting initiatives; one principle without the other will not be sufficient.
Clarifying what equity means for the effort concretely. The meaning of equity is often taken for granted, especially among progressive individuals who are committed to promoting it. Consequently, extensive discussion about its meaning, particularly within the context of the effort, does not occur, which in turn contributes to misaligned expectations and inaccurate measures of change. As mentioned before, equity means that everyone has fair opportunity to attain his or her fullest potential. In terms of an equity-promoting initiative, it is important to clarify what the concrete outcomes are that can be observed about fair opportunity, fullest potential, and everyone. It is imperative that all stakeholders clarify the entry point for promoting equity (e.g., local or state government, school district, workforce pipeline) and the type of inequity that leads the work. As mentioned before, individuals are not made of a singular but multiple social identities. However, it is hard enough to reduce the inequity for one population experiencing inequity due to a particularly salient demographic characteristic (e.g., race), let alone several populations with other salient demographic labels. Therefore, stakeholders must be clear about the population for which they expect to see change, so that the evaluation treats the outcomes for that population as intentional, while outcomes for other groups as unintentional, but nevertheless equally important to document and understand.
Asking the right questions. Developing evaluation questions is an essential and universal step in the evaluation process. In an equity-promoting effort, attention must be paid to:
For example, a good evaluation question would be, “To what extent was an equity lens implemented and what conditions were necessary to implement it?” versus “Was equity achieved by implementing the initiative?” Also, youth of color are frequently lumped together and labeled at-risk youth. With such an assumption, the evaluation could end up looking only at negative behaviors and how the initiative improved the behaviors, and neglect to identify positive behaviors, how the initiative helped to reinforce these behaviors, and the role of social capital and community in the youths’ lives. If such assumptions go unchallenged, evaluations will produce outcomes that continue to perpetuate biases about certain populations.
Using the basic tools of evaluation. A strong theory of change and a graphic illustration of how the theory is operationalized and expected to unfold, accompanied by a sound measurement framework (variables, data collection methods, data sources and frequency of data collection) and analysis plan, are essential. The graphic illustration is an especially useful tool to facilitate and push discussions among stakeholders about their vision for the path of change toward equity. The illustration must be explicit about the following:
The measurement framework, as mentioned before, is a matrix that lays out the variables, data collection methods, data sources, and frequency of data collection based on the illustrated pathway of change. It needs to include process and outcome measures, and collection of qualitative and quantitative data, in order to capture the complexities of the change process. The complexities must be documented and interpreted accurately, because sometimes the outcomes reflect a step in the right direction and other times they signal a step away from the anticipated direction for a variety of reasons, such as changes in the political landscape, economy, or people’s mindset about an issue. While this does not necessarily mean that the equity-promoting effort failed, it does require the evaluator and stakeholders to examine what happened, including what went wrong in the initiative’s implementation, or if external conditions such as national, state, or local elections and economic forces created a window of opportunity or put up a barrier that caused stakeholders to temporarily deviate from the original path of change. Quantitative or qualitative data alone will not fully capture these events and their contributing factors and consequences.
Managing expectations. Some federal and private funders underestimate the complexity of efforts designed to promote equity and, consequently, may have unrealistic expectations about the outcomes. These efforts take time to plan, implement, and evaluate, and they typically involve a whole host of organizations from across sectors to untangle and begin to dismantle the structures in place that perpetuate inequity. Evaluators must play an active role in helping funders and implementers have realistic and aligned expectations in order to ensure the use of right measures. For instance, in an initiative designed to promote economic equity, it is not realistic to see the needle move on poverty rates at the population level in three to five years—the typical grant period for community and systems change initiatives. What is more feasible are changes in local government policies and practices that increase access to job training and jobs for a particular group of people. An example of an outcome here would be increased pay, job promotion, or job retention for the intended group due to new skills or certification for a particular skill.
Also, in the evaluation of equity-promoting initiatives, the paradoxes of cross-sector collaboration and the complexity of understanding and dealing with power dynamics are part of the process (see Community Science’s earlier piece on “It Is Time to Take a Look at Collaborating for Equity and Justice”). Depending on how power is used or exploited, conflict should be an expected short-term outcome and therefore measured and documented.
With such complexities, the results are not so conclusive, and definitive studies are not possible. Funders, grantees, and evaluators should establish a process for feedback, learning, and adapting, which takes time and resources. All stakeholders should be clear from the outset about the amount of time and resources they have to dedicate to the process.
Communicating effectively. Finally, communication about the equity-promoting effort and its goals, evaluating findings, and lessons learned is usually left to the program staff and communications staff without any involvement from evaluators or other experts in strategic communications regarding equity. Equity is a complex concept, and if there is not a communications strategy to message it properly or manage the media, equity-promoting initiatives can create backlash, fear, or leave behind a negative sentiment that will make it harder to implement similar initiatives in the future. An effective communications strategy for equity-promoting efforts must take into consideration the precision of the language used to convey the goals and findings (so that expectations are managed appropriately) as well as the sensitivity of bringing attention to the experiences of certain communities without perpetuating biases about them. The communication also has to help people understand the distinction between individual-, institution-, and community-level changes.
Sounds Simple, but Remains a Challenge
After reading the above, you might ask, “Aren’t the above a part of any good evaluation and not just evaluation of equity-promoting efforts?” And you are right. The above principles should be integrated into the evaluation design for any initiative. However, certain professional traditions, norms, and fads challenge their implementation and hinder the advancement of our collective knowledge about equity work. These include:
Here are initial practices that evaluators can adopt to begin to counter the above traditions, norms, and fads:
Ultimately, evaluators have to see themselves as change agents who are responsible for ensuring that their work contributes to attaining equity for all. The above recommended practices will not be easy to adopt if evaluators do not see themselves as holding a piece of the puzzle or having the responsibility to help us all understand what it takes to make progress toward equity.