The Robert Wood Johnson Foundation (RWJF) Culture of Health Prize (the Prize) honors and elevates communities that are making great strides in their journey toward better health for all. The Prize is also designed to inspire and inform the healthy and equity journeys of other communities The Prize Assessment aims to learn about Prize winners and finalists and their journey toward health and understand if and how other communities are inspired and learning from Prize winners. The Community Science Team will use a mixed methods approach to achieve the Assessment’s goals, including a review of key Prize documents (i.e., application materials, past reports), surveys with Prize and Finalist stakeholders, and key informant interviews with key sectors engaged in health and equity efforts in Prize and Finalist communities. The findings will inform the Foundation’s comprehensive understanding of what catalyzes communities to pursue broad health, well-being and equity initiatives, the extent to which the Prize promotes an ongoing health narrative and action in the community, and how the Prize inspires other communities in their health journeys. (2019-2021)
The de Beaumont Foundation’s Public Health Reaching across Sectors (PHRASES) program was designed to address the need for effective strategies and framing and messaging techniques to help public health professionals effectively communicate the value of public health to potential partners and decisionmakers in other sectors. This increased communication has the ultimate goal of fostering cross-sector collaboration. To this end, PHRASES supported the creation of a tool kit designed to aid public health professionals to communicate public health’s value. The tool kit will be disseminated by 15 public health fellows who will engage their peers in a series of formal and informal presentations of the tool kit and other dissemination activities.
The de Beaumont Foundation contracted Community Science to conduct a process and outcome evaluation that explores the PHRASES dissemination strategy. Specifically, the evaluation will assess the extent to which the tool kit was effective in reaching public health professionals, the tool kit’s perceived value to the field, and the extent to which the PHRASES fellows contributed to the uptake of the tool kit by their peers. This will be achieved by tracking the fellows’ dissemination activities and querying recipients of the tool kits on their satisfaction with the tool kits, perceived usefulness of the tool kits, and intended and actual use of the tools. The results will provide the foundation with key insights regarding gaps in dissemination, required changes or additions to the tool kit, and case studies that highlight uptake and use of the tool kit. (2019-2021)
The National Eye Institute (NEI)’s National Eye Health Education Program designed the ¡Ojo con Su Visión! (Watch Out for Your Vision!) program to help raise awareness among Hispanics and Latinos about the importance of maintaining eye health. The ¡Ojo con Su Visión! program provides culturally and linguistically appropriate education resources that build capacity among community health workers and provides them with resources they can use to educate and disseminate eye health messages to their community members.
NEI, with assistance from contractors, is conducting a process and summative evaluation of the ¡Ojo con Su Visión! program to assess its effect on the target population. Community Science was contracted to provide strategic guidance to the evaluation by reviewing all evaluation materials and to make recommended improvements. To date, improvements have been made to the evaluation’s measurement framework, data collection protocols, and data collection instruments. Improvements for the data collection instruments include revisions to the Spanish translation, a decreased reading level to match the population, and standardized scale items to make them more intuitive for respondents. (2019-2020)
This multiyear project supports the Office of Infectious Disease and HIV/AIDS Policy to address Goal 3 of the National Vaccine Plan, which focuses on developing and disseminating communications that provide accurate, timely, transparent, and audience-appropriate information about vaccines. Achieving this goal is closely tied to efforts to better measure and track public confidence in vaccinations.
The team from Community Science developed and tested a Vaccine Confidence Index to support surveillance activities and advance research on adult immunizations. This required a review of the literature to identify existing vaccine confidence–related measures and items that could be used as the basis for a vaccination confidence–related index. In consultation with a national advisory group, the team identified appropriate constructs, indicators, and existing measures of vaccine confidence. A survey was completed to test the index, which showed racial and ethnic disparities in vaccine confidence for various vaccines. A follow-up study was conducted to explore the findings by conducting focus groups with African Americans and Hispanics to better understand underlying assumptions and barriers to receiving the influenza vaccine. The two studies identified the need to develop specific health messages that address the concerns and barriers experienced by specific racial and ethnic minority groups and the best mode of delivery for those messages. (2017-2020)
The Richmond Memorial Health Foundation (RMHF) launched the Collaboration, Community, and Peer Learning to Promote Health Equity in the Richmond Region initiative with the goal of supporting multi-entity, cross-sector collaboration; meaningfully including community members in projects that promote health equity; increasing understanding of health equity; using an intersectional approach that acknowledges race and racism; and informing RMHF’s learning around effective grantmaking. To this end, RMHF conceived an initiative to support eight community collaboratives to engage in a developmental process in which they improve their understanding of core racial equity concepts, increase their capacity to use a racial equity impact-assessment tool, and execute an equity-focused project that, in part, used a racial equity impact assessment.
Community Science was engaged to design and implement an evaluation and learning process to provide RHMF with insights about the initiative’s training and learning opportunities, grantmaking approach, and collaborative dynamics within the eight collaboratives. Our team not only conducted virtual focus groups and a survey of representatives from the eight collaboratives; we also designed and facilitated virtual meetings to review paths of change and operationalize effectiveness and impact; a webinar on how to use data within the context of racial equity impact assessment; and two in-person peer-learning sessions with grantees.
Among the insights generated by the evaluation and learning process are that nonprofit organizations require (1) consistent, frequent, and long-term technical assistance to help them understand and apply a racial equity lens to their health promotion and services; and (2) assistance to clearly define, practice, and sustain community engagement.
Community Science worked with the National Network of Consultants to Grantmakers (NNCG) to conduct a webinar about using research and evaluation to inform diversity, equity, and inclusion strategies, using RMHF’s case as an example, and also wrote about the experience in NNCG’s newsletter (see https://nncg.org/2019/10/using-research-and-evaluation-to-inform-dei-strategies/). (2017-2019)
Community Science was contracted to assess the reliability of the Success Measures Health Tools and their usability by housing and community development organizations (HCDOs) new to the practice of measuring health outcomes. The Success Measures Health Tools are a set of 65 data collection instruments that were developed to measure health outcomes in ways that are more accessible to HCDOs in terms of language and format. The Robert Wood Johnson Foundation funded this study in conjunction with the Health Outcomes Demonstration Project, a demonstration that provided 20 HCDOs support over a 26-month period to develop and implement an evaluation plan for a program focusing on health outcomes.
The grantees completed two rounds of data collection using select items from the suite of health tools. In this context, we explored the tools’ ability to produce data that were reliable and to produce data with minimal missing values across each of the HCDO sites. We also examined the grantees’ use of and experience with the Success Measures Health Tools. This included which tools grantees found most useful, which tools they found most complex, and what changes might be needed to improve the use of the tools for future HCDOs.
To complete this work, we used a mixed-methods analysis approach — we completed quantitative data analysis to examine the reliability and quality of data produced by the tools and conducted interviews, focus groups, and document reviews to answer questions related to grantees’ use and selection of tools, usefulness of tools for evaluation, and challenges using tools.
Our evaluation findings suggested that the majority of the tools in our sample were reliable and provided good quality data across the grantee sites. We also found that grantees valued the availability of the Success Measures Health Tools. The tools and accompanying technical assistance increased their understanding of the social determinants of health and how to measure health outcomes. In most cases, grantees reported that the tools were a good starting point for these concepts and were planning to continue measuring the health of their populations. Overall, the Health Outcomes Demonstration Project gave grantees an opportunity to explore health outcomes by doing and adding measured value to the already important services the HCDO grantees provided in their communities. (2017-2019)
The Transforming Communities Initiative (TCI) is a five-year effort to implement comprehensive strategies to address obesity, tobacco-free living, and the social determinants of health with the overall aim to improve health outcomes in Montgomery County, MD. The TCI supports the implementation of various public health strategies focusing on policy, systems, and environmental (PSE) changes to improve community-wide health and to prevent chronic disease.
Community Science works with the Institute for Public Health Innovation (IPHI) to implement the local-level evaluation of Montgomery County’s TCI. This includes examination of the overall TCI as it implements a set of coordinated, cross-sector, cross-system PSE strategies to address childhood obesity, tobacco use, and food insecurity; how the individual interventions or strategies are planned and implemented; how effectively they support PSE and health-behavior changes in targeted settings or with targeted populations; and the extent to which collaboration, community engagement and an equity focus improve as a result of the TCI. In addition, Community Science provides evaluation capacity building support to the TCI partners to encourage sustainability of data collection procedures and tools developed for the TCI.
Using qualitative and quantitative methods, Community Science’s team is examining the impact of the initiative through interviews with TCI partners and implementers; surveys with local school wellness councils, food security partners and tobacco coalition members; review of quarterly performance monitoring reports, review of existing data collected by the partners, and tallies of student walking/biking behaviors in elementary schools. The evaluation is helping IPHI and the TCI partners to improve strategy implementation, program monitoring and evaluation practices and sustainability planning. (2016-2021)
Community Science has been conducting a multiyear process and outcome evaluation of a leadership program designed to train the next generation of health equity–focused public health leaders. The study design follows participants through various stages of their training and 12 months after completion of the program.
As a result of this work, the Office of Minority Health (OMH) has trained over 100 students in the last four years who acknowledge being motivated to pursue a career in health equity, solidifying their existing plans to work in health equity and/or clarifying their paths toward a specific health equity–related career. At follow-up, many alumni credit the program with guiding the next steps in their professional and academic careers. The evaluation report has been used to fine-tune recruitment requirements, the content of the curriculum, and the inclusion of mentors and peer mentors as key components of the training program.
Using qualitative and quantitative methods, the Community Science Team examined the effect of the training through interviews with fellows, mentors, and organizations that hosted the fellows for the practicum. Community Science generated useful lessons about the recruitment and eligibility requirements that made the program more targeted and accessible to racial and ethnic minority students, as well as how to make the program more useful for fellows from various levels of education. In addition, the findings assisted OMH in developing materials to market the program to other federal agencies to sustain and expand the influence of the program. (2016-2020)
The National Partnership for Action to End Health Disparities (NPA) was a national initiative to mobilize a nationwide, comprehensive, community-driven, and sustained approach to combating health disparities, led by the US Department of Health and Human Services Office of Minority Health. The mission of the NPA was to increase the effectiveness of programs that target the elimination of health disparities through the coordination of partners, leaders, and stakeholders committed to action.
The goal of the evaluation of the NPA was to determine the extent to which the NPA, by creating multisector partnerships across different levels, was able to call attention to the social determinants of health and inform and support strategies to end racial and ethnic health disparities. Given the multilevel infrastructure of stakeholders involved in implementing the NPA, the evaluation employed a mixed-methods approach to examine this complex national initiative across various sectors and geographic regions. Data collection included stakeholder surveys to assess coalition functioning, interviews with coalition members and those participating in community activities, and extensive review of documents produced by the initiative’s 11 working collaboratives. Community Science provided timely, useful feedback on the progress of the NPA’s implementation and collected lessons learned in building the infrastructure required to maintain a national approach to ending health disparities.
Given that the NPA sought to make systems-level changes across multiple social sectors that address the social determinants of health, the evaluation team used a systems approach to identify components of the NPA that could be effectively assessed and to guide the assessment design overall. This strategy involved the development of impact case studies that illustrate the broader influence of the NPA over efforts to address health disparities at the local, regional, and national levels. The case studies included federal policy change, development of health equity professionals, changes in county and city policies and practices, and development of cross-sector partnerships. (2014-2019)
The federal Office of Minority Health (OMH) supports the NPA’s mission to end health disparities by leveraging private and public partnerships and integrating federal, state, tribal, and local community strategies and actions. The NPA was launched by the Secretary of the Department of Health and Human Services, Kathleen Sibelius, in April 2011. As part of Community Science’s role, we have:
To evaluate the NPA, Community Science is using a cross-case study approach which combines qualitative and quantitative methods to determine the effectiveness of the NPA in aligning (and influencing) federal, regional, state and local policies and activities in order to support NPA’s goals. We are conducting interviews and surveys and reviewing and analyzing archival data to weave together the multiple types of data needed to ensure a comprehensive assessment of the NPA’s implementation and effectiveness. We are also working with OMH’s key partners to extract data about how legislators, businesses, and state health officials have integrated the NPA’s values and strategies into their institutions and efforts. (2009 - 2019)
Community Science worked with the University of Colorado’s Center for Global Health to build its capacity for interdisciplinary graduate training through the development of a fund that will more fully integrate the School of Medicine’s Global Health Track with the Colorado School of Public Health’s Certificate Program in Global Health programs. The development of a cross-school integrated capacity in Global Health will provide students with a broad understanding of global health issues, needs and solutions, practical skills for working on interdisciplinary teams, and competency in appreciating and responding to cultural diversity in health-related settings. Community Science researched suitable funders, developed a budget framework, and wrote a proposal to help accelerate program development efforts. (2010)
The Summit Health Institute for Research and Education, Inc. (SHIRE) asked Community Science to design and conduct an evaluation that documents the program’s implementation and assesses the extent to which the project’s objectives were achieved. Community Science surveyed collaboration partners and conducted focus groups with Ward 8 residents, and submitted an annual report to inform the program’s ongoing improvement. (2009)
In 2006, the Robert Wood Johnson Foundation launched Aligning Forces for Quality, a $300 million national health care quality initiative designed to build the capacity of 14 local communities to collaborate and influence health care quality outcomes. The primary goals of the program are to improve 1) the quality of chronic care in inpatient and outpatients settings, 2) public reporting of performance measures, and 3) consumer engagement. Community Science was contracted to conduct an assessment of the consumer engagement technical assistance provided by the National Partnership for Women and Families. We conducted interviews with grantees, project stakeholders, and local consumer advocates; made site visits to selected grantee communities; and developed an assessment report that helped the foundation improve its approach and strategy for achieving the goals of the Aligning Forces for Quality initiative . (2008-2009)
Community Science was engaged by the foundation in 2007 to evaluate this particular ten-year objective. As part of the initial support, we helped the foundation convene and facilitate an advisory committee to develop a framework for improving patient-provider relations as a means to reducing health disparities. (2007-2008).
At around the same time, the foundation contracted with Community Science to evaluate its Health Literacy in Adult Education Settings initiative. This initiative aimed to infuse health literacy into the curricula of six adult education centers. Community Science conducted interviews center directors; conducted pre- and post-tests of students’ self-efficacy in health and health literacy knowledge; and surveyed the teachers. (2008–2009)
Guided by the framework mentioned above, we also conducted a retrospective and secondary analysis of the foundation’s discretionary grants related to patient-provider interactions. This report is a “live” document; we will update it every year to include new grants made by the foundation.
In 2008, the Connecticut Health Foundation, guided by the framework mentioned above, designed EQual, a two-year effort to assist private physician practices make quality improvements that will eventually lead to better patient-provider interactions. The first of its kind in Connecticut, this initiative is modeled after components of the Improvement Performance in Practice (IPIP) effort in Colorado. Community Science is evaluating EQual as part of our overall role to assess the degree to which the foundation achieves its ten-year objective. (2008-2011).
In 2006, The Colorado Trust funded 14 organizations to improve their cultural competency in order to strengthen their capacity to reduce health disparities. Community Science was engaged to evaluate the initiative, specifically assessing: 1) changes in cultural competency among grantees, 2) the influence of cultural competency changes on grantee interventions and short-term outcomes, 3) factors and conditions needed to bring about positive changes in organizational cultural competency, and 4) grantee progress and accomplishments over time. Community Science continues to evaluate the efforts in the 14 organizations. You can view the final report under Publications/Evaluations/Additional Programs at www.coloradotrust.org. (2006-2012)
Community Science evaluated phase I of the Community Action Grant Program, funded by the Center for Mental Health Services (CMHS), Substance Abuse and Mental Health Services Administration (SAMHSA), Department of Health and Human Services (DHHS). The evaluation assessed the extent to which phase 1 grantees were able to 1) identify an exemplary practice in mental health service delivery or administration and 2) conduct a consensus-building process to build support and commitment among key stakeholders in the community to implement the identified practice. The evaluation, which focused on 80 grants funded between 1997 and 2000 that had completed phase I projects, determined consensus-building outcomes, as well as factors that promoted or impeded consensus building. (2002)
The National Health Services Corps (NHSC) has been bringing health services to underserved communities for more than 25 years. Community Science assisted NHSC and its parent agency, the Bureau of Primary Health Care of the U.S. Department of Health and Human Services, to use NHSC's 25th anniversary celebration as an opportunity to increase the organization's role in eliminating disparities in access to health care and to focus public attention on the issue. Community Science facilitated a participatory strategic planning process and provided technical assistance for this effort. (1998)
The American Stop Smoking Intervention Study Trial (ASSIST), funded by the National Cancer Institute, is the largest public health tobacco-control effort in U.S. history. This initiative supported state and local coalitions in 23 states to control tobacco use through policy and other systemic changes. Community Science facilitated the design of the evaluation, including the development of a logic model, conceptual development and operationalization of systems change measures, and development of preliminary statistical models. We facilitated workgroups of leading scientists, National Cancer Institute staff, and practitioners. We also designed the final evaluation plan and developed an evaluation management plan. (1997)