Context. More than 30 million immigrants and refugees were living in the United States in 2007, and many of them had difficulties getting appropriate health and social services. To address this problem, in 2007, the Robert Wood Johnson Foundation (RWJF) issued a special solicitation seeking ideas for new community-based approaches to health and health care problems faced by immigrants and refugees. The response to the solicitation revealed:
- A need to strengthen the evaluation capacity of community-based organizations that work with immigrant and refugee communities
- Significant interest within these organizations in the issue of intimate partner violence
RWJF defines intimate partner violence (IPV) as actual or threatened acts of physical, sexual, psychological, financial and verbal harm, including stalking, by a current or former partner, boyfriend, girlfriend or someone wanting a romantic relationship.
In response to this need and interest the Vulnerable Populations team at RWJF developed the Strengthening What Works: Preventing IPV in Immigrant and Refugee Communities. The initiative funds organizations to plan and conduct an evaluation of their programs directed at preventing intimate partner violence in the communities they serve. The program also builds the evaluation capacity of organizations addressing intimate partner violence in these communities and it disseminates best practices.
RWJF engaged LTG Associates, an anthropologically-based consulting firm headquartered in Turlock, Calif., to serve as the national program office, providing direction and technical assistance to the initiative.
The right evaluators for the job. RWJF Senior Adviser for Evaluation Laura C. Leviton, Ph.D., suggested LTG Associates to oversee the program. "To me it was pretty clear that we wanted people with an ethnographic orientation, but also practical program experience," she says. "LTG has done ethnographic studies of a variety of programs, often for minority cultures. I knew their work and they have not let me down." Wendy Yallowitz, the RWJF program officer for Strengthening What Works said, "It was important that we partnered with the right group, one that was culturally competent but also apt in addressing various evaluation capacity needs for organizations of all sizes and types."
Nathaniel (Niel) Tashima, Ph.D., and Cathleen E. Crain, M.A., are the founders and co-managing partners of LTG Associates. Each brings strong anthropology education and experience to their work.
Tashima's background is a unique blend of community organizing and professional anthropology. He began his career working in the Asian-American and Pacific Islander communities of California. "On May 1, 1975," he remembers, "I opened a mental health program for arriving Vietnamese at Camp Pendleton, Calif. Saigon fell on April 30th and on May 1st we were in business. We received the very first Vietnamese refugees to come to the mainland. Ever since then, I have had an ongoing relationship with one or more refugee communities."
While working in these communities, Tashima received his bachelors from the University of California, San Diego, masters degree from San Diego State University and a doctorate from Northwestern University, all in psychological anthropology.
Professional anthropology "is a part of the discipline that has stood outside the academic world and pushed envelopes of methodology and activism in ways that more academic colleagues are less comfortable with," he says. "Through the professional anthropology side I've gotten very much engaged in research methodology, particularly for evaluation-looking at cultural and linguistic access issues and how research must be crafted in order to capture the desired information from participants. I've come to characterize this work as bringing the voice of communities into the policy and research evaluation conversation as partners instead of as subjects."
With his background and interests, Tashima says, "Walking into the RWJF work was ideal."
Crain's undergraduate degree, from McMaster University in Hamilton, Ontario, Canada, was in physical and cultural anthropology and her anthropology masters (also from McMaster) concentrated on medical and cultural systems. "I set out to be a physician and was seduced into anthropology by the promise of affecting the shape and functioning of systems," she says. "That's my passion: that systems and programs will work better and serve the people for whom they are intended. This passion has always been about access, appropriateness and quality of health and human services."
Prior to founding LTG Associates with Tashima, Crain ran several nonprofit organizations, was president of a rape crisis center that also dealt with domestic violence and was a substance abuse counselor. She has worked on refugee and immigrant issues for most of her career. "This program could not have been more custom-made for my passions and interests," she says.
The program. As RWJF staff began to plan Strengthening What Works, Leviton, with Wendy Yallowitz' concurrence, suggested the systematic screening and assessment method to identify appropriate projects to fund. The Foundation had used the method in its work in childhood obesity prevention. It is helpful, Leviton says, in situations where there appear to be "widely scattered and not very well understood strategies for dealing with a problem and where you need a methodical way of identifying them and figuring out whether they are likely to work." Leviton describes the process as follows:
- Find as many examples as possible through a nomination process, Web search, word-of-mouth and so on. (Tashima calls this an "environmental scan.")
- Review documentation on each example for logic, plausibility and adequate resources and activities supporting the effort.
- Perform an evaluability assessment on each example to determine whether the project is ready to be evaluated.
Following this method, 10 applicants, from a much larger original pool, received evaluability assessments. In November 2009, eight applicants received grants from RWJF for evaluation support related to their intimate partner violence prevention efforts. The grantees represent diverse immigrant and refugee populations in communities across the United States in Albuquerque, N.M.; Atlanta; Austin, Texas; Boston; Chicago; Oakland and San Francisco, Calif.; and St. Paul, Minn. (For a description of participating organizations, see the Grantee Overview section of the program's Web site.) In some cases, intimate partner violence is the sole focus of the organization, while in others, the agency offers a range of health and social services.
LTG Associates provides technical assistance to grantees as they plan and conduct their evaluations. This assistance includes site visits, conference calls, grantee meetings and a password-protected Web site with an archive of program documents and a venue for grantee discussion and questions. "This is a resource-rich project for the grantees," says Crain. "We intend them to get a lot of our time, have a lot of resources that are accessible and to have opportunities to work with one another."
A key member of the resource team is the project capacity consultant, a local evaluator who works with an individual grantee during the program and, when the program ends, continues the relationship as an ongoing resource to the organization. "The position is an experiment," says Crain, "an idea that we and the Foundation came up with early in the framing of the initiative."
During the first year, LTG staff helped grantees construct logic models, showing the path from their intervention to desired outcomes. The logic models also reflected each organization's theory of change and provided a common understanding of evaluation goals and the means to achieving them. Crain reports that some grantees discovered that "their interventions were slightly out of kilter with their theory of change and logic model and they were able to make adjustments to bring them into line."
They also helped grantees create or refine their data collection instruments, probing them to consider: "Are you asking the questions with these instruments that will get you the answers you're looking for?"
The multicultural and multilinguistic environment presents some challenges. Tashima offers an example: "One grantee designed its surveys in English and is using them with native Bhutanese speakers. A Cambodian woman, whose first language is Cambodian, is doing this with a Bhutanese translator. Ensuring accuracy in the development and utilization of evaluation tools can be very complicated—both good anthropology and good evaluation are needed. You go through two or three languages to get to the community and then come out through a language or two to get back to your evaluation instrument."
Building a culture of evaluation. A major focus of Tashima and Crain's work on the program is bringing "evaluation into the culture of the organization," as Tashima says. "It's a wonderful challenge because not everybody is enamored of evaluation. Most program people would prefer to be doing what they're doing, rather than evaluating. Part of our message is that they don't have to become evaluators themselves, but they have to become critical consumers. With a culture of evaluation it is not dependent upon one person and institutional memory is not lost when that person leaves."
"We don't expect any of these organizations to suddenly blossom as evaluators," says Crain. "Their purpose is to do intervention and education. We want them to learn evaluation and embrace evaluation as a tool that is critical to the improvement of the services to which they're devoted and to the continuous improvement of those services."
Tashima describes the starting point: "When we began the evaluability assessments it became clear that the organizations could say why things worked, but, in some cases, didn't have data to support this. These organizations are well grounded in their communities and they had a way of understanding how their programs operated and an evaluative judgment of their interventions and what they needed to modify and what they could leave alone."
Tashima emphasizes that the interventions being evaluated by the grantees were not created in response to Strengthening What Works, but were ongoing projects that the organizations considered important to their communities. Through participation in the program, he says, "They believe that what they're doing is even stronger because of the evaluation work and that they are able to describe in more concrete ways what the intervention does and how it affects participants."
A core belief: always a student, always a learner. Culture is a complex concept, according to Tashima, especially in refugee and immigrant communities, "where people find themselves in a completely new world and are trying to figure out their way in the various systems they need to navigate." Refugees can find some support from federal and state resettlement programs. Immigrants may have larger communities to tap into but not the governmental support to help them transition. The LTG staff understands these challenges and accounts for them in the initiative.
Crain says: "One of the central tenets of our discipline is that you are always a student, you are always a learner—and as a learner you are a servant to the improvement of those programs and those communities. That learner-servant role is one that we try to practice."
Crain believes that LTG's philosophic base is critical to its work on Strengthening What Works. "The philosophy that we are all participants in the improvement of services that will enrich or better people's lives is an important aspect of what we bring to this, along with our very strong belief that we are partners in a joint endeavor with the grantees, RWJF, LTG and all of the communities that are being served."
"I keep hearing the voices of my early mentors in my work," muses Tashima. "They are saying to me: 'Look for ways of engaging communities. Look for ways of allowing communities to speak to policy-makers and funders. Find a way to become invisible.'"
Connecting with mainstream academia. "The immigrant organizations we're working with," says Tashima, "have said they feel they stand outside the mainstream, and that the mainstream research does not address their communities' needs. They say that their communities are about maintaining the family and maintaining respect for people in a healthy relationship. Mainstream literature seems to say that the woman and children need to be removed from the abusive situation and create a new life. The cultural issues are not addressed by the mainstream literature.
"The grantees were very clear that the academic world was not hearing what they had to say, not listening to their approaches and valuing what community experience was trying to tell the academic world. So we set out turn that into an asset."
Crain elaborates: "One thing we have done is to bring together academics who have useful and valuable information for the interventions. At the last grantee meeting we included people who write extensively about intimate partner violence and about refugee and immigrant communities—people whose vision would be useful, informative and, perhaps, expand the grantees' notion of what academics could bring to them. It's important that there not be an immense gulf and that academics begin to appreciate the value of what is being done in the field, and that those in the field find a way to communicate with some of the potentially good partners on the academic side."
"Practice-based evaluation moves the grantees into an even playing field with the academic world," says Tashima. "Some of the evaluations the grantees constructed and are conducting can be submitted to peer-reviewed journals. Their alternative perspectives and alternative approaches to intimate partner violence can stand up to academic review, and the rigor of the evaluation will be transparent."
Midstream reflection. As of spring 2011, with grantees in the midst of data collection, Crain says, "We are excited about some of the things we're seeing that portend good outcomes and good evidence for the grantees' practices." While it is early to talk about emerging themes, Crain says that the next few months will confirm or deny some possible themes that they are noticing.
Says Tashima, "Going forward we proposed to the Foundation that we organize grantees in learning collaboratives. We'll bring together smaller groups of three or four grantees focused on one topic or process or methodology to talk about how it's being implemented in their situation and where they see it going. They can start their own conversation about how they can support one another. It's a transfer of a culture of evaluation to the grantees so they can be proactive in their use of this and have an immediate network to work with.
"We're hoping that, in this next year, we can bring these learning collaboratives together at least once, give them a launching pad for their own conversation and for them to become self-directed at that point. We have another year with them after this one to see if we can continue this process. I think it has great potential."
External communication will begin, through a variety of media, says Crain, such as "peer-reviewed journal articles, presentations, perhaps town hall-type meetings—whatever we and the grantees figure out is going to take the messages to the greatest number of useful audiences." Tashima notes the growing opportunity of social media. "We're trying to sort out how grantees could take advantage of this and also how we can use this for the learning collaboratives and for educational opportunities for the grantees."
Excitement about potential. The evaluators expect both methodological and programmatic outcomes. Methodologically, says Tashima, "If you follow the initiative from the environmental scans and evaluability assessments, through the technical assistance, development of formal evaluation methodologies that are rigorous, replicable, reliable and transparent and on to a point where grantees develop their own ideas and collaborate with one another, you have an interesting process that we can document and hopefully turn into a best practice. That is part of the exciting methodological process we're looking at."
"Individually the organizations are very strong," Crain notes, "and we think that Strengthening What Works is giving them tools to be even stronger and to demonstrate that strength both to funders and to the larger intimate partner violence arena."
Once the evaluation work is completed and communicated Tashima believes "it has the potential to change the conversation on intimate partner violence and who has a place in that conversation, creating a more equitable partnership among community-based organizations, academics and funders. That would be a wonderful, profound outcome of this initiative. If enough of these organizations collaborate with academic researchers a very different way of looking at the issue and understanding solutions will result."
RWJF perspective. According to Leviton, Tashima and Crain's approach sets LTG Associates apart from many evaluation firms. "They accept the communities as they are," she says, "and honor—and view as an asset—the cultural embeddedness of the programs that they're dealing with. Yet, Niel and Cathleen have a standard in mind for what they want to achieve. I am fully confident that they will expand knowledge about intimate partner violence prevention in these communities to a level that is better for all concerned.
"At the same time they are also building a cadre of community-based organizations that have evaluation experience that can be harvested for use by other initiatives in that community or other efforts of the agency. That is all so different from what we usually do in our evaluation work, that it is just remarkable."
Yallowitz adds, "By building the organizations' capacity to conduct evaluations and share their new knowledge with their communities, the program is not only strengthening each community's resources, but sustaining and improving the organizations' contributions to their communities."