Beneficiary Feedback in the Department's Evaluations - United States Department of State

There has been a growing interest in the development community in the subject of “beneficiary feedback.” Many bilateral and multilateral agencies such as DFID, the World Bank, UNHCR, SIDA and International Development Agency of Norway have embraced the concept and have been grappling to introduce it. UNHCR has also emphasized a beneficiary feedback approach and has conducted evaluations that have systematically sought the views and opinions of refugees and internally displaced persons. International NGOs such as World Vision and InterAction have been pushing for beneficiary feedback. However, the published material on the subject remains extremely limited. Evaluations that have used feedback mechanisms are not publicly available and remain buried in the bookshelves of the sponsoring organizations. Nor have the evaluators who have used them codified their experiences and explained the problems and challenges that they have faced. As a result, it will take time before the issues surrounding the beneficiary feedback approach are adequately conceptualized and tested, with practical guidance provided to practitioners.

There are two reasons why the Department should start a conversation on this subject. First, it should not overlook the ethical principle that the beneficiaries in whose name projects and programs are designed and implemented must have input into evaluations. They should be more than a source of information to evaluators. Second, and more importantly, feedback by beneficiaries can improve the quality of findings, conclusions and recommendations. Beneficiaries can often correct the flaws in findings and recommendations and provide a different perspective that can enhance the quality and credibility of evaluations. The purpose of this brief article is to start a conversation on the subject.

Conceptualizing Beneficiary Feedback in Evaluation

Although the expression is used in development literature, there is no agreed upon definition of beneficiary feedback. Some equate it with sharing evaluation findings and recommendations with beneficiaries to get their responses while others equate it with participatory evaluations in which beneficiaries are full partners in the entire evaluation enterprise. As conceptualized here, it is a systematic approach to give beneficiaries a voice in the evaluation process. The concept of beneficiary feedback can be explained as follows:

Beneficiary Feedback in Evaluations

Arrow increasing in size, lower left "No input from beneficiaries", middle "beneficiary feedback," upper right "participatory evaluations"

We can visualize a continuum from no inputs to full participation of beneficiaries. On the one extreme lie evaluations in which beneficiaries are only providers of information and have no say in the evaluation process. On the other extreme are participatory evaluations in which beneficiaries are equal partners. Beneficiary participation falls between these two extremes. The difference between participatory evaluation and a beneficiary feedback approach is that while in the former beneficiaries are represented in evaluation teams and are therefore engaged in planning, data collection and analysis, in the case of beneficiary feedback there is communication between the two, but the ultimate responsibility for the evaluation rests with the evaluators. Beneficiaries only provide feedback.

A beneficiary approach to development evaluation involves a one-way or two-way flow of information between beneficiaries and evaluators for the purpose of improving the evaluation process, findings and use. It is a structured and systematic approach that cuts across all stages of evaluation- from design to dissemination.¹
Leslie Groves

Who Are Beneficiaries?

Despite its wide use, the term beneficiary has been differently used in development discourse. In fact, those who see poverty alleviation as a “right” question the very use of the term and regard it as patronizing. Some experts would prefer to use the term only for direct beneficiaries while others like to include both direct and indirect beneficiaries. Moreover, the question has been raised as to whether the term should include groups that were not originally targeted in projects and programs but also benefited from interventions. It will take time before a consensus will emerge about the precise meaning of the term. For the purpose of this article, beneficiaries refer to (a) people for whose benefit a project or program is designed, such as refugees, internally displaced persons, patients suffering from HIV/AIDS, vulnerable women, farmers or others (called individual beneficiaries); (b) organizations, governmental or non-governmental, that receive assistance to enhance their organizational effectiveness, performance and delivery systems (called institutional beneficiaries). Examples include media organizations to which assistance is provided to strengthen and consolidate independent media, human rights NGOs that mobilize public opinion for basic civil and political rights or the organizations that are engaged in promoting policies and programs for climate change. In all these examples, while people benefit in the long run, there are no specifically targeted individuals for whose benefit the project or program is launched. Most experts do not treat the second category as beneficiaries in their analysis. This omission is not justifiable in the case of USG foreign assistance programs, however, as the bulk of them focus on institution building.

Beneficiary Feedback at Different Stages of Evaluation

Ideally, all projects and programs should have monitoring systems that gather information about beneficiaries and their perceptions, views and responses to the interventions. Such information can be gathered for both individual and institutional beneficiaries through a wide variety of data collection methods. Even when the information is not comprehensive, it will undoubtedly help managers in improving the performance and impacts of their projects and programs.

Beneficiary feedback can take place at four stages of evaluation, namely planning and designing; data collection; formulation of findings, conclusions and recommendations; and finally dissemination of evaluation reports. We discuss the feedback process for two categories of beneficiaries.

Exponents of the beneficiary approach suggest that at the planning stage, evaluation managers can seek the views and suggestions of individual beneficiaries about the nature and focus of the evaluation and the questions it seeks to answer. They can do it through interviews, group meetings and focus group discussions with beneficiaries. The problem is that this is not a realistic proposition for the Department’s evaluations. As projects and programs are located in foreign countries, evaluation managers do not have the time and resources to visit abroad and discuss the planned evaluations with a select sample of individual beneficiaries. Even the feedback on draft SOWs through written or electronic media is not practical in most cases because it is extremely difficult to reach many beneficiaries. Moreover, it is doubtful beneficiaries will have the time, technical expertise and incentive to provide inputs at the initial stage. The situation is different when there are well organized beneficiary organizations managers can rely on to communicate with beneficiaries at the planning stage.

In the case of institution building projects and programs, evaluation managers can share the evaluation questions and approach with the concerned organizations and seek their feedback. For example, if a project provided technical and financial assistance to a select number of democracy NGOs, a brief document containing the evaluation questions and approach may be shared with them and their comments and suggestions may be considered. In such cases, feedback is possible because the number of beneficiaries is limited and they can be approached through electronic communication. Moreover, given the high likelihood that these organizations focus on evaluating their own performance, they are likely to possess sufficient skills and expertise to make useful and relevant comments and suggestions.

Beneficiary feedback is more plausible at the data collection stage because most evaluation teams meet or should meet a selection of beneficiaries at this point. They usually conduct key informant interviews, surveys, group discussions or focus groups to obtain necessary data, information and opinions. Unfortunately, in most cases it remains a one-way communication; beneficiaries do not have opportunities to ask questions, challenge the focus of inquiry or review the data and information gathered. The beneficiary feedback approach requires that the beneficiaries be able to do so. There should be two-way communication. Evaluators not only seek information and opinions from the beneficiaries but also provide them with opportunities to ask questions and make suggestions.

Between September and October 2014, Cordaid with the assistance of Turkana Natural Resource Governance Hub carried out a qualitative baseline assessment in Turkana County to gain a better understanding of the perceived community effects and benefits of oil exploration. After the preliminary report was drafted, they presented their preliminary findings in three validation workshops. Workshops that were attended by beneficiaries gave opportunities to these stakeholders to critically examine the preliminary findings and make suitable suggestions. Participants also provided recommendations for improvement of community engagement. In total, 177 people participated in the workshops.
Validation of Findings by Stakeholders

How evaluators can obtain beneficiary feedback at this stage is bound to differ from intervention to intervention, by resources, by time available to evaluation teams and by teams’ technical expertise. In the case of institutional beneficiaries, evaluators can conduct qualitative interviews, conduct group interviews or even undertake an informal survey to discuss data collection. For example, a team examining the performance of a project providing training and technical assistance to women entrepreneurs in war-torn societies may call a meeting of the representatives of women getting training or technical assistance and discuss their data collection plans. Similarly, evaluators may call a meeting of NGOs receiving financial and technical assistance to seek advice about appropriate data collection methods. The team should be earnest in its approach and willing to make suitable changes if necessary. As far as individual beneficiaries are concerned, their views can be sought during interviews and group discussions. Questionnaires can include approaches that directly or indirectly provide beneficiaries’ views about the evaluation approach.

The third and most critical stage when beneficiary feedback can take place is when evaluators analyze collected data and information and formulate findings, conclusions and recommendations. At this stage, beneficiary feedback from institutional beneficiaries can be secured by holding workshops and focus group discussions during which tentative findings are presented and discussed. In an evaluation of a USAID media assistance program in Russia, evaluators used workshops as well as group meeting models for a program that had been providing training and technical assistance to nascent media outlets. They convened a meeting of media outlets that received or were receiving USAID assistance and explained that the main purpose of the evaluation was to learn lessons that can be used for improving USAID’s media assistance programs in the countries. The participants, who were highly educated, asked many pertinent questions and made useful suggestions about the sites to be visited and the economic and political contexts that should be taken into consideration. Evaluators also held a workshop at the end of fieldwork and shared findings, conclusions and lessons during which several participants provided useful explanations and suggestions.

Getting feedback from a large number of individual beneficiaries about findings, conclusions and recommendations is much more difficult. Often beneficiaries are not forthcoming. It requires time, patience and communication skills. One possible approach is to hold meetings with a small number of beneficiaries to get feedback, but this needs careful planning. For instance, in an evaluation of an area development project in Malawi, the evaluator met with the farmers about their use of improved variety of corn and the assistance they were receiving from extension workers. Focusing on five key findings and presenting them with the help of a translator, the evaluator encountered language and cultural barriers as well as a reluctance to criticize aspects the program in public. The evaluator might have gleaned more information by relying on a local person to hold the meeting. Evaluation teams may find it easier to seek information from individual beneficiary organizations, however. Evaluators will need to recognize that beneficiaries, as with all stakeholders, are an interested party whose views should be triangulated and validated. Socio-cultural context should be considered while seeking feedback from individual beneficiaries.

The last stage is the dissemination of evaluation reports. Practically, most funding agencies or their implementing partners do not share the evaluation reports with beneficiaries. The exponents of the beneficiary feedback approach view this current practice as unjustifiable. Since the beneficiaries have provided inputs into the evaluation, it is still more important they should be informed of the evaluation findings and even involved in the dissemination process. Therefore, evaluators should not only provide the beneficiaries with the evaluation reports but also seek their suggestions as to with whom else the report may be shared.

Table 1 gives examples of beneficiary feedback at different stages of the Department’s evaluations.

**Table 1:** **Beneficiary Feedback at Different Stages of Department’s Evaluations**
*Phase of Evaluation*	*Type of Beneficiaries*	*Mechanisms of Feedback*
Planning and Designing	Institutional Beneficiaries Individual Beneficiaries	Sharing of questions and evaluation approach through written or electronic communication where possible to seek inputs. Unless representative beneficiary organizations exist, not realistic.
Data Collection	Institutional Beneficiaries Individual Beneficiaries	Information and feedback sought through (a) key informant interviews, (b) surveys and censuses (c) focus groups, and (d) group meetings. As above
Findings, Conclusions and Recommendations	Institutional Beneficiaries Individual Beneficiaries	Feedback sought by holding (a) workshops or group meetings, and (b) written communication. Generally not feasible unless beneficiary organizations exist. In some cases, small meetings can be held with beneficiaries to discuss tentative findings and recommendations.
Dissemination	Institutional Beneficiaries Individual Beneficiaries	Evaluation reports may be shared through electronic or written communication and options. They may be asked to share them with other interested parties as well. If there are beneficiary organizations, reports or summaries can be shared.

Challenges in Using Beneficiary Feedback Approach

As the above discussion indicates getting beneficiary feedback is not simple and straightforward. In fact, the international community has not made much progress and is still grappling with the challenge of practicing it. Therefore evaluation managers should be cautious and consider several factors before attempting to use it.

First, evaluation managers should examine technical feasibility. They should consider whether beneficiaries are identifiable. The USG funds a large number of projects and programs that indirectly benefit people but do not have direct beneficiaries. In such cases, it is unrealistic to use this approach. Only in the case of identifiable individual or institutional beneficiaries is feedback is possible. They should also examine at what stage of an evaluation feedback is realistically feasible. As mentioned earlier, for many projects and programs, the feedback approach is not possible for all stages of the evaluation process. For example, in a large project whose individual beneficiaries are located in a wide geographical area, it is difficult to get feedback for SOWs unless there are beneficiary organizations. Finally, managers should explore whether evaluators with experience in using a feedback approach will be available.

Where evaluations have tight, unrealistic timeframes it may well be impossible to integrate a structured approach to beneficiary feedback in a meaningful manner. A structured approach requires time to prepare the process and design and implement it appropriately. Commissioners and evaluators need to be realistic and honest about what can and cannot be achieved within the resources allocated and will need to decide whether additional time should be allocated to enable a meaningful process of beneficiary feedback in evaluation.
Leslie Groves

Second, evaluation managers should consider the time factor. Evaluations pursuing a feedback approach require more time than other evaluations. For example, if feedback is sought from institutional beneficiaries for a draft SOW through correspondence, it may require an additional two to three weeks. Holding meetings with beneficiaries in the field to discuss evaluation findings and recommendations also requires additional time. It will also require that before leaving a country, the evaluation team frames tentative findings and recommendations that it can discuss with beneficiaries.

Third, the beneficiary feedback approach entails additional costs because evaluation teams put in additional time and effort. Teams have to stay longer in the field and need resources for holding meetings and workshops or interviews. If beneficiaries are scattered in different geographical areas, additional expenses must be covered.

Finally, evaluation managers should consider the utility of the feedback approach and weigh its benefits with reference to time and cost. They should grapple with questions such as: What additional information will be provided? Is it necessary? How will it be used?

Keeping these challenges in mind, bureaus and offices should explore incorporating beneficiary feedback into their evaluations on a pilot project basis if appropriate. For example, the beneficiary feedback approach can be used in short-term training projects in which trainees may suggest questions for evaluation and even provide inputs to the findings and recommendations. Those bureaus and offices interested in using a feedback approach may seek technical advice from the Office of U.S. Foreign Assistance Resources.

To summarize, there is a growing interest in the concept and practice of beneficiary feedback in evaluation. Simply defined, beneficiary feedback involves two-way communication between evaluators and beneficiaries. It implies that beneficiaries should not be a mere source of information and ideas to evaluators but should be consulted during the entire evaluation enterprise. Beneficiaries may include (a) people such as refugees, internally displaced persons, patients suffering from HIVAIDS, vulnerable women, farmers or others; and (b) organizations, governmental or non-governmental, that receive assistance. Beneficiary feedback can be sought during implementation as well as at different stages of evaluation: planning and designing, data collection, analysis (formulation of findings, conclusions and recommendations) and dissemination. The mechanisms for soliciting feedback for the two categories of beneficiaries can be different at different stages of an evaluation. Before adopting a beneficiary feedback approach, evaluation managers should consider factors such as technical feasibility, availability of additional time and resources and overall benefits of the approach.

Beneficiary Feedback in the Department’s Evaluations

Conceptualizing Beneficiary Feedback in Evaluation

Who Are Beneficiaries?

Beneficiary Feedback at Different Stages of Evaluation

Challenges in Using Beneficiary Feedback Approach

Suggested Readings

Beneficiary Feedback in the Department’s Evaluations

Conceptualizing Beneficiary Feedback in Evaluation

Who Are Beneficiaries?

Beneficiary Feedback at Different Stages of Evaluation

Challenges in Using Beneficiary Feedback Approach

Suggested Readings

U.S. Department of State

The Lessons of 1989: Freedom and Our Future