I. INTRODUCTION
The following Department of State Fiscal Year (FY) 2022 Annual Evaluation Plan was developed consistent with section 312(b) of the Foundations for Evidence-Based Policymaking Act (P. L. 115-435) (Evidence Act) and OMB guidance Memorandum M-19-23.
II. PROCESS
The Department analyzed evaluations conducted thus far for FY 2018-2022, determining that they align with the Joint Strategic Plan (JSP) strategic goals and objectives, and identifying areas to strengthen evaluation coverage. The Department also reviewed bureau evaluation plans for FY 2020, 2021, and 2022, using the Evaluation Registry (foreign assistance) and Evaluation Management System (diplomatic engagement), to identify ‘significant evaluations’ that may fill potential gaps in covering specific objectives. The FY 2018-2022 JSP can be found here.
The following chart presents a crosswalk of the JSP goals and objectives against the number of evaluations conducted during FY 2018 and FY 2019 (159 total). If the number of evaluations underneath a particular sub-objective equaled less than five percent (or eight in number) of total evaluations, that sub-objective was deemed to have a gap in evaluation coverage (denoted in the fourth column). If an evaluation is planned for FY 2022 that addresses one of those gaps, it has been included in the fifth column as an anticipated contribution to the pool of evidence.
The FY 2022 evaluation plan relied on an analysis of FY 2018 and 2019 completed evaluations, given the limitations of relying on FY 2020 and FY 2021 planned and ongoing evaluations. Including these ongoing and planned evaluations does not significantly alter the assessment of gaps, only showing that some gaps may be closing.
As the Department continues to implement the Evidence Act, it will be positioned to use the critical learning needs highlighted by the learning agenda to determine where evaluations can fill evidence gaps, and thus which evaluations are significant. While the learning agenda is under development, the FY 2022 evaluations identified in this plan will contribute to the totality of evidence related to the corresponding JSP sub-objectives. The FY 2022-2026 JSP is scheduled for completion in February 2022.
| Goals | No. of FY18/19 evaluations[1] (n=159) | No. of FY20/21 evaluations[2] (n=84) | Does a gap exist based on FY18/19 data? | Anticipated FY22 Contribution |
|---|---|---|---|---|
| Goal 1: Protect America’s Security at Home and Abroad
18% of all evaluations FY18-19 |
29 evaluations total | 30 evaluations total | ||
| 1.1. Counter the Proliferation of Weapons of Mass Destruction (WMD) and their Delivery Systems. | 4 evaluations | 5 evaluations | Yes | Potential Learning Agenda focus |
| 1.2 Defeat ISIS, Al Qaeda and other transnational terrorist organizations, and counter state-sponsored, regional, and local terrorist groups that threaten US National security interests. | 14 evaluations | 2 evaluations | No | N/A |
| 1.3 Counter instability, transnational crime, and violence that threaten US interests by strengthening citizen-responsive governance, security, democracy and rule of law. | 2 evaluations | 21 evaluations | Yes |
Outcome Evaluation on Implementation of Child Drug Use Treatment Protocols in India Effective Community Organizing and Mobilization: Three-Part Evaluation Series |
| 1.4 Increase capacity and strengthen resilience of our partners and allies to deter aggression, coercion, and malign influence by state and non-state actors. | 8 evaluations | 2 evaluations | No | N/A |
| 1.5 Strengthen US border security and protect US citizens abroad. | 1 evaluation | 0 evaluations | Yes | Potential Learning Agenda focus |
| Goal 2: Renew America’s Competitive Advantage for Sustained Economic Growth and Job Creation
4.4% of all evaluations FY18-19 |
7 evaluations total | 18 evaluations | ||
| 2.1 Promote American Prosperity by advancing bilateral relationships and leveraging international institutions and agreements to open markets, secure commercial opportunities, and foster investment and innovation to contribute to US job creation. | 1 evaluation | 9 evaluations | Yes | Potential Learning Agenda focus |
| 2.2 Promote Healthy, educated productive populations in partner countries to drive inclusive and sustainable development, open new markets and support US prosperity and security objectives. | 4 evaluations | 8 evaluations | Yes | Evaluation of Brazil’s Youth Ambassadors and English Immersion Camp |
| 2.3 Advance US economic security by ensuring energy security, combatting corruption, and promoting market-oriented economic and governance reforms. | 1 evaluation | 1 evaluation | Yes | Potential Learning Agenda focus |
| Goal 3: Promote American Leadership through Balanced Engagement
34% of all evaluations FY18-19 |
54 evaluations | 7 evaluations | ||
| 3.1 Transition nations from assistance recipients to enduring diplomatic, economic, and security partners. | 16 evaluations | 0 evaluations | No | Potential Learning Agenda focus |
| 3.2 Engage international fora to further American values and foreign policy goals while seeking more equitable burden sharing. | 14 evaluations | 0 evaluations | No | Evaluation of Child Protection for Refugee Children in Urban Settings in Africa (also 3.4) |
| 3.3 Increase partnerships with the private sector and civil society organizations to mobilize support and resources and shape foreign public opinion. | 23 evaluations | 3 evaluations | No | |
| 3.4 Project American values and leadership by preventing the spread of disease and providing humanitarian relief. | 16 evaluations | 4 evaluations | No | Evaluation of Child Protection for Refugee Children in Urban Settings in Africa (also 3.2) |
| Goal 4: Ensure Effectiveness and Accountability to the American Taxpayer
43% of all evaluations FY18-19 |
69 evaluations | 29 evaluations | ||
| 4.1 Strengthen the effectiveness and sustainability of our diplomacy and development investments. | 36 evaluations | 10 evaluations | No | N/A |
| 4.2 Provide modern and secure infrastructure and operational capabilities to support effective diplomacy and development. | 15 evaluations | 6 evaluations | No | N/A |
| 4.3 Enhance workforce performance, leadership, engagement and accountability to execute our mission efficiently and effectively. | 37 evaluations | 9 evaluations | No | N/A |
| 4.4 Strengthen security and safety of workforce and physical assets. | 0 | 4 evaluations | Yes | potential Learning Agenda focus |
|
[1] Evaluations completed in FY18/19 [2] Evaluations that are completed, on-going, or planned for this time period are included in this count. The count of evaluations included here may over- or under-represent the true count of evaluations that will be completed in that time period as a final data collection for FY 2020 will not take place until the fall/winter of 2020 and for FY 2021, that collection will not take place until the fall/winter of 2021. |
||||
III. SIGNIFICANT EVALUATIONS
The evaluations listed below are significant to the Department because they address gaps in evaluation coverage across priorities as identified in the joint strategic plan or they answer a key question for programming moving forward. For each evaluation that addresses a gap or an information need, the corresponding priority area (JSP goal and sub-objective) are listed. They are expected to be completed in FY 2022.
Outcome Evaluation on Implementation of Child Drug Use Treatment Protocols in India
Priority area addressed: Strategic Goal 1, sub-objective 1.3
The Department is supporting a five-year outcome evaluation study to evaluate the efficacy of the Child Intervention for Living Drug Free (CHILD) curriculum to treat substance use disorder in children under 12 years old. The evaluation has been implemented since 2017 in collaboration with the University of North Carolina and the Social Promotion for Youth and Masses (SPYM) – a local non-governmental organization in India, through data collections and motivational interviews. The Department expects to evaluate 1) how to alter intervention techniques to accommodate the child’s level of cognitive and emotional development; 2) how to effectively translate theoretical constructs into hands-on practices for use in the field; and 3) how to focus on pharmacology as a part of a holistic and systems approach to managing substance use disorders in children. Possible challenges to the evaluation may include low subject recruitment or the lack of reliable drug tests. The evaluation, once completed, will be used to strengthen the CHILD curriculum, which will then be peer reviewed and disseminated on a global scale as a part of the universal treatment curriculum. This evaluation also contributes to information needs in sub-objectives 2.3, 3.1, and 3.2, covering economic security, transition assistance, and burden sharing.
Evaluation of Brazil’s Youth Ambassadors and English Immersion Camp
Priority area addressed: Strategic Goal 2, sub-objective 2.2
This evaluation will study the extent to which the Youth Ambassadors (YA) program, as implemented by PAS Brazil, influences the attitudes, awareness, and actions of its participants and their close social network. It will examine both groups through surveys, interviews, and focus group discussions whether (a) the program causes changes in knowledge, skills, and attitudes that are aligned with program goals; (b) whether the program causes increase in participation or creation of a social impact projects; (c) whether the program participants share their experience, and to what extent, within their own social network. The findings from the evaluation will be used to improve future program design, to inform the next participants’ selection and to adjust the program’s monitoring tactics. The primary audience for this evaluation are program staff in PAS Brazil and policy makers across posts.
Refugee Child and Youth Protection in Urban Areas in Africa
Information need addressed: Strategic Goal 3, sub-objectives 3.2 and 3.4
This evaluation will look at international organization and non-governmental organization programs, State Department, and international organization and non-governmental organization advocacy efforts for protection of refugee children. Evaluation questions may include: (1) Is Department-funded assistance for child protection reaching those most in need of assistance in African urban areas, particularly adolescents and young adults? (2) Which Department-funded interventions aimed at refugee children and youth, implemented through both NGOs and IOs, have been most − and least − successful, and why? (3) What are lessons learned and best practices for the Department to follow? and (4) How well do funded activities support the goals and objectives outlined in the implementing or managing bureau’s Functional Bureau Strategy? This will be a mixed methods evaluation utilizing both new and existing qualitative and quantitative data. Anticipated challenges include locating and interviewing beneficiaries without putting them at risk and assessing the impact of Department-funded humanitarian assistance implemented through international organizations who have multiple funders. Unless doing so creates security or sensitivity concerns, the final evaluation report will be posted on public websites, and findings from the evaluation will be used to inform program design as well as funding and policy decisions.
Effective Community Organizing and Mobilization: Three-Part Evaluation Series
Priority area addressed: Strategic Goal 1, sub-objective 1.3
This evaluation will seek to understand the effectiveness of community organizing and mobilization in varying contexts through a series of three targeted iterative case studies of projects. It will seek to answer one overarching question: What makes for effective organizing and/or mobilization in democracy and rights-based programming, particularly in the following spaces: freedom of assembly and association, labor rights, and offshore programs in closed and closing environments? The objectives of this evaluation include: understanding the factors that contribute to effective community organizing and/or mobilization in democracy and rights-based programming in the three aforementioned spaces; determining the results and objectives realized by activists and mobilized groups; and exploring whether programmatic approaches led to sustainable effects. Existing data will be utilized (partner quarterly reports, performance indicator tables, evaluations, etc.) in addition to new data collected through interviews, focus group discussions and surveys. Content, thematic, and constant comparative analyses may be used. There may be challenges reaching project participants and implementing partner staff for the projects that have ended. Travel to certain countries may be limited due to the ongoing health crisis. Findings will be disseminated to other Department bureaus, implementing partners, and peer organizations, and through a web platform for the broader democracy and rights community. This evaluation will also provide information on sub-objective 3.3, increasing partnerships with the private sector and civil society organizations.