Department of State Evaluation Policy

January 29, 2015


Released by the Bureau of Budget and Planning and the Office of U.S. Foreign Assistance Resources

Contents

A. Evaluation Requirements
B. Evaluation Standards
C. Focus of Evaluations
D. Considerations for What a Bureau Evaluates
E. Types of Evaluations
E. Collaborative Evaluations
F. Evaluation Use
G. Bureau Evaluation Plans
H. Bureau Evaluation Coordinator
I. Evaluation Resources
J. Documenting and Disseminating Evaluation Reports

-------------------------------------------------------------------------------------------------------------------------------

The Department of State is committed to using performance management best practices, including evaluation, to achieve the most effective U.S. foreign policy outcomes and greater accountability to our primary stakeholders, the American people. Evaluation is the systematic collection and analysis of information about the characteristics and outcomes of programs, management processes, and delivery systems as a basis for judgments, to improve effectiveness and inform decisionmakers about current and future activities. Well designed evaluations contribute to a body of evidence for better decision making. A robust, coordinated evaluation function is essential to the Department’s capacity to monitor performance, make critical management and programmatic decisions, and improve management practices and services. It is also necessary to measure results; provide inputs for policy, planning, and budget decisions; and assure accountability.

This policy works in concert with the Department’s evaluation guidance. It draws on the evaluation principles and guidance developed by the American Evaluation Association, the Government Accountability Office and other U.S. government agencies, including USAID.

-------------------------------------------------------------------------------------------------------------------------------

Key Definitions

The following terms are defined to clarify their meaning as used in this policy.

Evaluation: Evaluation is the systematic collection and analysis of information about the characteristics and outcomes of programs, projects, and processes as a basis for judgments, to improve effectiveness, and/or inform decisionmakers about current and future activities. Evaluation is distinct from assessment, which may be designed to examine country or sector context to inform project design, or an informal review of projects.

Program: A set of interventions, activities, or projects that are typically implemented by several parties over a specified period of time and may cut across sectors, themes, and/or geographic areas. A program evaluation often involves an assessment or study of multiple activities that may cut across sectors and/or geographic areas.

Project: A project is a set of planned and then executed interventions identified through a design process, which are together intended to achieve a defined result, generally by solving an associated problem or challenge. The term project does not refer only or primarily to an implementing mechanism, such as a contract or grant. A set of projects makes up the portfolio of a program. A project evaluation is often carried out within the framework of a broader program.

Monitoring: Monitoring provides an indication of progress against goals and indicators of performance, reveals whether desired results are occurring, and confirms whether implementation is on track. In general, the results measured are the direct and near-term consequences of program activities, whereas evaluations document the achievement of outcomes and results and in some cases the value of continuing the investment.

A. Evaluation Requirements

All bureaus and independent offices are encouraged to conduct evaluations to improve their programs, projects, management processes, service delivery, or performance at a rate commensurate with the scale of their work, scope of their portfolio, size of their budget, etc. At a minimum, all bureaus and independent offices should undertake at least one evaluation per fiscal year. However, those who receive and directly manage program funds must conduct evaluations of their large programs once in their lifetime. Additionally, pilot programs should be evaluated before replicating. Bureaus and independent offices directly implementing programs should first and foremost have a program management plan in place. Bureaus and independent offices conducting extensive, multi-year evaluations should consult with the Office of U.S. Foreign Assistance Resources (F) and the Bureau of Budget and Planning (BP) on meeting the threshold requirements of one per year if an evaluation is occurring over multiple years. Bureaus and independent offices should consult the Evaluation Guidance for more specific criteria and guidelines.

Bureaus and independent offices may conduct internal, external, and/or collaborative evaluations. They can conduct evaluations with their own staff without contracting to outside firms or organizations. There are two requirements for such evaluations: (a) the bureau or office has trained evaluation staff with the requisite knowledge and experience commensurate with the complexity of the evaluation proposed, and (b) the evaluation staff is not accountable to the managers of the intervention or the management process to be evaluated.

GAO and OIG evaluations, studies, and audits, which are commissioned by outside entities, cannot be counted as evaluations for the purposes of this policy, but bureaus are encouraged to use such reports to inform the planning of evaluations, as applicable.

B. Evaluation Standards

Bureaus and independent offices should keep in mind the following standards while planning and managing evaluations:

Usefulness: The information, ideas, and recommendations generated by evaluations should serve the needs of the Department in general, and the commissioning units in particular. Evaluations should help the Department in improving its management practices and procedures as well as its ongoing programs and activities by critically examining their functioning and the factors that affect them. Evaluations should also be considered when formulating new policies and programs.

Methodological Rigor: Evaluations should be “evidence based,” meaning they should be based on verifiable data and information that have been gathered using the standards of professional evaluation organizations. The data can be both qualitative and quantitative.

Independence and Integrity: Bureaus should ensure that the evaluators are free from any pressure and/or bureaucratic interference. Independence does not, however, imply isolation from managers. In fact, active engagement of bureau staff and managers is necessary to conduct an evaluation as long as they are not in a position to improperly influence the outcome. This same requirement also applies to contractors and grantees.

C. Focus of Evaluations

The focus of evaluation efforts should be on the full spectrum of activities in the Department. In general, these can be categorized into two areas:

Management Processes and Service Delivery: The Department undertakes a vast array of services and processes that are critical to its functioning. Examples include managing facilities, procuring goods and services, ensuring diplomatic security, providing visas to legitimate visitors, recruiting and training staff, and communicating with legislative bodies and the like. Such services are being evaluated and should be evaluated to improve performance and outcomes. Regional and functional bureaus can profit from the evaluations of their own management structures, processes, or activities.

Specific Programs: In addition to its routine activities, the Department undertakes programs to achieve specific objectives. These programs, broadly defined to include projects, activities, and efforts, are time-bound and have separate resources budgeted for them. They fall under two categories: foreign assistance and diplomatic engagement. Foreign assistance programs are designed to achieve a wide range of objectives such as preventing conflict and stabilizing war-torn societies, rehabilitating refugees and internally displaced persons, supporting human rights and gender equality, and combating hunger, HIV/AIDS, and environmental degradation. Diplomatic engagement programs are designed for engaging other governments and global partners through a variety of efforts, such as public diplomacy; promoting economic and business relations; staffing bureaus, posts, and consulates to engage host country governments and international organizations; providing American citizens services; and providing grants to foreign countries to generate goodwill and friendship.

D. Considerations for What a Bureau Evaluates

The primary consideration in selecting a program, project, management operation, process, or service for evaluation should be the information needs of the commissioning bureau or independent office, which should prioritize those needs and then decide what should be evaluated. In addition, the bureau or independent office should examine if the proposed evaluation is technically feasible, i.e., relevant data can be gathered, fieldwork can be undertaken when necessary, and experts are available to conduct it. Moreover, the bureau or independent office must consider the availability of funds. Evaluations can be expensive, especially those requiring extensive fieldwork overseas. Therefore, if adequate funds are not available, evaluations that require considerable funding must be postponed for the next fiscal year. Finally, bureaus and offices should take into account political sensitivities and constraints. Absent the consent and support of senior officials, evaluations of highly politically sensitive issues should be considered carefully.

For specific guidance on when to evaluate a program, please consult the Evaluation Guidance.

E. Types of Evaluations

Bureaus and independent offices are free to conduct all kinds of evaluations depending upon their needs, resources, and preferences. However, the most common types of evaluations that they conduct, or are likely to conduct, include process evaluations, performance evaluations, outcome/impact evaluations, and special studies. Descriptions of each may be found in the Evaluation Guidance.

E. Collaborative Evaluations

This policy recognizes that bureaus and independent offices do not always directly implement a program. In many cases, they provide funds to other agencies, operating units, or international organizations to carry out a program. In such cases, there are two options: (a) ensure the implementing organization carries out evaluations of programs consistent with the policy and disseminates a final evaluation report, or (b) conduct collaborative evaluations with the implementing partners or organizations.

In general, bureaus and independent offices are encouraged to undertake collaborative evaluations with other bureaus, offices, U.S. government agencies, and bilateral or multilateral partners. A collaborative evaluation is one conducted jointly by more than one bureau, office, agency, or international partner for which a written agreement defining the roles and responsibilities for the collaboration is in place. Collaborative evaluations facilitate mutual learning among participating organizations, as well as reduce the costs to the bureau or office as they are shared among the participating organizations. Under the State Department evaluation policy, collaborative evaluations count as one full evaluation toward the policy’s evaluation requirement for each bureau or office that is party to the agreement.

F. Evaluation Use

Bureaus and independent offices must utilize evaluation findings for making decisions about policies, strategies, program priorities, and delivery of services, as well as for planning and budget formulation processes. For example, evaluation findings should be used to course-correct in interim years of a bureau’s multi-year strategic plan, or to shape that plan initially.

G. Bureau Evaluation Plans

All bureaus and independent offices are required to develop and submit a Bureau Evaluation Plan (BEP). Bureaus and independent offices should consult the Evaluation Guidance for specific timelines on when the BEP is due and for what information the BEP should include.

H. Bureau Evaluation Coordinator

Each bureau and office must identify a point of contact with decision making authority to serve as the Bureau Evaluation Coordinator to ensure that the evaluation function is fully operational and integrated into the planning and decision making process. He or she will serve as the main point of contact in the bureau on evaluation and will interact with BP and F on the bureau’s evaluation efforts and compliance with this policy.

I. Evaluation Resources

While fully recognizing that the Department faces funding constraints, managers should recognize the importance of evaluations and identify resources to conduct the evaluations. As far as programs are concerned, the Department’s grant and contract regulations allow performance monitoring and evaluation as allowable program costs.

J. Documenting and Disseminating Evaluation Reports

Bureaus and independent offices should maintain official copies of completed evaluation reports. Evaluations that are unclassified, but warrant administrative control and protection from public or other unauthorized disclosure, should be marked Sensitive But Unclassified (SBU) in accordance with 12 FAM 541. Classified evaluations must be marked with appropriate classification markings.

Unless reports, statements of work, and summaries of evaluation results are SBU, they will be posted internally, where they will be accessible to all State bureaus and independent offices for discussion and learning. Summaries of evaluations completed with foreign assistance funds will be posted publicly.