In order to ensure quality and sustainability of its programs, the Emergency Plan is committed to a continuous cycle of program improvement. With each year of implementation, PEPFAR's knowledge base of best practices and lessons learned grows. These drive funding decisions and adjustments to ongoing programs.
The collection and analysis of high-quality data in underresourced settings is thus a critical challenge. Given the range of challenges faced by health care personnel, it is difficult for them to focus on patient tracking and records management. The weakness of health management information system (HMIS) infrastructures compounds the challenge.
In order to support host nations in addressing these issues, U.S. Government (USG) in-country teams, in coordination with host governments and partners, determine strategic information (SI) activities and priorities for the upcoming fiscal year as the part of the Country Operational Plan (COP) process.
PEPFAR-supported activities at the country level focus on both the availability and the quality of data - both must be assured. Efforts to meet these challenges include:
Sporadic military hostilities gave way to a tense, unpredictable standoff when the AIS was launched in September 2004. Implementation of the survey required overcoming serious obstacles presented by the political-military crisis, which had divided the country into three zones, only one of which was fully controlled by government forces. The challenge of coordinating the implementation of a nationally representative high quality survey in such an environment was assured by an AIS Steering Committee, comprised of host country entities and international partners, with technical leadership provided by a PEPFAR implementing partner. Mapping for sampling and training in data and blood sample collection were completed in the first quarter of 2005, despite large-scale political demonstrations and violence. PEPFAR provided support for in all technical aspects of the survey, including preparation of questionnaires and manuals, training, data processing, and tabulation and analysis plans, even in the northern region where security agents had to be deployed with the technical experts to ensure safe passage.
Data and dry blood spot collection were successfully carried out nationwide between August 4 and October 20, 2005. Analysis of data from the field implementation of the survey is continuing. Despite ongoing tensions and the lack of unified leadership in the country, the response rate of the population in all three zones (government-controlled, rebel-controlled and buffer) was very high. All data were compiled and entered in the database in Abidjan, completing part of the AIS, for which the crisis conditions increased both the price tag and payoff in experience and knowledge gained.
Little is known about the dynamics of HIV/AIDS in crisis conditions, and yet such conditions are fairly common among those countries and populations most affected. The C�te d'Ivoire AIS, which could not have been implemented without Emergency Plan support, will provide high-quality information from such a setting. These data will be invaluable, informing USG HIV/AIDS programming and ensuring cost-effective use of Emergency Plan funding to mitigate the impact of the disease on highly vulnerable populations.
In the context of Emergency Plan support for host nations, coordination with other international partners is central to PEPFAR's strategic information efforts. One of the Three Ones principles for international coordination at the country level (discussed in the chapter on Strengthening Mulilateral Initiatives) is to support one national monitoring and evaluation (M&E) system. International partners are deeply involved in the responses of many nations hard-hit by HIV/AIDS, and uncoordinated efforts have the potential to handicap national responses. In the surveillance arena, the USG participated in and supported regional UNAIDS surveillance trainings on estimating HIV prevalence and incidence. With the World Health Organization (WHO), UNAIDS, and others, PEPFAR has supported development of new surveillance methods, monitoring of antiretroviral drug (ARV) resistance, and improved laboratory quality testing for HIV.
The USG has come together with WHO, UNAIDS, the UK Department for International Development (DfID), the World Bank, and the Global Fund to Fight AIDS, Tuberculosis and Malaria for a series of meetings on data harmonization. Among the results of these productive consultations were the first-ever joint analysis and release of data on treatment results in January 2005, improved estimates of treatment results by international partners (including PEPFAR), and initial steps toward improving estimates of prevention and care results. The organizations have also collaborated to produce coordinated guidance for reporting of future results.
USG strategic information staff have come together with staff from international organizations for joint training in such areas as monitoring and evaluation and management information systems. PEPFAR, along with UNAIDS, WHO, World Bank and Global Fund staff, offered two regional strategic information training meetings in Africa in fiscal year 2005, with the option of further meetings as needed. During these meetings, coordinated country monitoring and evaluation plans were discussed and outlined.
Working to ensure data quality, PEPFAR has worked with WHO and others to issue standards for the transfer of data among systems and data storage and reporting, and has worked with the Global Fund and the Health Metrics Network to develop a data quality diagnostic tool. Standardized data abstraction, storage, and access procedures will enable stakeholders in USG agencies and partner organizations to participate in antiretroviral treatment (ART) program evaluation and improvement. Consultations were held by international agencies and donors, including the USG, on use of HL7 standards in data transmission and common data storage and confidentiality procedures. These data collection standards will allow programs to efficiently transfer and better monitor progress and identify problems in order to refine and adapt implementation strategies. Analyses of these data will inform clinical decision-making, encourage improvement of ART program implementation, and assess impact of the USG's investment in treatment programs on individuals, families, and communities.
Ensuring accountability is predicated on obtaining high quality information on program results. In recognition of this joint objective, PEPFAR has partnered with the Global Fund and the Health Metrics Network to develop a framework for data quality. Three tools are linked together within this framework that target different stakeholders involved in data collection and results reporting: host country governments, grantees and prime partners, auditors, and USG or Global Fund managers. Initial pilots of the data quality harmonization framework and toolkit in 2005 have received favorable reviews from host country partners and grantees, which benefit greatly from standardized and coordinated approaches to results reporting and strategic information systems development.
Development of common indicators, to the extent possible, is another priority area for Emergency Plan strategic information efforts. PEPFAR is working with other international donors to create or improve indicator guides in such areas as treatment, counseling and testing, stigma, most at-risk populations, and orphans and vulnerable children (a joint effort with UNICEF).
The Emergency Plan and other key partners have collaborated within countries to share common data on program results and surveillance. At the international level, international partners have worked with host nations and each other to compare sub-recipients to help eliminate unnecessary duplication. With the Global Fund and UNAIDS, PEPFAR has established a Joint Monitoring and Evaluation Technical Assistance Facility. This facility, housed at UNAIDS, links countries requesting technical assistance with consultants who can assist them in the areas of need. The Global Fund and the Emergency Plan are also jointly supporting work by the U.S. Census Bureau to estimate infections averted in the 15 focus countries and other key countries.
The Emergency Plan has also provided support for the WHO in laboratory surveillance, development of ART patient monitoring and reporting guidance, management information systems standards, and monitoring and evaluation. For example, PEPFAR has supported WHO's Service Availability Mapping project - a census of health facilities that identifies which facility offers which HIV/AIDS services, critical information for international partner coordination.
Joint U.S. Government Planning
USG SI planning is accomplished in partnership between country strategic information teams and headquarters technical staff. Interagency teams in-country must create one SI plan as part of their annual COP submission. Headquarters SI personnel are also members of country core teams, supporting country teams through technical assistance, training, and work with international partners to assure common surveillance, MIS, and monitoring and evaluation guidelines and definitions.
In fiscal year 2005, the USG agencies implementing the Emergency Plan made further advances in integrating their surveillance, HMIS, targeted evaluation, and M&E activities. Building on success in 2005, both headquarters and country USG offices developed updated annual activity plans and joint budgets for strategic information for fiscal year 2006.
An important activity in fiscal year 2005 was the development of a number of SI technical working groups focused on developing standards and supporting country programs. Working groups address the following technical areas:
The Scientific Steering Committee, through its subcommittee on targeted evaluations, also guides headquarters participation in targeted evaluations. Twenty-five targeted evaluations, including some that had previously been initiated, were conducted in fiscal year 2005, and the number is expected to grow in the coming year.
In addition, fiscal year 2005 saw headquarters SI advisors help country teams to address challenges in Emergency Plan implementation, including target setting, COP submission, and results reporting. SI country liaisons helped to ensure submission of high quality data from the field.
A text-only version of following table is available.
In fiscal year 2005, the Country Operational Plan and Reporting System (COPRS) became fully operational. This web-based information system allows for the annual entry and updating of Emergency Plan COPs, annual and semiannual program results, and budget information by the focus country teams. Information can now be rapidly searched and reported by key factors and areas of interest to the program, including partner and partner type, program area, geographic coverage, gender issues, indicator targets and results, and budget.
The reporting burden task force formed in fiscal year 2005, which is discussed in the chapter on Implementation and Management, addressed and resolved many of the SI reporting issues raised by country teams.
As noted in the chapter on Prevention, PEPFAR created a Scientific Advisory Board to review data on the possible protective effect of male circumcision and to provide evidence-based guidance to Emergency Plan programs.
Surveillance Information to Track HIV Incidence, Prevalence, and Mortality
Surveillance information on HIV is important not only for countries to understand the dimensions of their national epidemics, but for assessing the impact of their interventions in response. PEPFAR supports such country efforts as behavioral surveillance surveys, demographic and health surveys (which have an HIV/AIDS module), AIDS indicator surveys, antenatal clinic surveillance, and most-at-risk population surveys.
In Kenya, the USG's investments in surveillance systems and surveys have been critical in producing data to monitor trends in HIV prevalence. Sentinel surveillance in pregnant women has been conducted annually since 1990, and the past year's analysis of the 2004 surveillance data demonstrated a continued drop in HIV prevalence among pregnant women from a high of 13.5 percent in 2000 to a national estimate of 7.4 percent in 2004. Additional USG support was provided in 2005 for the Kenya Demographic and Health Survey, the first national survey of HIV prevalence. Analysis of this survey's data provided further evidence of a decline in HIV prevalence in Kenya from 10 percent to 7 percent among adults 15 - 49 years of age from 1998 to 2003. Sentinel surveillance data were collected again in 2005, and will provide important measures of the potential impact of the USG's contributions to the prevention of HIV transmission in Kenya.
Behavioral surveillance is becoming more closely linked to HIV testing, with those surveyed offered testing, yielding additional data points on HIV prevalence. PEPFAR is also planning sentinel site surveillance to monitor treatment outcomes, as well as HIV-related morbidity and mortality. During 2005, the Emergency Plan drafted a protocol for ARV resistance surveys.
SAVVY (Sample Vital Registration with Verbal Autopsy) is an established best practice to collect sample vital registration information on national populations. In places where vital records systems are weak, SAVVY can improve the ability to estimate the levels and causes of morbidity and mortality through "verbal autopsies" conducted by interviewing surviving household members.
PEPFAR supports development of comprehensive curriculum and training materials to improve tuberculosis (TB)/HIV surveillance in TB clinics. This complements the successful electronic TB registers the Emergency Plan supports, which aid in tracking people living with HIV/AIDS (PLWHA) receiving TB care and treatment.
The USG supported testing of the BED-capture enzyme immunoassay (VED-CIA) to assess overall incidence trends in HIV epidemics. Testing efforts included working to make the assays available, training laboratory staff, and developing protocols to bring the assays to surveillance sites.
In recognition of the need to adapt surveillance strategies to the realities of particular countries, PEPFAR, in conjunction with international agencies, has developed tools and plans for surveillance in countries with concentrated epidemics.
Results of PEPFAR-supported surveillance efforts to date include improvements in HIV baseline estimates, methods for estimating and projecting incidence and prevalence, definition for AIDS case reporting, and methods for ART resistance surveillance.
Program Reporting and Monitoring Mechanisms
PEPFAR's central program monitoring process includes five-year strategic plans and annual COPs that set objectives, and annual and semi-annual reports on documentation of program results. Fiscal year 2005 saw the Emergency Plan's first full cycle of semi-annual program results reporting from the focus countries, and the quality and efficiency of reporting is improving with each new report. Results from the two fiscal year 2005 reports influenced program and budget decisions for fiscal year 2006.
PEPFAR's move to electronic reporting via the COPRS database is streamlining the process significantly, as country teams directly upload information that headquarters personnel can then download and review. The Emergency Plan conducted training and workshops on monitoring and evaluation for strategic information personnel in the focus countries - each of which now has an SI team. PEPFAR is also working to integrate monitoring into staff training curricula on an ongoing basis. In fiscal year 2005, common indicator guidance was developed for USG agencies to use in their results reporting.
All USG partners provide semi-annual program results reports to USG activity managers on progress and challenges. PEPFAR staff also conduct regular site visits to review partner activities and assess the quality of data reported by partners. In South Africa, many new local partners were engaged to rapidly scale up HIV/AIDS programming in 2005. The USG SI team prioritized strengthening the M&E capacity of these partners as an essential component of effective program implementation. Implementing partners all attended one or more M&E capacity building workshops in 2005 and benefited from individualized site visits from M&E experts. As a long term strategy, the South Africa team is leveraging other USG investments in the University of Pretoria, which implements an M&E track in the Masters of Public Health program. Through an innovative internship program, recent graduates of the Masters program are matched with PEPFAR partners that are in particular need of M&E support.
PEPFAR invested in the development of a data quality diagnostic tool as a practical guide for program managers to improve the quality of data collected according to Emergency Plan guidance. Initial piloting of the tool has proven its effectiveness in aiding program managers to generate and manage data for advocacy and program improvement decisions. The USG has leveraged WHO and Global Fund in support of the development of a broad set of data quality tools, with the intent of training host country partners in their use in fiscal year 2006.
The Emergency Plan M&E capacity-building technical working group has being working collaboratively with UNAIDS, the Global Fund, and WHO on the development of guidelines for the evaluation of M&E capacitybuilding in recipient countries. Application of these guidelines is planned for fiscal year 2006, with the identification of best practices for wide dissemination.
The strategic information generated by the MTG is discussed and approved in national consensus forums, involving national and international partners. MTG reports and analyses have been used as the most reliable sources of information about HIV prevalence and demographic impact. National advocacy groups and policy programs use MTG information for advocacy, resource mobilization, monitoring and evaluation, and for programmatic purposes such as estimating the number of people to be covered by specific interventions.
At the national level, the data generated by the MTG is used to draw a picture of the national HIV/AIDS situation, and has been used to inform key policies and programs to address the challenges of HIV/AIDS. This partnership - Emergency Plan funding, U.S. technical support, and strong leadership of Mozambican experts - has created a mechanism for the analysis and dissemination of HIV prevalence data and demographic impact. USG support for the work of the MTG has helped to provide the Mozambican government and civil society with a reliable and accurate source of data generation and analysis. In order to sustain and build capacity for analyzing and using improved data at more local levels, the national MTG has begun setting up provincial technical working groups that will help replicate the process throughout the country, broadening the availability and use of high quality strategic information in Mozambique.
Health Management Information Systems
Emergency Plan HMIS support pursues four key objectives:
HMIS developments in fiscal year 2005 include the application of USG and other health reporting technologies in developing country reporting and patient medical records monitoring systems. Technologies being piloted include smart cards, cellular phone reporting systems, and integrated patient management and prescription electronic records. Software tools for laboratory information systems have been inventoried, and PEPFAR has increased its use of geographical information systems. PEPFAR support for strategic information tools to help host nations improve clinical care management is described in the chapter on Building Capacity for Sustainability.
Examples of Emergency Plan-supported HMIS enhancements in 2005 include the following:
During fiscal year 2005, programs of the Emergency Plan were audited by the Inspector General (IG) of the Department of State, as well as the Inspectors General of other USG implementing agencies such as USAID, the Peace Corps and HHS. For example, the Department of State's IG inspected posts in Botswana, Malawi, Zambia and Zimbabwe. USAID completed audits of Emergency Plan programs in Ethiopia, Rwanda, Kenya, Zambia, Haiti, and Uganda, and the Peace Corps completed audits in Uganda, Kenya, Namibia, Mozambique and Zambia.
In addition to its data and financial auditing efforts, the Office of the U.S. Global AIDS Coordinator (OGAC) has engaged an independent contractor to conduct a special Program Audit to assess implementation of ABC prevention activities by Emergency Plan grantees and subgrantees.
The Emergency Plan has also funded the MEASURE Evaluation Project to provide:
These products will help PEPFAR develop systems and processes that contribute to long-term, sustainable, high-
quality HIV/AIDS monitoring and evaluation capacity in host nations.
Unlike many reporting systems, TRACnet accepts both phone and internet-based data entry. In Rwanda, internet access is limited, and an internetonly system would limit the reporting capabilities of specific centers. The fact that 85% of users input data via phone illustrates the importance of phonebased reporting capabilities. TRACnet has been deployed in 50 out of 53 health facilities offering ART in Rwanda, accounting for 95% of all ART patients.
The benefits of working with TRACnet to date include: allowing national decision-makers to quickly analyze and respond to time-sensitive information; enabling improved planning and response to critical needs down to the facility level; and tracking key program indicators to identify trends and view program impact over time.
In the future, TRACnet will be expanded to include prevention of mother-to-child transmission and counseling and testing modules, and will offer laboratory features to more sites. TRACnet will be interfaced with clinical information systems and will potentially be integrated with the National Health Information System.
The Emergency Plan is also the subject of several reviews by external entities, and PEPFAR is working with these reviewers to ensure successful evaluation and use of their findings for program improvement.
The Government Accountability Office (GAO) is reviewing HIV/AIDS prevention under PEPFAR. This review is being done at the request of the U.S. House of Representatives' Committees on International Relations and Government Reform. The review addresses PEPFAR's strategy for HIV/AIDS prevention, its interpretation and implementation; obligation, expenditure, and allocation of prevention funding; and the award process for implementing partners. During fiscal year 2005, GAO representatives traveled to four focus countries and regularly discussed prevention efforts with senior staff members of OGAC.
Congress has also mandated the U.S. Institute of Medicine (IOM), a non-governmental entity, to evaluate Emergency Plan implementation. The task of the IOM Committee for the Evaluation of PEPFAR Implementation is to examine a variety of measures of program success, in order to provide constructive information to the U.S. Congress. Evaluation measures are being derived from a variety of sources, including baseline information and information on activity design at the point of funding decisions, at the point of implementation, and at intervals thereafter. During fiscal year 2006, members of the Committee will conduct site visits to focus countries. In fiscal year 2005, the IOM Committee produced a report, entitled Healers Abroad, that provided recommendations relating to potential involvement of American volunteers in PEPFAR activities.
Identifying Best Practices
PEPFAR has established an SI Technical Working Group to guide improvements in this area. Under discussion for fiscal year 2006 is a shared USG staff website that will allow implementing agency personnel across all countries to collaborate with one another and share best practices.
Because of the Emergency Plan's focus on producing results through sustainable programs of high quality, identifying evidence-based best practices is a core principle. Determining key evaluation questions, monitoring methods and results, and evaluating programs and activities on the ground are critical elements in bringing about continuous program improvement.
Many promising proposals for targeted evaluations have been considered by the subcommittee on targeted evaluations of the PEPFAR Scientific Steering Subcommittee. This body of USG program implementers, evaluators, and research scientists conducted technical review of proposed targeted evaluations for fiscal year 2006, and its input was reflected in funding decisions.
As noted above, PEPFAR is also considering the possibility of joining with other donors to establish a centralized mechanism for technical assistance to countries implementing monitoring and evaluation.
The Second Annual Field Meeting on the Emergency Plan in Addis Ababa, Ethiopia, included many presentations on information collection and serialization by country teams, affording another important opportunity for dissemination of best practices to in-country teams as well as to host governments and other partners represented at the meeting. Key
Challenges and Future Directions
Many of the countries where PEPFAR is at work have historically suffered from weak health information systems, and thus have few personnel trained in the area. The SI challenges of these under-resourced nations remain immense.
Disruptions to national health systems include major setbacks to developing nations' efforts to monitor and evaluate programs, as these activities are often the first things to be abandoned during emergencies. During fiscal year
2005, Haiti and C�te d'Ivoire experienced difficult challenges due to natural disasters and civil unrest, complicating the reporting task of in-country teams.
Attribution of results of upstream Emergency Plan program support (as described at the end of the Prevention, Treatment, and Care chapters) for national programs is an ongoing challenge. There is typically a lack of USG presence at the sites where the upstream support is being put to use, making PEPFAR dependent on host government reports. In addition, the fact that Emergency Plan support for national efforts is provided in close collaboration with other donors makes attribution of results especially difficult. PEPFAR is working with host governments to refine attribution of results in these contexts.
Another key challenge is posed by the new joint USG/WHO ARV guidelines, which provide medical records standards. The Emergency Plan is working to ensure technical assistance and training in medical records systems as this change takes place.
As noted above, ensuring the data quality in resourceconstrained settings remains difficult, and PEPFAR is addressing this issue by providing training for country teams and partners on data quality tools.
PEPFAR has continued to work to build systems incountry for partner accountability and reporting. Some host nations have embraced database approaches analogous to PEPFAR's COPRS, and the USG is working with additional countries to facilitate their adoption of similar tools.
Another challenge in fiscal year 2006 will be the implementation of results reporting by a greatly expanded group of country teams - those in nations beyond the focus countries that receive more than $1 million in PEPFAR support.