|
Program Evaluation
Compared to other SSN programs, the CCTs have stronger M&E systems and invest in rigorously evaluating the impact of the program. Also, many CCT programs undertake regular assessments of key processes (such as compliance with the conditionality, client satisfaction) and of the targeting accuracy of the program.
The organizers believe that this combination of evaluations proves most useful for improving the design and management of the CCT programs. In isolation, impact evaluation provides limited value added for program improvement. The knowledge created is tailored to answer broader questions: whether the program achieves its higher-order objectives and by how much. It rarely tells, however, what program alternative works best, what implementation bottlenecks plague program performance, or how resources or processes can be deployed or reorganized to increase the impact of the program. When completed by other types of evaluation and/or a strong monitoring system, impact evaluation generates richer and more useful information that management can use to improve the program.
This session will introduce the most common types of evaluations used to assess CCT programs: impact evaluation; process or implementation evaluation (or qualitative evaluation); and assessment of targeting accuracy. The second part of the session will examine how these three types of evaluations can be used to produce information relevant for improving the performance of the program.
Questions for discussion
A. Cross-cutting themes: Whether, when and what to evaluate:
- What is the main objective of the evaluation (be it targeting, process, or impact evaluation)?
- What is results-based monitoring (RBM) and how does it differ from an impact evaluation?
- If budget, staff or time is an issue – and it always is! – what types of program evaluations are more useful?
- How often should they be done?
- What questions are appropriate for an external evaluation vs. an internal review?
- How do you organize and manage the evaluation process? (e.g. centralize it or allow different Ministries/units do their own evaluations)
- How to institutionalize program evaluation?
- How can one type of evaluation strengthen the others types?
B. Impact evaluation
- Is program achieving desired/stated objectives?
- How can one be sure that changes in outcomes of interest are “caused” by the program (and not something else)?
- What is the threshold for defining the success of the program? Minimum acceptable impact vs. statistical significance
- Treatment and control groups, estimators and evaluation design
- What types of robust evaluation designs are facilitated by the nature of the CCT programs
- How can process evaluation or targeting assessment strengthen impact evaluation?
C. Process evaluation
- What are the key questions addressed by process evaluation
- Data collection techniques
- Good examples, bad examples
- How can process evaluation strengthen the targeting assessment or impact evaluation?
D. Assessment of targeting accuracy
- How is eligibility for program benefits determined?
- Geographic or Household level (or both)?
- Evaluating targeting accuracy: Inclusion errors (leakage) & Exclusion errors (under coverage)
- Evaluating performance of targeting (impact on poverty relative to other feasible methods and transfer schemes)
- Relationship with the quality control review of eligibility process
- How can a targeting assessment strengthen either impact evaluation or process evaluation?
Panelist
- Emmanuel Skoufias, World Bank
Chair
Rapporteur
- Aline Coudouel, World Bank
- Rapporteur Notes:
Related Materials
|
