By:
Imagine that the hospital you’re working at has decided to train all of the staff on the maternity ward in proper infection control practices. Or that in a rural community, training is taking place to instruct community health workers in newborn care… How do we know if these trainings are effective? How do we draw a clear link between a training for a midwife, and a healthy baby born under her care?
I-TECH’s Dr. Gabrielle O’Malley, a University of Washington professor and evaluation specialist, leans forward emphatically. “It sounds easy, right?” – It isn’t. In fact, O’Malley explains, the question is one of the simplest, and most difficult, faced by the people who fund, design, and implement health care worker training programs worldwide. Because health care workers are trained, and practice, in complex settings, measuring the impact of instruction they receive can be challenging.
Needs are changing; output indicators that count the number of people trained don’t tell us enough. Funders and implementers are interested in outcome indicators – for example: What are the measureable improvements in patient health? The Training Evaluation Framework and Tools (TEFT)1 was developed at ITECH2 at the University of Washington to help with this challenge. TEFT looks at 9 types of training outcomes that can be evaluated at 3 levels – the individual, the organization and the population – and it addresses the complex factors impacting training outcomes to help users refine the scope of the evaluation and decide on design, methods and indicators.
TEFT has recently been released; evidence on its effectiveness is emerging. A case study illustrates how to use the tool: Anil is a well-respected monitoring and evaluation (M&E) officer for Health1, a local non-governmental organization (NGO).3 Health1 provides in-service training for doctors and nurses; they are learning how to use their country’s new HIV guidelines.
They have a formal plan for monitoring and reporting on their training activities that includes counting the number of trainings held and the number of providers trained, and evaluating trainings using pre- and post-training knowledge assessments. For nearly all trainings evaluated, they have been able to show increased knowledge of proper HIV guidelines.
Yet Anil has been hearing more about the need for outcome-level data to assess the impact of programs. The emphasis on outcomes raises some questions: What does this mean for Health1’s monitoring and evaluation plan? How might he modify their current training evaluation to include more information about what happens after training? How does he demonstrate Health1’s training outcomes and impacts within a reasonable evaluation budget?
| Steps |
| 1: Visualize Anticipated Outcomes |
| 2: Identify influential situational factors: supplies, records, policies, etc. |
| 3: Refine scope of evaluation |
| 4: Define evaluation questions, objectives and indicators |
| 5: Design and methods |
| 6: Plan Evaluation |
Anil turns to TEFT’s website to explore these questions. His goal is to go through TEFT’s 6 steps to have a greater focus on outcomes. TEFT helps Anil develop a framework that shows anticipated outcomes at the individual, facility and population level and identifies a chain of causation between the training and its outcomes. He identifies the most useful and feasible method for evaluating the outcomes and in discussion with his program’s stakeholders makes a clear evaluation plan. His framework sheds light on the process.
After the evaluation, the data are used to inform funders’ decisions about the best places to invest their limited capacity-building resources. The results show outcomes of increased CD4 count at the individual and facility level and reduced HIV morbidity. Given strong evidence of program success, their funder is ready to award further funding to the program.
While TEFT won’t answer all your questions or draft your evaluation protocol for you, the tools and resources that make up the 6-step process of planning can help you move your initial broad question – Was the training effective? – into a concrete plan for finding the best answers for your situation. As one user put it, “Using the TEFT steps to walk us through the development of an outcome evaluation plan was a great team activity…The tools are easy to use and promote a thoughtful and implementable evaluation plan.”
1 The TEFT is available free to any user online at www.go2itech.org/resources/TEFT
2 International Training and Education Center for Health
3 Anil and Health1 are fictional entities; any semblance to actual people or organizations is coincidental and not intentional.
| Frances Petracca is a Senior Quality Improvement Advisor for I-TECH and an author of TEFT. The International Training and Education Center for Health (I-TECH) is a center in the University of Washington’s Department of Global Health, in partnership with the University of California, San Francisco. I-TECH has offices in Africa, India, and Haiti, and projects in more than twenty countries. I-TECH’s 700 worldwide staff work with local ministries of health, universities, non-governmental organizations (NGOs), medical facilities, and other partners to support the development of a skilled health work force and well-organized national health delivery systems. |  |
Leave a Reply