Levels of Outcomes
Moore's expanded framework identifies a set of objectives for CME activities ranging from participation (I sat in on a lecture) to improved health of patient communities1:
| Outcome Level | Description | Source of objective data |
| Participation | Number of physician who attend an event | Attendance records |
| Satisfaction | Degree to which an event meets participant expectations | Evaluations |
| Declarative Knowledge | Degree to which participants state what the activity intended them to learn | Pre- and post-tests of knowledge |
| Procedural knowledge | Degree to which participants state how to do what the activity intended them to learn | Pre- and post-tests of knowledge |
| Competence | Degree to which participants can show in an educational setting how to do what the activity intended them to be able to do | Observation in an educational setting |
| Performance | Degree to which participants do what the CME activity intended them to be able to do | Observation of performance in patient care setting |
| Patient health | Degree to which the health status of patients improves due to changes in the practice behavior of participants | Health status measures, e.g. from charts or EHR |
| Community health | Degree to which the health status of a community of patients improves due to changes in the practice behavior of participants | Epidemiological data |
We achieve levels 1 and 2 through standard processes of attendance taking and activity evaluation. Declarative and procedural knowledge can both be demonstrated through basic assessment techniques.
It would be great to ensure competence, but this isn't practical for most education environments. The good news is that if we follow the ACCME process of defining a need based on an identified measurable gap, we can measure both competence and performance with appropriate follow-up. The final levels of patient and community health then flow directly from repeated implementation of the same process to close known performance gaps.
Objectives to Outcomes
Options for Evaluating Activities
Evaluations can be conducted using various methods and techniques:
- The simplest level is satisfaction and the degree to which the learner’s expectations were met
- At the other extreme is research in which the evaluator attempts to employ experimental design so as to gain an understanding of the actual outcomes of the program that can be attributed directly to the education intervention.
- It is very important to identify clearly at which level you wish to conduct the evaluation
- Formats are also varied from using the simple paper/online questionnaire, to performance-based tests
Analyzing Evaluation Data to Drive the Education Process
Methods for analyzing evaluation data can vary, and depend on the goals of the assessment effort. In CME, we're specifically interested in whether our intended outcomes match the actual outcomes of a given even or activity, and if that activity contributes towards the learning and development of it's target audience. The process will include:
- Determining the goals and objectives for learning
- Developing an assessment plan
- Select or design and implement a data collection approach
- Tabulate, analyze, and share findings
- Review the process to see how it's working
Reporting can be complex, or as simple as tallying the responses on a one-page “happiness index” and displaying the results to the activity planning team. A useful report includes clear information presented in a way that satisfactorily answers the evaluative questions in a manner that is as free from bias as possible.
Generally, we include major findings and a summary of recommendations that can be used to plan future events
Once activity has been completed, the evaluations will need to be summarized and a report returned to the Course Director for the activity for use in future planning. If possible and/or practical, a post-activity assessment should be done as a follow-up to the determine if the learner made any changes in their practice in response to their participation in the activity.
How CME can Help
The division of CME has a long track record in the design, implementation, and analysis of evaluations. We can provide background information on the evaluation process, consulting on the design of evaluations for specific activities, and analysis of data to inform planning future activities. We can also improve the efficiency of the process by providing customized online evaluations for your regularly scheduled series or live event.
We can tailor online evaluations for quarterly assessment of regularly scheduled series, more frequent RSS mini-evaluations, individual sections of seminars and conferences, or single-event activities.
Here are some examples:
Short version - could be used on a more frequent basis
Mini-survey - could be distributed for each event