The Purpose of Evaluations
We evaluate activities to see if they are effective in doing what they're intended to do, i.e. do learners achieve the stated objectives of the activity? We also use evaluation data to inform the planning process of future educational events, and programmatic evaluation allows us in the division of CME to gauge how effective we are in achieving our goals and the mission of the unit as a whole.
According to the American Medical Association (AMA), for an activity to be designated as Category 1, an evaluation mechanism is required to assess the quality of the activity and its relevance to stated needs and objectives; however, evaluation is not merely a "compliance" activity, it is an important feedback mechanism that provides data on how well needs are met, how future education activities might be designed, and as a guide for decisions related to the intent of education efforts.
UAB CME policy is to conduct evaluations for regularly scheduled series on a quarterly basis. The timing of those evaluation points can vary, i.e. they can occur at the end of each quarter on a calendar year basis, follow the academic year; occur at some regular midpoint during a quarter, etc. All live activities and enduring must include an evaluation.
The Joint Accreditation Standard
Criteria #2 and #11 specifically address evaluation. These criteria require that:
Criterion #2:The provider gathers data or information and conducts a program-based analysis on the degree to which its CE mission—as it relates to changes in skills/strategy, or performance of the healthcare team, and/or patient outcomes—has been met through the conduct of CE activities/educational interventions.
Criterion #11:The provider analyzes changes in the healthcare team (skills/strategy, performance) and/or patient outcomes achieved as a result of its IPCE activities/educational interventions.
So, it's important that CME take an active role in design, structure, development, and evaluation of education activities. We're always interested in partnering with you to make sure that your education efforts are effective in addressing the needs of your constituents, and that the content provides them with information that helps close a recognized practice gap.
Given that it's mandated, evaluation is mostly viewed as a compliance activity rather than one that fosters continuous improvement - that's a pity. Used thoughtfully, frequent evaluation can tell us how participants perceive the value of an activity, what their learning experience is like, and what additional needs they have. It can also be used as an engagement tool to facilitate a more active role for learners in the education process.
How to Implement Evaluations
Evaluations should be conducted at the conclusion of an educational activity. For a regularly scheduled series, it is up to the provider to determine when and how the RSS will be evaluated in order to ensure that their RSS meet compliance expectations. The UAB policy is to conduct evaluations at minimum on a quarterly basis, but shorter more regular evaluations can also provide valuable information on the usefulness of activity content to participants.
Options for Evaluating Activities
Evaluations can be conducted using various methods and techniques:
- The simplest level is satisfaction and the degree to which the learner’s expectations were met
- At the other extreme is research in which the evaluator attempts to employ experimental design so as to gain an understanding of the actual outcomes of the program that can be attributed directly to the education intervention.
- It is very important to identify clearly at which level you wish to conduct the evaluation
- Formats are also varied from using the simple paper/online questionnaire, to performance-based tests
Evaluations can be implemented on paper, electronically in Qualtrics, or can be built into the activity record in the CME system. Paper implementation typically yields higher returns but are more difficult to aggregate and analyze. Custom-built evaluations in Qualtrics provide the most flexibility and allow for identification of feedback important to specific aspects of an education activity. We can also conduct mid-year or year-end focus groups with a subset of participants to get more in-depth feedback.
Analyzing Evaluation Data
Methods for analyzing evaluation data can vary, and depend on the goals of the assessment effort. In CME, we're specifically interested in whether our intended outcomes match the actual outcomes of a given even or activity, and if that activity contributes towards the learning and development of it's target audience. The process will include:
- Determining the goals and objectives for learning
- Developing an assessment plan
- Select or design and implement a data collection approach
- Tabulate, analyze, and share findings
- Review the process to see how it's working
Reporting can be complex, or as simple as tallying the responses on a one-page “happiness index” and displaying the results to the activity planning team. A useful report includes clear information presented in a way that satisfactorily answers the evaluative questions in a manner that is as free from bias as possible.
Generally, we include major findings and a summary of recommendations that can be used to plan future events
Once an activity has been completed, the evaluations will need to be summarized and a report returned to the Course Director for the activity for use in future planning. If possible and/or practical, a post-activity assessment should be done as a follow-up to the determine if the learner made any changes in their practice in response to their participation in the activity.
How CME can Help
The division of CME has a long track record in the design, implementation, and analysis of evaluations. We can provide background information on the evaluation process, consulting on the design of evaluations for specific activities, and analysis of data to inform planning future activities. We can also improve the efficiency of the process by providing customized online evaluations for your regularly scheduled series or live event.
We can tailor online evaluations for quarterly assessment of regularly scheduled series, more frequent RSS mini-evaluations, individual sections of seminars and conferences, or single-event activities.
Here are some examples:
Short version - could be used on a more frequent basis
Mini-survey - could be distributed for each event
Evaluation Question Bank - an inventory of questions that may be relevant to your activity