Outline
- When to start the process
- initial requirements
- marketing and promotion considerations
- statements
- disclosure
- Managing commercial support
- pre-event considerations
- event considerations and logistics
- post-event considerations
- managing exhibitors
- pre-event considerations
- event considerations and logistics
- post-event considerations
- Tracking attendance
- certificates and transcripts
- Evaluation
- methods
- summary
- Closeout:
- participant list - data needs
- evaluation
- budget
The ACCME Standard
ACCME's criteria #3 and #11 specifically address evaluation. These criteria require that:
Criterion #3:The provider (that's us in CME) generates activities/educational interventions that are designed to change competence, performance, or patient outcomes as described in its mission statement
Criterion #11:The provider analyzes changes in learners (competence, performance, or patient outcomes) achieved as a result of the overall program’s activities/educational interventions.
So, it's important that CME take an active role in design, structure, development, and evaluation of education activities. We're always interested in partnering with you to make sure that your education efforts are effective in addressing the needs of your constituents, and that the content provides them with information that helps close a recognized practice gap.
Given that it's mandated, evaluation is mostly viewed as a compliance activity rather than one that fosters continuous improvement - that's a pity. Used thoughtfully, frequent evaluation can tell us how participants perceive the value of an activity, what their learning experience is like, and what additional needs they have. It can also be used as an engagement tool to facilitate a more active role for learners in the education process.
How to Implement Evaluations
Evaluations should be conducted at the conclusion of an educational activity. For a regularly scheduled series, it is up to the provider to determine when and how the RSS will be evaluated in order to ensure that their RSS meet the compliance expectations of the ACCME. The UAB policy is to conduct evaluations on a quarterly basis, but shorter more regular evaluations can also provide valuable information on the usefulness of activity content to participants.
Options for Evaluating Activities
Evaluations can be conducted using various methods and techniques:
- The simplest level is satisfaction and the degree to which the learner’s expectations were met
- At the other extreme is research in which the evaluator attempts to employ experimental design so as to gain an understanding of the actual outcomes of the program that can be attributed directly to the education intervention.
- It is very important to identify clearly at which level you wish to conduct the evaluation
- Formats are also varied from using the simple paper/online questionnaire, to performance-based tests
Analyzing Evaluation Data
Methods for analyzing evaluation data can vary, and depend on the goals of the assessment effort. In CME, we're specifically interested in whether our intended outcomes match the actual outcomes of a given even or activity, and if that activity contributes towards the learning and development of it's target audience. The process will include:
- Determining the goals and objectives for learning
- Developing an assessment plan
- Select or design and implement a data collection approach
- Tabulate, analyze, and share findings
- Review the process to see how it's working
Reporting can be complex, or as simple as tallying the responses on a one-page “happiness index” and displaying the results to the activity planning team. A useful report includes clear information presented in a way that satisfactorily answers the evaluative questions in a manner that is as free from bias as possible.
Generally, we include major findings and a summary of recommendations that can be used to plan future events
Once an activity has been completed, the evaluations will need to be summarized and a report returned to the Course Director for the activity for use in future planning. If possible and/or practical, a post-activity assessment should be done as a follow-up to the determine if the learner made any changes in their practice in response to their participation in the activity.