Objective: Online curricula are used increasingly for educating physicians, and evaluating educational outcomes can help improve their effectiveness. It is unknown how specific educational outcomes associate with each other among learners using online curricula. We set out to study how two educational outcomes, learner satisfaction and knowledge, and the learner’s year of training and training hospital, were associated with one another among learners accessing a widely used online curriculum.
Methods: Using data from the 2006–2007 academic year, learner satisfaction was compared with pretest knowledge, posttest knowledge, changes in knowledge, module topic, year of training, and training hospital among 3229 residents at 73 internal medicine residency training programs. A multivariable model was used to calculate the odds ratio of learner satisfaction relative to changes in knowledge.
Results: Module topic, year of training, and hospital type were associated with learner satisfaction. Second-year residents were more satisfied with training modules (mean rating 4.01) than first- and third-year residents (mean ratings 3.97 and 3.95, respectively; P < 0.05). Learner satisfaction was greater among community hospital residents than university hospital residents (mean rating 4.0 vs 3.92; P < 0.05). Learner satisfaction was greater in residents with high pretest and high posttest knowledge (P < 0.05). In multivariate analyses, greater gains in knowledge were associated with greater learner satisfaction (P < 0.05).
Conclusions: Greater learner satisfaction is associated with greater baseline knowledge, greater knowledge after completing a curriculum, and greater improvement in knowledge while enrolled in a curriculum.
1. Cook DA, Levinson AJ, Garside S, et al. Internet-based learning in the health professions: a meta-analysis. JAMA 2008; 300: 1181–1196.
2. Curran V, Lockyer J, Sargeant J, et al. Evaluation of learning outcomes in Web-based continuing medical education. Acad Med 2006; 81 (10 Suppl): S30–S34.
3. Curran VR, Fleet L. A review of evaluation outcomes of web-based continuing medical education. Med Educ 2005; 39: 561–567.
4. Wong G, Greenhalgh T, Pawson R. Internet-based medical education: a realist review of what works, for whom and in what circumstances. BMC Med Educ 2010; 10: 12.
5. Kirkpatrick DL. Evaluating Training Programs: The Four Levels. San Francisco: Berrett-Koehler; 1994.
6. Cohen PA. Student ratings of instruction and student achievement: a meta-analysis of multisection validity studies. Rev Educ Res 1981; 51: 281–309.
7. Shea J, Bellini LM. Evaluations of clinical faculty: the impact of level of learner and time of year. Teach Learn Med 2002; 14: 87–91.
8. Cannon GW, Keitz SA, Holland GJ, et al. Factors determining medical students’ and residents’ satisfaction during VA-based training: findings from the VA learners’ Perceptions Survey. Acad Med 2008; 83: 611–620.
9. Davis DA. Does CME work? An analysis of the effect of educational activities on physician performance or health care outcomes. Int J Psychiatry Med 1998; 28: 21–39.
10. Sargeant J, Curran V, Jarvis-Selinger S, et al. Interactive on-line continuing medical education: physicians’ perceptions and experiences. J Contin Educ Health Prof 2004; 24: 227–236.
11. Gold JP, Begg WB, Fullerton DA, et al. Evaluation of Web-based learning tools: lessons learned from the Thoracic Surgery Directors Association Curriculum Project three-year experience. Ann Thorac Surg 2005; 80: 802–810.
12. Dornan T, Arno M, Hadfield J, et al. Student evaluation of the clinical “curriculum in action.” Med Educ 2006; 40: 667–674.
13. Cobb SC. Internet continuing education for health care professionals: an integrative review. J Contin Educ Health Prof 2004; 24: 171–180.
14. Cook DA. The failure of e-learning research to inform educational practice, and what we can do about it. Med Teach 2009; 31: 158–162.
15. Cook DA, Beckman TJ, Thomas KG, et al. Adapting Web-based instruction to residents’ knowledge improves learning efficiency: a randomized controlled trial. J Gen Intern Med 2008; 23: 985–990.
16. Levinson AJ. Where is evidence-based instructional design in medical education curriculum development? Med Educ 2010; 44: 536–537.
17. Mayer RE. Applying the science of learning to medical education. Med Educ 2010; 44: 543–549.
18. Sisson SD, Hughes MT, Levine D, et al. Effect of an Internet-based curriculum on post-graduate education: a multicenter intervention. J Gen Intern Med 2004; 19: 503–507.
19. Sisson SD, Rastegar DA, Rice TN, et al. Multicenter implementation of a shared graduate medical education resource. Arch Intern Med 2007; 167: 2476–2480.
20. Sisson SD, Hill-Briggs F, Levine D. Using education theory to improve medical education website design. BMC Med Educ 2010; 10: 30.
21. Knowles MS, Holton EF, Swanson RA. The Adult Learner. 5th ed. Woburn, MA: Butterworth-Heinemann; 1998.
22. Fitzgibbons JP, Bordley DR, Berkowitz LR, et al. Redesigning residency education in internal medicine: a position paper from the Association of Program Directors in Internal Medicine. Ann Intern Med 2006; 144: 920–926.
23. Weinberger SE, Smith LG, Collier VU. Redesigning training for internal medicine. Ann Intern Med 2006; 144: 977–932.
24. Larson EB, Fihn SD, Kirk LM, et al. The future of general internal medicine: report and recommendations from the Society of General Internal Medicine (SGIM) Task Force on the domain of general internal medicine. J Gen Intern Med 2004; 19: 69–77.
25. Meyers FJ, Weinberger SE, Fitzgibbons JP, et al. Acad Med 2007; 82: 1211–1219.
26. Sisson SD, Dalal D. Internal medicine residency training on topics in ambulatory care: a status report. Am J Med 2011; 124: 86–90.
27. Curran VR, Fleet LJ. A comparative evaluation of the effect of Internet-based CME delivery format on satisfaction, knowledge and confidence. BMC Med Educ 2010; 10: 10.
28. Cook DA, Gelula MH, Dupras DM, et al. Instructional methods and cognitive and learning styles in web-based learning: report of two randomized trials. Med Educ 2007; 41: 897–905.
29. Levinson AJ. Where is evidence-based instructional design in medical education curriculum development? Med Educ 2010; 44: 536–537.