Review Article

Best Practices for Survey Use in Medical Education: How to Design, Refine, and Administer High-Quality Surveys

Authors: Tanya Nikiforova, MD, MS, Andrea Carter, MD, MS, Emmanuelle Yecies, MD, MS, Carla L. Spagnoletti, MD, MS

Abstract

Surveys are a frequently used method to collect data in medical education research. As such, it behooves medical educators involved in scholarly work to understand the best practices in the selection, development, implementation, and reporting of surveys used when conducting research and curriculum development projects. This review article prepares the reader to name the steps of designing and administering high-quality surveys in medical education research, identify the characteristics of both well-written and poorly written survey questions, and apply the principles of survey design to write and revise surveys for use in their own research.

 

This content is limited to qualifying members.

Existing members, please login first

If you have an existing account please login now to access this article or view purchase options.

Purchase only this article ($25)

Create a free account, then purchase this article to download or access it online for 24 hours.

Purchase an SMJ online subscription ($75)

Create a free account, then purchase a subscription to get complete access to all articles for a full year.

Purchase a membership plan (fees vary)

Premium members can access all articles plus recieve many more benefits. View all membership plans and benefit packages.

References

1. Phillips AW, Friedman BT, Utrankar A, et al. Surveys of health professions trainees: prevalence, response rates, and predictive factors to guide researchers. Acad Med 2017;92:222–228.
 
2. Artino AR, Phillips AW, Utrankar A, et al. The questions shape the answers: assessing the quality of published survey instruments in health professions education research. Acad Med 2018;93:456–463.
 
3. Phillips AW, Artino MA Jr. Lies, damned lies, and surveys. J Grad Med Educ 2017;9:677–679.
 
4. Annals of Emergency Medicine. Guidelines and preferences for specific research study designs. https://www.annemergmed.com/content/designs. Accessed November 25, 2020.
 
5. Rickards G, Magee C, Artino AR Jr. You can’t fix by analysis what you’ve spoiled by design: developing survey instruments and collecting validity evidence. J Grad Med Educ 2012;4:407–410.
 
6. Artino AR Jr, La Rochelle JS, Dezee KJ, et al. Developing questionnaires for educational research: AMEE guide no. 87. Med Teach 2014;36:463–474.
 
7. Sullivan GM, Artino AR Jr. How to create a bad survey instrument. J Grad Med Educ 2017;9:411–415.
 
8. Krosnick JA. Survey research. Annu Rev Psychol 1999;5:537–567.
 
9. Streiner DL, Normal GR, Cariney J. Health Measurement Scales: A Practical Guide to Their Development and Use. New York: Oxford University Press; 2015.
 
10. Barnette JJ. Effects of stem and Likert response option reversals on survey internal consistency: if you feel the need, there is a better alternative to using those negatively worded stems. Educ Psychol Meas 2000;60:361–370.
 
11. Likert RA. A technique for the measurement of attitudes. Arch Psychol 1932;140:1–55.
 
12. Liu M, Lee S, Conrad FG. Comparing extreme response styles between agree-disagree and item-specific scales. Public Opin Q 2015;79:952–975.
 
13. Saris WE, Krosnick JA, Revilla M, et al. Comparing questions with agree/ disagree response options to questions with item-specific response options. Surv Res Methods 2010;4:61–79.
 
14. Revilla MA, Saris WE, Krosnick JA. Choosing the number of categories in agree–disagree scales. Sociol Methods Res 2014;43:73–97.
 
15. Krosnick JA, Holbrook A, Berent M, et al. The impact of “no opinion” response options on data quality: non-attitude reduction or an invitation to satisfice? Public Opin Q 2002;66:371–403.
 
16. Bishop GF. Experiments with the middle response alternative in survey questions. Public Opin Q 1987;51:220–232.
 
17. Liu M, Keusch F. Effects of scale direction on response style of ordinal rating scales. J Off Stat 2017;33:137–154.
 
18. Sonderen E, Sanderman R, Coyne JC. Ineffectiveness of reverse wording of questionnaire items: let’s learn from cows in the rain. PLoS One 2013;8:e68967.
 
19. The GenIUSS Group. Best Practices for Asking Questions to Identify Transgender and Other Gender Minority Respondents on Population-Based Surveys. Los Angeles: The Williams Institute; 2014.
 
20. Cull WL, O’Connor KG, Sharp S, et al. Response rates and response bias for 50 surveys of pediatricians. Health Serv Res 2005;40:213–226.
 
21. Grava-Gubins I, Scott S. Effects of various methodologic strategies: survey response rates among Canadian physicians and physicians-in-training. Can Fam Physician 2008;54:1424–1430.
 
22. Palla B, Callahan N, Miloro M. One survey to rule them all. J Oral Maxillofac Surg 2020;10:31215–5.
 
23. Christof VM. Improving web survey efficiency: the impact of an extra reminder and reminder content on web survey response. Int J Soc Res Methodol 2017;20:317–327.
 
24. Phillips AW, Reddy S, Durning SJ. Improving response rates and evaluating nonresponse bias in surveys: AMEE guide no. 102. Med Teach 2016;38:217–228.
 
25. Dillman DA, Smyth JD, Christian LM. Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method, 3rd ed. New York: John Wiley & Sons;2002.
 
26. Halbesleben JR, Whitman MV. Evaluating survey quality in health services research: a decision framework for assessing nonresponse bias. Health Serv Res 2013;48:913–930.
 
27. American Association for Public Opinion Research. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys, 9th ed. Lenexa, KS: American Association for Public Opinion Research; 2016.
 
28. Groves RM. Nonresponse rates and nonresponse bias in household surveys. Public Opin Q 2006;70:646–675.
 
29. Meyer HS, Durning SJ, Sklar D, et al. Making the first cut: an analysis of Academic Medicine editors’ reasons for not sending manuscripts out for external peer review. Acad Med 2018;93:464–470.
 
30. Artino AR Jr, Durning SJ, Sklar DP. Guidelines for reporting survey-based research submitted to Academic Medicine. Acad Med 2018;93:337–340.
 
31. McKenzie J, Wood ML, Kotecki JE, et al. Establishing content validity: using qualitative and quantitative steps. Am J Health Behav 1999;23:311–318.
 
32. Willis GB, Artino AR. What do our respondents think we’re asking? Using cognitive interviewing to improve medical education surveys. J Grad Med Educ 2013;5:353–356.
 
33. Artino AR, Durning SJ, Creel AH. AM Last Page: reliability and validity in educational measurement. Acad Med 2010;85:1545.