ChatGPT In Assessment: A Comparison of Artificial Intelligence (AI)-Generated and Human-Generated One Best-Type Multiple Choice Questions
DOI:
https://doi.org/10.51253/pafmj.v76i6.12930Keywords:
Artificial Intelligence, Educational Measurement, Educational Technology, Medical Education, Machine LearningAbstract
Objective: To compare the quality of Multiple-Choice Questions (MCQs) created by humans and Chat GPT using post-hoc item analysis.
Study Design: Quasi-Experimental Research Design.
Place and Duration of Study: Department of Medicine, Azra Naheed Medical College, The Superior University, Lahore Pakistan, from Feb to Aug 2024.
Methodology: A total of 100 participants were given two sets of 50 questions. One set was created using Chat GPT while the other was crafted by human teachers. Duration of generation of MCQs was noted. Post hoc item analysis was done (difficulty index, discrimination index and distractor analysis). Student scores were compared. Independent sample t-test was applied to compare mean difficulty index, discrimination index and student scores. Two proportions z-test to compare mean distractor efficiency. The results were reported as means, frequencies and percentages and analyzed using SPSS version 27.
Results: Total of 100 MCQs were evaluated. Chat GPT generated MCQs in 25 minutes, significantly faster than humans' 205 minutes. The Difficulty Index (Chat GPT: 30.3, humans: 34.7, p=0.39), Discrimination Index (Chat GPT: 0.21, humans: 0.19, p=0.514), student scores (Chat GPT: 19.4, humans: 18.46, p=0.125) and Distractor efficiency (Chat GPT: 75%, humans 69.5%, p=0.385) were statistically non-significant, indicating similar quality.
Conclusion: The quality of MCQs produced by Chat GPT and Human Educators is comparable; however, the duration required by Chat GPT is significantly less. A mixed approach, where subject matter experts qualitatively evaluate the MCQs generated by Chat GPT, can enhance the overall assessment quality.
Downloads
References
1. Nassar A, Reid S, Kahnamoui K, Tuma F, Waheed A, McConnell M. Burnout among academic clinicians as it correlates with workload and demographic variables. Behav Sci 2020; 10(6): 94.
https://doi.org/10.3390/bs10060094
2. Woolhandler S, Himmelstein DU. Administrative work consumes one-sixth of U.S. physicians’ working hours and lowers their career satisfaction. Int J Health Serv 2014; 44(4): 635-642. https://doi.org/10.2190/hs.44.4.a
3. Collignon SE, Chacko J, Wydick Martin M. An alternative multiple-choice question format to guide feedback using student self-assessment of knowledge. Decis Sci J Innov Educ 2020; 18(3): 456–480.
https://psycnet.apa.org/doi/10.1111/dsji.12213
4. Siddiqui S. Categorized and correlated multiple-choice questions: A tool for assessing comprehensive physics knowledge of students. Educ Sci 2022; 12(9): 575.
https://doi.org/10.3390/educsci12090575
5. Mondal H, Marndi G, Behera JK, Mondal S. ChatGPT for teachers: Practical examples for utilizing artificial intelligence for educational purposes. Indian J Vasc Endovasc Surg 2023; 10: 200-205.
https://doi.org/10.4103/ijves.ijves_37_23
6. Nasution N. Using artificial intelligence to create biology multiple-choice questions for higher education. Agric Environ Educ 2023; 2(1): 4–8.
https://doi.org/10.29333/agrenvedu/13071
7. Klang E, Portugez S, Gross R, Kassif Lerner R, Brenner A, Gilboa M, et al. Advantages and pitfalls in utilizing artificial intelligence for crafting medical examinations: a medical education pilot study with GPT-4. BMC Med Educ 2023; 23: 772.
https://doi.org/10.1186/s12909-023-04752-w
8. Leo J, Kurdi G, Matentzoglu N, Parsia B, Sattler U, Forge S, et al. Ontology-based generation of medical, multi-term MCQs. Int J Artif Intell Educ 2019; 29: 145–188.
https://doi.org/10.1007/s40593-018-00172-w
9. Cheung BHH, Lau GKK, Wong GTC, Lee EYP, Kulkarni D, Seow CS, et al. ChatGPT versus human in generating medical graduate exam multiple choice questions—A multinational prospective study (Hong Kong SAR, Singapore, Ireland, and the United Kingdom). PLoS One 2023; 18(8): e0290691.
https://doi.org/10.1371/journal.pone.0290691
10. Khilnani AK. Potential of large language model (ChatGPT) in constructing multiple-choice questions. GAIMS J Med Sci 2023; 3(2): 1–3. https://doi.org/10.5281/zenodo.7751267
11. Kıyak YS. A ChatGPT prompt for writing case-based multiple-choice questions. Rev Esp Educ Med 2023; 4(3): 98-103.
https://doi.org/10.6018/edumed.587451
12. Olney AM. Generating multiple choice questions from a textbook: LLMs match human performance on most metrics. Proceedings of the AIEDLLM1 Workshop: Empowering Education with LLMs; 2023 Jul 7; Tokyo, Japan. Memphis, TN: Institute for Intelligent Systems, University of Memphis. 2023. Available from: https://olney.ai/category/2023/07/07/mcq.html
13. Agarwal M, Sharma P, Goswami A. Analysing the applicability of ChatGPT, Bard, and Bing to generate reasoning-based multiple-choice questions in medical physiology. Cureus 2023; 15(6): e40977.
https://doi.org/10.7759/cureus.40977
14. Ilgaz HB, Çelik Z. The significance of artificial intelligence platforms in anatomy education: An experience with ChatGPT and Google Bard. Cureus 2023; 15(9): e45301.
https://doi.org/10.7759/cureus.45301
15. Rao C, Prasad HLK, Sajitha K, Permi H, Shetty J. Item analysis of multiple-choice questions: Assessing an assessment tool in medical students. Int J Educ Psychol Res 2016; 2: 201.
https://doi.org/10.4103/2395-2296.189670
16. Javaid M, Haleem A, Singh RP, Khan S, Khan IH. Unlocking the opportunities through ChatGPT Tool towards ameliorating the education system. BenchCouncil Trans Benchmarks Stand Eval 2023; 3(2): 100115.
https://doi.org/10.1016/j.tbench.2023.100115
17. Aya AH, Khalifa M, Abdelsalheen M, Afifi S. A systematic review of coastal vulnerability assessment methods in the Egyptian context: The case study. HBRC J 2023; 19(1): 161–181.
https://doi.org/10.1080/16874048.2023.2241964
18. Paranjape K, Schinkel M, Panday RN, Car J, Nanayakkara P. Introducing artificial intelligence training in medical education. JMIR Med Educ 2019; 5(2): e16048.
19. Chin C, Osborne J. Students' questions: A potential resource for teaching and learning science. Stud Sci Educ 2008; 44(1): 1–39.
https://doi.org/10.1080/03057260701828101
20. Halagatti M, Gadag S, Mahantshetti S, Hiremath CV, Tharkude D, Banakar V. Artificial intelligence: the new tool of disruption in educational performance assessment. In: Smart Analytics, Artificial Intelligence and Sustainable Performance Management in a Global Digitalised Economy. Vol. 110A. Emerald Group Publishing Limited; 2023: 261-287.
https://EconPapers.repec.org/RePEc:eme:csefzz:s1569-37592023000110a014
21. United Nations Educational, Scientific and Cultural Organization (UNESCO). Artificial Intelligence in Education: Challenges and Opportunities for Sustainable Development. Paris: UNESCO; 2019.
https://unesdoc.unesco.org/ark:/48223/pf0000366994
22. Johnson D, Goodman R, Patrinely J, Stone C, Zimmerman E, Donald R, et al. Assessing the accuracy and reliability of AI-generated medical responses: an evaluation of the Chat-GPT model. Res Sq 2023; rs.3.rs-2566942.
https://doi.org/10.21203/rs.3.rs-2566942/v1
23. Haleem A, Javaid M, Qadri MA, Suman R. Understanding the role of digital technologies in education: A review. Sustain Oper Comput 2022; 3: 275–285.
https://doi.org/10.1016/j.susoc.2022.05.004
24. Choi W. Assessment of the capacity of ChatGPT as a self-learning tool in medical pharmacology: A study using MCQs. BMC Med Educ 2023; 23: 864.
Downloads
Published
License
Copyright (c) 2026 Muhammad Moeed Haidar Naqvi, Masood Jawaid, Syeda Tahseen Fatima, Asad Ali, Tayyaba Azhar
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
How to Cite
