The ChatGPT Artificial Intelligence Chatbot: How Well Does It Answer Accounting Assessment Questions?

Author First name, Last name, Institution

David A. Wood, Brigham Young University
Muskan P. Achhpilia, Radboud University Nijmegen
Mollie T. Adams, Missouri State University
Sanaz Aghazadeh, Louisiana State University
Kazeem Akinyele, University of Wisconsin–Oshkosh
Mfon Akpan, Methodist University
Kristian D. Allee, University of Arkansas at Fayetteville
Abigail M. Allen, Brigham Young University
Elizabeth D. Almer, Portland State University
Daniel Ames, Bryant University
Viktor Arity, RMIT University
Dereck Barr-Pulliam, University of Louisville
K. Asli Basoglu, University of Delaware
Andrew Belnap, The University of Texas at Austin
Jeremiah W. Bentley, University of Massachusetts Amherst
Terje Berg, Norwegian University of Science and Technology
Nathan R. Berglund, Mississippi State University
Erica Berry, University of San Diego
Avishek Bhandari, University of Wisconsin–Whitewater
Md Nazmul Hasan Bhuyan, North Carolina Agricultural and Technical State University
Paul W. Black, Auburn University
Eva Blondeel, Ghent University
David Bond, University of Technology Sydney
Annika Bonrath, University of Duisburg-Essen
A. Faye Borthick, Georgia State University
Erik S. Boyle, Idaho State University
Marianne Bradford, North Carolina State University
Duane M. Brandon, Auburn University
Joseph F. Brazel, North Carolina State University
Bryan G. Brockbank, Oklahoma State University

Document Type

Article

Source of Publication

Issues in Accounting Education

Publication Date

4-18-2023

Abstract

ChatGPT, a language-learning model chatbot, has garnered considerable attention for its ability to respond to users’ questions. Using data from 14 countries and 186 institutions, we compare ChatGPT and student performance for 28,085 questions from accounting assessments and textbook test banks. As of January 2023, ChatGPT provides correct answers for 56.5 percent of questions and partially correct answers for an additional 9.4 percent of questions. When considering point values for questions, students significantly outperform ChatGPT with a 76.7 percent average on assessments compared to 47.5 percent for ChatGPT if no partial credit is awarded and 56.5 percent if partial credit is awarded. Still, ChatGPT performs better than the student average for 15.8 percent of assessments when we include partial credit. We provide evidence of how ChatGPT performs on different question types, accounting topics, class levels, open/closed assessments, and test bank questions. We also discuss implications for accounting education and research.

DOI Link

10.2308/issues-2023-013

ISSN

0739-7983

Publisher

American Accounting Association

First Page

1

Last Page

28

Disciplines

Business | Computer Sciences

Keywords

artificial intelligence, chat bots, GPT-3, accounting education, assessment, crowdsourcing

Recommended Citation

Wood, David A.; Achhpilia, Muskan P.; Adams, Mollie T.; Aghazadeh, Sanaz; Akinyele, Kazeem; Akpan, Mfon; Allee, Kristian D.; Allen, Abigail M.; Almer, Elizabeth D.; Ames, Daniel; Arity, Viktor; Barr-Pulliam, Dereck; Basoglu, K. Asli; Belnap, Andrew; Bentley, Jeremiah W.; Berg, Terje; Berglund, Nathan R.; Berry, Erica; Bhandari, Avishek; Bhuyan, Md Nazmul Hasan; Black, Paul W.; Blondeel, Eva; Bond, David; Bonrath, Annika; Borthick, A. Faye; Boyle, Erik S.; Bradford, Marianne; Brandon, Duane M.; Brazel, Joseph F.; and Brockbank, Bryan G., "The ChatGPT Artificial Intelligence Chatbot: How Well Does It Answer Accounting Assessment Questions?" (2023). All Works. 5795.
https://zuscholars.zu.ac.ae/works/5795

Indexed in Scopus

no

Open Access

yes

Open Access Type

Bronze: This publication is openly available on the publisher’s website but without an open license