The imitation game: Detecting human and AI-generated texts in the era of ChatGPT and BARD

Document Type

Article

Source of Publication

Journal of Information Science

Publication Date

1-1-2024

Abstract

The potential of artificial intelligence (AI)-based large language models (LLMs) holds considerable promise in revolutionising education, research and practice. However, distinguishing between human-written and AI-generated text has become a significant task. This article presents a comparative study, introducing a novel dataset of human-written and LLM-generated texts in different genres: essays, stories, poetry and Python code. We employ several machine learning models to classify the texts. Results demonstrate the efficacy of these models in discerning between human and AI-generated text, despite the dataset’s limited sample size. However, the task becomes more challenging when classifying GPT-generated text, particularly in story writing. The results indicate that the models exhibit superior performance in binary classification tasks, such as distinguishing human-generated text from a specific LLM, compared with the more complex multiclass tasks that involve discerning among human-generated and multiple LLMs. Our findings provide insightful implications for AI text detection, while our dataset paves the way for future research in this evolving area.

ISSN

0165-5515

Publisher

SAGE Publications

Disciplines

Computer Sciences

Keywords

Artificial intelligence, BARD, GPT, LLM, natural language processing, plagiarism detection

Scopus ID

85185694448

Indexed in Scopus

yes

Open Access

yes

Open Access Type

Green: A manuscript of this publication is openly available in a repository

Link to Full Text

Share

COinS