Evaluating the Efficacy of Large Language Models in Identifying Phishing Attempts

Document Type

Conference Proceeding

Source of Publication

International Conference on Human System Interaction, HSI

Publication Date

1-1-2024

Abstract

Phishing, a prevalent cybercrime tactic for decades, remains a significant threat in today's digital world. By leveraging clever social engineering elements and modern technology, cybercrime targets many individuals, businesses, and organizations to exploit trust and security. These cyber-attackers are often disguised in many trustworthy forms to appear as legitimate sources. By cleverly using psychological elements like urgency, fear, social proof, and other manipulative strategies, phishers can lure individuals into revealing sensitive and personalized information. Building on this pervasive issue within modern technology, this paper will aim to analyze the effectiveness of 15 Large Language Models (LLMs) in detecting phishing attempts, specifically focusing on a randomized set of '419 Scam' emails. The objective is to determine which LLMs can accurately detect phishing emails by analyzing a text file containing email metadata based on predefined criteria. The experiment concluded that the following models, ChatGPT 3.5, GPT-3.5-Turbo-Instruct, and ChatGPT, were the most effective in detecting phishing emails.

ISBN

[9798350362916]

ISSN

2158-2246

Publisher

IEEE

Disciplines

Computer Sciences

Keywords

Bidirectional Encoder Representations from Transformers (BERT), General Pretrained Transformer (GPT), Large Language Models (LLMs), Natural Processing Language (NPL), Phishing Email Detection, Social Engineering

Scopus ID

85201565266

Indexed in Scopus

yes

Open Access

yes

Open Access Type

Green: A manuscript of this publication is openly available in a repository

Link to Full Text

Share

COinS