Evaluation of Large Language Models in Simulating Real-World Engineering Scenarios and Decision Making: A Comparative Study of ChatGPT-4o, Gemini, and Perplexity

Document Type

Conference Proceeding

Source of Publication

Lecture Notes of the Institute for Computer Sciences Social Informatics and Telecommunications Engineering Lnicst

Publication Date

4-1-2026

Abstract

Integrating real-world simulations and decision-making scenarios through case studies has been shown to enhance students’ critical thinking and problem-solving skills. Prior research highlighted a gap in faculty capacity to create or access such materials, an issue particularly acute in developing countries. This research explores the potential of three prominent LLMs, namely ChatGPT-4o, Gemini 2.0, and Perplexity, to generate high-quality case studies for graduate engineering education. In this study, we define a high-quality engineering case study as one that is narratively rich, embeds socio-technical and multidisciplinary perspectives, presents an open-ended practical problem, includes plausible and data rich role-playing scenarios, and supports active learning and reflective thinking. Our evaluation employs a multi-layered methodology combining: (1) a linguistic analysis using standard NLP metrics; (2) automated mixed-method assessments using four LLMs; and (3) mixed-method evaluation by three Subject Matter Experts (SMEs). Our findings revealed that while Perplexity produced the most readable content, it failed along with Gemini to meet the minimum required word count. The SMEs unanimously ranked ChatGPT-4o as superior across all performance dimensions. This study reveals several LLM limitations, such as limited narrative flow, lack of depth, compelling storytelling and academic rigor, failure to meet explicit instructional requirements, and instances of fake or inaccurate citations. These challenges reinforce the necessity for a “human-in-the-loop” approach. Our study offers a balanced and evidence-based perspective on the role of LLMs in augmenting, rather than replacing, human expertise in case study development.

ISBN

[9783032166340]

ISSN

1867-8211

Publisher

Springer Nature Switzerland

Volume

676 LNICST

First Page

365

Last Page

393

Disciplines

Computer Sciences

Keywords

Applied Computing, Case Studies, Computer-managed Instruction, Critical Thinking, Decision-making, Emerging Technologies, Engineering Education, Experiential Learning, Generative AI, Large Language Models, Natural Language Processing, NLP

Scopus ID

105039223757

Indexed in Scopus

yes

Open Access

no

Link to Full Text

Share

COinS