Claude AI and Literature Reviews: An Experiment in Utility and Ethical Use

Abstract

Generative large language models such as ChatGPT and Claude have sparked debate throughout higher education about their potential and threats to the current pedagogical models for research. This paper reviews the literature on the benefits and drawbacks of using large language models in teaching and research and compares and contrasts two literature reviews on the subject, one written by a human author and one produced by Claude. The comparison explores areas where the generative artificial intelligence excels, such as quickly summarizing points of agreement across sources, as well as its limitations, such as struggling with synthesis, providing incomplete citations, and hallucinating false information. While Claude was able to identify key themes and sources, the areas where it struggles show that without human intervention (providing context and analysis), the tool cannot produce a literature review that would stand on its own. However, the experiment demonstrates potential for these models to augment and accelerate research workflows when leveraged responsibly alongside human scholars.

Main research questions 

1) Does providing Claude AI with the full text of a list of sources allow it to write a passable literature review? 
2) What are some of the benefits and drawbacks of using LLMs in writing, research, and pedagogy?

What was already known 

This is a fast-developing area of research and there is very little agreement currently about the effectiveness and ethics of using generative AI in research. This study is designed as an experiment intended to dive further into the effectiveness and ethics, as well as the practicality, of using generative AI in a research setting.

Novel methodology 

A few studies that used comparable methodologies turned up in a search of the literature, with the majority of papers using generative AI to produce a literature review and then critiquing it on its own merits rather than comparing it with a human-generated review on the same topic.