# Six AI Tools Took On a 102-Paper Literature Review. None Passed All Four Stages URL: https://edufabtech.com/news/six-ai-tools-literature-review-benchmark-manchester-study Section: News · Artificial Intelligence Published: 2026-11-07 Publisher: EduFabTech (https://edufabtech.com) > Researchers from the University of Manchester and University of Washington tested six AI tools against a 102-article benchmark review, finding library access sharply increased article discovery but synthesis still contained unsupported claims. A study posted to arXiv on 3 November 2026 tested six AI tools against a full scientific literature review, rather than against a single isolated task, and found that each one broke down somewhere in the process. The paper, by Abdullah Al Kalbani, David M. Schultz, Alexandra K. Anderson-Frey, Zhonghua Zheng and Catherine McGuire, comes from the University of Manchester and the University of Washington, and uses a published weather-science review as its answer key. Literature reviews are usually tested on AI tools one piece at a time — can a model find relevant papers, can it summarise one correctly, can it draft a coherent paragraph. The arXiv preprint instead ran six tools through all four stages that a human reviewer actually performs: finding relevant articles, interpreting them, synthesising across them, and proposing future research. The tools were ChatGPT Deep Research (GPT-5.6 Sol), Gemini Deep Research (Gemini 3.1 Pro), the Claude web interface (Claude Opus 5), Codex (GPT-5.6 Sol), Claude Code (Claude Opus 5) and NotebookLM (Gemini 3.5). The benchmark was a real review already in print: Schultz, together with George Young and Daniel J. Kirshbaum, had published a review in Monthly Weather Review (American Meteorological Society, 2025) covering 102 peer-reviewed articles on the "Spanish plume" weather pattern, published between 1968 and 2023. Because the authors of the new study already knew which 102 articles mattered and what the review concluded, they could score the AI tools against a known, checkable answer rather than against their own judgment of what counted as a good result. ![A four-stage pipeline (Discover, Interpret, Synthesize, Propose) shows where each of the three fully-tested tools broke down, from 82/102 papers found with library access to only 1 partial match on proposed research directions.](https://edufabtech.com/uploads/2026/11/six-ai-tools-literature-review-benchmark-manchester-study-2.png) *A four-stage pipeline (Discover, Interpret, Synthesize, Propose) shows where each of the three fully-tested tools broke down, from 82/102 papers found with library access to only 1 partial match on proposed research directions. — EduFabTech · Own work* ## Finding the papers was the easiest part, and library access changed everything In the first stage, five of the tools searched for relevant literature without being handed the corpus (NotebookLM, which works from uploaded documents rather than open search, was not part of this stage). According to the preprint, those five tools collectively surfaced 57 of the 102 articles, with the best single tool finding 42. That changed sharply once the tools were given institutional library access: Codex's discovery rose from 42 to 82 articles, and Claude Code's discovery rose from 15 to 79, the authors report. ## Interpretation: right on the headline cases, wrong on the borderline ones Three of the six tools — Codex, Claude Code and NotebookLM — were assigned by the researchers to attempt all four stages. Given the complete set of 102 PDFs, all three correctly flagged all 18 articles that discussed the Spanish plume substantially, the study found. But the same three tools misclassified between 40% and 79% of the 84 articles that only mentioned the topic in passing, incorrectly treating many of them as substantial discussions. ## Synthesis held up on accuracy but not on insight The three tools then wrote syntheses of the literature, which two domain experts scored blind: Schultz himself, as lead author of the benchmark review, and an independent severe-convection researcher who had not worked on the original study. Both rated the syntheses on five criteria — writing quality, depth and breadth, veracity, insight, and usefulness to a reviewer — across ten themes, each on a five-point scale, for 50 ratings per synthesis per reviewer. The preprint describes the results as "largely accurate" but offering "little insight," with syntheses that sometimes contained unsupported claims, quotation errors, and exaggerated statements of consensus. ## Proposing new research directions fared better, within limits For the final stage, the three tools suggested directions for future research. NotebookLM's four suggested gaps were ones that individual articles in the corpus had already named themselves, according to the study, while Codex and Claude Code drew their own inferences from across the literature and came closer to the benchmark review's own recommended research directions — NotebookLM matched only one of them, and only partly. Claude Code's suggestions stayed the most closely tied to the Spanish plume literature specifically, even though all three tools also proposed research questions broader than that topic alone. ![A horizontal bar chart compares article-discovery counts before and after institutional library access, showing Codex rising from 42 to 82 and Claude Code from 15 to 79 out of the 102-paper corpus.](https://edufabtech.com/uploads/2026/11/six-ai-tools-literature-review-benchmark-manchester-study-3.png) *A horizontal bar chart compares article-discovery counts before and after institutional library access, showing Codex rising from 42 to 82 and Claude Code from 15 to 79 out of the 102-paper corpus. — EduFabTech · Own work* ## What it means for researchers and students The authors are explicit that no single tool performed best across all four stages, and they frame their results as a case for assistance rather than replacement: AI tools "can assist with discovering and interpreting scientific literature, particularly when institutional access is available, but cannot replace expert judgment in evaluating evidence or synthesizing research," the preprint states. For students and early-career researchers building their first literature reviews, the practical implication is that a tool's confidence is not a reliable signal — the study notes that these tools show equally strong confidence whether or not their output is accurate. The paper closes with eight recommendations, including: run searches through institutional subscription access rather than the open web; separately search for older literature, since tools favour recent articles; write detailed prompts with explicit inclusion and exclusion criteria; verify every quotation and claim against the original source; and treat a tool's own judgment of an article's importance as a starting point for human review, not a final answer. Because the underlying benchmark review and the new evaluation both work from the same fixed, published corpus, other researchers can rerun the same test as newer tool versions appear — the paper is a reproducible yardstick rather than a one-time snapshot of what large language models could do in late 2026. ## Sources 1. Abdullah Al Kalbani, David M. Schultz, Alexandra K. Anderson-Frey, Zhonghua Zheng, Catherine McGuire. (2026). Can Large Language Models Review the Scientific Literature? A Test Against a 102-Article Benchmark. arXiv preprint https://doi.org/10.48550/arXiv.2611.05459 2. David M. Schultz, George Young, Daniel J. Kirshbaum. (2025). The Spanish Plume Elevated Mixed Layer: A Review of Its Use and Misuse within the Scientific Literature. Monthly Weather Review (American Meteorological Society) https://doi.org/10.1175/MWR-D-24-0139.1 --- Source page: https://edufabtech.com/news/six-ai-tools-literature-review-benchmark-manchester-study Quote with a link to the source page. © EduFabTech.