Same Data, Different AI 'Analysts': Verdicts Swing 34-66 Points by Persona Alone, PNAS Finds

In July 2026, the Proceedings of the National Academy of Sciences published a study that ran the many-analyst design — one dataset and one hypothesis, handed to many independent analysts who work alone and compare notes only afterward — on fleets of artificial intelligence agents instead of human researchers. One of its three datasets was lifted directly from a landmark 2018 human many-analyst study; the other two were built new for this project. The question was whether AI "analysts" would disagree with each other the way human research teams famously do when given identical data. They did — and the size of the disagreement turned out to depend on something as simple as the personality the researchers assigned to the AI before it opened the spreadsheet.

The paper, by Martin Bertran and Riccardo Fogliato of Amazon and Zhiwei Steven Wu of Amazon and Carnegie Mellon University, is titled "Many AI analysts, one dataset: Navigating the agentic data science multiverse". It builds directly on a line of research called many-analyst studies, in which independent teams are given one dataset and one hypothesis and asked to test it however they see fit. Those studies have repeatedly found that teams reach different conclusions from identical numbers, not because anyone is cheating, but because every analysis pipeline embeds dozens of small, defensible choices.

What is new is doing this with machines, at a scale no human-coordination study could match. The authors ran each AI analyst through a complete pipeline, covering data cleaning, model specification and inference, on a fixed dataset and hypothesis, then had a separate AI "auditor" check every run for methodological validity. Of 4,946 total runs, 3,303, or 67%, passed that compliance screening, according to the published paper. The three datasets spanning distinct domains were a soccer referee-bias dataset originally used in Silberzahn and colleagues' 2018 many-analyst study of refereeing bias, a randomized controlled trial of AI-assisted programming run by METR (metr-rct, with its hypothesis deliberately reversed from METR's own published conclusion to limit contamination from the models' training data), and the American National Election Studies time series (anes-views).

Same data, different verdicts, by design

The AI analysts were built on four large language models — Claude Sonnet 4.5, Claude Haiku 4.5, Qwen3 Coder 480B, and Qwen3 235B A22B — and were assigned one of five personas before touching the data: Standard, Negative (primed to find the hypothesis implausible), Positive (primed to find it plausible), Confirmation Seeking, and Strong Confirmation Seeking. Everything else about the task, including the dataset, the hypothesis and the instructions to run a valid analysis, stayed the same.

Across the three datasets, the paper reports, "AI analyst-produced analyses exhibit substantial dispersion in effect sizes, p-values, and conclusions," and that dispersion "can be traced to identifiable analytic choices in preprocessing, model specification, and inference that vary systematically across LLM and persona conditions," as the authors write in the published paper. On the ANES-views dataset, effect-size estimates from valid, auditor-approved runs spanned from negative to positive, meaning some compliant analyses of the same data supported the hypothesis and others contradicted it outright.

The persona effect: 34 to 66 percentage points

The sharpest finding concerns what the authors call "steerable" outcomes. Reassigning only the persona, not the data, not the hypothesis, not the model's actual competence, shifted how often an AI analyst's conclusion supported the hypothesis under test. Comparing the most skeptical persona (Negative) against the most confirmation-seeking (Strong CS), the paper states that "the Negative-to-Strong CS gap ranges from 34 percentage points (anes-views) to 66 percentage points (metr-rct)." That is the difference between a dataset that looks like it barely supports a claim and one that looks like it strongly does, produced from identical numbers by changing a few words of framing in the instructions given to the analyst.

The authors are explicit that this steerability showed up "even among methodologically sound runs" — the auditor had already screened out analyses that broke the rules of valid inference. The disagreement was not sloppiness; it was the ordinary flexibility of data analysis, amplified by the fact that an AI analyst can be instructed, explicitly or through a persona, to lean one way before it starts.

Where the method comes from: 73 humans, one dataset

The many-analyst design did not start with AI. It traces most visibly to a 2022 PNAS study led by Nate Breznau and Eike Mark Rinke, in which 73 independent research teams analyzed identical cross-national survey data to test whether rising immigration reduces public support for government social policy. The teams' results, the paper reports, ranged from large negative to large positive effects of immigration on support for social policy, and more than 95% of the variance in those numerical results remained unexplained even after the authors coded every identifiable decision each team made in its workflow.

That finding reframed how methodologists talk about "researcher degrees of freedom": the problem is not that any one team did bad work, but that the space of defensible analytic choices is large enough to generate almost any numerical answer, and prior expertise does little to predict which answer a given team will land on. The 2026 AI-analyst paper sits in that tradition deliberately. It uses the same multi-analyst logic, but produces the dispersion mechanically and at a fraction of the coordination cost, which is why the authors call it an "agentic data science multiverse."

Significance moved. Effect sizes mostly didn't.

One complication from the human-analyst literature is worth carrying into how a reader interprets any of this: statistical significance and effect-size magnitude can tell different stories about the same dispersion. A 2023 PNAS reanalysis of the Breznau dataset by Maya Mathur, Christian Covington, and Tyler VanderWeele, "Variation across analysts in statistical significance, yet consistently small effect sizes," found that 25% of the 73 teams' estimates were significant and negative, 58% were nonsignificant, and 17% were significant and positive, a dramatic split by the standard that most readers use to judge a result. But when the authors looked at the actual point estimates instead of pass or fail significance tests, 90% of the standardized estimates fell within the range −0.037 to 0.037, which the paper describes as more than fivefold smaller than conventional benchmarks for a "small" effect. Teams disagreed loudly about whether an effect existed; they mostly agreed, in magnitude, that it was tiny.

The 2026 AI-analyst paper does not repeat that particular decomposition, but its own numbers invite the same caution. A 34-to-66-point swing in hypothesis-support rate is a verdict-level statistic, built from how often an analysis crossed a significance threshold or produced a conclusion in one direction. It is not, by itself, a statement about how large or how certain the underlying effect is. Readers who see a headline figure like that should ask the same follow-up Mathur and colleagues asked of the human data: what did the actual effect-size estimates look like across those runs, not just which side of a threshold they landed on.

What this means for reading a single study

None of this is an argument that any individual finding is wrong. It is an argument that any individual finding is one draw from a space of defensible analyses that could have gone differently, and that the size of that space is often invisible to a reader who only sees the published numbers. The 2026 paper's authors propose a specific fix, writing in the published version that "AI-generated analyses should be accompanied by multiverse-style reporting and full disclosure of the prompts used, on par with code and data." Multiverse-style reporting, also called specification-curve analysis in some fields, means showing the distribution of results across many reasonable analytic choices rather than a single preferred one.

For a reader evaluating any quantitative study, human- or AI-assisted, several questions follow directly from this body of work:

• Is this the only analytic specification the authors tried, or do they report how results vary across reasonable alternatives?
• Is the headline a significance verdict (did it cross a threshold) or an effect-size estimate (how big was it, with what interval)? These can diverge sharply, as the Mathur reanalysis showed.
• If an AI system was used anywhere in the data-cleaning, modeling, or inference pipeline, were the prompts or instructions disclosed, given that the 2026 study found persona framing alone moved verdicts by dozens of percentage points?
• Would a differently motivated, equally competent analyst, human or AI, have been likely to reach the same conclusion from the same data?

The limits of the new evidence

The 2026 study covers three datasets and four language models, screened by an AI auditor whose own judgment was not independently validated against human expert review in the published paper. A 67% compliance pass rate also means roughly a third of all runs were discarded for failing basic methodological screening before the dispersion analysis even began, a reminder that AI analysts can produce invalid work as well as valid-but-divergent work, and that the two failure modes are not the same problem. The authors' own framing treats this as a demonstration of a mechanism, not a claim that any particular published AI-assisted finding is unreliable.

What the paper adds to the record is a controlled, repeatable version of something the human many-analyst literature had already shown with far more labor: that a single analysis, however carefully done, is a sample from a distribution of plausible analyses, and that distribution is worth asking about before treating one number as the answer.