A 25,163-Person Registered Report on Word Priming, and the Deviations It Logged Along the Way

In January 2026, Nature Human Behaviour published a registered report with 324 listed authors from 126 research labs in 41 geopolitical regions, measuring a textbook cognitive effect — semantic priming — in 19 languages at once. The paper, "Measuring the Semantic Priming Effect Across Many Languages", is worth reading not just for what it found but for how openly it shows its own seams. The Stage 1 protocol was "accepted in principle" by the journal on 15 July 2022, before any data were collected, and the published paper carries a public list of every place where the executed study departed from that plan. For anyone trying to judge whether a piece of research evidence can be trusted, that list is as instructive as the result itself.

Semantic priming is the finding that people recognize a word faster when it follows a related word (DOG before CAT) than an unrelated one (BUS before CAT). It has been studied for close to 50 years, largely in English, and the corresponding author, Erin M. Buchanan of Harrisburg University of Science and Technology, led a team assembled through the Psychological Science Accelerator, a network built for exactly this kind of large, distributed data collection. The stated goal was to replace a patchwork of small, mostly English-language priming studies with one coordinated, cross-linguistic dataset built from matched stimuli.

The scale is the headline number: 35,904 participants opened the study link, 31,645 completed at least one trial, and under the study's stricter preregistered accuracy rule, 25,163 participants across 19 languages were retained for the two preregistered analyses. Data collection reached 30 languages or dialects in total, but only 19 — including Czech, Danish, English, German, Japanese, Korean, Russian, Simplified Chinese, Spanish and Turkish — met the minimum sample size the team had set in advance.

Planning the Sample Size Before Seeing Any Data

The detail most relevant to evaluating evidence sits earlier than the results: how the researchers decided how much data to collect. Instead of the common convention of using roughly 30 participants per item, the team used a method called accuracy in parameter estimation (AIPE), which sets a target width for a confidence interval and samples until an item reaches it. They derived that target from two existing datasets — the English Lexicon Project and the original Semantic Priming Project — simulating how standard errors shrink as sample size grows, which produced a minimum of 50 participants and a maximum of 320 participants per word pair.

That translated into a planned range of 741 to 4,741 participants per language, and the study used an adaptive algorithm that kept sampling under-measured word pairs while reducing the sampling probability of pairs that had already reached the target precision. In practice, this produced what the authors describe as more than 21 million lexical decision trials, collected by 126 labs that had each completed ethics review — either locally (81 labs), through Harrisburg University's review board (31 labs), or by documenting that no review was required (14 labs).

What the Two Preregistered Hypotheses Actually Found

The design table, fixed before data collection, specified exactly how each hypothesis would be judged. Hypothesis 1 asked whether semantic priming is a non-zero effect, to be supported only if the lower bound of the 95% confidence interval for the pooled priming effect exceeded zero. The result: a standardized priming effect of b0 = 0.12 (SE = 0.001, 95% CI [0.11, 0.12]), equivalent to a raw response-latency advantage of 30.61 milliseconds (95% CI [29.78, 31.45]) for related word pairs over unrelated ones. That raw figure sits close to the 23–25 millisecond effect reported by the original Semantic Priming Project, which the new study was explicitly designed to extend beyond English.

Hypothesis 2 asked whether the size of priming varies across languages, with a preregistered decision rule: support the hypothesis only if adding a random intercept for language improved model fit by at least 2 points of AIC. It did, by a wide margin — the AIC difference was 97.84 for the full dataset, 135.01 and 126.07 under two different outlier-exclusion rules the team also ran. But the practical size of that variation was small: the standard deviation of the random effect was 0.02 (95% CI [0.01, 0.03]), and the paper reports a pseudo-R² of .0160 for the model, describing the cross-language variation in the discussion as "likely small." Only Portuguese and Simplified Chinese produced confidence intervals in the forest plot that did not overlap with each other; most of the 19 languages' effects fell inside the same narrow band.

A Deviation That Preregistration Didn't Prevent

The paper's "Deviation List" is where the preregistration process shows its limits rather than its strengths, and that is precisely what makes it useful to read. The protocol had defined participant accuracy as the number of correct answers divided by the number of trials seen. During data processing, the team discovered the same formula could reasonably be read a second way — correct answers divided only by the trials a participant actually answered, excluding timeouts. The two readings are not interchangeable: the stricter, as-written formula retained 25,163 participants for analysis, while the alternative reading would have retained 28,162. Rather than quietly picking whichever number looked better, the authors used the stricter preregistered formula for their primary analysis and reported the alternative version in their supplementary materials, flagging the ambiguity explicitly in the manuscript.

Other logged deviations are smaller but equally concrete: for Korean, a shuffling step meant for pushing down the similarity of "unrelated" word pairs accidentally produced mismatched related-unrelated sets, requiring 33 new trial combinations to be added after the fact; a stimulus-writing bug meant some participants' early trial blocks had to be excluded from analysis; and in Arabic, specific word pairs were dropped from the stimulus set at the request of local labs because of their taboo content, which reduced the total number of possible items in that language alone. None of these changed the study's two headline conclusions, but each is named, dated in effect, and tied to the specific language or procedure it touched — rather than folded invisibly into a single, tidier-looking final dataset.

Why the Reliability Numbers Matter as Much as the Headline Effect

Buried past the hypothesis tests is a number that complicates how confidently any single word pair's priming score should be trusted. Item-level reliability — how consistently a given word pair's priming effect reproduces when the data are split in half — averaged .56 across languages. Participant-level reliability, the consistency of priming within a single person's own responses, averaged just .08. The paper connects this to two earlier studies it cites, by Tom Heyman and colleagues and by Keith Hutchison and colleagues, which had already found low reliability for item-level priming effects in the original, smaller Semantic Priming Project; the new, much larger dataset improved on those earlier estimates but did not erase the underlying pattern, noting that languages with the highest item reliability in this study were Japanese, English and Russian.

That combination — a robust, non-zero pooled effect alongside comparatively weak reliability at the level of individual items and individual participants — is a reminder that a statistically solid headline finding and a trustworthy item-by-item measurement are different claims, answered by different statistics within the same paper. A reader who only takes in the confidence interval for Hypothesis 1 would miss that the study's own authors flag its item- and participant-level numbers as noisier, and recommend that anyone building future experiments on this dataset weight that caveat accordingly.

Reading a Registered Report as a Reader, Not Just a Citation

What makes this paper a useful model for evaluating evidence generally is less its specific finding about word recognition than its structure. The Stage 1 in-principle acceptance date, the preregistration links, the sample-size derivation, and the deviation list are all dated and cross-referenced against each other, so a reader can check whether a given analysis was planned in advance or added afterward. The accuracy-formula ambiguity in particular shows that preregistration does not eliminate interpretive choices in research — it just forces them into the open, with both readings reported side by side rather than one silently chosen. For a dataset this size, built from a continuous lexical decision task programmed in the open-source tool lab.js and shared in full on GitHub and the Open Science Framework, that openness is what lets another researcher re-run the disputed step themselves rather than take the authors' word for which version was right.