A Count of 248 Research-Assessment Reform Plans Finds Values Stated, Operational Change Lagging

When more than 800 research organisations sign a pledge to stop over-relying on publication counts and journal metrics, the pledge itself is easy to count. What the organisations actually do afterwards is harder to see. A new study in Scientometrics tries to close that gap by reading, term by term, the implementation documents that signatories were required to file.

The study, by Alexander Rushforth, Nino Gogadze, Tornike Skhirtladze and Janne Pölönen, examines the Coalition for Advancing Research Assessment, known as CoARA, which grew out of an Agreement on Reforming Research Assessment published on 20 July 2022 and formally launched on 1 December 2022, when more than 300 organisations attended its constitutive assembly, according to the European Commission's account of the launch. Signatories commit to moving away from mechanical use of citation counts and journal rankings and toward qualitative, peer-based evaluation that also recognises open data, software, teaching and other contributions. Each is required to publish an action plan within a year of signing, setting out how it will change its own hiring, promotion and evaluation practices.

The research team coded 248 of these publicly available action plans, submitted by signatories across 35 countries and 19 transnational organisations, tracking how often specific open-science terms appeared and whether they were tied to concrete operational commitments or left as general statements of intent.

Words that appear, and words that don't

The umbrella term "open science" appeared in 167 of the 248 plans, or 68%, making it by far the dominant framing. "Open research" was used in 20% of plans and "open scholarship" in just 2%, suggesting most institutions default to a shared vocabulary rather than developing their own. Beyond that headline term, the authors report a skewed distribution: most of the specific open-science practices they searched for, such as recognition of open data or software in evaluation criteria, were mentioned in fewer than 20% of documents.

Where the plans came from also varied by region. Submissions were relatively evenly split between Nordic countries (23% of plans), Southern Europe (24%) and Western and Central Europe (21%), with Eastern Europe contributing a smaller share, 15%. By institution type, universities dominated the sample at 66% (164 of 248), followed by research centres and research funders at roughly 10.5% each (26 apiece), academies and learned societies at 6.5% (16), research assessment agencies at 3.6% (9) and other non-profits at 2.8% (7).

Signing up is not the same as showing up

A companion working paper by two of the same authors, Pölönen and Rushforth (2025), puts the action-plan sample into a wider institutional context. Merging CoARA's signatory list with the European Tertiary Education Register and other public sources across 3,439 European higher-education institutions in 40 countries, they find that 450 institutions, or 13%, have signed the Agreement. Uptake is far from uniform: 27% of PhD-awarding universities have signed, rising to 52% among research-intensive institutions ranked in the Leiden Ranking, while participation is consistently lower among teaching-focused institutions and among institutions in several Eastern European and other "widening" systems still building research capacity. The same paper finds that 36% of institutions have signed at least one reform initiative when CoARA is counted alongside the San Francisco Declaration on Research Assessment (DORA), the HR Excellence in Research framework and audits by the European Quality Assurance Register for Higher Education (EQAR), with 14% signed up to more than one.

Values ahead of practice

Taken together, the two studies point to the same conclusion from different angles: joining the coalition and stating its values is comparatively easy; rewriting the criteria that actually decide who gets hired, tenured or promoted is not. The Scientometrics paper describes this as "real but uneven follow-through" — plans consistently restate the Agreement's language, but references to specific operational changes, such as rewritten promotion criteria, altered hiring-panel instructions or revised evaluation forms, are markedly rarer than the values statements that precede them.

That gap matters for how the reform should be read from outside Europe, where the great majority of signatories and, in turn, of the action plans analysed are based. CoARA's own membership figures list signatories in more than 40 countries, some well beyond Europe, but the dataset behind both studies is concentrated enough that its authors describe their regional findings as a read on European reform dynamics specifically, not a global one.

What to look for in an action plan

For a researcher trying to judge whether their own institution's plan amounts to more than a restatement of values, both studies point to the same checklist: does the document name a specific body responsible for revising hiring or promotion criteria, does it set a timeline, and does it commit to reporting back. CoARA's own agreement requires signatories to provide a progress update by 2027 or within five years of signing, whichever comes first, which gives the next round of content analysis a natural point of comparison.

For now, the count stands at 248 documents read closely enough to say what they contain, out of a coalition that, by its own tally, now numbers more than 800 signing organisations.