Valencia researchers: a 184M encoder beats a 123B LLM at value detection

Two researchers at the Universitat Politècnica de València ran one value-detection task through five models, and the smallest one won. A 184M-parameter DeBERTa-v3-base encoder scored .314 macro-F1, against .241 for the best zero-shot large language model they tested. The fourth version of the paper went up on arXiv on 7 September, and it’s accepted to Findings of ACL: EMNLP 2026.

The task comes from ValueEval’24 at Touché, which annotated roughly 3,000 texts of 400 to 800 words. Every sentence is labeled on its own against the 19 refined Schwartz values, among them Tradition, Hedonism and Security: societal. That’s hard, because the cues that justify a label frequently sit in the text around it. Just under half the test sentences carry no value at all.

Víctor Yeste and Paolo Rosso varied one factor at a time: how much surrounding text a model sees, and whether it also gets retrieved definitions of the values. That’s what makes the numbers below readable, so here’s the test macro-F1 across three of those conditions.

Model	Sentence only	Full document	Document + retrieval
DeBERTa-v3-base (184M, supervised)	.260	.285	.314
DeBERTa-v3-large (435M, supervised)	.242	.280	.294
Gemma 3 12B IT (zero-shot)	.198	.181	.202
Qwen2.5-72B-Instruct (zero-shot)	.215	.171	.194
Mistral-Large 123B (zero-shot)	.208	.202	.220

More context helped the supervised encoders but hurt the prompted ones. Replacing the sentence with its full document added .025 macro-F1 for the base encoder and .038 for the large one. The same swap cost Gemma .017 and Qwen .044.

here, a few hundred million supervised parameters outperform 123B unsupervised ones

Yeste and Rosso, arXiv:2605.22641

The authors are careful about what that comparison shows, since the encoders were trained on this task and the language models weren’t. It’s still a number worth having if you’re weighing a small tuned model against a large general one. Scale wasn’t monotonic among the factors they varied either, because DeBERTa-v3-large didn’t reliably improve on base.

The moral knowledge base is small enough to read by hand: 58 curated chunks, made up of 19 value definitions, 25 annotation guidelines and 14 entries on how neighboring values differ. Retrieval picks the two nearest chunks by dense sentence-embedding similarity. Adding them improved every model family and every context, by .013 to .030, but the reason turned out to differ by family.

Then the researchers swapped those chunks for two drawn at random from the same 58. Most of the encoder’s gain survived: random chunks scored .311 against .314 for retrieved ones, so relevance was worth .003 and that isn’t significant. Gemma behaved the other way, gaining .013 to .024 from relevance and nothing reliable from random text. That points to a trained encoder benefiting from the value ontology being visible in its input at all, whichever chunks arrive.

None of this makes the task easy, though. The best score anywhere for Humility, Self-direction: thought and Conformity: interpersonal stays below .18 F1. Retrieval also churns predictions rather than quietly adding accuracy: among sentences whose label set changed, 23% became exactly correct and 20% stopped being correct.

The study covers one benchmark, one English label space, and one genre of political news and manifestos. The language model scores are zero-shot, which the paper calls a floor rather than a ceiling. Code and prompts are on GitHub, and the best encoder checkpoint is on Hugging Face, where the released seed scores .3224 macro-F1.

The condition they didn’t run is still the one worth watching. Few-shot prompting with training-split examples could make performance scale with model size, and the authors say so plainly. If it did, supervision and scale would reinforce each other here instead of competing.