A shared task of the ELOQUENT lab · CLEF 2027

Fragments
Can a language model learn a language from its specification alone?

Each year we publish a new fragment of a language in Montague's sense: a grammar, invented words, rules that map sentences to logic, and a few small models. Systems parse it, interpret it and reason in it. Every answer is checked by a parser, a model checker or a theorem prover. No fragment has ever been seen in training.

parseinterpretinferautomatic scoring

lexicon.txt PN: wonbis, mislup N: petfat, zukka, zispam IV: mippos, wutbu TV: dorom, gigup Det: gesdam, zusdit, tamwup grammar.txt S -> NP VP NP -> Det N | PN | Det N RC RC -> that VP | that NP TV VP -> IV | TV NP | not VP semantics.txt [S NP VP] = NP(VP) [VP TV NP] = \x. NP(\y. TV(x,y)) gesdam = \P.\Q. forall x (P(x) -> Q(x)) zusdit = \P.\Q. exists x (P(x) & Q(x)) tamwup = \P.\Q. ~exists x (P(x) & Q(x)) models.json M1: D={a,b} petfat={a,b} zukka={b} mippos={a} dorom={(a,b)} wonbis=b mislup=a ...

Why

A model in practical use is often handed rules it has never seen: a style guide, a domain ontology, an in-house format. We want to know whether it follows such a specification or falls back on what it already knows. Most benchmarks cannot tell these apart, because the grammar, the words or the logic were in the training data. COGS and SLOG train the grammar in; MTOB gives a model an informal grammar book; RuleTaker and ProofWriter state rules over a language the model already speaks. If the language is familiar, a right answer does not show the rules were used.

Fragments removes the confound by construction. The grammar is new, the words are invented, and in one variant a familiar-looking word is given a contradicting meaning, so a system that trusts appearances goes visibly wrong.

One sentence, three item types

The same sentence from the practice fragment, as it appears in each of the three item types. Answers are one JSON object per line.

parse

gesdam zispam gigup wonbis

answer

{"grammatical": true,
 "tree": "[S [NP [Det gesdam] [N zispam]]
          [VP [TV gigup] [NP [PN wonbis]]]]"}

checked by the parser

interpret

gesdam zispam gigup wonbis

answer

{"lfs": ["! [X] : (zispam(X) =>
            gigup(X,wonbis))"],
 "truth": [{"M1": false, "M2": false}]}

checked on M1, M2 and on models we withhold

infer

premises: gesdam zispam gigup wonbis
          zusdit zispam mippos
conclusion: zusdit zispam gigup wonbis

answer

{"label": "entailment",
 "lfs": {...},
 "certificate": {"proof": "..."}}

formulas checked; label re-derived from them by Prover9

Any logical form equivalent to the gold counts; nothing is string-matched. For a neutral inference the certificate is two models of the premises in which the conclusion differs. For ambiguous sentences the gold is a set of readings.

Four variants of every item

Every item has twins with the same abstract tree. Consistency across them is what the task measures.

nonce

Invented words, the default. Rules out lexical memorisation.

gesdam zispam gigup wonbis

english

The same trees with English words. Shows how much comes from familiar-language priors.

every cat likes Tam

scrambled

The same rules in an unnatural word order. Is the grammar read or assumed?

wonbis gigup zispam gesdam

counter-English

Nonce words, but the determiners meaning every and no have swapped meanings. A system that trusts how words look fails here; the drop from nonce to counter is the headline number per system.

gesdam = \P.\Q. ~exists x (P(x) & Q(x))

What is hard, in order

Items are tagged by phenomenon level and scores are reported per level. FraCaS sections give the template.

1
Quantification: every, some, no 2027
2
Negation and monotonicity 2027
3
Relative clauses and scope ambiguity, where the gold is a set of readings 2027
4
Pronouns and donkey sentences, with a dynamic semantics
5
Plurals and comparatives
6
Intensional contexts, with worlds in the models

How to take part

Give the four specification files and the item file of each variant to your system, and return one JSON line per item: one answer file per variant, plus a short submission.json that declares your track and system type. That is the whole interface. We ship a prompting baseline that does exactly this, so a submission can be as small as one model run, and a validator that checks your files before you send them and explains every problem by line number. Details: the participant guide and the submission format.

TrackWhat it means
model alonethe model's outputs from the specification in its context
model with toolsthe model with a prover, or with code it writes itself

Each submission declares its system type (prompted model, prover loop, program synthesis, neuro-symbolic, fine-tuned) and comes with a short experiment report. Results are reported by type rather than on one leaderboard. This is not a competition: several submissions per team and negative results are welcome. In an optional builder/breaker round, modelled on Voight-Kampff, you may submit new items with gold; items that pass our checker are run against all other systems.

Scoring

ItemChecked byMetric
parsethe organisers' parsergrammaticality accuracy; derivation match
interpretmodel checker on released and withheld modelstruth-value agreement on every model; set equality of readings for ambiguous items
inferProver9 and Mace4 on the submitted formulaslabel accuracy and certificate validity, reported separately

Gold is never annotated; it is computed from the specification. The generator is public. The test fragment comes from a fresh, secret seed, which we publish when the test window closes, so anyone can regenerate and check the gold. The specification itself, scored as a system, gets 100% by construction and appears in every table as the ceiling, next to a human baseline from a linguistics seminar.

Timeline

January 2027
Practice fragment with full gold, scoring scripts and baseline released
April 2027
Test fragment released; two-week submission window; break submissions
May 2027
Scores returned; experiment reports due
14–17 Sept 2027
CLEF 2027, Bucharest: results and the builder/breaker round

Code and data

The generator, scorer, baseline and the practice fragment are in the task repository. The test fragment, from a fresh and secret seed, is released only in the test window.

RepositoryELOQUENT lab

Organisers

No Data Task team: Maria Barrett, Stergios Chatzikyriakidis, Jussi Karlgren, Bruno Nadalic Sotic, Aarne Talman.

ELOQUENT 2027 lab organisers: Maria Barrett, Sarah Bouaraba, Stergios Chatzikyriakidis, Diandra Fabre, Federica Gamba, Lorraine Goeuriot, Rohit Raj Gunti, Jindřich Helcl, Jussi Karlgren, Jindřich Libovický, Josiane Mothe, Philippe Mulhem, Bruno Nadalic Sotic, Lucie Poláková, Yves Scherrer, Didier Schwab, Pavel Šindelář, Aarne Talman, Markarit Vartampetian, Erik Velldal, Eline Visser, Lilja Øvrelid.

Contact the ELOQUENT organisers:

eloquent-clef2027-organizers@googlegroups.com