A shared task of the ELOQUENT lab · CLEF 2027
Each year we publish a new fragment of a language in Montague's sense: a grammar, invented words, rules that map sentences to logic, and a few small models. Systems parse it, interpret it and reason in it. Every answer is checked by a parser, a model checker or a theorem prover. No fragment has ever been seen in training.
parseinterpretinferautomatic scoring
A model in practical use is often handed rules it has never seen: a style guide, a domain ontology, an in-house format. We want to know whether it follows such a specification or falls back on what it already knows. Most benchmarks cannot tell these apart, because the grammar, the words or the logic were in the training data. COGS and SLOG train the grammar in; MTOB gives a model an informal grammar book; RuleTaker and ProofWriter state rules over a language the model already speaks. If the language is familiar, a right answer does not show the rules were used.
Fragments removes the confound by construction. The grammar is new, the words are invented, and in one variant a familiar-looking word is given a contradicting meaning, so a system that trusts appearances goes visibly wrong.
The same sentence from the practice fragment, as it appears in each of the three item types. Answers are one JSON object per line.
parse
gesdam zispam gigup wonbis
answer
{"grammatical": true,
"tree": "[S [NP [Det gesdam] [N zispam]]
[VP [TV gigup] [NP [PN wonbis]]]]"}
checked by the parser
interpret
gesdam zispam gigup wonbis
answer
{"lfs": ["! [X] : (zispam(X) =>
gigup(X,wonbis))"],
"truth": [{"M1": false, "M2": false}]}
checked on M1, M2 and on models we withhold
infer
premises: gesdam zispam gigup wonbis
zusdit zispam mippos
conclusion: zusdit zispam gigup wonbis
answer
{"label": "entailment",
"lfs": {...},
"certificate": {"proof": "..."}}
formulas checked; label re-derived from them by Prover9
Any logical form equivalent to the gold counts; nothing is string-matched. For a neutral inference the certificate is two models of the premises in which the conclusion differs. For ambiguous sentences the gold is a set of readings.
Every item has twins with the same abstract tree. Consistency across them is what the task measures.
Invented words, the default. Rules out lexical memorisation.
gesdam zispam gigup wonbis
The same trees with English words. Shows how much comes from familiar-language priors.
every cat likes Tam
The same rules in an unnatural word order. Is the grammar read or assumed?
wonbis gigup zispam gesdam
Nonce words, but the determiners meaning every and no have swapped meanings. A system that trusts how words look fails here; the drop from nonce to counter is the headline number per system.
gesdam = \P.\Q. ~exists x (P(x) & Q(x))
Items are tagged by phenomenon level and scores are reported per level. FraCaS sections give the template.
Give the four specification files and the item file of each variant to your system, and return one JSON line per item: one answer file per variant, plus a short submission.json that declares your track and system type. That is the whole interface. We ship a prompting baseline that does exactly this, so a submission can be as small as one model run, and a validator that checks your files before you send them and explains every problem by line number. Details: the participant guide and the submission format.
| Track | What it means |
|---|---|
| model alone | the model's outputs from the specification in its context |
| model with tools | the model with a prover, or with code it writes itself |
Each submission declares its system type (prompted model, prover loop, program synthesis, neuro-symbolic, fine-tuned) and comes with a short experiment report. Results are reported by type rather than on one leaderboard. This is not a competition: several submissions per team and negative results are welcome. In an optional builder/breaker round, modelled on Voight-Kampff, you may submit new items with gold; items that pass our checker are run against all other systems.
| Item | Checked by | Metric |
|---|---|---|
| parse | the organisers' parser | grammaticality accuracy; derivation match |
| interpret | model checker on released and withheld models | truth-value agreement on every model; set equality of readings for ambiguous items |
| infer | Prover9 and Mace4 on the submitted formulas | label accuracy and certificate validity, reported separately |
Gold is never annotated; it is computed from the specification. The generator is public. The test fragment comes from a fresh, secret seed, which we publish when the test window closes, so anyone can regenerate and check the gold. The specification itself, scored as a system, gets 100% by construction and appears in every table as the ceiling, next to a human baseline from a linguistics seminar.
The generator, scorer, baseline and the practice fragment are in the task repository. The test fragment, from a fresh and secret seed, is released only in the test window.
No Data Task team: Maria Barrett, Stergios Chatzikyriakidis, Jussi Karlgren, Bruno Nadalic Sotic, Aarne Talman.
ELOQUENT 2027 lab organisers: Maria Barrett, Sarah Bouaraba, Stergios Chatzikyriakidis, Diandra Fabre, Federica Gamba, Lorraine Goeuriot, Rohit Raj Gunti, Jindřich Helcl, Jussi Karlgren, Jindřich Libovický, Josiane Mothe, Philippe Mulhem, Bruno Nadalic Sotic, Lucie Poláková, Yves Scherrer, Didier Schwab, Pavel Šindelář, Aarne Talman, Markarit Vartampetian, Erik Velldal, Eline Visser, Lilja Øvrelid.
Contact the ELOQUENT organisers: