Skip to main content

18. Experimental Setup

This section is dry and necessary. Reviewers read it to decide whether to believe your results. Make that check easy.

Where the line falls. New authors often blur this section with Methods. Methods says what your contribution is. Experimental setup gives everything a reader needs to re-run it and trust the numbers: data versions, baselines, metrics, hyperparameters, seeds, compute. The test is simple. If a detail is part of the idea, it goes in Methods. If it is a knob you turned to get a number, it goes here. Hyperparameter values are setup. If your paper's contribution is a dataset, much of the construction lives in Dataset Construction instead, but you still report baselines, metrics, and splits here.

What belongs here​

  • Datasets. Name, size, splits, source, license. If you built it, point to the construction details (usually a separate section or appendix). If you used a standard benchmark, cite the version (GLUE vs. SuperGLUE; SQuAD 1.1 vs. 2.0).
  • Baselines. What you compare to and why those. The honest answer is "the most competitive prior methods we could run." Reviewers will check whether you ducked a strong baseline.
  • Metrics. Which ones, and why each fits. If you use a non-standard metric, justify it, and report the standard ones too where you can.
  • Implementation details. Model size, framework, hardware, training time: enough for another researcher to reproduce the work.
  • Hyperparameters and selection procedure. How did you choose them: grid search, prior work, default? Did you tune on the validation set or the test set? (It must be the validation set.)
  • Statistical practice. Do you average results over seeds? How many? Are differences statistically significant? Card et al.'s "With Little Power Comes Great Responsibility" is mandatory reading here.

Make the numbers mean something​

A metric is a proxy. The only question that matters is whether it tracks what you care about. If you report a number because it is standard, without asking whether it is valid for your claim, your results may not mean what they say. Three habits keep you honest. (Significance, seeds, and power are covered above and in the phrasebook below. This part asks whether you are measuring the right thing at all.)

Check that your metric measures your claim. Automatic metrics can diverge sharply from human judgment. A structured review of 284 reported correlations found BLEU's agreement with human quality ratings far weaker and more variable than its ubiquity suggests (Reiter, Computational Linguistics 2018). Accuracy misleads under class imbalance. Overlap metrics like BLEU and ROUGE reward surface similarity over meaning. Choose a metric because it is valid for what you claim, and report a second when one cannot carry the claim. Remember Goodhart (Chapter 5): once a number becomes the target, it stops measuring what you wanted.

Use human evaluation when metrics cannot reach. For generation and other subjective tasks, automatic scores are necessary but not sufficient, and human evaluation is the standard. Yet papers run it so inconsistently that comparison across them breaks down (van der Lee et al., INLG 2019). Do it properly. Define each criterion precisely; fluency, adequacy, and faithfulness are different things. Use enough items and annotators for adequate power. Report inter-annotator agreement (Chapter 16), pay fairly (Chapter 23), and release the protocol and the raw ratings. Eyeballing a handful of outputs is "vibes-based evaluation" (Chapter 5), and it is not human evaluation.

Treat an LLM judge as an instrument to be calibrated. Scoring outputs with a language model is cheap and increasingly common. It also carries documented biases: a position bias toward whichever answer it sees first, a verbosity bias toward longer answers, and a self-enhancement bias toward its own outputs (Zheng et al., NeurIPS 2023). Before you trust an LLM judge, validate it against human ratings on a sample. Control for position by swapping answer order, report the exact model and prompt, and be wary of letting a model grade its own family's outputs. An unvalidated LLM judge is a metric of unknown validity, the problem this section opened with.

Phrasebook: the moves of an experimental setup​

The reporting sentences of an experimental setup are formulaic. That is good news: the moves are learnable, and the goal is clarity. As in the Dataset Construction and Methods phrasebooks, most entries below are adaptable templates with [bracketed] slots. Fill every bracket with a specific, and never report a number you did not compute. A few entries in the last group are real, verified sentences from the methodology papers that argue for honest reporting; quote them when you need the authority. Two entries sit in the open under each move; expand Show more for the rest.

Datasets, benchmarks, and versions​

  • "We evaluate on [dataset] ([cite]), using the [official / standard] train/dev/test split of [sizes]." Names the data, the citation, and the exact split in one sentence.
  • "We use [benchmark] version [N.N] ([cite]) and report on the [public dev / held-out test] set, because [reason]." Pins the version and the evaluation set. "SQuAD" without "1.1 or 2.0" is a hole.
Show 5 more
  • "For [dataset] no canonical split exists; we create one by [method] and release it so others can compare." Discloses a self-made split and commits to releasing it.
  • "We report on [N] of the [M] tasks in [benchmark], excluding [tasks] because [reason]." Honest about partial coverage instead of quietly dropping tasks.
  • "All datasets are English unless noted; [dataset] additionally covers [languages]." States the language scope, which English-default papers routinely omit.
  • "We use the [filtered / deduplicated] version of [dataset] from [cite], not the original, to avoid [issue]." Names which release you used, since versions of the same dataset differ.
  • "Dataset statistics are in Table [N]; construction details are in [Dataset Construction / Appendix N]." Points back without re-narrating the collection.

Baselines and what you compare to​

  • "We compare against [A] ([cite]), [B] ([cite]), and [C] ([cite]) — the strongest published methods on [task] that we could run." States the comparison set and, implicitly, that you did not duck a strong one.
  • "Where published numbers are not comparable (different [split / metric]), we re-run the baseline under our setting; otherwise we cite the reported score and mark it." Pre-empts the "is this a fair comparison?" objection.
Show 5 more
  • "[Baseline] is the current best published result on [benchmark]; we treat it as the bar to clear." Names the bar explicitly, so the reader knows what beating it means.
  • "We include a [majority / random / simple] baseline to establish the floor on [task]." The lower bound that contextualises every other number.
  • "For a controlled comparison, all baselines share the same [backbone / data / tokeniser]; only [the variable] changes." Signals the comparison isolates one factor.
  • "We omit [baseline] because [it is inapplicable / code is unavailable / it needs resources we lack]; Section [N] discusses the gap." Name the missing comparison before a reviewer does.
  • "Scores marked † are from the original papers; all others are from our runs." A one-line provenance convention for a mixed results table.

Metrics, and why each one​

  • "We report [metric] as the primary measure because [it matches the task / prior work uses it], with [secondary metric] as a [complementary / diagnostic] check." Justifies the metric instead of printing it bare.
  • "Because the labels are imbalanced, we report [macro-F1 / per-class F1] rather than accuracy." Ties the metric choice to a property of the data.
Show 5 more
  • "[Metric] is computed with [official script / library + version] for comparability with [cite]." Pins the implementation; metric definitions drift between toolkits.
  • "We report [generation metric] but read it cautiously; Section [N] adds the human evaluation it cannot replace." Flags the known limits of automatic generation metrics.
  • "For [task] we follow the official evaluation of [benchmark]: [metric] under [protocol]." Defers to the benchmark's protocol rather than inventing one.
  • "We also report [latency / FLOPs / parameters], since our claim concerns [cost], not only accuracy." Match the metric to the actual claim.
  • "Significance of these differences is assessed in the [seeds and significance] paragraph below." A forward pointer that keeps the metric prose uncluttered.

Human and model-based evaluation​

  • "Because automatic metrics do not capture [quality dimension], we run a human evaluation: [N] annotators rate [N] outputs for [criterion] on a [scale]; inter-annotator agreement is [κ / α = value]." Reports a human evaluation with the details that make it credible.
  • "We use [model + version] as an automatic judge, validated against human ratings on [N] items (agreement [value]), and swap answer order to control for position bias." Reports an LLM judge as a calibrated instrument.
Show 3 more
  • "Annotators were [native speakers / domain experts] paid [rate]; the full protocol and the raw ratings are released." The fair-pay-and-transparency sentence human evaluation now needs.
  • "We validate [metric] against human judgment on [task], finding [correlation]; we therefore [trust / caution against] it for our claim." Establishes a metric's validity instead of assuming it.
  • "The judge's exact prompt is in Appendix [N]; scores shift by [amount] across prompt variants, so we report [how you handled it]." Pins the judge's configuration and reports its sensitivity.

Implementation, hyperparameters, and compute​

  • "We implement all models in [framework + version] and train on [hardware]; a single run takes [time]." The reproducibility basics in one sentence.
  • "Hyperparameters were selected by [grid / random / Bayesian] search over [ranges] on the validation set; the space and the chosen values are in Table [N]." States the search, and that it used validation data.
Show 5 more
  • "We adopt the hyperparameters of [cite] without further tuning, changing only [the few you changed]." Honest about inherited settings rather than implying a fresh search.
  • "Model sizes are [N] parameters for [model] and [M] for [baseline]; Table [N] lists them in full." Reports scale, which the reader needs to read any comparison fairly.
  • "All choices were made on validation data; the test set was used once, for the final reported numbers." The val/test-separation sentence reviewers look for.
  • "Total compute is approximately [GPU type] × [hours], including hyperparameter search and failed runs." The honest compute figure, not just the final model.
  • "Code, configs, and weights [will be released upon acceptance / are at the anonymised link]." The release sentence, kept anonymity-safe for double-blind review.

Seeds, variance, and significance (with real-paper anchors)​

In this group a number becomes a claim, or fails to. Report variation instead of a lone point estimate. Test whether a difference survives noise before you build an argument on it.

  • "All results are mean ± standard deviation over [N ≥ 3] seeds; Table [N] reports both." Turns a point estimate into a distribution, the minimum bar for a non-deterministic method.
  • "In this paper we show that reporting a single performance score is insufficient to compare non-deterministic approaches." The thesis that justifies reporting over seeds instead of a lone number. Reporting Score Distributions Makes a Difference (Reimers & Gurevych, EMNLP 2017)
Show 5 more
  • "test-set performance scores alone are insufficient for drawing accurate conclusions about which model performs best." Grounds the practice of also reporting validation performance and the search budget. Show Your Work: Improved Reporting of Experimental Results (Dodge et al., EMNLP 2019)
  • "Underpowered experiments make it more difficult to discern the difference between statistical noise and meaningful model improvements, and increase the chances of exaggerated findings." The reason to run a power analysis before claiming a small gain. With Little Power Comes Great Responsibility (Card et al., EMNLP 2020)
  • "We test significance with [paired bootstrap / approximate randomisation] ([cite]); differences marked * are significant at p < [0.05]." Name the test, not just the p-value.
  • "Our gain over [baseline] is [X] points (p = [value]); we report it as [significant / not] and over-claim neither way." States the result and its uncertainty in the same breath.
  • "Variance across seeds is [large / small] for [setting]; we therefore [report all runs / caution against fine-grained ranking]." Treats variance as information, not embarrassment.

Common mistakes​

  • The "see appendix" punt. Dataset sizes, baselines, metrics, and number of seeds belong in the main text. Pushing all of this to the appendix tells reviewers you do not want them to see it.
  • Single-seed results. One run with one seed is an anecdote. Average over at least three seeds wherever feasible, and report variance.
  • Test-set tuning. If you made hyperparameter choices while looking at test numbers, your results are invalid. Be religious about val/test separation.
  • Cherry-picked baselines. If a stronger baseline exists and you can run it, run it. Reviewers will notice the missing comparison.
  • No license / no provenance. Especially for new datasets or scraped corpora. The ACL Responsible NLP Checklist asks about this directly.
  • A metric that does not fit the claim. Reporting BLEU, ROUGE, or accuracy because it is standard, without asking whether it tracks what you claim.
  • An unvalidated LLM judge. Scoring outputs with a model and reporting the numbers as fact, without checking them against humans or controlling for position and verbosity bias.

A reproducibility minimum​

Before you submit, make sure the paper or its appendix answers all of these:

  • Exact training data and splits used.
  • Number of model parameters.
  • Hyperparameter ranges searched and selection criterion.
  • Number of random seeds used and how results are aggregated.
  • Total compute used (GPU type × hours).
  • Software versions (or a frozen requirements.txt link).
  • Whether code and weights will be released.

If any is missing, reviewers will ask. Answer it in the paper before you have to answer it in a rebuttal.

Five examples to study​

  1. GLUE and SuperGLUE benchmark papers (Wang et al.). How to describe a benchmark in detail.
  2. Don't Stop Pretraining (Gururangan et al., ACL 2020). Clean baseline structure across domains.
  3. MMLU (Hendrycks et al., ICLR 2021). Defines a benchmark and how it should be evaluated.
  4. Linguistic Knowledge and Transferability of Contextual Representations (Liu et al., NAACL 2019). 17 probing tasks described compactly with clear setup details.
  5. Show Your Work (Dodge et al., EMNLP 2019). Read it for its argument, and study its own experimental setup as a model.

Further reading​