16. Dataset Construction
If your paper's contribution is a dataset, a benchmark, or any other resource, this is your methods section. Reviewers read it with more suspicion than any other. A modelling paper that hides a detail loses a little credibility. A resource paper that hides how the data was made loses all of it, because the data is the contribution. Everything a reader needs to trust your numbers, reuse your data, and reproduce your pipeline lives here. Write it as if a skeptical reviewer will try to rebuild the dataset from your description alone, because the good ones will.
This chapter covers resource papers only: new datasets, benchmarks, annotated corpora, shared-task data, evaluation suites. The test is one question: is the resource itself the contribution? If yes, this section is your methods. The construction is the work a modelling paper describes in Chapter 17. If your contribution is a model or a method that only uses existing data, skip this section. Write Methods instead, and cover the data with one sentence in Experimental Setup that points to the original release. Retelling someone else's collection story is not your job. A paper that ships both a dataset and a model writes both sections, dataset first, since the model is trained and evaluated on it.
Where it goes, and what it replaces
In a resource paper this section usually comes right after Related Work. Its title varies (Data, Dataset, Dataset Construction, Data Collection, or Corpus), but its contents do not. Whatever you call it, it takes the slot a modelling paper gives to Methods. Baselines and metrics then go in Experimental Setup.
Most *ACL data and reproducibility criteria ask reviewers of resource papers a direct question. Are the collection and annotation documented well enough to judge data quality and to reuse the resource? Treat that question as the spec for this section.
What the section must report
A complete dataset section answers six questions. Skip one and a reviewer will assume the worst about it.
1. The task and the labels
State what the data is for before how you made it. What is the unit (sentence, document, turn, image–caption pair)? What is the label space, and what does each label mean? Give a one-line definition per label, with a borderline example. That stops half your annotation disagreements from looking like noise. If the task is novel, justify your label scheme here. A footnote listing three schemes you rejected is no justification.
2. Languages and language varieties
Be specific to the point of pedantry. "Multilingual" and "low-resource" are not data descriptions. Name every language, give its ISO 639-3 code, and say which script and which variety or dialect (Modern Standard Arabic is not Algerian Arabic; Nigerian Pidgin is not English). Note code-switching if your data contains it. Most real-world text from multilingual communities does, and pretending otherwise misleads everyone downstream. AfriSenti (Muhammad et al., EMNLP 2023) and MasakhaNER (Adelani et al., TACL 2021) are good models. Each lists its languages, families, and scripts instead of waving at "African languages." For typological breadth as a deliberate design goal, see TyDi QA (Clark et al., TACL 2020).
3. Data source and collection
Where did the raw text come from, and how did you get it? A reviewer needs provenance: the platform or corpus, the time window, the sampling method, the query terms or seed lists, and the size before filtering. State the license or terms of service under which you collected the data and may redistribute it. Say whether you redistribute the text itself or only IDs/links; platform terms often forbid the former. If the data involves people, as social-media and elicited data always do, say what consent and anonymisation you applied. Document every filtering and deduplication step, because each one shapes the distribution your models will learn.
4. Annotation
Resource papers are won or lost here. Report:
- Who annotated, and why they are qualified. Native speakers? Trained linguists? Crowdworkers? For any language other than high-resource English, native-speaker or fluent annotation is required, and reviewers will ask. Give the number of annotators.
- The guidelines. Summarise them and put the full version in an appendix or repository. Without written guidelines, a label scheme is a vibe.
- The process. Pilot rounds, annotator training, how many annotators saw each item, how you adjudicated disagreements, how the guidelines changed after early rounds.
- Compensation and ethics. Pay rate relative to local minimum wage, recruitment, and IRB/ethics approval where it applies. Underpaying annotators is an ethical failure and a growing desk-reject risk. See also Ethical Considerations.
- The tooling. Name the annotation interface or platform. It affects which errors are even possible.
5. Quality and inter-annotator agreement
To a reviewer, a dataset with no reported agreement is a dataset of unknown quality. Report an inter-annotator agreement statistic, and say which one and why. Use Cohen's κ for two annotators, Fleiss' κ for more, and Krippendorff's α when you have missing labels or non-nominal scales. Use raw percentage agreement only as a supplement. Interpret the number instead of just printing it, and say what you did about low-agreement items (adjudication, expert review, exclusion). Sometimes agreement is low because the phenomenon is subjective, as in much of pragmatics, sentiment, and toxicity. Then say so, and treat the disagreement as signal.
6. Statistics and splits
Put the statistics in a table. At minimum report total size, the train/dev/test split sizes, the label distribution per split, and basic shape (sentence/document length, vocabulary size, number of annotators). A class that appears only in test is a bug. Then explain the part most papers botch: how you made the splits and why they are leak-free. Random split? Stratified? Split by document, speaker, or time so that no entity straddles train and test? Deduplicated across splits? For benchmarks meant to test generalisation, contamination control is now a core concern. Say what you did to keep test data out of common pretraining corpora.
Phrasebook: the moves of a dataset section
The reporting sentences of a dataset section are formulaic, so you can learn the moves. As in the Related Work phrasebook, most entries below are adaptable templates with [bracketed] slots. They are not quotations. A few are real, verified sentences from well-known resource papers, marked with a citation. Fill every bracket with a specific, and never report a statistic you did not compute. Two entries sit in the open under each move; expand Show more for the rest.
Source and provenance
- "We collected [N] [units] from [source] posted between [start] and [end], using [sampling method / query terms]." Provenance in one sentence: how many, from where, when, and how sampled.
- "We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage." A real provenance sentence that gives the source (Wikipedia), the method (crowdworkers), and the format at once. SQuAD: 100,000+ Questions for Machine Comprehension of Text (Rajpurkar et al., EMNLP 2016)
Show 7 more
- "The raw corpus contains [N] [units]; after [filtering steps] and deduplication, [M] remain." Always report the before-and-after of filtering; the gap tells the reader how aggressive you were.
- "Data was gathered under [license / platform terms], which permit [redistribution / ID-only release]; we release [what you can actually share]." States the legal basis and what you can and cannot redistribute.
- "To protect privacy we removed [PII types] and replaced [usernames / handles] with placeholders." Names the anonymisation, which social and elicited data require.
- "Texts were drawn from [domain/genre A], [genre B], and [genre C] to ensure genre diversity." Justifies coverage when the source is heterogeneous.
- "We seeded collection with [seed terms / lexicons] curated by native speakers of [language]." The honest way to describe targeted collection for a specific language.
- "Documents shorter than [N] tokens or in a language other than [target] were discarded; language ID used [tool]." Names the filters and the tool, so the distribution is reproducible.
- "We sampled to [balance / preserve] the natural distribution of [variable]; Section [N] reports the resulting skew." States the sampling intent and points to the evidence.
Annotation process
- "Each [unit] was independently annotated by [k] [native-speaker / expert] annotators following written guidelines (Appendix [N]); disagreements were resolved by [adjudication / majority / expert]." The core sentence: how many annotators, who, by what rules, and how ties broke.
- "Annotators were [recruited from / employed via] [source], paid [rate], and completed [training / a qualification round] before annotating." Recruitment, pay, and training in one sentence. Reviewers now require all three.
Show 6 more
- "Guidelines were refined over [N] pilot rounds; the final version is released with the data." Signals an iterative, documented process rather than an ad-hoc one.
- "All annotators are native or fluent speakers of [language]; [N] of them are [trained linguists / domain experts]." Establishes annotator qualification, which is non-negotiable for non-English data.
- "Annotation was carried out in [tool / platform], which constrained labels to [the valid set] and recorded [timing / provenance]." Names the interface, which shapes which errors are possible.
- "A separate group of [k] annotators validated a random [N]% sample; [M]% of labels were confirmed." Describes a second-pass validation, a strong quality signal.
- "We obtained [IRB / ethics] approval ([reference]) for the collection and annotation involving human participants." The one-line ethics statement reviewers look for.
- "Annotators could flag items as [out-of-scope / offensive / unintelligible]; flagged items were [handled how]." Shows you planned for the messy items instead of forcing labels.
Agreement and quality
- "Inter-annotator agreement was [κ / α] = [value] ([metric name]), which we interpret as [substantial / moderate] for [this task]." Report the number, name the metric, and interpret it. Never print a bare coefficient.
- "We use Krippendorff's α rather than κ because [missing labels / ordinal scale / more than two annotators]." Justifying your choice of statistic shows you understand it.
Show 6 more
- "Agreement is lower on [subcategory] (α = [value]), reflecting the genuine subjectivity of [phenomenon]; we treat this disagreement as signal and release all raw annotations." The mature way to report low agreement on a subjective task.
- "Items with no majority label ([N], [P]%) were [adjudicated by an expert / excluded]; we report results both with and without them." Shows the disagreement did not silently disappear.
- "We report agreement per language, since pooling across languages can mask low agreement in [language]." Disaggregated agreement, essential for multilingual resources.
- "A domain expert reviewed a [N]-item sample and found an estimated label error rate of [P]%." An independent quality estimate beyond the agreement coefficient.
- "To distinguish annotator disagreement from annotator error, we [analysis]." Separates noise from genuine ambiguity, which reviewers appreciate.
- "Raw (pre-adjudication) annotations are released alongside the gold labels so others can study disagreement." Releasing raw labels is increasingly expected and strengthens the contribution.
Statistics and splits
- "Table [N] reports per-split sizes, label distribution, and [length / vocabulary] statistics." Point to a table instead of narrating numbers in prose.
- "We split [units] into train/dev/test ([sizes]) by [document / speaker / time], so that no [entity] appears in more than one split." States the split axis and the leakage guarantee in one sentence.
Show 6 more
- "Splits are stratified by [label / language / domain] to preserve the class distribution across train, dev, and test." Names the stratification, which prevents distribution skew between splits.
- "The label distribution is imbalanced ([majority class] = [P]%); we report [macro-F1 / per-class] accordingly." Reports imbalance and the metric choice it forces.
- "We deduplicated within and across splits using [exact / near-duplicate method], removing [N] items." Documents deduplication, a common silent source of leakage.
- "To limit contamination, we [held out a fresh test set / checked overlap with common pretraining corpora]." The contemporary contamination-control statement for benchmarks.
- "Each test instance carries [k] reference labels; we report [agreement / human ceiling] of [value]." Gives the human upper bound, which contextualises model scores.
- "Appendix [N] gives the full per-language / per-class breakdown." Pushes exhaustive tables to the appendix while signalling they exist.
Licensing, consent, and release
- "The dataset is released under [license] at [URL / DOI]; the annotation guidelines and code are included." The release sentence: license, location, and what is included.
- "Because [platform] terms forbid redistributing [content], we release [tweet IDs / hydration scripts] rather than the text." The honest workaround when terms of service constrain release.
Show 5 more
- "We provide a datasheet (Appendix [N]) following Gebru et al. (2021), documenting motivation, composition, collection, and recommended uses." Commits to a recognised documentation standard.
- "A data statement (Bender and Friedman, 2018) describing speaker and annotator demographics accompanies the release." The NLP-specific documentation standard for who produced the language.
- "We document known limitations and intended uses, and explicitly discourage [misuse]." Pairs the release with a use statement, which reviewers increasingly expect.
- "For the anonymous submission, the data is available via [anonymised link]; the permanent release will be at [home]." Handles double-blind release without revealing authorship.
- "All personal identifiers were removed and the release was reviewed for [PII / harmful content] before publication." States the pre-release safety pass.
Common mistakes
- "We collected tweets." From where, when, how many, in what language, under what license, sampled how? Each unstated detail is a hole a reviewer will fall into.
- No inter-annotator agreement. The fastest way to lose a resource-paper reviewer. If the task is subjective, report the low number and discuss it. Silence is worse.
- "Annotated by annotators." Who? Native speakers? Paid how much? One annotator per item with no agreement number gives you one person's opinions. That is not an annotated dataset.
- English passed off as universal. A dataset in one language is a contribution. Calling it "a benchmark for language understanding" without the qualifier is overclaiming. Name the language in the abstract and the title.
- Leaky splits. Random splits when the same speaker, document, or near-duplicate appears in both train and test. Your reported numbers are then partly memorisation, and a reviewer who notices will not be gentle.
- No license, or an impossible one. Releasing scraped platform text that the platform's terms forbid you to redistribute. Sort this out before you submit, not after acceptance.
- Statistics without splits. A bare "50,000 examples" with no per-split, per-class breakdown hides the problems reviewers look for.
- The vanishing appendix. "Guidelines available upon request" is not documentation. Release them.
A reporting checklist
Before you submit, confirm the dataset section answers every one of these:
- Task, unit, and label definitions, with a borderline example per label.
- Every language with ISO 639-3 code, script, and variety; code-switching noted.
- Source, time window, sampling, and size before filtering.
- License / terms for collection and redistribution; consent and anonymisation.
- Who annotated, their qualification, the guidelines, the pay, and the ethics approval.
- Annotators per item, adjudication procedure, and pilot/training rounds.
- Inter-annotator agreement: which metric, the value, and an interpretation.
- A statistics table: total, per-split sizes, per-split label distribution, basic shape.
- How you made the splits and why they are leak-free; deduplication and contamination control.
- Release: license, URL/DOI, datasheet or data statement, guidelines, and code.
Five examples from real resource papers
- SQuAD: 100,000+ Questions for Machine Comprehension of Text (Rajpurkar et al., EMNLP 2016). A textbook crowdsourced-construction section. It gives the source, the collection interface, and a human-performance ceiling that puts every later result in context.
- A large annotated corpus for learning natural language inference (Bowman et al., EMNLP 2015). The SNLI paper spells out the elicitation prompt and the validation pass that turned crowd labels into gold.
- TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (Clark et al., TACL 2020). Treats typological diversity as a design requirement and picks the collection method to avoid translation artefacts.
- AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languages (Muhammad et al., EMNLP 2023). Lists 14 languages across four families, documents native-speaker annotation, and handles code-switching and platform-terms release honestly.
- MasakhaNER: Named Entity Recognition for African Languages (Adelani et al., TACL 2021). Community-driven annotation by native speakers across ten languages, with per-language agreement and a clear release.
Further reading
- Emily M. Bender and Batya Friedman, "Data Statements for Natural Language Processing" (TACL 2018). The NLP standard for documenting who produced your data and who it represents.
- Timnit Gebru et al., "Datasheets for Datasets" (CACM 2021). The general documentation checklist; pair it with the data statement.
- Ron Artstein and Massimo Poesio, "Inter-Coder Agreement for Computational Linguistics" (Computational Linguistics, 2008). The reference on which agreement metric to use and how to interpret it.
- The companion phrasebooks in Chapter 12: The Abstract and Chapter 13: The Introduction. You still have to sell your dataset's scale and contribution there, in one move sentence.