Skip to main content

10. The Introduction

If the abstract sold the reader on opening the paper, the introduction has to sell them on reading past page two. This is the section that gets the most rewrites in any paper I have ever advised on.

The classic five-paragraph *ACL introduction​

Most well-written ACL papers follow a structure roughly like this:

Paragraph 1 — The world the paper lives in. Two to four sentences setting the broader context. Why does this kind of problem matter? Avoid "Recently, X has attracted attention…" — instead, point at a real phenomenon, deployment scenario, or scientific question.

Paragraph 2 — The specific gap. What is missing, broken, or unknown? This is where you cite prior work and explain its limits briefly (the full treatment goes in Related Work). End this paragraph with the research question.

Paragraph 3 — What we do. "In this paper, we…" Describe the move, then the method, in plain language. A diagram of the approach (Figure 1) usually lives in this section; if you have one, refer to it here.

Paragraph 4 — What we find. Preview the main results. Two or three concrete bullets or a short paragraph. Do not be coy; reviewers should not have to dig for your numbers.

Paragraph 5 — Contributions, explicit. Many papers end the intro with a bulleted contributions list. This is genre-appropriate in *ACL. Three to five bullets, parallel grammar, each a real contribution and not a restatement of the method.

Phrasebook: phrases for each paragraph​

The five subsections below mirror the five paragraphs above. Each lists verbatim sentences taken from the introduction body of highly-cited papers — not from their abstracts. (Chapter 9 has a separate phrasebook drawn from the same papers' abstracts; an introduction restates the same five moves at greater length, and comparing the two versions side by side is itself instructive — notice how the abstract compresses what the introduction unfolds.) Many of the same papers recur because their introductions are exemplary; study the move, not the sentence.

These are patterns to internalise, not templates to fill in word-for-word. Lifting a phrase verbatim is plagiarism; absorbing the rhetorical move — "the human-vs-model contrast opener," "the however-plus-enumerated-failures gap," "the contributions list framed as deliverables" — and writing your own version is craft. Read a subsection, name the underlying move, close the page, then write. The italic glosses describe the move; the quotes are evidence that it works in real published papers. Inline citations have been elided from the quotes for readability.

Under each paragraph below, two examples sit in the open so you can skim the move at a glance; expand Show more for the rest.

Paragraph 1 — Opening the paper (setting the scene)​

  • "Language model pre-training has been shown to be effective for improving many natural language processing tasks." — Flat, factual opener: states what the field already has, no hype. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al., NAACL 2019)
  • "Recurrent neural networks, long short-term memory and gated recurrent neural networks in particular, have been firmly established as state of the art approaches in sequence modeling and transduction problems such as language modeling and machine translation." — Opens by naming the incumbent paradigm it intends to displace. Attention Is All You Need (Vaswani et al., NeurIPS 2017)
Show 16 more
  • "The human ability to understand language is general, flexible, and robust. In contrast, most NLU models above the word level are designed for a specific task and struggle with out-of-domain data." — Human-vs-model contrast in the first two sentences sets up the entire motivation. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (Wang et al., EMNLP Workshop 2018)
  • "A unique feature of human intelligence is the ability to seamlessly combine task-oriented actions with verbal reasoning." — Opens from cognitive science to motivate a machine-learning method. ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al., ICLR 2023)
  • "The semantic concepts of entailment and contradiction are central to all aspects of natural language meaning, from the lexicon to the content of entire texts." — Asserts the importance of the phenomenon before introducing the dataset. A large annotated corpus for learning natural language inference (Bowman et al., EMNLP 2015)
  • "Reading Comprehension (RC), or the ability to read text and then answer questions about it, is a challenging task for machines, requiring both understanding of natural language and knowledge about the world." — Defines the task in sentence one so any reader can follow. SQuAD: 100,000+ Questions for Machine Comprehension of Text (Rajpurkar et al., EMNLP 2016)
  • "Summarization is the task of condensing a piece of text to a shorter version that contains the main information from the original." — One-sentence definitional opener. Get To The Point: Summarization with Pointer-Generator Networks (See et al., ACL 2017)
  • "Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks." — States the prevailing assumption the paper will complicate. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (Lan et al., ICLR 2020)
  • "Recent years have featured a trend towards pre-trained language representations in NLP systems, applied in increasingly flexible and task-agnostic ways for downstream transfer." — Names the trend the paper extends. Language Models are Few-Shot Learners (Brown et al., NeurIPS 2020)
  • "Pre-trained word representations are a key component in many neural language understanding models." — Plain statement of what the field currently relies on. Deep Contextualized Word Representations (Peters et al., NAACL 2018)
  • "Pre-trained neural language models have been shown to learn a substantial amount of in-depth knowledge from data." — Opens with the capability the gap sentence will qualify. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., NeurIPS 2020)
  • "Current state-of-the-art representation learning methods for language can be viewed as learning denoising autoencoders." — Reframes the whole field under one lens to set up the contrast. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators (Clark et al., ICLR 2020)
  • "Training a machine learning model to perform natural language processing (NLP) tasks often requires that the model can process text in a way that is amenable to downstream learning." — Starts from the first principles of the problem. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al., JMLR 2020)
  • "Large language models achieve impressive zero- and few-shot results on a variety of natural language processing tasks and show several emergent capabilities." — Concedes the incumbent's strengths first, before the "however." Toolformer: Language Models Can Teach Themselves to Use Tools (Schick et al., NeurIPS 2023)
  • "Large unsupervised language models (LMs) trained on very large datasets acquire surprising capabilities. However, these models are trained on data generated by humans with a wide variety of goals, priorities, and skillsets." — Opener that turns on a "However" toward the problem by the second sentence. Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafailov et al., NeurIPS 2023)
  • "Despite its importance to empirical evaluation, relatively little attention has been paid to statistical power in NLP." — Opens on the gap itself — "Despite X, little attention to Y." With Little Power Comes Great Responsibility (Card et al., EMNLP 2020)
  • "Chomsky (2023), Chomsky et al. (2023), and Moro et al. (2023) make very broad claims to the effect that large language models (LLMs) are equally capable of learning possible and impossible human languages." — Opens by naming the specific claim the paper will test. Mission: Impossible Language Models (Kallini et al., ACL 2024)
  • "Open-domain question answering (QA) is a task that answers factoid questions using a large collection of documents." — Defines the task plainly in the first sentence. Dense Passage Retrieval for Open-Domain Question Answering (Karpukhin et al., EMNLP 2020)

Paragraph 2 — The gap (and the research question)​

  • "The fundamental constraint of sequential computation, however, remains." — Names the single core limitation in one short sentence. Attention Is All You Need (Vaswani et al., NeurIPS 2017)
  • "Training is computationally expensive, limiting the amount of tuning that can be done, and is often done with private training data of varying sizes, limiting our ability to measure the effects of the modeling advances." — Gap framed as a measurement problem, not a quality complaint. RoBERTa: A Robustly Optimized BERT Pretraining Approach (Liu et al., 2019)
Show 17 more
  • "While more effective than conventional language-model pre-training due to learning bidirectional representations, these masked language modeling (MLM) approaches incur a substantial compute cost because the network only learns from 15% of the tokens per example." — "While X, Y" concession-pivot that quantifies the inefficiency. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators (Clark et al., ICLR 2020)
  • "However, a major limitation to this approach is that while the architecture is task-agnostic, there is still a need for task-specific datasets and task-specific fine-tuning." — Classic "However, a major limitation…" naming the precise dependency. Language Models are Few-Shot Learners (Brown et al., NeurIPS 2020)
  • "The rapid rate of progress and diversity of techniques in this burgeoning field can make it difficult to compare different algorithms, tease apart the effects of new contributions, and understand the space of existing methods for transfer learning." — Gap as field-level confusion, motivating a systematic survey-experiment. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al., JMLR 2020)
  • "However, these methods typically focus on particular types of end tasks (e.g. span prediction, generation, etc.), limiting their applicability." — Gap as narrow applicability of prior objectives. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (Lewis et al., ACL 2020)
  • "However, learning high quality representations can be challenging. They should ideally model both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy)." — Gap stated as two enumerated desiderata prior work misses. Deep Contextualized Word Representations (Peters et al., NAACL 2018)
  • "The major downside of fine-tuning is that the new model contains as many parameters as in the original model." — One-sentence gap pinning the exact cost. LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., ICLR 2022)
  • "They cannot easily expand or revise their memory, can't straightforwardly provide insight into their predictions, and may produce 'hallucinations'." — Three-fold enumerated gap in one sentence. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., NeurIPS 2020)
  • "Existing datasets for RC have one of two shortcomings: (i) those that are high in quality are too small for training modern data-intensive models, while (ii) those that are large are semi-synthetic and do not share the same characteristics as explicit reading comprehension questions." — The gap as a dilemma: "two shortcomings (i)/(ii)." SQuAD: 100,000+ Questions for Machine Comprehension of Text (Rajpurkar et al., EMNLP 2016)
  • "Though these systems are promising, they exhibit undesirable behavior such as inaccurately reproducing factual details, an inability to deal with out-of-vocabulary (OOV) words, and repeating themselves." — "Though promising… they exhibit" plus enumerated failures. Get To The Point: Summarization with Pointer-Generator Networks (See et al., ACL 2017)
  • "However, in our view, existing NLI corpora do not permit such an assessment. They are generally too small for training modern data-intensive, wide-coverage models, many contain sentences that were algorithmically generated, and they are often beset with indeterminacies of event and entity coreference that significantly impact annotation quality." — Gap as three concrete deficiencies in existing resources. A large annotated corpus for learning natural language inference (Bowman et al., EMNLP 2015)
  • "However, scaling up model size alone has not proved sufficient for achieving high performance on challenging tasks such as arithmetic, commonsense, and symbolic reasoning." — Gap as a ceiling that scale alone cannot break. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Wei et al., NeurIPS 2022)
  • "However, these models often express unintended behaviors such as making up facts, generating biased or toxic text, or simply not following user instructions." — Gap framed as enumerated misbehaviours. Training language models to follow instructions with human feedback (Ouyang et al., NeurIPS 2022)
  • "While RLHF produces models with impressive conversational and coding abilities, the RLHF pipeline is considerably more complex than supervised learning, involving training multiple LMs and sampling from the LM policy in the loop of training, incurring significant computational costs." — "While X, Y is considerably more complex" — gap as procedural pain. Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafailov et al., NeurIPS 2023)
  • "However, existing approaches either rely on large amounts of human annotations or limit tool use to task-specific settings only, hindering a more widespread adoption of tool use in LMs." — "either … or …" dichotomy naming two failure modes. Toolformer: Language Models Can Teach Themselves to Use Tools (Schick et al., NeurIPS 2023)
  • "However, collecting such instruction data is costly and often suffers limited diversity given that most human generations tend to be popular NLP tasks." — Gap as a data-collection bottleneck. Self-Instruct: Aligning Language Models with Self-Generated Instructions (Wang et al., ACL 2023)
  • "Is having better NLP models as easy as having larger models?" — The gap posed as a one-sentence research question (the paragraph also notes "further model increases become harder due to GPU/TPU memory limitations and longer training times"). ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (Lan et al., ICLR 2020)
  • "One might expect such strong claims to be supported by extensive formal analysis and/or experimental evidence. However, as far as we are aware, this is not the case." — Gap as a strong claim left unsupported by evidence. Mission: Impossible Language Models (Kallini et al., ACL 2024)

Paragraph 3 — What we do ("In this paper, we…")​

  • "In this paper, we improve the fine-tuning based approaches by proposing BERT: Bidirectional Encoder Representations from Transformers." — Names the artefact and expands the acronym in the move sentence. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al., NAACL 2019)
  • "In this work we propose the Transformer, a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output." — "Propose + name + what it removes + what it relies on." Attention Is All You Need (Vaswani et al., NeurIPS 2017)
Show 19 more
  • "We present a replication study of BERT pretraining, which includes a careful evaluation of the effects of hyperparmeter tuning and training set size." — Frames the contribution as a replication study rather than a new model (the misspelling of "hyperparameter" is in the original). RoBERTa: A Robustly Optimized BERT Pretraining Approach (Liu et al., 2019)
  • "As an alternative, we propose replaced token detection, a pre-training task in which the model learns to distinguish real input tokens from plausible but synthetically generated replacements." — "As an alternative, we propose" positions the move against the gap just stated. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators (Clark et al., ICLR 2020)
  • "To address these problems, we present two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT." — "To address these problems, we present…" ties the move directly to the gap. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (Lan et al., ICLR 2020)
  • "In this paper, we test this hypothesis by training a 175 billion parameter autoregressive language model, which we call GPT-3, and measuring its in-context learning abilities." — Move sentence carries the headline scale (175B) and frames the work as a hypothesis test. Language Models are Few-Shot Learners (Brown et al., NeurIPS 2020)
  • "The basic idea underlying our work is to treat every text processing problem as a 'text-to-text' problem, i.e. taking text as input and producing new text as output." — States the unifying idea before any architecture. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al., JMLR 2020)
  • "In this paper, we present BART, which pre-trains a model combining Bidirectional and Auto-Regressive Transformers." — Name + one-clause statement of what it combines. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (Lewis et al., ACL 2020)
  • "In this paper, we introduce a new type of deep contextualized word representation that directly addresses both challenges, can be easily integrated into existing models, and significantly improves the state of the art in every considered case across a range of challenging language understanding problems." — Move sentence answers the two challenges raised in the gap. Deep Contextualized Word Representations (Peters et al., NAACL 2018)
  • "We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture." — Move sentence doubles as a method summary. LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., ICLR 2022)
  • "In this paper, we address the question: can we train a better dense embedding model using only pairs of questions and passages (or answers), without additional pretraining?" — The move stated as the precise research question. Dense Passage Retrieval for Open-Domain Question Answering (Karpukhin et al., EMNLP 2020)
  • "We endow pre-trained, parametric-memory generation models with a non-parametric memory through a general-purpose fine-tuning approach which we refer to as retrieval-augmented generation (RAG)." — Names the mechanism, then the term, in one sentence. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., NeurIPS 2020)
  • "In this work, we introduce Self-Instruct, a semi-automated process for instruction-tuning a pretrained LM using instructional signals from the model itself." — Name + category ("semi-automated process") + mechanism. Self-Instruct: Aligning Language Models with Self-Generated Instructions (Wang et al., ACL 2023)
  • "In this paper, we combine the strengths of these two ideas in a way that avoids their limitations. Specifically, we explore the ability of language models to perform few-shot prompting for reasoning tasks, given a prompt that consists of triples: ⟨input, chain of thought, output⟩." — General move sentence, then a "Specifically" sentence that gets concrete. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Wei et al., NeurIPS 2022)
  • "We focus on fine-tuning approaches to aligning language models. Specifically, we use reinforcement learning from human feedback (RLHF) to fine-tune GPT-3 to follow a broad class of written instructions." — "We focus on… Specifically, we use…" narrows from approach to mechanism. Training language models to follow instructions with human feedback (Ouyang et al., NeurIPS 2022)
  • "In this paper, we show how to directly optimize a language model to adhere to human preferences, without explicit reward modeling or reinforcement learning." — Move framed by what it does without (the painful parts of the gap). Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafailov et al., NeurIPS 2023)
  • "To address the need for a large and high-quality reading comprehension dataset, we present the Stanford Question Answering Dataset v1.0 (SQuAD), consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage." — Move sentence packed with the dataset's defining facts. SQuAD: 100,000+ Questions for Machine Comprehension of Text (Rajpurkar et al., EMNLP 2016)
  • "To facilitate research in this direction, we present the General Language Understanding Evaluation (GLUE) benchmark: a collection of NLU tasks including question answering, sentiment analysis, and textual entailment, and an associated online platform for model evaluation, comparison, and analysis." — "To facilitate research…, we present…" enumerates the deliverables. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (Wang et al., EMNLP Workshop 2018)
  • "In this paper we present an architecture that addresses these three issues in the context of multi-sentence summaries." — Terse move sentence that points back at the three issues just listed. Get To The Point: Summarization with Pointer-Generator Networks (See et al., ACL 2017)
  • "To address this, this paper introduces the Stanford Natural Language Inference (SNLI) corpus, a collection of sentence pairs labeled for entailment, contradiction, and semantic independence." — "To address this, this paper introduces X, a Y" template. A large annotated corpus for learning natural language inference (Bowman et al., EMNLP 2015)
  • "In the present paper, we provide extensive new experimental evidence to inform the claim that LLMs are equally capable of learning possible and impossible languages in the human sense." — Move framed as supplying the evidence the gap said was missing. Mission: Impossible Language Models (Kallini et al., ACL 2024)

Paragraph 4 — Previewing the results​

  • "BERT advances the state of the art for eleven NLP tasks." — One-sentence headline; the number does the work. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al., NAACL 2019)
  • "The Transformer allows for significantly more parallelization and can reach a new state of the art in translation quality after being trained for as little as twelve hours on eight P100 GPUs." — Result preview pairs quality with a concrete training-cost figure. Attention Is All You Need (Vaswani et al., NeurIPS 2017)
Show 19 more
  • "We find that BERT was significantly undertrained and propose an improved recipe for training BERT models, which we call RoBERTa, that can match or exceed the performance of all of the post-BERT methods." — Preview leads with the diagnostic finding ("undertrained"), then the consequence. RoBERTa: A Robustly Optimized BERT Pretraining Approach (Liu et al., 2019)
  • "As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and SQuAD benchmarks while having fewer parameters compared to BERT-large." — Headline result balanced against the efficiency win ("fewer parameters"). ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (Lan et al., ICLR 2020)
  • "Broadly, on NLP tasks GPT-3 achieves promising results in the zero-shot and one-shot settings, and in the few-shot setting is sometimes competitive with or even occasionally surpasses state-of-the-art." — Honestly hedged preview ("sometimes," "occasionally"). Language Models are Few-Shot Learners (Brown et al., NeurIPS 2020)
  • "By combining the insights from our exploration with scale and our new 'Colossal Clean Crawled Corpus', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more." — Preview names the ingredients of the win, then the breadth. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al., JMLR 2020)
  • "LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times." — Two efficiency numbers, no prose padding. LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., ICLR 2022)
  • "It not only outperforms BM25 by a large margin (65.2% vs. 42.9% in Top-5 accuracy), but also results in a substantial improvement on the end-to-end QA accuracy compared to ORQA (41.5% vs. 33.3%) in the open Natural Questions setting." — "Not only … but also …" with paired before/after numbers. Dense Passage Retrieval for Open-Domain Question Answering (Karpukhin et al., EMNLP 2020)
  • "Our RAG models achieve state-of-the-art results on open Natural Questions, WebQuestions and CuratedTrec and strongly outperform recent approaches that use specialised pre-training objectives on TriviaQA." — Names the exact datasets and the kind of baseline beaten. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., NeurIPS 2020)
  • "On seven Semantic Textual Similarity (STS) tasks, SBERT achieves an improvement of 11.7 points compared to InferSent and 5.5 points compared to Universal Sentence Encoder." — Two deltas against two named baselines. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (Reimers & Gurevych, EMNLP 2019)
  • "Applying our method to the vanilla GPT3, we demonstrate a 33% absolute improvement over the original model on Super-NaturalInstructions, on par with the performance of InstructGPT." — Numeric delta plus a comparison to a stronger (paid-data) system. Self-Instruct: Aligning Language Models with Self-Generated Instructions (Wang et al., ACL 2023)
  • "For instance, prompting a PaLM 540B with just eight chain-of-thought exemplars achieves state-of-the-art accuracy on the GSM8K benchmark of math word problems, surpassing even finetuned GPT-3 with a verifier." — Result with the sample size (eight) baked in. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Wei et al., NeurIPS 2022)
  • "On our test set, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having over 100x fewer parameters." — Punchy comparison: smaller model preferred over one 100x larger. Training language models to follow instructions with human feedback (Ouyang et al., NeurIPS 2022)
  • "Our experiments show that DPO is at least as effective as existing methods, including PPO-based RLHF, for learning from preferences in tasks such as sentiment modulation, summarization, and dialogue, using language models with up to 6B parameters." — Preview states the bar ("at least as effective") and the scope. Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafailov et al., NeurIPS 2023)
  • "Our best model achieves an F1 score of 51.0%, which is much better than the sliding window baseline (20%)." — Number, baseline, delta — minimal but complete. SQuAD: 100,000+ Questions for Machine Comprehension of Text (Rajpurkar et al., EMNLP 2016)
  • "We find that unified multi-task trained models slightly outperform comparable models trained on each task separately." — Honest, low-key preview ("slightly") for a benchmark paper. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (Wang et al., EMNLP Workshop 2018)
  • "Therefore we apply our model to the recently-introduced CNN/Daily Mail dataset, which contains news articles (39 sentences on average) paired with multi-sentence summaries, and show that we outperform the state-of-the-art abstractive system by at least 2 ROUGE points." — Result as a minimum delta ("at least 2"). Get To The Point: Summarization with Pointer-Generator Networks (See et al., ACL 2017)
  • "We conduct experiments on a variety of different downstream tasks, demonstrating that after learning to use tools, Toolformer, which is based on a pretrained GPT-J model with 6.7B parameters, achieves much stronger zero-shot results, clearly outperforming a much larger GPT-3 model and several other baselines on various tasks." — Preview foregrounds beating a far larger baseline. Toolformer: Language Models Can Teach Themselves to Use Tools (Schick et al., NeurIPS 2023)
  • "On ALFWorld and WebShop, two or even one-shot ReAct prompting is able to outperform imitation or reinforcement learning methods trained with task instances, with an absolute improvement of 34% and 10% in success rates respectively." — Two named environments, two deltas, low-shot framing. ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al., ICLR 2023)
  • "Although retrospective analysis of power involves challenges, we present evidence that underpowered experiments are widespread in NLP research." — Preview of a diagnostic finding about the field itself, not a leaderboard number. With Little Power Comes Great Responsibility (Card et al., EMNLP 2020)
  • "What we find is that these models indeed struggle to learn impossible languages, shown through three core experiments." — States the finding and signposts the three experiments that follow. Mission: Impossible Language Models (Kallini et al., ACL 2024)

Paragraph 5 — The explicit contributions list​

Not every great paper has one — Attention Is All You Need, GPT-3, T5, and many model papers state their contribution in prose instead. But when the work is a benchmark, dataset, or multi-part system, an enumerated list is the genre-standard close, and reviewers skim it to decide what you are claiming. The examples below show the common framings; the last two show variants where the slot is filled by design desiderata or empirical findings rather than a literal "contributions" list.

  • The concept / consequence / result triad. "We demonstrate the importance of bidirectional pre-training for language representations." / "We show that pre-trained representations reduce the need for many heavily-engineered task-specific architectures." / "BERT advances the state of the art for eleven NLP tasks." — Three parallel bullets: an idea, its practical consequence, the headline number. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al., NAACL 2019)
  • "In summary, we offer" + deliverables. "(i) A suite of nine sentence or sentence-pair NLU tasks…; (ii) An online evaluation platform and leaderboard…; (iii) An expert-constructed diagnostic evaluation dataset; (iv) Baseline results for several major existing approaches to sentence representation learning." — Each item is a concrete artefact the reader can use. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (Wang et al., EMNLP Workshop 2018)
Show 14 more
  • Bolded-lead "improves in several ways." "More challenging tasks…", "More diverse task formats…", "Comprehensive human baselines…", "Improved code support…", "Refined usage rules…" — Each bullet opens with a bolded noun phrase naming the axis of improvement. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (Wang et al., NeurIPS 2019)
  • "In sum, our contributions are the following." "(i) We release a suite of 9 cross-lingual benchmark tasks covering 40 typologically diverse languages." / "(ii) We provide an online platform and leaderboard for the evaluation of multilingual models." / "(iii) We provide a set of strong baselines… and release code to facilitate adoption." / "(iv) We provide an extensive analysis of limitations of state-of-the-art cross-lingual models." — Release / platform / baselines / analysis — a complete resource-paper template. XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (Hu et al., ICML 2020)
  • "Our contributions are twofold." "First, we demonstrate that with the proper training setup, simply fine-tuning the question and passage encoders on existing question-passage pairs is sufficient to greatly outperform BM25." — Counts the contributions up front, then enumerates. Dense Passage Retrieval for Open-Domain Question Answering (Karpukhin et al., EMNLP 2020)
  • Contributions as an inline numbered list of design changes. "Our modifications are simple, they include: (1) training the model longer, with bigger batches, over more data; (2) removing the next sentence prediction objective; (3) training on longer sequences; and (4) dynamically changing the masking pattern applied to the training data." — Whole list packed into one sentence. RoBERTa: A Robustly Optimized BERT Pretraining Approach (Liu et al., 2019)
  • The method / evidence / release triad. "we introduce Self-Instruct, a method for inducing instruction following capabilities with minimal human-labeled data" / "we demonstrate its effectiveness via extensive instruction-tuning experiments" / "we release a large synthetic dataset of 52K instructions and a set of manually-written novel tasks for building and evaluating future instruction-following models." — Method, then proof, then the artefact. Self-Instruct: Aligning Language Models with Self-Generated Instructions (Wang et al., ACL 2023)
  • Contributions framed as "key advantages." "LoRA possesses several key advantages." → "A pre-trained model can be shared and used to build many small LoRA modules for different tasks." / "LoRA makes training more efficient and lowers the hardware barrier to entry by up to 3 times when using adaptive optimizers." / "Our simple linear design allows us to merge the trainable matrices with the frozen weights when deployed, introducing no inference latency." / "LoRA is orthogonal to many prior methods and can be combined with many of them, such as prefix-tuning." — Reframes contributions as reader-facing benefits. LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., ICLR 2022)
  • The numbered method / experiments / analysis triad. "(1) we introduce ReAct, a novel prompt-based paradigm to synergize reasoning and acting in language models for general task solving;" / "(2) we perform extensive experiments across diverse benchmarks to showcase the advantage of ReAct… over prior approaches that perform either reasoning or action generation in isolation;" / "(3) we present systematic ablations and analysis to understand the importance of acting in reasoning tasks, and reasoning in interactive tasks." — The most common three-part shape in modern *ACL papers. ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al., ICLR 2023)
  • The single-sentence contribution. "Our main contribution is Direct Preference Optimization (DPO), a simple RL-free algorithm for training language models from preferences." — When the work is one clean idea, one sentence is enough. Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafailov et al., NeurIPS 2023)
  • "Firstly / Secondly" conceptual advances. "Firstly, instead of using a fixed forward or backward factorization order as in conventional AR models, XLNet maximizes the expected log likelihood of a sequence w.r.t. all possible permutations of the factorization order…" / "Secondly, as a generalized AR language model, XLNet does not rely on data corruption. Hence, XLNet does not suffer from the pretrain-finetune discrepancy that BERT is subject to." — Contributions stated as ideas, each contrasted with the incumbent. XLNet: Generalized Autoregressive Pretraining for Language Understanding (Yang et al., NeurIPS 2019)
  • "To summarize, our main contributions are." "BigBird satisfies all the known theoretical properties of full transformer… we show that under standard assumptions regarding precision, BigBird is Turing complete." / "Empirically, we show that the extended context modelled by BigBird benefits variety of NLP tasks." / "Lastly, we introduce a novel application of attention based models where long contexts are beneficial: extracting contextual representations of genomics sequences like DNA." — Theory, then empiry, then a surprising new application. Big Bird: Transformers for Longer Sequences (Zaheer et al., NeurIPS 2020)
  • Contributions framed as research "goals." "Our goals in doing so are: 1. to enable research progress toward building high-quality question answering systems in roughly the world's top 100 languages; and 2. to encourage research on models that behave well across the linguistic phenomena and data scenarios of the world's languages." — A resource paper states the community outcomes it hopes to enable. TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (Clark et al., TACL 2020)
  • Contributions enumerated as the experiments themselves. "In Experiment 1, we train GPT-2 models on our set of defined possible and impossible languages… We find that models trained on possible languages learn more efficiently." / "In Experiment 2, we more closely examine a set of languages that exhibit count-based verb marking rules…" / "In Experiment 3, we dive deeper into the internal mechanisms that models may develop… using causal abstraction analysis." — Each contribution is one experiment plus its finding. Mission: Impossible Language Models (Kallini et al., ACL 2024)
  • Variant — design desiderata, not contributions. "The use of tools should be learned in a self-supervised way without requiring large amounts of human annotations…" / "The LM should not lose any of its generality and should be able to decide for itself when and how to use which tool…" — The bullets state goals the proposed method must satisfy; the contribution is then "the method that meets them." Toolformer: Language Models Can Teach Themselves to Use Tools (Schick et al., NeurIPS 2023)
  • Variant — findings as bolded headers. "Labelers significantly prefer InstructGPT outputs over outputs from GPT-3." / "InstructGPT models show improvements in truthfulness over GPT-3." / "InstructGPT shows small improvements in toxicity over GPT-3, but not bias." — In place of a contributions list, the intro previews results as a series of bolded one-line findings. Training language models to follow instructions with human feedback (Ouyang et al., NeurIPS 2022)

Common mistakes​

  • The history-of-NLP introduction. "Since the days of Chomsky…" No. Start where the problem starts.
  • Forward references everywhere. "As we will show in §5.3…" Three of these is fine; thirty makes the paper feel like an index.
  • Contributions that are not contributions. "We use a transformer architecture" is not a contribution. "We provide the first evidence that X" is.
  • Method dump in the intro. The intro previews; the methods section delivers. Keep the intro at a level a smart non-specialist can follow.
  • The bait-and-switch. Promising one thing in the intro and delivering another in the experiments. Reviewers catch this and they punish it.

The contributions list, done right​

Bad:

  • We propose a transformer-based model.
  • We train it on a large dataset.
  • We achieve state-of-the-art results.

Good:

  • We introduce MethodName, the first approach to X that does not require Y.
  • We release a benchmark of 12,000 examples covering Z, with human-verified labels.
  • We show empirically that the standard assumption A does not hold in setting B, and identify the conditions under which it does.

Notice: each bullet promises something the reader could not have done themselves.

Five examples from real ACL papers​

  1. BERT (Devlin et al., NAACL 2019). The contributions list at the end of §1 is a textbook example: each bullet is precise and verifiable.
  2. Linguistic Knowledge and Transferability of Contextual Representations (Liu et al., NAACL 2019). Clean five-paragraph structure with a sharp research question.
  3. On the Dangers of Stochastic Parrots (Bender, Gebru, McMillan-Major, Mitchell, FAccT 2021). Not technically *ACL but field-defining; study how it frames the problem space in paragraphs 1–2.
  4. Stanza: A Python NLP Toolkit for Many Human Languages (Qi et al., ACL 2020 Demo). A resource paper that motivates itself in two paragraphs without overclaiming.
  5. Don't Stop Pretraining (Gururangan et al., ACL 2020). The intro tells you exactly what experiment you are about to read and what it found.

Further reading​

  • Kate Turabian, A Manual for Writers, chapter on argument structure.
  • The "CARS" model (Create A Research Space) by John Swales — particularly useful for paragraph 2 (establishing the gap). Search "Swales CARS model."