Skip to main content

13. The Introduction

The abstract persuades the reader to open the paper. The introduction must persuade them to read past page two. In every paper I have advised on, it is the section that gets the most rewrites.

The classic five-paragraph *ACL introduction​

Most well-written ACL papers follow roughly this structure:

Paragraph 1: The world the paper lives in. Two to four sentences of broader context. Why does this kind of problem matter? Avoid "Recently, X has attracted attention…". Point instead at a real phenomenon, deployment scenario, or scientific question.

Paragraph 2: The specific gap. What is missing, broken, or unknown? Cite prior work here and explain its limits briefly; the full treatment goes in Related Work. End the paragraph with the research question.

Paragraph 3: What we do. "In this paper, we…" Describe the move, then the method, in plain language. A diagram of the approach (Figure 1) usually sits here; if you have one, refer to it in this paragraph.

Paragraph 4: What we find. Preview the main results in two or three concrete bullets or a short paragraph. Do not be coy. Reviewers should not have to dig for your numbers.

Paragraph 5: Contributions, explicit. Many papers end the introduction with a bulleted contributions list, which suits the *ACL genre. Write three to five bullets in parallel grammar. Each must be a real contribution, not a restatement of the method.

Phrasebook: phrases for each paragraph​

The five subsections match the five paragraphs. Each quotes sentences from the introduction body of highly cited papers, never from their abstracts. Chapter 12 has a separate phrasebook drawn from the same papers' abstracts. An introduction restates the same five moves at greater length. Compare the two versions side by side to see how the abstract compresses what the introduction unfolds. Many papers recur because their introductions are exemplary. Study the move, not the sentence.

Learn the patterns; do not copy the words. Lifting a phrase verbatim is plagiarism. Absorbing the rhetorical move and writing your own version is craft. Examples of moves are "the human-vs-model contrast opener," "the however-plus-enumerated-failures gap," and "the contributions list framed as deliverables". Read a subsection, name the move, close the page, then write. The italic glosses describe the move; the quotes show that it works in published papers. We removed inline citations from the quotes for readability.

Each paragraph shows two examples in the open; expand Show more for the rest.

Paragraph 1: opening the paper (setting the scene)​

  • "Language model pre-training has been shown to be effective for improving many natural language processing tasks." Flat, factual opener: states what the field already has, no hype. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al., NAACL 2019)
  • "Recurrent neural networks, long short-term memory and gated recurrent neural networks in particular, have been firmly established as state of the art approaches in sequence modeling and transduction problems such as language modeling and machine translation." Opens by naming the incumbent paradigm it intends to displace. Attention Is All You Need (Vaswani et al., NeurIPS 2017)
Show 16 more

Paragraph 2: the gap (and the research question)​

  • "The fundamental constraint of sequential computation, however, remains." Names the single core limitation in one short sentence. Attention Is All You Need (Vaswani et al., NeurIPS 2017)
  • "Training is computationally expensive, limiting the amount of tuning that can be done, and is often done with private training data of varying sizes, limiting our ability to measure the effects of the modeling advances." Frames the gap as a measurement problem instead of a quality complaint. RoBERTa: A Robustly Optimized BERT Pretraining Approach (Liu et al., 2019)
Show 17 more
  • "While more effective than conventional language-model pre-training due to learning bidirectional representations, these masked language modeling (MLM) approaches incur a substantial compute cost because the network only learns from 15% of the tokens per example." "While X, Y" concession-pivot that quantifies the inefficiency. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators (Clark et al., ICLR 2020)
  • "However, a major limitation to this approach is that while the architecture is task-agnostic, there is still a need for task-specific datasets and task-specific fine-tuning." Classic "However, a major limitation…" naming the precise dependency. Language Models are Few-Shot Learners (Brown et al., NeurIPS 2020)
  • "The rapid rate of progress and diversity of techniques in this burgeoning field can make it difficult to compare different algorithms, tease apart the effects of new contributions, and understand the space of existing methods for transfer learning." Gap as field-level confusion, motivating a systematic survey-experiment. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al., JMLR 2020)
  • "However, these methods typically focus on particular types of end tasks (e.g. span prediction, generation, etc.), limiting their applicability." Gap as narrow applicability of prior objectives. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (Lewis et al., ACL 2020)
  • "However, learning high quality representations can be challenging. They should ideally model both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy)." Gap stated as two enumerated desiderata prior work misses. Deep Contextualized Word Representations (Peters et al., NAACL 2018)
  • "The major downside of fine-tuning is that the new model contains as many parameters as in the original model." One-sentence gap pinning the exact cost. LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., ICLR 2022)
  • "They cannot easily expand or revise their memory, can't straightforwardly provide insight into their predictions, and may produce 'hallucinations'." Three-fold enumerated gap in one sentence. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., NeurIPS 2020)
  • "Existing datasets for RC have one of two shortcomings: (i) those that are high in quality are too small for training modern data-intensive models, while (ii) those that are large are semi-synthetic and do not share the same characteristics as explicit reading comprehension questions." The gap as a dilemma: "two shortcomings (i)/(ii)." SQuAD: 100,000+ Questions for Machine Comprehension of Text (Rajpurkar et al., EMNLP 2016)
  • "Though these systems are promising, they exhibit undesirable behavior such as inaccurately reproducing factual details, an inability to deal with out-of-vocabulary (OOV) words, and repeating themselves." "Though promising… they exhibit" plus enumerated failures. Get To The Point: Summarization with Pointer-Generator Networks (See et al., ACL 2017)
  • "However, in our view, existing NLI corpora do not permit such an assessment. They are generally too small for training modern data-intensive, wide-coverage models, many contain sentences that were algorithmically generated, and they are often beset with indeterminacies of event and entity coreference that significantly impact annotation quality." Gap as three concrete deficiencies in existing resources. A large annotated corpus for learning natural language inference (Bowman et al., EMNLP 2015)
  • "However, scaling up model size alone has not proved sufficient for achieving high performance on challenging tasks such as arithmetic, commonsense, and symbolic reasoning." Gap as a ceiling that scale alone cannot break. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Wei et al., NeurIPS 2022)
  • "However, these models often express unintended behaviors such as making up facts, generating biased or toxic text, or simply not following user instructions." Gap framed as enumerated misbehaviours. Training language models to follow instructions with human feedback (Ouyang et al., NeurIPS 2022)
  • "While RLHF produces models with impressive conversational and coding abilities, the RLHF pipeline is considerably more complex than supervised learning, involving training multiple LMs and sampling from the LM policy in the loop of training, incurring significant computational costs." "While X, Y is considerably more complex". Gap as procedural pain. Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafailov et al., NeurIPS 2023)
  • "However, existing approaches either rely on large amounts of human annotations or limit tool use to task-specific settings only, hindering a more widespread adoption of tool use in LMs." "either … or …" dichotomy naming two failure modes. Toolformer: Language Models Can Teach Themselves to Use Tools (Schick et al., NeurIPS 2023)
  • "However, collecting such instruction data is costly and often suffers limited diversity given that most human generations tend to be popular NLP tasks." Gap as a data-collection bottleneck. Self-Instruct: Aligning Language Models with Self-Generated Instructions (Wang et al., ACL 2023)
  • "Is having better NLP models as easy as having larger models?" The gap posed as a one-sentence research question (the paragraph also notes "further model increases become harder due to GPU/TPU memory limitations and longer training times"). ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (Lan et al., ICLR 2020)
  • "One might expect such strong claims to be supported by extensive formal analysis and/or experimental evidence. However, as far as we are aware, this is not the case." Gap as a strong claim left unsupported by evidence. Mission: Impossible Language Models (Kallini et al., ACL 2024)

Paragraph 3: what we do ("In this paper, we…")​

  • "In this paper, we improve the fine-tuning based approaches by proposing BERT: Bidirectional Encoder Representations from Transformers." Names the artefact and expands the acronym in the move sentence. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al., NAACL 2019)
  • "In this work we propose the Transformer, a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output." "Propose + name + what it removes + what it relies on." Attention Is All You Need (Vaswani et al., NeurIPS 2017)
Show 19 more
  • "We present a replication study of BERT pretraining, which includes a careful evaluation of the effects of hyperparmeter tuning and training set size." Frames the contribution as a replication study rather than a new model (the misspelling of "hyperparameter" is in the original). RoBERTa: A Robustly Optimized BERT Pretraining Approach (Liu et al., 2019)
  • "As an alternative, we propose replaced token detection, a pre-training task in which the model learns to distinguish real input tokens from plausible but synthetically generated replacements." "As an alternative, we propose" positions the move against the gap just stated. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators (Clark et al., ICLR 2020)
  • "To address these problems, we present two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT." "To address these problems, we present…" ties the move directly to the gap. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (Lan et al., ICLR 2020)
  • "In this paper, we test this hypothesis by training a 175 billion parameter autoregressive language model, which we call GPT-3, and measuring its in-context learning abilities." Move sentence carries the headline scale (175B) and frames the work as a hypothesis test. Language Models are Few-Shot Learners (Brown et al., NeurIPS 2020)
  • "The basic idea underlying our work is to treat every text processing problem as a 'text-to-text' problem, i.e. taking text as input and producing new text as output." States the unifying idea before any architecture. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al., JMLR 2020)
  • "In this paper, we present BART, which pre-trains a model combining Bidirectional and Auto-Regressive Transformers." Name + one-clause statement of what it combines. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (Lewis et al., ACL 2020)
  • "In this paper, we introduce a new type of deep contextualized word representation that directly addresses both challenges, can be easily integrated into existing models, and significantly improves the state of the art in every considered case across a range of challenging language understanding problems." Move sentence answers the two challenges raised in the gap. Deep Contextualized Word Representations (Peters et al., NAACL 2018)
  • "We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture." Move sentence doubles as a method summary. LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., ICLR 2022)
  • "In this paper, we address the question: can we train a better dense embedding model using only pairs of questions and passages (or answers), without additional pretraining?" The move stated as the precise research question. Dense Passage Retrieval for Open-Domain Question Answering (Karpukhin et al., EMNLP 2020)
  • "We endow pre-trained, parametric-memory generation models with a non-parametric memory through a general-purpose fine-tuning approach which we refer to as retrieval-augmented generation (RAG)." Names the mechanism, then the term, in one sentence. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., NeurIPS 2020)
  • "In this work, we introduce Self-Instruct, a semi-automated process for instruction-tuning a pretrained LM using instructional signals from the model itself." Name + category ("semi-automated process") + mechanism. Self-Instruct: Aligning Language Models with Self-Generated Instructions (Wang et al., ACL 2023)
  • "In this paper, we combine the strengths of these two ideas in a way that avoids their limitations. Specifically, we explore the ability of language models to perform few-shot prompting for reasoning tasks, given a prompt that consists of triples: ⟨input, chain of thought, output⟩." General move sentence, then a "Specifically" sentence that gets concrete. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Wei et al., NeurIPS 2022)
  • "We focus on fine-tuning approaches to aligning language models. Specifically, we use reinforcement learning from human feedback (RLHF) to fine-tune GPT-3 to follow a broad class of written instructions." "We focus on… Specifically, we use…" narrows from approach to mechanism. Training language models to follow instructions with human feedback (Ouyang et al., NeurIPS 2022)
  • "In this paper, we show how to directly optimize a language model to adhere to human preferences, without explicit reward modeling or reinforcement learning." Move framed by what it does without (the painful parts of the gap). Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafailov et al., NeurIPS 2023)
  • "To address the need for a large and high-quality reading comprehension dataset, we present the Stanford Question Answering Dataset v1.0 (SQuAD), consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage." Move sentence packed with the dataset's defining facts. SQuAD: 100,000+ Questions for Machine Comprehension of Text (Rajpurkar et al., EMNLP 2016)
  • "To facilitate research in this direction, we present the General Language Understanding Evaluation (GLUE) benchmark: a collection of NLU tasks including question answering, sentiment analysis, and textual entailment, and an associated online platform for model evaluation, comparison, and analysis." "To facilitate research…, we present…" enumerates the deliverables. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (Wang et al., EMNLP Workshop 2018)
  • "In this paper we present an architecture that addresses these three issues in the context of multi-sentence summaries." Terse move sentence that points back at the three issues just listed. Get To The Point: Summarization with Pointer-Generator Networks (See et al., ACL 2017)
  • "To address this, this paper introduces the Stanford Natural Language Inference (SNLI) corpus, a collection of sentence pairs labeled for entailment, contradiction, and semantic independence." "To address this, this paper introduces X, a Y" template. A large annotated corpus for learning natural language inference (Bowman et al., EMNLP 2015)
  • "In the present paper, we provide extensive new experimental evidence to inform the claim that LLMs are equally capable of learning possible and impossible languages in the human sense." Move framed as supplying the evidence the gap said was missing. Mission: Impossible Language Models (Kallini et al., ACL 2024)

Paragraph 4: previewing the results​

  • "BERT advances the state of the art for eleven NLP tasks." One-sentence headline; the number does the work. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al., NAACL 2019)
  • "The Transformer allows for significantly more parallelization and can reach a new state of the art in translation quality after being trained for as little as twelve hours on eight P100 GPUs." Result preview pairs quality with a concrete training-cost figure. Attention Is All You Need (Vaswani et al., NeurIPS 2017)
Show 19 more
  • "We find that BERT was significantly undertrained and propose an improved recipe for training BERT models, which we call RoBERTa, that can match or exceed the performance of all of the post-BERT methods." Preview leads with the diagnostic finding ("undertrained"), then the consequence. RoBERTa: A Robustly Optimized BERT Pretraining Approach (Liu et al., 2019)
  • "As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and SQuAD benchmarks while having fewer parameters compared to BERT-large." Headline result balanced against the efficiency win ("fewer parameters"). ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (Lan et al., ICLR 2020)
  • "Broadly, on NLP tasks GPT-3 achieves promising results in the zero-shot and one-shot settings, and in the few-shot setting is sometimes competitive with or even occasionally surpasses state-of-the-art." Honestly hedged preview ("sometimes," "occasionally"). Language Models are Few-Shot Learners (Brown et al., NeurIPS 2020)
  • "By combining the insights from our exploration with scale and our new 'Colossal Clean Crawled Corpus', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more." Preview names the ingredients of the win, then the breadth. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al., JMLR 2020)
  • "LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times." Two efficiency numbers, no prose padding. LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., ICLR 2022)
  • "It not only outperforms BM25 by a large margin (65.2% vs. 42.9% in Top-5 accuracy), but also results in a substantial improvement on the end-to-end QA accuracy compared to ORQA (41.5% vs. 33.3%) in the open Natural Questions setting." "Not only … but also …" with paired before/after numbers. Dense Passage Retrieval for Open-Domain Question Answering (Karpukhin et al., EMNLP 2020)
  • "Our RAG models achieve state-of-the-art results on open Natural Questions, WebQuestions and CuratedTrec and strongly outperform recent approaches that use specialised pre-training objectives on TriviaQA." Names the exact datasets and the kind of baseline beaten. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., NeurIPS 2020)
  • "On seven Semantic Textual Similarity (STS) tasks, SBERT achieves an improvement of 11.7 points compared to InferSent and 5.5 points compared to Universal Sentence Encoder." Two deltas against two named baselines. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (Reimers & Gurevych, EMNLP 2019)
  • "Applying our method to the vanilla GPT3, we demonstrate a 33% absolute improvement over the original model on Super-NaturalInstructions, on par with the performance of InstructGPT." Numeric delta plus a comparison to a stronger (paid-data) system. Self-Instruct: Aligning Language Models with Self-Generated Instructions (Wang et al., ACL 2023)
  • "For instance, prompting a PaLM 540B with just eight chain-of-thought exemplars achieves state-of-the-art accuracy on the GSM8K benchmark of math word problems, surpassing even finetuned GPT-3 with a verifier." Result with the sample size (eight) baked in. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Wei et al., NeurIPS 2022)
  • "On our test set, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having over 100x fewer parameters." Punchy comparison: smaller model preferred over one 100x larger. Training language models to follow instructions with human feedback (Ouyang et al., NeurIPS 2022)
  • "Our experiments show that DPO is at least as effective as existing methods, including PPO-based RLHF, for learning from preferences in tasks such as sentiment modulation, summarization, and dialogue, using language models with up to 6B parameters." Preview states the bar ("at least as effective") and the scope. Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafailov et al., NeurIPS 2023)
  • "Our best model achieves an F1 score of 51.0%, which is much better than the sliding window baseline (20%)." Number, baseline, delta. Minimal but complete. SQuAD: 100,000+ Questions for Machine Comprehension of Text (Rajpurkar et al., EMNLP 2016)
  • "We find that unified multi-task trained models slightly outperform comparable models trained on each task separately." Honest, low-key preview ("slightly") for a benchmark paper. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (Wang et al., EMNLP Workshop 2018)
  • "Therefore we apply our model to the recently-introduced CNN/Daily Mail dataset, which contains news articles (39 sentences on average) paired with multi-sentence summaries, and show that we outperform the state-of-the-art abstractive system by at least 2 ROUGE points." Result as a minimum delta ("at least 2"). Get To The Point: Summarization with Pointer-Generator Networks (See et al., ACL 2017)
  • "We conduct experiments on a variety of different downstream tasks, demonstrating that after learning to use tools, Toolformer, which is based on a pretrained GPT-J model with 6.7B parameters, achieves much stronger zero-shot results, clearly outperforming a much larger GPT-3 model and several other baselines on various tasks." Preview foregrounds beating a far larger baseline. Toolformer: Language Models Can Teach Themselves to Use Tools (Schick et al., NeurIPS 2023)
  • "On ALFWorld and WebShop, two or even one-shot ReAct prompting is able to outperform imitation or reinforcement learning methods trained with task instances, with an absolute improvement of 34% and 10% in success rates respectively." Two named environments, two deltas, low-shot framing. ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al., ICLR 2023)
  • "Although retrospective analysis of power involves challenges, we present evidence that underpowered experiments are widespread in NLP research." Preview of a diagnostic finding about the field itself, not a leaderboard number. With Little Power Comes Great Responsibility (Card et al., EMNLP 2020)
  • "What we find is that these models indeed struggle to learn impossible languages, shown through three core experiments." States the finding and signposts the three experiments that follow. Mission: Impossible Language Models (Kallini et al., ACL 2024)

Paragraph 5: the explicit contributions list​

Attention Is All You Need, GPT-3, T5, and many other model papers state their contribution in prose. For a benchmark, dataset, or multi-part system, an enumerated list is the genre-standard close, and reviewers skim it to see what you claim. The examples below show the common framings. In the last two, design desiderata or empirical findings fill the slot instead of a literal "contributions" list.

  • The concept / consequence / result triad. "We demonstrate the importance of bidirectional pre-training for language representations." / "We show that pre-trained representations reduce the need for many heavily-engineered task-specific architectures." / "BERT advances the state of the art for eleven NLP tasks." Three parallel bullets: an idea, its practical consequence, the headline number. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al., NAACL 2019)
  • "In summary, we offer" + deliverables. "(i) A suite of nine sentence or sentence-pair NLU tasks…; (ii) An online evaluation platform and leaderboard…; (iii) An expert-constructed diagnostic evaluation dataset; (iv) Baseline results for several major existing approaches to sentence representation learning." Each item is a concrete artefact the reader can use. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (Wang et al., EMNLP Workshop 2018)
Show 14 more
  • Bolded-lead "improves in several ways." "More challenging tasks…", "More diverse task formats…", "Comprehensive human baselines…", "Improved code support…", "Refined usage rules…" Each bullet opens with a bolded noun phrase naming the axis of improvement. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (Wang et al., NeurIPS 2019)
  • "In sum, our contributions are the following." "(i) We release a suite of 9 cross-lingual benchmark tasks covering 40 typologically diverse languages." / "(ii) We provide an online platform and leaderboard for the evaluation of multilingual models." / "(iii) We provide a set of strong baselines… and release code to facilitate adoption." / "(iv) We provide an extensive analysis of limitations of state-of-the-art cross-lingual models." Release / platform / baselines / analysis. A complete resource-paper template. XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (Hu et al., ICML 2020)
  • "Our contributions are twofold." "First, we demonstrate that with the proper training setup, simply fine-tuning the question and passage encoders on existing question-passage pairs is sufficient to greatly outperform BM25." Counts the contributions up front, then enumerates. Dense Passage Retrieval for Open-Domain Question Answering (Karpukhin et al., EMNLP 2020)
  • Contributions as an inline numbered list of design changes. "Our modifications are simple, they include: (1) training the model longer, with bigger batches, over more data; (2) removing the next sentence prediction objective; (3) training on longer sequences; and (4) dynamically changing the masking pattern applied to the training data." Whole list packed into one sentence. RoBERTa: A Robustly Optimized BERT Pretraining Approach (Liu et al., 2019)
  • The method / evidence / release triad. "we introduce Self-Instruct, a method for inducing instruction following capabilities with minimal human-labeled data" / "we demonstrate its effectiveness via extensive instruction-tuning experiments" / "we release a large synthetic dataset of 52K instructions and a set of manually-written novel tasks for building and evaluating future instruction-following models." Method, then proof, then the artefact. Self-Instruct: Aligning Language Models with Self-Generated Instructions (Wang et al., ACL 2023)
  • Contributions framed as "key advantages." "LoRA possesses several key advantages." → "A pre-trained model can be shared and used to build many small LoRA modules for different tasks." / "LoRA makes training more efficient and lowers the hardware barrier to entry by up to 3 times when using adaptive optimizers." / "Our simple linear design allows us to merge the trainable matrices with the frozen weights when deployed, introducing no inference latency." / "LoRA is orthogonal to many prior methods and can be combined with many of them, such as prefix-tuning." Reframes contributions as reader-facing benefits. LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., ICLR 2022)
  • The numbered method / experiments / analysis triad. "(1) we introduce ReAct, a novel prompt-based paradigm to synergize reasoning and acting in language models for general task solving;" / "(2) we perform extensive experiments across diverse benchmarks to showcase the advantage of ReAct… over prior approaches that perform either reasoning or action generation in isolation;" / "(3) we present systematic ablations and analysis to understand the importance of acting in reasoning tasks, and reasoning in interactive tasks." The most common three-part shape in modern *ACL papers. ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al., ICLR 2023)
  • The single-sentence contribution. "Our main contribution is Direct Preference Optimization (DPO), a simple RL-free algorithm for training language models from preferences." When the work is one clean idea, one sentence is enough. Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafailov et al., NeurIPS 2023)
  • "Firstly / Secondly" conceptual advances. "Firstly, instead of using a fixed forward or backward factorization order as in conventional AR models, XLNet maximizes the expected log likelihood of a sequence w.r.t. all possible permutations of the factorization order…" / "Secondly, as a generalized AR language model, XLNet does not rely on data corruption. Hence, XLNet does not suffer from the pretrain-finetune discrepancy that BERT is subject to." Contributions stated as ideas, each contrasted with the incumbent. XLNet: Generalized Autoregressive Pretraining for Language Understanding (Yang et al., NeurIPS 2019)
  • "To summarize, our main contributions are." "BigBird satisfies all the known theoretical properties of full transformer… we show that under standard assumptions regarding precision, BigBird is Turing complete." / "Empirically, we show that the extended context modelled by BigBird benefits variety of NLP tasks." / "Lastly, we introduce a novel application of attention based models where long contexts are beneficial: extracting contextual representations of genomics sequences like DNA." Theory, then empiry, then a surprising new application. Big Bird: Transformers for Longer Sequences (Zaheer et al., NeurIPS 2020)
  • Contributions framed as research "goals." "Our goals in doing so are: 1. to enable research progress toward building high-quality question answering systems in roughly the world's top 100 languages; and 2. to encourage research on models that behave well across the linguistic phenomena and data scenarios of the world's languages." A resource paper states the community outcomes it hopes to enable. TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (Clark et al., TACL 2020)
  • Contributions enumerated as the experiments themselves. "In Experiment 1, we train GPT-2 models on our set of defined possible and impossible languages… We find that models trained on possible languages learn more efficiently." / "In Experiment 2, we more closely examine a set of languages that exhibit count-based verb marking rules…" / "In Experiment 3, we dive deeper into the internal mechanisms that models may develop… using causal abstraction analysis." Each contribution is one experiment plus its finding. Mission: Impossible Language Models (Kallini et al., ACL 2024)
  • Variant: design desiderata, not contributions. "The use of tools should be learned in a self-supervised way without requiring large amounts of human annotations…" / "The LM should not lose any of its generality and should be able to decide for itself when and how to use which tool…" The bullets state goals the proposed method must satisfy; the contribution is then "the method that meets them." Toolformer: Language Models Can Teach Themselves to Use Tools (Schick et al., NeurIPS 2023)
  • Variant: findings as bolded headers. "Labelers significantly prefer InstructGPT outputs over outputs from GPT-3." / "InstructGPT models show improvements in truthfulness over GPT-3." / "InstructGPT shows small improvements in toxicity over GPT-3, but not bias." In place of a contributions list, the intro previews results as a series of bolded one-line findings. Training language models to follow instructions with human feedback (Ouyang et al., NeurIPS 2022)

Common mistakes​

  • The history-of-NLP introduction. "Since the days of Chomsky…" No. Start where the problem starts.
  • Forward references everywhere. "As we will show in §5.3…" Three of these is fine; thirty makes the paper feel like an index.
  • Contributions that are not contributions. "We use a transformer architecture" is not a contribution. "We provide the first evidence that X" is.
  • Method dump in the intro. The intro previews; the methods section delivers. Keep the intro at a level a smart non-specialist can follow.
  • The bait-and-switch. Promising one thing in the intro and delivering another in the experiments. Reviewers catch this and punish it.

The contributions list, done right​

Bad:

  • We propose a transformer-based model.
  • We train it on a large dataset.
  • We achieve state-of-the-art results.

Good:

  • We introduce MethodName, the first approach to X that does not require Y.
  • We release a benchmark of 12,000 examples covering Z, with human-verified labels.
  • We show empirically that the standard assumption A does not hold in setting B, and identify the conditions under which it does.

Each good bullet promises something the reader could not have done themselves.

Five examples from real ACL papers​

  1. BERT (Devlin et al., NAACL 2019). The contributions list at the end of §1 is a textbook example: each bullet is precise and verifiable.
  2. Linguistic Knowledge and Transferability of Contextual Representations (Liu et al., NAACL 2019). Clean five-paragraph structure with a sharp research question.
  3. On the Dangers of Stochastic Parrots (Bender, Gebru, McMillan-Major, Mitchell, FAccT 2021). Not *ACL, but field-defining. Study how it frames the problem space in paragraphs 1–2.
  4. Stanza: A Python NLP Toolkit for Many Human Languages (Qi et al., ACL 2020 Demo). A resource paper that motivates itself in two paragraphs without overclaiming.
  5. Don't Stop Pretraining (Gururangan et al., ACL 2020). The intro tells you exactly what experiment you are about to read and what it found.

Further reading​

  • Kate Turabian, A Manual for Writers, chapter on argument structure.
  • John Swales's "CARS" model (Create A Research Space), useful for paragraph 2 (establishing the gap). Search "Swales CARS model."