5. Decoding the jargon
*ACL reviews, hallway conversations and social media threads are thick with terms a newcomer is assumed to know: desk reject, Reviewer 2, salami-slicing, SOTA, stochastic parrots. Nobody defines them for you, and asking can feel like admitting you do not belong.
This chapter collects the jargon, in-jokes and named failure modes of research culture, weighted toward NLP and machine learning. For each term we give a short definition, its origin where one is documented, and how people use it, since tone carries as much meaning as the words. We mark which terms have a citable origin and which are folklore. Knowing which is which is part of fitting in.
Many of these terms are critiques, a warning the rest of the guide also makes. Aim them at patterns and at your own work, never at named people. The author whose paper you wave away as "cargo-cult science" may be reviewing your next submission.
Several of these terms used to be gossip and are now policy. Salami-slicing, self-plagiarism, dual submission and undisclosed AI writing each have a written rule in the ARR call for papers, and the penalty for most of them is a desk reject. Where a term has consequences, we spell them out.
The publishing game
The vocabulary of academic incentives is faintly cynical, because the incentives are.
Publish or perish. The pressure to publish steadily to get hired, funded and promoted. The phrase appears in academic writing by 1928 and was popularized by Logan Wilson's 1942 book The Academic Man (overview). People say it wearily, usually to explain why a field rewards quantity over depth.
Least publishable unit (LPU). The smallest sliver of a research project that can still get past peer review on its own. The term traces to William Broad's 1981 Science article "The Publishing Game: Getting More for Less" (reference). It is always pejorative. A reviewer who smells an LPU asks "where is the rest of this paper?"
Salami-slicing, now "thinly sliced contributions." Carving one study into several LPUs to inflate a publication count: the dataset in one paper, the obvious model on it in a second, the error analysis in a third (overview). For decades this cost only reputation. Since 2026 it breaks written rules. ICML named the problem "thinly sliced contributions," meaning "submission of a large number of papers with small variations on the same theme," and tied it to the flood of generated papers (ICML 2026 blog). ARR followed: "ARR now follows ICML in discouraging thinly sliced contributions," so "any concurrently submitted papers on a related topic with an overlapping set of authors must cite each other and discuss the differences in the related work section," with anonymized copies uploaded as supplementary material, and "there can be no overlap in stated contributions" (ARR call). Papers that break this "may be desk rejected." ICML goes further: violations "may result in desk rejection of all papers by any of the authors of an offending paper," including a group leader who did not know. EMNLP 2026 added a ban. Under its paper integrity policy, "all authors involved may be ineligible to commit their paper(s) to EMNLP 2026 and EMNLP 2027." That is two years out of a flagship venue for every name on the paper. Two honest papers from one project are fine, if each stands alone and each cites the other. Three papers that only make sense together are one paper. See Chapter 4 on what a contribution is.
Self-plagiarism, or text recycling. Reusing your own published text without saying so. ARR caps it: re-use of text from other publications is allowed up to "no more than 10% total tokens," and "self-plagiarism (significant overlap in content with other submissions or publications by the same authors)" is a listed reason for desk rejection (ARR call). Extending a workshop paper into a conference paper is normal. Pasting it in is self-plagiarism. Rewrite, cite the earlier version, and say what is new.
Dual (or multiple) submission. Having the same paper under review at two venues at once. "ARR precludes multiple submissions" and "will not consider any paper that is under review in a journal or another conference," with no exception for workshops that publish proceedings (ARR call). The one legal form of parallelism is the ARR system itself: review once, then commit to one venue. Dual submission earns a desk reject and a reputation that follows you.
Preprint, and the old anonymity period. Posting your paper on arXiv before or during review. Until 2024, *ACL banned non-anonymous preprints in the month before a deadline and during review, the anonymity period. "Beginning February 15, 2024, there is no anonymity period or limitation on posting or discussing non-anonymous preprints while the work is under peer review" (ARR call; the change). The paper you submit must still be anonymous. You can also tick a binding "no non-anonymous preprint" box, which earns eligibility for some awards and priority for borderline papers. Tick it and you must keep to it or be desk rejected.
Predatory venues and paper mills. A predatory journal or conference takes a fee and publishes almost anything, with little or no review (overview). A paper mill sells ready-made papers and authorship slots (overview). Both prey on researchers whose institutions count publications without checking venues. They are a live hazard where *ACL conferences are not recognised for promotion. A line from either on your CV is a liability: a reviewer who finds one reads the rest of the CV differently. Every *ACL venue is in the ACL Anthology. If a "conference on NLP" is not, ask why.
The file drawer problem. Studies that found nothing stay in the drawer, so the published record over-represents positive results. Rosenthal named it in 1979; the general term is publication bias (overview). *ACL pushes against it by inviting negative results and reproductions as contributions (Chapter 3). The Insights workshop exists for the papers that used to stay in the drawer.
The Matthew effect. "For unto every one that hath shall be given." Robert Merton's 1968 name for the way credit flows to the already credited: the famous co-author gets the citation, the known lab gets the benefit of the doubt (overview). Anonymous review exists to blunt it. Preprints and the end of the anonymity period arguably sharpen it. That argument lies behind the "anonymous preprint" incentive above.
The h-index. A single number summarizing a researcher's output: you have an h-index of h if h of your papers each have at least h citations. Physicist Jorge Hirsch proposed it in 2005 (PNAS). People like it because it is simple. They distrust it because self-citation, salami-slicing and citation trading can game it, and because career length and field size bias it.
Citation cartel. A group of authors, journals or venues that cite each other heavily to pump everyone's metrics beyond what the scholarship warrants (overview). The lighter, individual version is self-citation inflation. Across one large study, roughly a third of citations in a paper's first three years are self-citations, concentrated among the least-cited work. Use the term only as a documented accusation.
Gift (or honorary) authorship. Listing someone as an author who did not contribute substantively. Often it is a senior figure added by default, sometimes a junior researcher added as a favor. The practice is widespread and well documented, and in extreme cases authorship has been bought and sold (Chapman et al., 2019, "Games academics play and their consequences"). The mirror problem, leaving off someone who did contribute, is ghost authorship.
Impact factor. The average citations per article for a journal over a window. People routinely misuse it as a proxy for the quality of a single paper in the journal. Citations concentrate in a few articles, so the average says little about any one of them (a much-cited illustration: 89% of Nature's 2004 impact factor came from a quarter of its papers). *ACL is conference-driven, so this matters less here than in journal fields. The gaming instinct is the same.
AI writing assistance, and "AI slop." Using a language model to draft, rewrite or generate parts of a paper. The ACL policy, which ARR follows, allows "assistance purely with the language of the paper" without disclosure. Any other use "must be disclosed in the Responsible NLP Checklist" with details in the Acknowledgements, and "generative AI tools do not qualify for authorship" (ARR call). ICML's 2026 policy uses AI slop for low-quality generated papers submitted in bulk. It treats them as abuse of peer review, with the same penalties as thin slicing. The rules bind reviewers too. A reviewer "is not permitted to use generative assistance to create the first draft" of a review and may not upload your manuscript to a non-private tool (reviewer guidelines). Chapter 24 covers what to disclose and how.
Goodhart's law. "When a measure becomes a target, it ceases to be a good measure." Charles Goodhart stated the original, about monetary policy, in 1975; the crisp modern phrasing is Marilyn Strathern's, from 1997 (overview). It is the principle behind most of this section and behind benchmark-chasing below.
Inside the review machine
*ACL documents its review process in detail, and folklore has grown around it. Learn the documented rules and treat the folklore as folklore.
ARR (ACL Rolling Review), and "commitment." A centralized reviewing system that produces reviews and meta-reviews tied to no single venue. Authors then "commit" a reviewed paper to a specific conference (ACL, EMNLP, NAACL), which makes its own accept/reject call (process). So an ARR review is not an acceptance. The two steps are separate.
Soundness, Excitement, Overall Assessment, and Confidence. The ARR review form scores a paper on Soundness (is it technically valid?), Excitement (is it interesting? The form treats this as independent of soundness) and an Overall Assessment, each on a 1–5 scale. A separate Confidence score records how well the reviewer knows the area (reviewer guidelines). A paper can be sound but unexciting, or exciting but shaky, and the form lets reviewers say so.
Area chair (AC), senior area chair (SAC), and the meta-review. The AC oversees a cluster of papers, manages reviewers and runs the author–reviewer discussion. The AC also writes the meta-review, which names the strongest pros and cons and judges whether the pros outweigh them. The SAC calibrates meta-reviews across an area (AC guidelines). Both recommend; program chairs and the committing venue decide.
Desk reject. A rejection by an editor or chair before peer review starts. The usual cause is a broken hard rule: over the page limit, no Limitations section, an anonymity violation, bad formatting, or a paper out of scope. It is avoidable and entirely on you, and it is the fastest way to waste a submission cycle.
The Responsible NLP Checklist. The form you fill at submission, question by question, saying whether the paper reports limitations, risks, licences, compute, annotator pay and AI assistance (checklist). Reviewers read it. ARR lists "incorrect, incomplete or misleading filing of the responsible NLP checklist" among the common desk-reject reasons of recent cycles (ARR call). Answer it honestly and early. A reviewer will quote back any "yes" you cannot point to in the paper.
Revise and resubmit, and the "new paper" test. ARR lets you fix a paper and send it back to the same reviewers with a summary of changes, which is what resubmission means there. Calling the fix a new paper will not free you from bad reviews: "to count as a new paper rather than a revision, the submission must be on a completely different topic," and "further improving the proposed methods, extending the set of experiments or analysis, extending a short paper to a long one, or even shifting the topic substantially" does not qualify (ARR call). You may ask for new reviewers, but you must still answer the old reviews point by point. Chapter 28 has the strategy.
Borderline. A paper whose scores sit on the accept line, where the meta-review and the committing venue's quota decide. Most papers are borderline. Here a clear contributions list, an honest Limitations section and a calm rebuttal count most, and here the "anonymous preprint" incentive above applies.
Rebuttal / author response. The window in which authors reply to reviews: correcting misreadings, reporting a quick extra experiment, conceding fair points. Tone wins or loses rebuttals. Address the substance and do not lecture the reviewer. ("Happy to address in the rebuttal" is the standard phrase for a concern you plan to answer there.)
"Not novel." The most common and laziest rejection rationale: the claim that the contribution has been done before. ARR constrains it. A reviewer claiming a method is not novel must cite the prior work, and reproductions, re-implementations and analyses count as contributions (reviewer guidelines). A "not novel" with no citation is a point for your rebuttal.
"Championing" a paper. Informal shorthand for an AC or reviewer who argues hard to rescue a strong paper from negative scores. It is folklore. ARR documents no "champion reviewer" role, however often the word comes up in committee talk.
AI reviewing. Reviews written by a language model, either against the rules (a human reviewer pasting your paper into a chatbot, which ARR forbids) or by design. EMNLP 2026 ran the first large opt-in AI Reviewing Experiment. In it, 7,921 submissions opted in and each was assigned to one of three systems. Only the authors saw the AI review, after the rebuttal period; human reviewers and chairs never did, and it carried no score or recommendation. The chairs' own pilot found the AI reviews "overly optimistic" and unable to tell "which errors/weaknesses are critical" (update). Expect more of these experiments. Expect the area chair to challenge a review that reads like one.
Reviewer 2. The folk archetype of the hostile, dismissive referee. Reviewer 2 wants a different paper from the one you wrote, demands their own work be cited, and rejects on vibes. The figure is pure folklore with no origin event, only a shared grievance. The 28,000-member "Reviewer 2 Must Be Stopped" group and a genre of academic humor keep it alive (scholarly treatment). Use it to commiserate, never as a description in a real review.
Ways to fool yourself
Most of these named failure modes were coined outside NLP, and all of them occur inside it. Each names a way a result can look real when it is not. Richard Feynman's line covers them all: the easiest person to fool is yourself.
p-hacking. Trying analyses, subsets or metrics until something clears the significance bar, then reporting only that. The researchers behind the Data Colada blog (Simonsohn, Nelson, and Simmons) coined the term in 2014 (overview). In ML the equivalent is tweaking splits, seeds or hyperparameters until a baseline loses. That is why Chapter 18 insists on val/test separation and seeds.
The garden of forking paths. Coined by Andrew Gelman and Eric Loken in a 2013 working paper whose title nods to a Borges story (paper). It names a subtler problem than p-hacking. Even a single analysis can be invalid if the choices that produced it depended on the data, because many other reasonable choices were available. The authors distinguish it from "p-hacking": forking paths needs no conscious fishing, only ordinary judgement applied after seeing the data.
HARKing. Hypothesizing After the Results are Known: presenting a post-hoc finding as if it had been your prediction all along. Norbert Kerr coined it in 1998, and "The HARK Side of Deep Learning" (2019) applied it to machine learning. The fix is the honesty Chapter 17 asks for. Say what you set out to test, and what you found by accident.
Cargo-cult science. Work that mimics the outward forms of science (the equations, the tables, the significance tests) while missing the essential thing. Feynman named that thing as a "leaning over backwards" honesty about everything that might prove you wrong (Feynman, 1974 Caltech commencement address, "Cargo Cult Science"). The epithet is sharp and contested, since some argue it slides too easily into attacks on character. Use it on practices, never on persons.
Clever Hans / shortcut learning. Clever Hans was a horse in the 1900s that seemed to do arithmetic but was reading its trainer's body language. In ML the analogy names a model that scores well by exploiting a dataset artifact instead of learning the task. Lapuschkin et al. popularized it in 2019 with "Unmasking Clever Hans predictors". A model "pulls a Clever Hans" when a strong number hides a spurious shortcut. Finding these is the job of Chapter 20.
Graduate student descent. A joke on gradient descent: hyperparameter tuning done by hand, by a grad student running configs until something works. It is folklore, but it names a real reproducibility hazard. Numbers found this way are easy to HARK and hard to repeat.
Cherry-picking. Reporting the run, the seed, the subset or the example that makes the method look best, and leaving out the rest. It is the quiet cousin of p-hacking. It is also why Chapter 18 asks for variance over seeds and Chapter 20 asks for the failure cases as well as the successes.
Weak (or strawman) baselines. Comparing against a baseline you did not tune, an old number copied from a paper with a different setup, or a model nobody would deploy. The gain is then an artifact of the comparison. The ARR reviewer guidelines push the other way too: reviewers should not demand comparisons to closed commercial models when "test contamination and a general lack of information about 'closed' models" make them meaningless (reviewer guidelines). Tune the baseline as hard as the method. Sennrich and Zhang's Revisiting Low-Resource NMT (ACL 2019) is the standard demonstration of what happens when you do.
Data leakage. Information from the test set reaching training, directly or through a feature, a duplicate, or a preprocessing step fitted on everything. The resulting number will not survive new data. Benchmark contamination, below, is leakage at pretraining scale.
LLM-as-judge. Using a language model instead of humans to score another model's outputs. It is cheap and increasingly common. The ARR guidelines name "LLM-only evaluation without validation" as a methodological weakness: if a model is the evaluator, "is their reliability in this context sufficiently validated?" (reviewer guidelines). Validate the judge against a human sample, report the agreement, and say what the judge cannot see.
Overclaiming, and construct validity. Concluding more than the measurement supports. A few QA benchmarks are not "reasoning," one model size is not "LLMs," and a benchmark score is not "understanding" unless you have shown the benchmark measures it. The ARR guidelines list these as weaknesses, for example "concluding from benchmark evaluation that LLMs generally 'understand' language, without validating that construct" (reviewer guidelines). Write the claim the data supports, then stop.
Ablation. Removing one component of a system to measure its contribution. The term comes from experimental biology, where you lesion a part to learn its function. Ablation is the standard cure for a failure mode. It shows which of your design choices earns its place (Chapter 20).
The replication (or reproducibility) crisis. The finding, across many empirical fields, that a large share of published results do not hold up when re-run. In NLP it shows up as numbers that cannot be reproduced from the paper. The field responded with reproducibility checklists, and Chapter 25 treats release and documentation as part of the contribution.
The lore of NLP and machine learning
Terms born in our own subfield, mostly in the last decade, mostly on the internet.
"X is all you need." A snowclone, a fill-in-the-blank cliché, spawned by the title of the 2017 Transformer paper "Attention Is All You Need", which itself echoed the Beatles. The template now signals, half-ironically, "this one idea is the whole story." Do not title a paper this way. The joke was fresh in 2017.
The bitter lesson. Richard Sutton's 2019 essay (overview). It argues that across the history of AI, general methods that scale with computation (search and learning) have repeatedly beaten approaches that build in human knowledge by hand. The result keeps surprising researchers, hence "bitter" in the name. People invoke it in every debate about whether cleverness or compute will win the next round.
"Scale is all you need." The maximalist reading of the bitter lesson: bigger models and more data drive progress more than new ideas do. Scaling optimists say it approvingly; skeptics say it witheringly. The argument is still open.
Stochastic parrots. A model that stitches together plausible language from statistical patterns in its training data "without any reference to meaning." The phrase comes from Bender, Gebru, McMillan-Major, and Mitchell's 2021 paper "On the Dangers of Stochastic Parrots", a landmark critique of large language models. People use it to puncture overclaims about machine "understanding." Detractors use it as a strawman. Know the paper before you use the phrase.
SOTA, and SOTA-chasing. SOTA = state of the art, the current best published number on a benchmark. SOTA-chasing is treating a higher number as the whole point of a paper. ARR rejects this as a review standard: "SOTA results are neither necessary nor sufficient for a scientific contribution" (reviewer guidelines). A paper can contribute through efficiency, analysis or fairness without topping a leaderboard. SOTA-chasing is Goodhart's law applied to a leaderboard.
Benchmark contamination. Test data leaking into a model's training set, which makes high scores meaningless. It grows more likely as models are pretrained on scraped web text that includes the benchmarks. This is why resource papers now argue that their test set is clean (Chapter 16).
GPU-poor vs GPU-rich. Informal labels for the compute gap between groups with vast accelerator fleets and everyone else (students, most academics, most of the Global South). The terms entered wide use through 2023 industry commentary (context). The divide shapes which research questions a given budget can even ask.
"Vibes-based evaluation." Folklore, and a self-aware confession: judging a model by eyeballing a few outputs instead of measuring anything. Sometimes it admits that no good metric exists yet. More often it is a warning sign. If your evaluation is vibes, read Chapter 18.
Concurrent work. Research released so close to yours that neither could have cited the other. Under *ACL norms the status excuses a missing citation for work posted within roughly the last three months. Claim it honestly and you lose nothing. Claim someone's earlier work as concurrent and a reviewer will notice (Chapter 14).
The Bender Rule. Name the language you work on, even when it is English. Emily Bender made the point in talks from 2011 and wrote it up in 2019. Papers on English say "language" while papers on Hausa say "Hausa," which lets the field mistake English for the general case (The Gradient). Reviewers now cite it. Put the language in the abstract and, if the paper is about one language, in the title.
Low-resource, under-resourced, and the resource classes. A low-resource language is one with little labelled or unlabelled data for NLP. The term is relative and loosely used. The State and Fate of Linguistic Diversity and Inclusion in the NLP World (Joshi et al., ACL 2020) made it concrete with six classes, from "the left-behinds" with almost no data to "the winners." Say which class you mean. Many prefer under-resourced: the resources are missing because nobody built them, and the language itself lacks nothing.
The curse of multilinguality. Add more languages to one model at fixed size and per-language quality eventually drops. Unsupervised Cross-lingual Representation Learning at Scale (Conneau et al., ACL 2020), the XLM-R paper, named and measured it. It is why "just use the multilingual model" is no plan for a low-resource language, and it motivates language-adaptive fine-tuning and language-family models.
Scaling laws, and Chinchilla. Empirical power laws relating model loss to parameters, data and compute, set out in Scaling Laws for Neural Language Models (Kaplan et al., 2020). Chinchilla is shorthand for the 2022 correction in Training Compute-Optimal Large Language Models (Hoffmann et al.). Most large models had been under-trained on data for their size, so "Chinchilla-optimal" means the parameter-to-token ratio that paper recommends. The term comes up whenever someone argues about whether a model is too big for its data.
Emergent abilities, and the mirage. Capabilities that seem to switch on at a certain model scale and are absent in smaller models. The term comes from Emergent Abilities of Large Language Models (Wei et al., 2022). Are Emergent Abilities of Large Language Models a Mirage? (Schaeffer et al., 2023) replied that many "emergences" are artifacts of discontinuous metrics such as exact match, and vanish under smooth ones. Use the word with the metric attached.
Hallucination. A fluent output that the input and the facts do not support: an invented citation, a wrong date, a confident summary of a sentence that was never there. The word is borrowed from psychiatry, disputed as a metaphor, and now unavoidable. For a paper, the practical version is the hallucinated reference, the subject of the next entry.
Unverifiable references. A bibliography entry that no academic database can find, usually the product of letting a language model write the related work. Machines now screen for it. EMNLP 2026 "has partnered with GPTZero to detect references that do not exist in academic databases in all submitted papers". Every entry in every bibliography is classified, the area chairs check flagged entries, and "papers with any unverifiable reference may be desk rejected, and all authors involved in the submission may be ineligible to commit their paper(s) to EMNLP 2026 and EMNLP 2027" (paper integrity policy). One fabricated reference can mean a desk reject and two years out. The first enforcement round was large. In the ARR May 2026 cycle, EMNLP "rejected 258 submissions written by 1,166 authors." A paper was rejected if "there was at least one reference to a paper that did not exist with the given title" or "at least three references had major errors in their author lists," and the chairs recommended barring from EMNLP 2027 the 35 authors with more than one such paper (EMNLP statement). Acceptance is no protection. ACL 2026 desk-rejected papers for hallucinated references at the camera-ready stage, and the ACL executive ruled that "affected papers must go through the normal resubmission process" (EMNLP 2026 note). The defence is mechanical. Take every entry in your .bib from the Anthology, a DOI, or a paper you have open; see Chapter 25 and Chapter 24.
Further reading
- The peer-review machinery, from the source: the ACL Rolling Review reviewer and area-chair guidelines. Read them before you review, and before you rebut.
- Feynman, "Cargo Cult Science" (1974). Still the best short statement of what scientific integrity demands.
- Chapman et al., "Games academics play and their consequences" (2019). A sober catalogue of the incentive pathologies in the first section above.