Skip to main content

2. Understanding the field

You have a supervisor, a vague topic, and a laptop. You do not yet have a research question, and you should not pretend to. Spend the first months of a PhD reading widely: the whole area around your topic, beyond the twelve papers closest to it. By the end you should be able to name the people who shape your subfield and say where they publish. You should be able to list its open problems and sketch three questions you could spend a year on. The most useful tool for getting there is the review paper, both read and written.

What you are trying to learn​

Reading widely has deliverables. Have four things on paper by the time you stop.

The people. A few dozen names recur in your subfield's author lists, citations, and programme committees. Learn them and the groups they lead, and read their last three papers, because their reviewers will be your reviewers.

The venues. Learn where your subfield publishes and where it argues. Chapter 1 lays out the *ACL family; your job here is narrower. Find the workshop your area treats as home, the shared tasks it runs, and the journals that take its long work.

The subareas, methods, and data. Draw the map: the tasks people work on, the two or three method families that dominate each, and the datasets and benchmarks everyone evaluates on. If you cannot name the standard benchmark for a task, you do not know the task yet.

The open problems. These are the point of the exercise. They hide in the last paragraph of good papers, in Limitations sections, in the "future work" of surveys, and in the gap between what a benchmark measures and what the field claims. Write each one down the moment you see it, with the paper that made you see it.

Where to look​

Start with the ACL Anthology, which is free, searchable, and complete back to the 1960s. Use a general search engine after it. The cs.CL listing on arXiv shows what is coming before it is reviewed. Read it daily for a month and you will learn the rhythm of the field. Semantic Scholar and Connected Papers turn one paper into its citation neighbourhood. That is the fastest way to find the work a method came from and the papers that superseded it. For the foundations, Jurafsky and Martin's Speech and Language Processing is free online and kept current.

Read each paper in passes. Keshav's "How to Read a Paper" gives the standard three-pass method: title, abstract, headings and conclusion first; figures, tables and the gist second; a full, critical read only for the papers that survive the first two. Most papers you meet in your first year should get one pass.

The field also studies itself, and those papers are a shortcut to understanding it. Read these three early:

Start with review papers​

A review paper compresses five years of a subfield into twenty pages. A good one gives you the taxonomy of approaches, the standard datasets, the strongest methods, and a list of challenges that someone with real command of the area believes are open. Three reviews of your area, from different years, teach you more in a week than three months of primary papers read in citation order.

To find them, search the Anthology for survey, review, primer and tutorial alongside your topic. Computational Linguistics and TACL publish the long ones; the main conferences accept surveys as a contribution type (Chapter 3); ACM Computing Surveys takes many of the rest. Read the most recent one first, then the one it cites as its predecessor.

An editorial note​

(This paragraph is in the first person because it is advice I give my own students.) Do not just read the reviews. While you read them, draft one of your own, framed by your own problem. Keep a document with one row per paper: task, data, method, headline number, and the limitation the authors admit or you spot. Group the rows as you go. Within a few weeks you will have the skeleton of a survey of your narrow area. You will also find which parts of the published reviews you did not understand, because you could not write them down. Francis Bacon put the principle in one sentence in Of Studies (1625): "Reading maketh a full man; conference a ready man; and writing an exact man." Reading fills you; writing makes you exact, since you cannot write a sentence you have not understood. The same document becomes your related work section (Chapter 14), and sometimes a publishable survey in its own right.

The kinds of review paper NLP writes​

NLP uses four kinds of review. They look alike from a distance, but reviewers judge each by different standards. Decide which one you are writing before you start.

The narrative survey​

The narrative survey is the common form: an expert's organised map of a subfield, built from the papers they judge to matter. Its contribution is the organisation. A survey that summarises eighty papers in the order you found them is an annotated bibliography, and reviewers will say so. A survey that proposes a taxonomy, places every method in it, and shows where the cells are empty is a research contribution.

To write one, build the taxonomy first and let it drive the reading. Make each section answer a question a newcomer would ask ("how do people get labels for this?", "what breaks when the input is long?"). Naming a method family is not enough. Add comparison tables: methods against the properties that distinguish them, datasets against size, language and licence. End with open problems specific enough to become someone's thesis, not "more work is needed on low-resource languages."

Five that show the form:

The systematic review​

A systematic review replaces the expert's judgment about which papers matter with a protocol anyone can rerun: a stated search query over stated sources and dates, explicit inclusion and exclusion rules, and a count of papers at each step. The authors code each included paper against a fixed scheme and report counts instead of impressions. The form comes from medicine; the PRISMA 2020 statement is the reporting standard, and more NLP reviewers now know it.

Write the protocol before you read a single paper, and report it in full so a reader can reproduce your paper set. Most of the work is in the coding scheme. Decide what you will record about each paper (did it report seeds? how many annotators? which languages?) and record it for every paper, even when the answer is "not stated", because "not stated" is often the finding. Systematic reviews suit questions about practice ("how does the field evaluate X?") better than questions about ideas.

Four from NLP:

The critical review​

A critical review surveys a literature to argue that it is doing something wrong. It has a thesis, it gathers the evidence from the papers themselves, and it ends with recommendations. It is the hardest of the four to write well. A thesis without a systematic reading behind it is an opinion, and reviewers will call it a position paper with no new insight.

Combine the other two forms. Read with a protocol so no one can accuse you of cherry-picking, then organise the findings around the argument instead of the methods. Quote the papers you criticise exactly. Criticise the practice and leave the authors alone.

Four that earn their argument:

The tutorial or primer​

A primer teaches a method rather than mapping a field. Its reader is a newcomer who must use the method next month. So it builds from first principles, fixes one notation and keeps it, and works examples. It does not claim coverage, and it does not argue. Many primers start life as the "background" chapter of a thesis or the notes for a tutorial at a conference.

Three to study:

Which should you write first?​

Write a narrative survey of your own narrow problem, for yourself, and keep it alive. If the area is small enough to enumerate (a few hundred papers from a clear query), make it systematic from the start; the protocol costs a day and the result is publishable. Leave the critical review until you have something to be critical about, which takes a year of reading and at least one paper of your own. Leave the primer until you have taught the method to someone.

Publish only when the document answers a question other people are asking. The main conferences accept surveys, and Findings takes many of them. TACL and Computational Linguistics take the long ones. Workshops suit a survey of a narrow area. Wherever you send it, signal the type in the title and abstract so reviewers look for a taxonomy instead of experiments.

Common mistakes​

  • The annotated bibliography. Summaries, one per paper, in the order you found them. There is no argument and no taxonomy, so there is no contribution.
  • The survey of everything. "A survey of large language models" cannot be done well in twenty pages by one student. Narrow until the paper set is one you have read.
  • No stated method. Even a narrative survey should say how you found the papers and why you chose those. Without it, every omission looks like ignorance.
  • Missing the origin or the latest. Reviewers in the area will check for the paper that started it and the three from the last cycle. Use the citation graph for the first and arXiv for the second.
  • Out of date on submission. A survey takes months and the field does not wait. Date your search, and rerun it the week before you submit.
  • Citations you have not read. Surveys cite hundreds of papers and are where fabricated references appear most often, especially when a language model helped. Every entry must be a paper you opened. See Chapter 24.

Further reading​

  • S. Keshav, "How to Read a Paper": the three-pass method, three pages long.
  • The PRISMA 2020 statement: the reporting standard for systematic reviews, with the flow diagram NLP reviewers now expect.
  • Chapter 3 on where surveys sit among *ACL paper types, and Chapter 14, which turns the document from this chapter into a section of your paper.
  • Chapter 30 for the reading list that belongs beside this one.