I have taken a lot of notes on research papers since I started doing research. I want to explore why I did that, and whether I should keep doing it.

In particular, I wrote brief summaries and reflections on many of the papers I read when I was researching reward learning and deep learning as a CHAI intern and Master’s student. I took these notes by hand, on A5 pieces of paper and arranged them across pinboards in my office like a detective. I now have hundreds and hundreds of these hand-written slips archived in piles on my desk. Why did I do that?

Since then, I haven’t been reading papers so voraciously (with the exception of titles). I’ve taken notes on fewer than one hundred papers since the start of my doctorate. There’s a stack of blank A5 slips on my desk, but my pinboard sits empty. This has happened for a mix of reasons, but I now have an opportunity to spend more time reading. Should I try to pick back up the habit of note-taking? Should I make any changes?

Let’s explore!

§Recollection

One plausible justification is that I take notes for the purpose of better remembering what I read. Hypothetically, simply writing down what I read will make me remember it for longer. Maybe physically locating the paper on the pinboard will let me associate a physical location with the ideas in the paper.

The problem with this theory is that, now, many years after reading hundreds of papers with this method, I remember only a tiny fraction of the papers I read. On the rare occasion that I flick through the archive, there are some papers I have completely forgotten having read until I see them. There are others that I can’t remember having ever seen at all even after I see them.

If I wanted a memory aid for recalling all the papers I’ve ever read, I would probably be better off turning my notes into flash cards and drilling them with a spaced repetition system like Anki or, my preferred approach, Bayesian spaced repetition. Spaced repetition is highly regarded and it served me well personally when I was drilling German vocabulary on exchange. Not that I remember the vocab now. As my Bayesian app will tell you, my probability of recalling these items has atrophied since I stopped using it. But there was a time when I was juggling thousands of vocab items, which is proof that I can scale this approach to the number of papers I expect to be reading, if I want.

Of course, I’m not sure I need or want to remember every paper I ever read. Which brings me to the next theory.

§Reference

Maybe I take notes in order that I don’t have to remember the papers I have read. The notes act as a summary that I can refer back to if I merely remember that I read about a paper, not exactly what was and wasn’t in it. The thing is, again after many years, I have barely used my notes in this way.

I do sometimes look at the notes. For example, when I’m writing a paper, I sometimes refer back to my notes board (or notes pile) and find papers I had forgotten were relevant and are worth incorporating into the story. More broadly, when I want to remember what was in a paper, the summaries I write are a helpful starting point.

However, this theory needs to reckon with the fact that I could potentially achieve the same outcome by keeping a list of the titles of all the papers I read during a long research project, and consulting that list, including consulting the papers themselves if I forgot what was in them. Papers often have summaries, in the form of their abstract, or I could start from someone else’s summary, or ask a language model. That would all be much faster than writing my own summaries.

§Trust

The thing is, you can’t always trust what you read in papers. Authors sometimes overstate, or understate, the relevance of their findings to my work. Sometimes they contain claims that aren’t justified at all, because they are undermined by some subtle (or not so subtle) errors in reasoning. So another theory is that my summaries are valuable artefacts because I’ve filtered them to emphasise the parts that are relevant and de-emphasise the parts that are irrelevant or untrue.

This is something I used to believe. Recently, I realised it’s a bit naive. I’m also an author. I also sometimes overstate or understate the relevance of findings to my work. Over the years, I have been known to sometimes write down claims that aren’t justified at all, being undermined by some subtle (or not so subtle) errors in reasoning.

It’s not like I can afford to place zero trust in things others have written and absolute trust in things I have written. At the same time, I can’t afford to verify everything including that I just verified everything, that’s an infinite loop.

§Conversation

Interesting! We have encountered a conceptual separation between the person writing the summaries (myself in the past) from the person reading the summaries (myself in the present). By reading the notes, it’s kind of like present-me, the summary reader, is having a conversation with past-me, the summary writer.

In “Communicating with slip boxes”, the prolific German sociologist Niklas Luhmann explains his experience having a sustained “conversation” with a collection of boxes of thousands of manually-hyperlinked paper slips amassed over decades. He reports that his Zettelkasten system was a fruitful conversation partner because over time it surfaced surprising connections between ideas encountered at different times or in seemingly different contexts.

That’s all good in theory. But this isn’t what has happened for me, at least historically. As a conversation, things have been almost entirely one-sided, with the person writing the summaries doing all of the talking and present-me not spending much time listening. Moreover, I haven’t invested the time to laboriously interlink my paper summaries or draw connections between disparate ideas.

§Automation

Some people take the time to build a physical Zettelkasten. Digital slip boxes are more popular. Personal wiki software can easily automate querying and indexing. This substantially lowers the friction of interacting with the system, and enabling smoother conversation.

These days, one can take this much further. In some sense, language models are the ultimate form of the corpus loquens. Talking with a language model is a convoluted means of querying an extremely large body of notes. And, at times, they can make surprising connections between disparate ideas.

But you’re not talking with a past version of yourself when you talk with a language model. It’s more comparable to talking with another person. It removes the need for me to take notes on what I read—the model can summarise the papers for me as I read them. Actually, it can summarise them whether or not I read them.

§Creation

There may come a day when I can’t contribute anything to paper summaries over what you can get from a language model. There may come a day when research itself is automated and I have nothing left to contribute at all. But it is not this day.

Today, I think I could still have a contribution to make. It’s a contribution that no current language model, or even no other researcher, would make. If that’s true, I’ll only be able to make that contribution by leaning into being me, which includes applying my unique perspective on ideas I encounter.

Under this view, what matters is that the notes come through me. As I read papers, certain ideas resonate with what I have read and thought before. I write those down as a means of letting connections surface. I ignore other ideas that evoke nothing from me in particular. Still other ideas sound wrong, whether immediately or in a subtle way I can’t put my finger on. I pause to think out loud about why and to try to make progress towards a better version of the ideas myself.

Thus, my notes aren’t just a summary of the contents of the paper. They are more like the outcome of a collision between myself and the paper. Taking notes is how I systematically dwell on the ideas long enough to elicit a reaction with whatever is happening in my mind.

Ah, that must be it.

§Conclusion

This theory of the purpose of note-taking seems pretty compelling to me. I’m happy to try it again now that I’m getting back into reading.

In fact, I should probably lean into it. If I’m summarising as a means of synthesising the key ideas in my own words rather than as an accurate reference, I don’t need to worry so much about being faithful to the source, or about being comprehensive.

I should also consider more effectively achieving the goals of the reference and conversation theories. I can keep a log of the papers I read without committing to write detailed notes on every one. I can schedule deliberate reviews over the notes I write, making for a more sustained, two-way conversation, and surfacing more connections over time.

As for changes, I am considering going digital, but I’m torn. There are obvious advantages, especially in the language model era. And, I already handle my title-reading habits and literature collection digitally, so it’s within reach. But there is still something romantic about the paper-and-pinboard approach, and it gives both a physical sense of progress and a natural point for reviewing notes (when clearing the board). Maybe I’ll try adapting both to my newly-discovered goals, and see what works.