Every data team knows it should document its work, and almost every data team's documentation is a graveyard — a wiki full of pages written once in a burst of good intentions, read by no one, and wrong within a month because nothing that changed got reflected back. The usual conclusion drawn from this is that documentation is a nice idea that never survives contact with reality, and people quietly give up on it. I think that conclusion is wrong. The documentation fails not because documenting is futile but because most of it is written for the wrong reader, at the wrong moment, and kept in the wrong place. Fix those three things and you can write documentation people genuinely use — I've watched it happen. It just requires treating documentation as an act of communication rather than an act of compliance, and those are very different jobs.

Write for the reader, not for yourself

The first and biggest mistake is writing documentation to prove you documented, rather than to help a specific person do a specific thing. Documentation written as a box-tick is shaped by the author's sense of completeness — every field described, every detail recorded — and it's exhausting to write and useless to read, a wall of exhaustive facts with no sense of what the reader actually needs. Documentation written for a reader starts from a question: who's going to read this, and what are they trying to do? The new engineer trying to understand how a pipeline works needs something completely different from the analyst trying to figure out what a column means, and both differ from the future-you debugging this at 3 a.m. next year. Pick the reader, imagine their moment of need, and write to that. Aim at everyone and you help no one.

Answer the questions people actually have

The most useful documentation answers the questions people really ask, which are rarely the ones exhaustive docs address. For most data work, those questions are:

  • What is this, and why does it exist? The purpose — the thing that's obvious to you now and completely opaque to anyone arriving later, including yourself.
  • What does this actually mean? For data especially: what a field represents, what a metric includes and excludes, the difference between two similar-looking things. The definitions that stop people guessing wrong.
  • How do I use it / run it / fix it? The practical "how do I do the thing," including the gotchas — the traps you learned the hard way and would spare the next person.
  • Who do I ask when this doesn't help? A named human as the fallback. Documentation doesn't have to be complete if it reliably points to the person who can fill the gap.

Notice how much shorter that is than "document everything." Good documentation is often less than the exhaustive version, because it's focused on real needs rather than completeness for its own sake.

Put it where the work is

Documentation kept somewhere people have to remember to visit will not be read, full stop. The closer it lives to the thing it describes — a description attached to the dataset itself, a comment in the code, a note in the tool people already use — the more it gets read, because it's there at the moment of the question rather than filed in a wiki nobody opens. Proximity beats completeness. A short, correct note where the reader already is beats a comprehensive document three clicks away that they'll never find.

Keep it alive, or accept it will lie

The other reason documentation fails is decay: it's written once and never updated, so it drifts from the truth until it's actively misleading — worse than nothing, because people trust it and get burned. The only real answer is to make updating documentation part of the work of changing the thing, not a separate task nobody prioritises. If changing a pipeline includes updating its description, the docs stay honest. If it doesn't, they rot, and no amount of good intentions saves them. Better a small amount of documentation you actually maintain than a comprehensive set you let go stale.

A small before-and-after

To make the difference concrete: the box-tick version of documenting a table reads "fct_sales: fact table. Columns: sale_id (int), amount (decimal), cust_id (int), status (varchar)…" — technically accurate, completely useless, because it tells the reader nothing they couldn't see themselves. The written-for-a-reader version reads: "Sales fact table — one row per completed sale. amount is net of returns (that's the trap; use gross_amount if you want pre-returns). status of 'P' means pending and should usually be excluded from revenue. Owner: the finance-data team; ask Marijke if something looks off." Same table. One is a schema dump; the other is the hard-won knowledge that stops the next person getting the number wrong. Only one of them was worth writing.

The through-line

I once wrote about documenting a migration so humans could actually follow it, and the lesson has only deepened for me since: documentation is communication, and it obeys the same rule as all communication — it's not about what you said, it's about what the reader received. A document that's technically complete but that no one reads, or understands, or can find, communicated nothing, however conscientiously it was written. The whole craft is imagining the person on the other end — their question, their moment, their level of context — and writing precisely what helps them, where they'll find it, kept true enough to trust. Do that and documentation stops being a chore you resent and a graveyard you ignore, and becomes what it was always meant to be: the way your team's hard-won knowledge outlives the person who happened to learn it first.