There is no shortage of GDPR advice, and almost none of it is written for the person who actually has to build the thing. It's written for lawyers, for executives, for compliance officers — all necessary, all pitched at the level of principle and policy. But the data engineer sitting in front of a pipeline needs something different: the specific, concrete places where GDPR stops being an abstract principle and starts biting into your architecture, forcing real design decisions. Having spent time on a heavily-governed EU project, I want to write that version — the bits that actually bite the person who builds.
Let me go through the ones that reliably cause real engineering pain, because forewarned is a much better state than finding these out during an audit.
The right to erasure versus everything you built to never lose data
Here's the big one, the collision that catches everyone. GDPR gives individuals the right to have their personal data erased — the "right to be forgotten." Simple principle. Now consider how you, as a data engineer, have spent your entire career building systems: to never lose data. Immutable raw layers. Append-only logs. Backups on backups. Historical archives you can reprocess from. Every instinct and best practice you have is oriented around durability — data, once captured, is precious and permanent.
The right to erasure drives a truck straight through that. Suddenly you must be able to reach into all those places you carefully designed to hold data forever — the immutable bronze layer, the backups, the archives, the downstream copies that sprawled out to other systems — and surgically remove one person's data from all of them, on request, provably. The two design goals are in direct tension: build for permanence, then be required to selectively forget. This is the bit that bites hardest, and it has to be designed for up front, because retrofitting "find and delete one person everywhere" onto a system built to never delete is genuinely painful. If you take one thing from this post: design for erasability before you have petabytes of un-erasable history.
Data minimisation versus "capture everything, it might be useful"
The second collision is with the data engineer's other deep instinct: hoard. Storage is cheap, data might be useful later, so the reflex is to capture everything and keep it forever. GDPR's data minimisation principle says the opposite: collect only the personal data you actually need for a specific purpose, and keep it only as long as you need it.
That bites directly into pipeline design. Every "let's also capture this field, might come in handy" now needs a real justification — a legitimate purpose — or it's a liability rather than an asset. And "keep it only as long as needed" means building retention and deletion into your pipelines as a first-class feature: data that ages out and is removed automatically, rather than accumulating forever by default. The engineer's instinct to keep everything becomes, under GDPR, exactly the instinct to discipline, and the discipline has to be built into the plumbing, not bolted on by policy.
Purpose limitation versus the reusable data platform
Here's a subtle one that catches sophisticated teams. You build a lovely central data platform, and the whole point of it is reuse — land data once, use it for many things. GDPR's purpose limitation says personal data collected for one purpose can't freely be used for a different one without proper basis. That tension is real: the platform's value proposition (reuse this data everywhere) rubs against the regulation's constraint (this data was collected for X, you can't just also use it for Y).
Practically, it means you can't treat all personal data in your platform as a free-for-all buffet for any use case. You have to track what data was collected for what purpose, and gate uses against that — which is metadata and lineage work that the "just make all the data available" platform vision quietly skips. The reusable platform is still right; it just can't be indiscriminately reusable where personal data is concerned.
Pseudonymisation: the technique that actually helps
Not everything is a collision — GDPR also hands you a genuinely useful technique, and it's worth knowing because it reduces your pain. Pseudonymisation — replacing identifying fields with tokens, holding the mapping separately and securely — lets you do a lot of your analytical work on data that no longer directly identifies people. It's not anonymisation (you can still re-link with the key, so it's still personal data under GDPR), but it meaningfully reduces risk and, done well, shrinks how much of your estate is handling raw identities at all.
For an engineer, this is the constructive move: separate the who from the what as early in the pipeline as you sensibly can, so that most of your processing runs on pseudonymised data and the sensitive linkage is concentrated in one well-guarded place rather than smeared across everything. It's more work up front and it makes almost every other GDPR obligation easier to meet.
GDPR doesn't bite the data engineer at the level of principle. It bites at the level of design — where "right to be forgotten" meets your immutable layers, where "minimise" meets your hoarding instinct, where "purpose limitation" meets your reusable platform. Handle those in the architecture, early, and the principles mostly take care of themselves.
The backup problem, and the trick that solves it
The erasure-versus-immutability collision has one especially nasty corner that deserves its own answer: backups. It's one thing to delete a person's data from your live systems. But you also have backups — by design, copies of everything, held precisely so you can restore after a disaster. How do you honour "erase this person" when their data also sits in a snapshot from three months ago that you can't, and shouldn't, casually rewrite?
Deleting selectively from backups is often genuinely impractical — they're frequently immutable snapshots by design, and surgically editing them undermines the very integrity that makes them worth keeping. So the pragmatic technique that's emerged, and the one worth knowing, is crypto-shredding: encrypt personal data with per-subject (or per-scope) keys, and when you need to erase someone, delete the key rather than hunting down every copy of the data.
The logic is neat. Data encrypted with a key you've destroyed is, for all practical purposes, gone — unrecoverable ciphertext, indistinguishable from noise, even if the encrypted bytes still physically sit in some backup. You've made the data effectively erased everywhere at once by removing the one thing that made it readable, without reaching into every immutable copy. It reframes "delete this person from everywhere" — a nightmare — into "delete this person's key" — a tractable operation.
It's not a silver bullet: you have to design for it from the start (encrypting at the right granularity, managing the keys properly), and it's more architecture up front. But it's the cleanest answer I know to the specific horror of "right to erasure meets immutable backups," and it's exactly the kind of thing that has to be designed in early rather than bolted on. If your architecture holds personal data in immutable stores — and most do — this is the technique to reach for before the erasure requests start arriving, not after.
The honest engineering takeaway
The theme running through all of these is that GDPR is not, for the engineer, a policy you comply with by reading a document. It's a set of architectural requirements that conflict with several deep data-engineering instincts, and the only real way to meet them is to design for them from the start: build erasability into systems you'd instinctively make permanent, build minimisation and retention into pipelines you'd instinctively make hoard, build purpose-tracking into platforms you'd instinctively make indiscriminately open, and pseudonymise early to shrink the whole problem.
Do that and GDPR is a manageable, even clarifying, set of constraints that make you build more thoughtfully. Ignore it until it's a legal problem and you'll be trying to retrofit "selectively forget one person across a system designed to never forget anyone" under time pressure, which is precisely the nightmare the whole regulation should have prompted you to design away. The bits that bite are all bits you can defang in advance — but only in advance. After the fact, they just bite.