I run a data platform team on Scrum, and I'll admit something the agile purists won't like: a fair amount of the framework, applied to data work unmodified, grates. Scrum was designed for building software — for teams shipping features in predictable increments — and data work is a different animal in ways that matter. A pipeline migration doesn't slice into neat two-week features. A data investigation genuinely can't be estimated, because you don't know what you'll find until you look. And "potentially shippable increment" is a slippery idea when the thing you're building is a warehouse that's only useful once several unglamorous pieces all exist at once. So dropping the full ceremony set on a data team and insisting they perform it produces a lot of quiet resentment and cargo-cult ritual. But — and this matters — the answer isn't to abandon agile. It's to keep the parts that genuinely help data work and adapt or drop the parts that don't. Here's my honest sorting.

What genuinely earns its keep

  • The daily standup — if it stays short. A quick, daily "what did I do, what am I doing, what's blocking me" is genuinely valuable for a data team, because data work is riddled with hidden blockers — waiting on access, waiting on a source system, waiting on someone else's answer. Surfacing those daily gets them unstuck faster. The value is the blocker-surfacing, not the ritual; keep it to the promised few minutes or it curdles into a status meeting nobody needs.
  • The retrospective. This is the ceremony I'd defend hardest, and the one teams drop first. Regularly stopping to ask "what's working, what isn't, what do we change" is how a team actually improves, and it matters more in data work because so much of the friction is process — the handoffs, the environments, the recurring "why is this always painful." Protect the retro.
  • A visible backlog and board. Making the work visible — what's queued, in progress, done — helps a data team just as much as a software one, especially because data requests otherwise arrive from everywhere and vanish into someone's inbox. A shared board turns chaos into something you can prioritise.

What needs bending

  • Estimation. Story points on a data investigation are mostly fiction, because you can't size what you haven't discovered. For exploratory work I lean on timeboxing instead — "spend two days finding out, then we'll decide" — rather than pretending we can estimate the unknowable. For well-understood build work, ordinary estimation is fine. Match the technique to how knowable the task actually is.
  • The sprint boundary. Two-week sprints can work, but data work often has a rhythm that doesn't respect them — a migration that takes as long as it takes, an incident that ignores your sprint plan. Hold the sprint as a useful planning cadence, not a sacred wall the work must be forced through.
  • "Potentially shippable increment." Reframe it for data as "a demonstrable step of progress." Sometimes that's a working report; sometimes it's a validated pipeline stage nobody outside the team can see yet but which genuinely moves things forward. Insisting every increment be user-visible pushes teams to build the visible thing before the necessary thing.

What to quietly let go

Some ceremony is pure overhead for a small, stable data team, and pretending otherwise just breeds cynicism. Elaborate estimation rituals, ceremony performed because the framework says so rather than because it helps, process weight that costs more attention than it returns — let it go. The test is simple and I apply it constantly: does this practice help us deliver better, or are we doing it because a certification told us to? If it's the latter, it's theatre, and theatre erodes a team's trust in the whole idea of process.

A worked example: the un-estimable investigation

Here's the situation that breaks naive Scrum on a data team, and how I handle it. A stakeholder asks why a key metric dropped last quarter. In planning, someone wants a story-point estimate. But nobody can estimate this — the answer might be a five-minute find (a broken feed) or a five-day archaeology dig through source systems, and you cannot know which until you start. So I don't estimate it. I timebox it: "we'll spend one day investigating, then report back what we found and what it'll take to fix." That respects the genuine uncertainty instead of pretending it away, gives the stakeholder a firm checkpoint, and stops the team from committing to a number that was always fiction. The lesson generalises — when work is genuinely unknowable, timebox the discovery and estimate the fix only once you actually know what it is.

The deeper point

Here's what I've learned leading a data team through this, and it connects to something I wrote about leading a data platform team in its early months: agile is a set of tools for helping a team work well, not a religion to be observed. The moment you're serving the framework instead of the framework serving you, you've got it backwards. The best agile data teams I've seen are unapologetically pragmatic — they've kept the standups and retros that genuinely help, bent estimation and sprints to fit the reality of data work, and dropped the ceremony that was only ever there to satisfy a rulebook.

The counter-view is fair and worth holding: this pragmatism can slide into using "data work is different" as an excuse to abandon all discipline, and a team with no cadence, no visible backlog, and no reflection is not agile — it's just disorganised with better vocabulary. So keep the discipline; it's the ceremony-for-its-own-sake you're pruning, not the rigour. The goal was never to do Scrum correctly. It was to build good data products with a team that isn't quietly miserable — and the framework is only worth as much as it moves you toward that.