Every data pipeline needs something to conduct it — to press "go" on the right steps, in the right order, at the right time, to wait for one thing before starting the next, to retry the flaky bit, and to tell someone when it all falls over at 3 a.m. That job is orchestration, and it's the unglamorous nervous system of any data platform. In the Azure world you have at least three reasonable ways to do it — Azure Data Factory, Synapse Pipelines, and hand-rolled scheduling around notebooks or scripts — and the choice matters more than people expect, because picking the wrong one means either fighting a heavyweight tool for a simple job or painfully outgrowing a lightweight one. Here's how I actually decide.

The three options, honestly

Azure Data Factory (ADF) is the mature, general-purpose orchestrator. It's a visual, pipeline-based tool for moving and transforming data at scale, with a large library of connectors, built-in scheduling and triggers, monitoring, retry logic, and the ability to coordinate complex multi-step workflows with dependencies. It's the default for serious, production data movement across a varied estate, and it earns that default.

Synapse Pipelines are, essentially, the same orchestration engine as ADF, built into the Synapse Analytics workspace. If your world already lives in Synapse — serverless and dedicated SQL, Spark, the lot — then Synapse Pipelines let you orchestrate it all from one integrated place without wiring up a separate service. The capability closely mirrors ADF; the difference is mostly whether you want orchestration living inside your Synapse workspace or as a standalone service spanning more than just Synapse.

A notebook on a schedule — a Databricks notebook or a Python script triggered by a simple scheduler — is the lightweight option. For a self-contained transformation that's mostly code, sometimes the honest answer is that you don't need a full orchestration platform; you need a script to run on a timer and shout if it fails. Reaching for ADF to run one nightly notebook is using a cargo crane to lift a suitcase.

How I choose

The decision comes down to a few honest questions:

  • How complex is the workflow? A single job on a schedule? A notebook trigger is plenty. Many steps with dependencies, branching, retries, and coordination across different services? That's what ADF and Synapse Pipelines are built for, and doing it by hand in scripts is reinventing an orchestrator badly.
  • Where does the rest of your work live? If you're all-in on Synapse, Synapse Pipelines keep everything in one workspace. If you're orchestrating across a broad estate — many sources, services, and destinations beyond Synapse — ADF's standalone breadth fits better. If you're Databricks-centric, notebook-native scheduling (or Databricks' own workflow tooling) may be the most natural home.
  • How much connectivity do you need? ADF's deep library of connectors is a genuine differentiator when you're pulling from many varied systems. If you're just running code against data already in your lake, you may not need any of it.
  • Who's maintaining it, and how visible must it be? The visual, monitored pipeline tools give you a clear operational picture — what ran, what failed, how long it took — that hand-rolled scripts only provide if you build it yourself. If several people need to see and reason about the orchestration, the platform tools pay for their overhead in legibility.

The mistake in both directions

The failure I see most is going too heavy: teams stand up elaborate ADF pipelines to run a single scheduled script, then spend their time maintaining orchestration machinery vastly bigger than the job requires. But the opposite mistake is just as real and more dangerous long-term: hand-rolling scheduling and dependency logic in scripts for a workflow that has quietly grown complex, until you've built a fragile, undocumented, unmonitored orchestrator held together with cron jobs and hope — and reinvented, badly, the thing ADF gives you for free. Both mistakes come from choosing the tool by habit rather than by fit.

Build it to be reused, whichever you pick

One piece of advice that applies regardless of which tool you land on: parameterise from the start. The moment you have a second pipeline that looks almost like the first — same shape, different table or source — resist copy-paste. A parameterised, reusable pipeline that takes the varying bits as inputs will save you enormous grief the day a change has to be applied everywhere, because you make it once instead of in twenty near-identical copies. This is the same discipline as writing a function instead of pasting a block of code, and orchestration tools reward it: ADF and Synapse Pipelines both support parameters and reusable components precisely so you don't hand-build a hundred slightly-different pipelines you'll later have to maintain in parallel. The tool you choose matters less than whether you built it to change cheaply.

The rule of thumb

Match the orchestrator to the real complexity and location of the work. Notebook-on-a-schedule for genuinely simple, self-contained, code-heavy jobs. Synapse Pipelines when your world is Synapse and you want orchestration integrated into it. Azure Data Factory for serious, complex, multi-source data movement across a broad estate that needs real connectors, monitoring, and coordination. And revisit the choice as things grow — the lightweight option that was right for one nightly job is often the wrong one a year later when that job has become fifteen interdependent ones. Orchestration is the plumbing nobody admires until it fails, so choose it deliberately, keep it as simple as the work honestly allows, and give it enough structure that when it does break at 3 a.m., it tells you exactly where — because it will break eventually, and the only question is whether it'll be able to explain itself.