I started my data career elbow-deep in SSIS packages, generating them at scale with BIML, and I have a great deal of affection for the old machinery. I also know exactly how it ends when an organisation decides to replace it all at once. The "great migration" gets funded with enthusiasm, a parallel rewrite begins, the new platform is always three months from ready, the old pipelines keep needing changes because the business doesn't pause for your architecture project — and eighteen months later you're running two estates, paying for both, and praying nobody asks when it'll be done. The big-bang ETL migration is one of the most reliable ways to spend a lot of money and finish nothing. So don't do it. Retire legacy ETL the way you'd remove a load-bearing wall: one supported seam at a time, with the building standing the whole while.
The pattern has a name borrowed from software — the strangler fig, after the vine that grows around a tree, slowly takes over its structure, and leaves the original to wither once it's no longer needed. Applied to ETL, it means the new platform grows around the old one, taking over one data flow at a time, until the legacy estate is doing nothing and can be switched off without ceremony. Here's the order of operations that actually works.
1. Inventory before you touch anything
You cannot retire what you haven't mapped. Before a single package moves, build an honest inventory of the legacy estate: every job, what it feeds, what feeds it, how often it runs, who depends on the output, and — crucially — which ones are actually still used. Almost every estate I've inventoried contained pipelines feeding reports nobody had opened in two years. The first win of any migration is discovering how much of it you can simply delete. Retire the dead wood before you migrate a line of the living.
2. Rank by value and risk, not by convenience
With the map in hand, resist the urge to start with whatever's easiest. Rank the pipelines on two axes: business value and migration risk. The ideal first candidates are high-value, low-risk — something that matters enough to prove the new platform's worth, but simple enough that you'll actually succeed. Save the terrifying, tangled, business-critical monster for when your team has migrated a dozen simpler flows and knows exactly what it's doing. You want your early moves to build confidence and credibility, not to gamble the whole programme on the hardest thing first.
3. Run the two estates side by side, on purpose
This is the heart of the strangler pattern and the part people flinch at: for a while, you deliberately run old and new in parallel. The new pipeline is built, and it runs alongside the legacy one feeding the same target — so you can compare outputs and prove equivalence before you cut over. Yes, that's temporary duplication. It is also what lets you migrate without betting the business on a big-bang switch. The duplication is the price of safety, and it's far cheaper than the cost of getting a critical cutover wrong.
4. Validate by reconciliation, then cut the seam
Before you retire a legacy pipeline, prove the new one produces the same result — reconcile row counts, spot-check totals, run them in parallel for a defined period until you trust the new output. Only then do you cut the seam: point the consumers at the new pipeline and turn the old one off (not delete — off, so you can revert). Once it's been dark and unmissed for a sensible interval, you delete it for real. Each retired pipeline is a small, complete, reversible win — the opposite of the all-or-nothing cutover.
5. Modernise the shape while you're in there — but only a little
A migration is a tempting moment to redesign everything, and that temptation has killed as many projects as the big bang itself. The discipline is to modernise the shape where it's cheap and obviously right — moving from row-by-row logic to set-based, landing raw data in a lakehouse bronze layer, adopting a medallion structure as you go — without turning every pipeline migration into a ground-up redesign. Lift-and-improve, not lift-and-perfect. You can always refine a pipeline once it's on the new platform and no longer on fire. Get it across first.
The order, in one breath
Inventory and delete the dead. Rank by value and risk. Build the new beside the old. Reconcile, cut the seam, turn the legacy off, then delete it. Repeat until the old estate is doing nothing, and switch it off. At no point is the business running on a half-finished platform, and at every point you have a working system and a clear next step. This is the same philosophy I laid out for the specific case of retiring SSIS onto Fabric — the platform details change, the order of operations doesn't.
Is there ever a case for the big bang?
Occasionally — a small estate, a hard vendor deadline, a legacy platform being switched off for you whether you're ready or not. When the old thing is going away on a fixed date outside your control, incremental has less to offer and you may simply have to run the risky cutover. But those are the exceptions, and even then I'd carve off whatever I could migrate early to shrink the scary weekend. For the overwhelming majority of ETL modernisations, the strangler fig wins for one unglamorous reason: it's the approach where you're allowed to stop. Budgets get cut, priorities shift, the person who championed the project leaves — and with the incremental approach, a half-done migration is still a pile of completed, valuable wins and a smaller legacy estate. A half-done big bang is just two systems and an apology. Build so that stopping early still leaves you better off, and you'll find you rarely have to.