The technical part of moving off Azure Data Factory is not the hard part. Copy activities become Auto Loader. Data flows become PySpark. Triggers become jobs or workflows. None of that is where these projects die.
They die on the third pipeline nobody can explain.
Undocumented logic in old pipelines
An estate that has been running for four or five years accumulates logic in places nobody documented: a stored procedure called from a lookup activity, a derived column with a nested iif that encodes a business rule from a meeting in 2021, a copy that quietly filters out one region because of a data quality issue that was fixed two years ago.
The person who wrote it has left. The report it feeds is used by someone in finance who will notice immediately if the number moves, and who was not asked about the migration.
So the migration team does the sensible thing: they reproduce the pipeline exactly, including the parts they do not understand. Now you have the same undocumented logic, in a new language, and you have paid to move it.
Inventory before architecture
Before designing anything, I build a table with one row per pipeline:
| Column | Why |
|---|---|
| Pipeline name | the key for the rest of the row |
| Trigger and schedule | tells you what runs, and how often |
| Sources and sinks | the dependency graph |
| Last successful run | anything stale is a candidate for deletion |
| Downstream consumer | the hard one |
| Business owner | a named person, not a team |
| Undocumented logic | flag it now, do not discover it at cutover |
The last three columns are where the value is. The first four you can generate from the ADF REST API in an afternoon.
Every estate I have inventoried has had pipelines that could simply be switched off. Not migrated. Deleted. That work is free and it shrinks the project.
Sequence so every step ships
Never a big bang. The sequence I use:
- Landing first. Point Auto Loader at the same sources ADF is reading. Both run. Nothing downstream changes yet.
- Silver next, in parallel. The new Delta tables are built alongside the old ones. Still nothing downstream changes.
- Reconcile. Row counts, sums on the numeric columns that matter, and a spot check on the specific reports the business owner named. This is the step that gets cut under pressure and it is the step that makes the cutover safe.
- Move consumers one at a time, starting with the one with the most tolerant owner.
- Switch off the ADF pipeline only after its last consumer has moved and a full cycle has passed.
Each step is independently shippable and independently revertible. These projects do get paused. If one stops halfway, you are left in a working state, not a broken one.
Unity Catalog is a naming decision, not a technical one
The catalog and schema layout you choose will outlive the migration, and changing it later means rewriting every grant and every reference. Decide it deliberately: catalog per environment or per domain, schema per source system or per business area, and a grant model written down before anyone runs a GRANT.
This is worth an afternoon of argument at the start. It is worth a quarter of pain later.
The hard part
Knowing both sides. Reading a data flow with six nested derived columns and recognising the business rule inside it, then knowing what that becomes in PySpark without a semantic change. That intersection of old Microsoft estate and modern lakehouse is narrower than either skill alone, and it is the skill these projects run short of.