In February 2025 I wrote about building an event-driven ETL system in Microsoft Fabric. The metaphor was air traffic control: notebooks as flights, Azure Service Bus as the control tower, the Bronze/Silver/Gold medallion layers as the runway sequence. The whole system existed because Fabric has core-based execution limits that throttle how many Spark jobs run simultaneously on a given capacity SKU.
The post was about working around a constraint. You could not just fire all your notebooks at once. You needed something to manage the queue.
More than a year on, it is worth being honest about what held up and what has changed.
The original architecture, briefly
Three components:
Azure Service Bus acted as the message queue. When a source system finished loading raw data, it dropped a message onto the bus. Each message represented one flight waiting for clearance.
A capacity monitor notebook ran on a short schedule. It checked how many notebooks were currently executing and compared that count against the available core capacity. If capacity was available, it pulled the next message from the queue and triggered the appropriate notebook via the Fabric REST API.
The processing notebooks were standard Bronze, Silver, and Gold Spark notebooks. They ran as normal Fabric notebooks with no awareness of the orchestration layer above them. On completion, they acknowledged the Service Bus message.
The deliberate design choice was to keep the notebooks clean and put the complexity in the orchestration layer. A notebook should not need to know whether it is being called by a scheduled job, a pipeline, or a service bus monitor. That separation held up well.
What has changed in Fabric
Two relevant changes since February 2025:
Native job queueing
Fabric now queues Spark jobs automatically when capacity is exhausted rather than rejecting them outright. Jobs queue in FIFO order and wait up to 24 hours before expiring. The platform starts them automatically as capacity becomes available.
There is a hard constraint that limits how far this goes: the queue depth is bounded by the SKU’s CU allocation. It is not unlimited. A sudden burst of 100+ notebooks would exceed both the concurrent execution limit and the queue depth, and the excess jobs would be rejected rather than queued — the same failure mode as before native queueing existed.
So native FIFO queueing helps if your workload arrives gradually. It does not change the original problem if your trigger pattern involves large simultaneous batches. The Service Bus buffer sits outside Fabric and has no queue depth constraint. That distinction is why the architecture is still relevant.
Job-level bursting controls
Fabric capacities support bursting at up to 3x the nominal CU allocation. You can now disable bursting for specific Spark jobs, giving finer-grained control over which jobs are allowed to consume burst headroom. Useful for ring-fencing critical workloads. This is an additive improvement to the platform regardless of which orchestration approach you use.
What held up
The decoupled architecture held up. Keeping orchestration logic out of the processing notebooks made them easier to test, modify, and redeploy independently. A notebook that does not know how it was triggered is easier to reason about than one that contains scheduling and queueing logic alongside its data transformation. Nothing about that changed.
Azure Service Bus held up as a reliable messaging backbone. At-least-once delivery, dead-letter queues, message peek-lock, and configurable time-to-live are production-grade features. There were no reliability issues with the messaging layer over the period.
The Bronze, Silver, Gold medallion structure held up. That is sound data architecture independent of the orchestration tool above it.
What I would do differently today
Not much, given the workload pattern. The original system was designed for burst scenarios, and the burst scenario has not changed. The native queue depth limit tied to CU allocation means there is still no Fabric-native replacement for an external buffer that absorbs an unbounded number of incoming messages and feeds them into Fabric at a controlled rate.
The one thing I would reconsider is the capacity monitor notebook’s polling loop. It runs on a short schedule and checks active job counts before pulling the next message. That works, but it adds latency and a scheduling dependency. Whether the Fabric REST API now exposes enough observability to build a leaner version of that loop is worth investigating — but that is an implementation detail, not a reason to replace the architecture.
The honest production reality
The system is still running. A working production system with understood failure modes and people who know how to debug it has a high replacement threshold. The architecture from February 2025 has not needed replacing because the problem it solved has not stopped existing.
The question I get asked: is there a more native way to do this now? The answer is no, not for the burst-buffering problem specifically. Fabric pipelines handle sequential orchestration well. Native FIFO queueing handles gradual workloads within its depth limit. Neither absorbs an unbounded burst and controls submission into a capacity-constrained runtime. Service Bus still does that job, and it still does it well.
The air traffic controller is still in the tower. The runway is a little wider than it was, but the traffic has grown too.
