Author: Jens Vestergaard

Azure Databricks at FabCon 2026: What Got Announced and What It Actually Means

Posted by Jens Vestergaard on in Programming 0 Comment

FabCon 2026 in Atlanta last week was bigger in many ways. For the first time, the Microsoft Fabric Community Conference ran alongside SQLCon, which meant two large communities shared the same convention center, the same coffee queues and the general atmosphere of too-many-sessions-not-enough-sleep that defines any conference worth attending.

Databricks showed up with several announcements. Some of them are incremental. A few are worth more than a passing glance, depending on where you sit on the data and BI stack.

One thing that still catches people off guard: Azure Databricks has been a first-party Azure service since 2017. Not a partner product, not a third-party integration. A first-party Azure service, alongside Power BI, Excel, Teams, Azure OpenAI, Copilot Studio and the Power Platform. When Microsoft talks about a unified data and AI platform on Azure, Databricks is part of the architecture. The announcements this week make that more visible.

Here is what they shipped, and what I think it means in practice.

Lakeflow Connect Free Tier: 100 Million Records a Day, at No Cost

Bad pipelines are one of the constants of BI work. The data arrives late, the connectors are fragile, someone is maintaining a web of custom scripts because there was never time to do it properly, and the people building reports spend half their week cleaning up after problems they did not cause.

Databricks announced a Lakeflow Connect Free Tier, and the headline number is worth taking seriously: 100 million records per workspace per day, at no charge. That is 100 free DBUs per day, included with every workspace, before standard Lakeflow Connect pricing applies.

What it connects to out of the box:

  • Databases: SQL Server, Oracle, Teradata, PostgreSQL, MySQL, Snowflake, Redshift, Synapse, BigQuery
  • SaaS applications: Dynamics 365, Salesforce, ServiceNow, Workday, Google Analytics

For databases, the ingestion runs on Change Data Capture, which means it reads the transaction log incrementally rather than scanning full tables. Data lands in Delta format on Azure Data Lake Storage. Unity Catalog governance applies from the moment the first record arrives, so access control and lineage are not something to sort out later.

Databricks quotes 25x faster pipeline builds and 83% ETL cost reduction. I would take vendor benchmarks with the usual scepticism, but the direction is clear: the intent is to make data ingestion a problem you configure rather than one you maintain. For a BI team currently paying for third-party Dynamics or Salesforce connectors, or running CSV exports on a schedule, this is worth a practical test.

Lakebase Is Generally Available: A Postgres Database Inside the Lakehouse

This one sits closer to architecture and engineering than it does to daily BI work, but it changes some assumptions that are worth understanding.

Azure Databricks Lakebase is now generally available in 14 Azure regions. It is a managed, serverless Postgres service that runs inside your lakehouse, on the same storage as your Delta tables.

The problem it addresses is one data architects have been working around for years: operational data and analytical data have historically lived on separate platforms, connected by pipelines that were always someone’s responsibility and frequently nobody’s priority. Lakebase puts an operational database directly in the same governed environment as the rest of the data platform.

Key characteristics:

  • Full Postgres compatibility, with support for extensions including pgvector and PostGIS
  • Compute and storage separated, with scale-to-zero and sub-second startup
  • Branching and instant restore for development and testing workflows
  • High availability with automatic failover across availability zones

The use cases Databricks highlights: transactional analytics on operational data, AI agent state management, and customer personalization and feature serving. For data engineers building pipelines that feed AI applications, Lakebase removes the need to run a separate operational database outside the Databricks platform just to give an agent somewhere to write state.

It is available to test today in 14 regions. If you have been looking for a Postgres layer that sits inside the lakehouse without architectural compromise, now is a reasonable time to look at the documentation.

The Excel Add-in Is in Public Preview: Governed Lakehouse Data in the Tool Most People Actually Use

This is the announcement that will get the most immediate attention from analysts and business users, and probably the one that causes the most internal conversations about data governance.

The Azure Databricks Excel Add-in is in public preview. It connects Excel directly to Unity Catalog tables and Metric Views. From inside Excel, you can browse the catalog, build pivot tables from governed semantic definitions, and filter and analyze data without writing SQL. It works on Excel for Windows, macOS and the web.

The problem it addresses is one every BI developer and governance specialist knows well: business users need data in Excel. So someone exports a CSV. Or the business user pulls their own export. Within 24 hours there are four versions of the file in four different places, none of them current, all of them cited in separate meetings. The analyst who originally produced them has no idea which version is being used.

The add-in replaces that pattern with a live connection to the same tables that power your Power BI reports and your analytics models. The data is current. The access rules in Unity Catalog apply here too, so a user who cannot query a table in Databricks cannot query it through the add-in either.

For analysts who work primarily in Excel, this is a genuine change in how a typical Tuesday works. For governance teams, it removes a whole class of ungoverned data copy that currently exists because there was no better option.

Genie Gets More Capable: Agent Mode, Genie Code and Databricks One

Genie is Databricks’ conversational analytics experience: you ask a data question in plain language and get back a chart, a table or a narrative answer. Databricks reported this week that 98% of Databricks SQL warehouse customers are using AI/BI, with monthly active Genie users up more than 300% year-over-year. The numbers are moving fast enough to suggest this has passed the experimental phase.

Three updates this week.

Genie Agent Mode

Standard Genie answers one question at a time. Genie Agent Mode takes a more complex business question, builds a research plan, runs multiple queries, tests intermediate results, refines its approach and then delivers a complete answer with supporting tables, charts and narrative context.

The difference becomes concrete quickly. Standard Genie handles: “What were total sales in Q3?” Genie Agent Mode handles: “Revenue in the Southeast dropped in Q3. Why did that happen, and what does the pattern suggest for Q4?” That is not a single query. It is an investigation, and Agent Mode runs it without someone having to direct every step.

For analytics managers sitting on a queue of complex ad hoc requests that only a senior analyst can currently answer, this is the update worth spending time with.

Genie Code

Genie Code is aimed at data practitioners, not end users. It is an agentic development assistant that runs inside Databricks notebooks, SQL editors and Lakeflow pipelines.

The distinction from a general-purpose AI coding assistant is that Genie Code understands your data context through Unity Catalog. It knows your tables, your lineage, your governance policies and your business semantics. With that, it can build pipelines and dashboards from natural language prompts, debug Lakeflow failures, generate queries grounded in your actual schema, and handle routine operational monitoring.

For senior BI developers and data engineers who spend part of every week on repetitive work that requires knowing the platform well, having an assistant that actually knows prod.gold.customer_activity is a different experience from hitting tab on a general-purpose tool that has never seen your schema.

Databricks One and Databricks One Mobile

Databricks One now includes a unified multi-agent chat experience powered by Genie. Business users can ask questions across the full data estate without needing to know which Genie space to route to. When a question goes beyond what existing spaces can answer, Databricks One can bring in additional agents to investigate. AI/BI dashboards and Databricks Apps are surfaced in the same interface.

Databricks One Mobile brings this to iOS and Android: Genie, dashboards and apps from a phone. Business users can ask data questions without being at a desk.

Genie in Microsoft Teams: Data Answers Where the Decisions Actually Happen

For organizations already using Microsoft 365, this is probably the most immediately deployable announcement.

You can now connect Genie to Microsoft Teams via Copilot Studio. The setup connects a Genie space to a Teams agent through the Copilot Studio connector, which handles the API and MCP logic. Once connected, users can ask data questions directly in a Teams conversation and get answers backed by your lakehouse data.

The part that makes this credible to security teams and BI leaders: every conversation runs through OAuth, authenticated against the user’s own identity. If a user does not have SELECT access to a table in Unity Catalog, Genie will not surface that data in Teams. The access model you already manage in Unity Catalog carries through to every Teams conversation.

For data governance managers who have spent years explaining why pasting screenshots of reports into Teams messages is not the same as having a governed answer, this changes the practical alternative. The question gets answered where it was asked, with the right access controls applied, and nothing leaves the governed environment.

For business users, it means getting a trusted data answer without leaving the tool they already have open.

What I Am Taking Away From This Week

The pattern across all of these announcements is one I have been watching build for a couple of years. Operational data, analytical data and AI have historically lived on separate platforms, and the work connecting them got called integration. That work is expensive, slow, and usually the first thing cut when a project runs over budget.

What Databricks is building is a single platform where all of it sits together, governed by Unity Catalog, accessible from Excel, Teams, a notebook, a mobile app or a SQL query. Whether the individual pieces fit together as neatly in production as they do in the announcement demos is something I will be watching as they move from preview toward GA.

If you were at FabCon this week, the Databricks session was Thursday March 19th in room C302 and should be available on demand if you missed it.

The next major Databricks gathering is Data + AI Summit, June 15 to 18, 2026, in San Francisco. 25,000 attendees, 800+ sessions, and the most complete view of where the platform is heading. Worth putting on the calendar.

What caught your attention this week at FabCon? Drop a comment below. I would like to hear what people are actually planning to test.

MVP Summit 2026

Posted by Jens Vestergaard on in Programming 0 Comment

Today is day one of the Microsoft MVP Summit 2026. The event runs until the 26th, and the core of it happens on the Microsoft campus in Redmond, Washington. For the second year in a row, I’m joining from my home office.

The Summit is an invitation-only event, open to active Microsoft MVPs and Regional Directors. You sign an NDA and spend a few days getting direct access to the product teams building the tools you use and advocate for every day. Real roadmap conversations, early previews, and the chance to make your voice heard in rooms where decisions are still being made. Around 3,000 MVPs attending, from all over the world.

It is a great event and I wouldn’t want to miss out. I should say that plainly, because what follows is honest and not a complaint.

The remote experience

Attending remotely works. The virtual sessions run well, the content is real, and I come away with things worth knowing. I’m not going to pretend otherwise.

But here’s the thing: the sessions are maybe half of what makes the Summit worth attending. The other half is the people. Three thousand of the most experienced, most generous, most technically opinionated people in the Microsoft ecosystem, in the same place for three days. The conversations that happen between sessions, at dinner, in the corridors, over a coffee at the venue. That is where a lot of the real value is, and that does almost not exist in a virtual format. It’s not the organizers fault, it’s inherently the format.

Product Group Day

The specific part that really stings to miss is Product Group Day.

It is in-person only. No virtual stream, no recording, no alternative. It is where MVPs get direct, unscripted time with the engineering teams, and where the feedback that actually matters gets delivered face-to-face. It is the most unique piece of the whole event.

Still here

The time zone gap between Copenhagen and Redmond means most sessions land in the late evening and push into the early hours. That cuts both ways: the regular work day stays intact, which is good, and somewhere around the second session after midnight the tiredness kicks in, which is less good.

But I will be there, taking notes, looking for the things worth acting on.

And I’m already making a mental note about 2027.

Exploring Fabric Ontology

Posted by Jens Vestergaard on in Programming 1 Comment

Note: The Fabric Ontology is currently in preview as part of the Fabric IQ workload. Features and behaviour may change before general availability.

I have been spending a little time with the Microsoft Fabric data agent documentation lately, and one pattern keeps showing up, and it is not just in the official guidance but in community posts from people who have actually tried to deploy these things: the demo runs beautifully. The AI answers questions in plain English, leadership gets excited, the pilot gets approved. Then it hits production. Real users send real questions. The answers start drifting. Numbers that should match do not. The same question returns different results on different days. Trust evaporates faster than it was built.

And almost every time, the root cause is the same thing: the semantic foundation was not solid enough before anyone pointed an agent at it.

That is exactly the problem the Fabric Ontology is designed to address. It is the piece I think most teams will underestimate right up until the moment they need it.

Why the Data Agent Gets It Wrong

Generative AI is genuinely good at working with language and meaning. What it cannot do is fill in documentation that was never written.

Most enterprise databases were built for systems, not for consumption. Column names follow technical conventions an engineer settled on years ago. Business logic lives in a stored procedure nobody has touched since SQL Server 2014. Which customer table is the authoritative customer table? Documented nowhere. The abbreviation cust_rev_ytd_adj was obvious to the person who named it. To everyone else, including an AI agent, it is a puzzle.

When you connect an agent to that data and ask it to answer business questions, you are asking it to decode a language it was never given a dictionary for. It is not going to find meaning that was never documented. Someone has to build that foundation deliberately, before the agent gets anywhere near it.

This is not a new problem. It is the same problem that made undocumented semantic models painful for analysts, made onboarding new BI developers slow, and made “what does ‘active customer’ mean?” a recurring meeting agenda item. The AI just made it impossible to paper over.

What the Fabric Ontology Actually Is

The Fabric Ontology operates above the table and column level, at the concept level, the level where business people actually think and where agreements actually need to live.

Three building blocks:

Entity types are the real-world objects your business runs on. Customer, Order, Product, Shipment, Store. Defined once, with a stable name, description, and identifiers. Not four slightly different customer tables with different primary keys depending on which source system populated them first.

Properties are named, typed facts about an entity. Instead of a column called cust_rev_ytd_adj, you publish a Customer property called Adjusted Year-to-Date Revenue with a declared unit, a data type, and a binding to the underlying source column. Something a new analyst can understand without asking someone who remembers the original intent. Something an AI agent can reason about without guessing.

Relationships are explicit, directional, typed links between entities with cardinality rules. Customer places Order. Order contains Product. Shipment originates from Plant. Made reusable and visible, rather than buried in join logic spread across three different pipelines and a Power BI measure that no one wants to open.

Those concept definitions then bind to your actual data in OneLake: lakehouse tables, Eventhouse streams, Power BI semantic models. The data bindings handle schema drift, enforce data quality checks, and track provenance at the concept layer.

The result: a shared vocabulary that both people and AI agents can reason over. When an agent is grounded in a well-defined ontology, it is not reverse-engineering meaning from raw tables. It is working from a context that someone owns and maintains.

The Ontology Graph: Relationships as Queryable Data

The Fabric Ontology also builds an ontology graph from your data bindings and relationship definitions: a queryable instance graph where entity instances are nodes and relationships are edges, each carrying metadata and data source lineage, refreshed on a schedule.

For anyone who has spent time making implicit relationships explicit and queryable, this is worth understanding. Context that previously only existed as join logic, tracing which customers are tied to which orders and which products trace back to which suppliers, becomes something you can traverse, analyze, and govern. Path finding, centrality analysis, community detection: graph algorithms applied to your actual business data.

On top of that sits a Natural Language to Ontology (NL2Ontology) query layer that converts plain-language questions into structured queries across your bound sources, routing automatically to GQL for graph queries or KQL for Eventhouse. Not a best-effort guess at what a column might mean. Consistent answers that follow the definitions you published in your ontology.

Three Things That Actually Matter Before You Build the Agent

I have not shipped a production data agent grounded in a Fabric Ontology end-to-end yet. The feature is still in preview and I am still working through it. But the guidance is consistent enough across documentation and early community experience that I think these three things are worth naming before you start.

Build the semantic foundation first

This is the step that gets rushed. The agent is only as reliable as the context it has to work with. If your semantic model has undocumented measures, ambiguous column names, and definitions that three different teams would answer three different ways, an ontology built on top of that inherits all of it.

Before connecting an agent to your data:

  • Audit your semantic model. Are measure names self-explanatory? Are the terms your business uses defined anywhere?
  • Generate a Fabric Ontology from your semantic model as a starting point. Fabric can auto-generate one to give you something concrete to refine, rather than starting from a blank canvas.
  • Write descriptions for the columns and measures that currently only make sense to the person who created them.
  • Resolve the definitions that lack consensus before rollout, not after. “What counts as an active customer?” is not a technical question. It is a business alignment question. It needs an answer before the agent encounters it at 9am from a business stakeholder.

Keep each agent’s scope narrow

The temptation is to build one agent that answers everything. It almost always underperforms. The more data an agent has to reason over per question, the harder it is for it to return consistent answers.

A sales agent. An inventory agent. A finance agent. Each one is easier to configure, easier to test, and easier for the people who rely on it to trust, because the scope is legible and the owner is clear.

Start with one domain. The one where trust matters most and the semantic definitions are clearest. Do it properly. Let that one earn credibility before expanding.

Write the instructions like you are briefing a smart new colleague

Data agents are probabilistic: they use statistical reasoning to determine the most likely answer. Business users expect deterministic behavior, meaning the same question should return the same answer every time. Detailed agent instructions are the primary lever for closing that gap.

Think of it as the standing brief you would write for a new analyst on their first day: here is what matters, here is how we define things, here is what belongs out of scope, and here is what to do when a question is ambiguous.

For your most critical business questions, Fabric data agents support sample questions with pre-defined SQL, DAX, or KQL behind them, removing the probabilistic element entirely for those specific scenarios. Use it. Treat the instructions as a living document and update them as you learn how people in your organization actually phrase questions.

Where I Am On This

The hard part of building reliable AI over enterprise data is not the model. It is the semantic gap between raw data structures and the meaning business users expect the model to already know. The Fabric Ontology looks like the most direct thing Microsoft has shipped to address that gap at the platform level. That is what makes it worth paying attention to, even while it is in preview.

I am still early in exploring this and plan to dig further as it moves toward GA. If you have already started building with it, whether you found a workflow that clicked, hit a wall, or worked around something unexpected, I would genuinely like to hear about it in the comments.

Materialized Lake Views: Now Generally Available

Posted by Jens Vestergaard on in Programming 0 Comment

I’m excited to share that materialized lake views, announced at the FabCon Atlanta conference, are now generally available in Microsoft Fabric.

Reference:
Materialized lake views are also now generally available, simplifying medallion architecture implementation in Spark SQL and PySpark and enabling always up-to-date pipelines with no manual orchestration. 
FabCon and SQLCon 2026: Unifying databases and Fabric on a single data platform | Microsoft Azure Blog

As a SQL developer working with large-scale analytics, I’m always looking for ways to simplify data pipelines and boost query performance. Recently, I’ve been exploring materialized lake views in Microsoft Fabric, and I want to share how they can transform the way we manage and serve data in a lakehouse environment.

Why Materialized Lake Views?

If you’ve ever built reporting datasets or complex aggregations in a lakehouse, you know the pain: Spark notebooks for transformations, pipelines for scheduling, and a lot of manual orchestration to keep everything fresh and consistent. Materialized lake views change the game by letting you define your transformations in SQL, then letting Fabric handle execution, storage, and refresh.

The result? Fast, query-ready assets persisted as Delta tables in OneLake, automatically refreshed on a schedule or when source data changes. No more worrying about refresh logic or execution order—just focus on your SQL.

When Should You Use Them?

I reach for materialized lake views when I need:

  • Frequently accessed aggregations (like daily sales or monthly metrics)
  • Complex joins across large tables that need consistent, up-to-date results
  • Declarative data quality rules (think: “drop rows where sales_amount <= 0”)
  • Reporting datasets that combine multiple sources and need automatic refresh
  • Medallion architecture (bronze → silver → gold) defined in SQL

They’re not for everything. For one-off queries, simple fast transformations, or non-SQL logic (like ML or Python), I stick with notebooks or other tools.

How Do They Work?

It’s simple: write a SQL query, and Fabric materializes the result as a Delta table. When source data changes, Fabric figures out the optimal refresh—incremental, full, or skip. You query the view like any other table, and built-in monitoring lets you track refreshes, data quality, and dependencies.

My Favorite Features

  • Automatic refresh optimization: Only new or changed data is processed when possible.
  • Built-in data quality: Add constraints right in your SQL, like:
  CONSTRAINT valid_sales CHECK (sales_amount > 0) ON MISMATCH DROP
  • Dependency management: Fabric handles execution order for you, even across chained views.
  • Monitoring: Track refresh status, data quality, and lineage in one place.

Real-World Example

Here’s how I create a daily sales summary that always stays fresh:

CREATE MATERIALIZED LAKE VIEW daily_sales AS
SELECT 
    DATE(order_date) as sale_date,
    region,
    SUM(amount) as total_sales,
    COUNT(*) as order_count
FROM orders 
GROUP BY DATE(order_date), region;

Limitations

Right now, cross-lakehouse lineage and execution aren’t supported, and for high-frequency streaming or non-SQL logic, you’ll want other tools.

Final Thoughts

Materialized lake views have made my data pipelines simpler, faster, and more reliable. If you’re building analytics in Microsoft Fabric, give them a try as you’ll spend less time orchestrating and more time delivering insights.

Planning in Microsoft Fabric IQ for SQL Developers

Posted by Jens Vestergaard on in Programming 0 Comment

If you work with SQL in Fabric, you already know the pattern. Reporting data lands in Fabric SQL or Lakehouse tables, semantic models sit on top, and Power BI turns that into something business users understand. Planning has always been the odd one out, usually living in a separate tool that IT feeds with exports and integrations. Not anymore…

Announced today at FabCon today (March 18th 2026)
If you haven’t already, check out Arun Ulag’s hero blog “FabCon and SQLCon 2026: Unifying databases and Fabric on a single, complete platform” for a complete look at all of our FabCon and SQLCon announcements across both Fabric and our database offerings. 

With Planning in Microsoft Fabric IQ, that separation disappears. Planning now sits directly on top of the same Fabric SQL and semantic models that developers and data engineers already maintain.

Ref: Introducing Planning in Microsoft Fabric IQ: From historical data to forecasting the future | Microsoft Fabric Blog | Microsoft Fabric

For SQL professionals, this unlocks a much cleaner architecture, less integration work and more predictable data flows.

Planning Uses Your Existing Semantic Models

Planning in Fabric IQ reads business logic directly from Power BI semantic models. SQL developers no longer have to replicate definitions or maintain special “planning exports.”

If your measures, reference tables and dimensions are modeled correctly, Planning uses them as its foundation. This eliminates drift between planning logic and reporting logic, something that has always been painful in disconnected systems.

Writeback Lands in Fabric SQL

The biggest operational change for SQL developers is this. Planning writeback does not land in a proprietary planning database. It lands in Fabric SQL tables that you control and can query.

This means:

  • Forecasts and budgets are stored as regular tables
  • Versioning and governance policies work the same way as other data
  • You can join planning data with operational data without a round trip to another tool
  • Downstream BI reports update automatically because the data lives in the same environment

A typical structure might look like:

SQL

SELECT
    d.CustomerKey,
    f.ForecastAmount,
    f.Version,
    a.ActualSales
FROM dbo.FinanceForecast f
    JOIN dbo.DimCustomer d ON d.CustomerKey = f.CustomerKey
    JOIN dbo.SalesActuals a ON a.CustomerKey = f.CustomerKey
WHERE f.Version = '2026-Base'

No more external APIs, sync jobs or file drops.

OneLake Makes Planning Data Instantly Available

Shortcuts and mirroring remove a major pain point for SQL developers. Data used by planners does not need a dedicated pipeline or internal copy. If the data is already in OneLake, Planning can use it immediately.

This avoids:

  • Daily ETL loads
  • Redundant staging areas
  • Manual reconciliation work

For SQL developers, this is more predictable, more consistent and easier to operate.

A Single Environment From Actuals to Forecasts

Traditionally, SQL teams have had to maintain two parallel worlds.
One world holds actuals and historical performance.
Another world holds budgets and planning data from a separate system.

Planning in Fabric IQ merges these worlds:

  • Actuals remain in Lakehouse and Fabric SQL
  • Plans and scenarios write back into Fabric SQL
  • Semantic models unify both
  • Power BI reports read everything from the same data estate

This reduces the number of moving parts and makes lineage, governance and validation much easier.

Planning as a New Input to AI and Automation

Planning introduces a new type of data for SQL developers to work with: intent data.
Targets, constraints, scenario assumptions and expected outcomes become tables that intelligent agents can read.

This shifts planning from an isolated workflow to a central part of automated decision support. For SQL developers, this means new opportunities to model features, feed scoring pipelines and support decision logic with richer context than just historical data.

Why SQL Developers Should Care

Planning in Fabric IQ is worth paying attention to because it simplifies several long standing operational challenges:

  • No more pipeline maintenance to feed external planning tools
  • No more reconciliation work between planning and reporting
  • No more duplicated metrics in separate systems
  • No separate planning database that IT cannot fully control
  • Writeback is now a standard Fabric SQL operation
  • Planning logic aligns with semantic model logic

This is a major improvement for anyone who has had to support planning workflows while also maintaining clean, governed SQL environments.

Getting Started

Planning in Fabric IQ is available in preview. To experiment:

  1. Open a Fabric workspace
  2. Connect Planning to an existing semantic model
  3. Observe how writeback lands in Fabric SQL
  4. Integrate planning data with your existing T SQL workload

From there, it becomes clear that planning is no longer an external dependency. It is part of the platform, and SQL developers can finally treat it like any other governed dataset.