Community

Trusting Fabric Data Agent output

Posted by Jens Vestergaard on in Community 1 Comment

In two previous posts I went down the path of getting a semantic model ready for AI: descriptions on every measure, an instructions file, the schema tidied up enough that a Fabric data agent has something real to read. That work has a satisfying endpoint. The model looks ready.

Ready is not the same as right.

A data agent will answer almost any question you put to it. It answers the ones it understands and it answers the ones it does not understand, in the same calm tone and with the same neatly formatted number. The demo question works in the meeting, everyone nods, and the agent goes into circulation. Three weeks later someone asks it for “sales outside the US last year” and it returns a confident figure that quietly excludes returns. Nobody checks, because checking is the exact thing the agent was supposed to remove.

I have done enough BI to distrust a number I have not reconciled. So before I let an agent answer questions on behalf of other people, I want to do to it what I would do to any new measure: feed it inputs where I already know the output, and count how many it gets right.

The instinct is borrowed from unit testing

Earlier this year I wrote about unit testing DAX with Semantic Link: a known input, a known expected value, an assertion that passes or fails. The value of that pattern is not the assertion itself. It is the discipline of writing down what you expect before you run anything, so the tool cannot talk you into accepting whatever it happens to produce.

A data agent is harder to test than a measure, because the input is a sentence and the output is a sentence. But the shape is identical. A question, an expected answer, a verdict. If I can compute the expected answer myself, I can score the agent against it.

Computing the ground truth, not guessing it

The part that matters, and the part it is tempting to skip, is that the expected answers have to be correct independently of the agent. If I let the agent propose its own expected answers, I am grading its homework with its own answer key.

So the expected column comes from somewhere I trust. For each question I write the DAX myself and evaluate it against the model with Semantic Link, the same way I built the assertions in the unit testing post. The result of that DAX becomes the expected answer. It is the same instinct as the lakehouse validation post, where I ran DAX against the source to confirm the model agreed with it. Here the DAX is the source of truth and the agent is the thing under test.

A handful of questions the business actually asks is enough to start. Total sales last year. Sales outside the US. The top product in a category. The questions people ask in Teams and then argue about.

The evaluation set

The Fabric data agent SDK has an evaluation module that does exactly this, which saved me from building a harness by hand. It is in preview, and it needs an F2 or higher capacity (or P1 or higher), so it is not something you run on a trial tenant. You install it in a notebook:

%pip install -U fabric-data-agent-sdk

The ground truth is a dataframe with two columns, question and expected_answer:

import pandas as pd

df = pd.DataFrame(
    columns=["question", "expected_answer"],
    data=[
        ["Total sales for 2013", "19,968,887.95"],
        ["Top product by sales for Canada in 2013", "Mountain-200 Black, 42"],
        ["Which category had the highest sales for Canada in 2013", "Bikes (Total Sales: 938,654.76)"],
    ],
)

I keep this in a CSV in the lakehouse so it lives in source control next to the model, rather than in a notebook cell that gets overwritten the next time someone opens it.

Running it and reading the score

from fabric.dataagent.evaluation import evaluate_data_agent

evaluation_id = evaluate_data_agent(
    df,
    "RetailSalesAgent",
    table_name="sales_eval",
    data_agent_stage="production",
)

The function asks the agent each question, compares the answer to the expected value, and writes the verdicts to a Delta table. Two tables, actually: the summary, and a sales_eval_steps table with the reasoning the agent used to get there. The steps table is where the interesting failures live, because it shows you the DAX the agent generated, not just the number it landed on.

The summary is the figure I look at first:

from fabric.dataagent.evaluation import get_evaluation_summary

get_evaluation_summary("sales_eval")

It comes back with the total number of questions and an accuracy figure, split into three buckets: true, false, and unclear. The third bucket is the honest one. It means the judge could not decide whether the agent was right, which is a result in itself and not something to round away.

Then the row level, filtered down to the ones that did not pass:

from fabric.dataagent.evaluation import get_evaluation_details

get_evaluation_details(evaluation_id, "sales_eval", get_all_rows=False, verbose=True)

That returns the question, the expected answer, the actual answer and the verdict per row, plus a link to the full thread so I can read what the agent actually did to get there.

What the failures looked like

I ran twenty questions against the retail model from the documentation posts. Most passed. The ones that did not were more useful than the ones that did, and they fell into a few recognisable shapes.

  • The ambiguous column, picked wrong. Two columns could plausibly answer a question about “region”, and the agent chose the one carried over from the source system rather than the reporting one. The number was real. It was the wrong number.
  • Wrong grain. A question about monthly sales came back aggregated to the year. Correct total, wrong shape, presented as if it were the answer asked for.
  • Wrong filter context. “Sales outside the US” came back including the US, because the agent built the filter as a positive selection it did not fully invert. This is the one that worried me most, because the figure looks reasonable and only the reconciliation catches it.
  • Plausible but false. A measure that wraps four others got interpreted from its name, and the agent confidently returned a figure that no DAX in the model actually produces.

None of these are exotic. They are the same mistakes a new analyst makes in their first week, which is the right way to think about an agent: capable, fast, and in need of supervision until it has earned otherwise.

The part I have not solved

The judge in the middle is itself a language model. By default the SDK uses a built-in critic prompt to decide whether the actual answer matches the expected one, and you can replace it with your own through the critic_prompt parameter and the {query}, {expected_answer} and {actual_answer} placeholders. That is genuinely useful when the agent says 19,968,887.95 and my expected value reads 19.97M. They are the same answer in different clothes, and a strict string comparison would fail it for no good reason.

But it cuts both ways. A lenient critic will pass answers that are close enough in wording and wrong in substance. A strict one will fail answers that are right. And because the critic is itself non-deterministic, the same evaluation can land in slightly different places on different runs. The “unclear” bucket is where that uncertainty collects, and there is no getting around reading those rows yourself.

So I do not have a clean, fully automated grade I would trust without looking. What I have is a repeatable harness that turns a vague feeling of “the agent seems fine” into a number, a list of the specific questions it got wrong, and the agent’s own reasoning for each one. That is enough to stop a bad answer reaching a stakeholder, and enough to re-run after every model change so that a description edit or a new measure does not silently break something that used to work.

Ready was never the finish line. The evaluation set is.

Power BI RLS and Fabric Data Agent considerations

Posted by Jens Vestergaard on in Community 1 Comment

In the last post I walked through auditing a semantic model before connecting it to an AI tool like Fabric Data Agent. Descriptions, naming, explicit measures, star schema: the things that decide whether a Fabric data agent generates an accurate query or a confident wrong one. I left one thing out on purpose, because it deserved its own post and because I got it wrong the first time I thought about it.

Security.

Here is the setup. I inherited a model. It had row-level security on it. Sales people saw their own region, the finance role saw everything, and the reports had been running like that for two years without a complaint. So when the request came in to connect a data agent to the workspace, I ticked the security box in my head and moved on. RLS was handled.

It was handled for people browsing reports. That is not the same thing as handled for an agent answering questions over the model, and the gap between those two sentences is where the trouble lives.

What I got wrong first

My first assumption was the dramatic one: the agent runs as some service identity, ignores the report layer, and reads everything. That is wrong, and the reason it is wrong is the useful part.

A Fabric data agent honors the security on the underlying data. Microsoft is explicit: the agent honors all user permissions to the data, including row-level security and column-level security, and for a Power BI semantic model the querying user only needs Read permission on the model, not workspace access, as documented in Fabric data agent sharing and permission management. The data agent concepts page repeats it in the limitations section: access through a data agent is governed by Read permission, and RLS and CLS still apply. So the agent is not a backdoor. It queries as the person asking the question, and that person’s RLS role filters the rows.

Good. So where is the problem? The problem is everything that decides what “the person’s RLS role” actually evaluates to. RLS is only as strong as the identity model around it, and that is where I had two real holes.

Hole one: RLS only binds Viewers

This is the one that catches people, and it caught me.

Row-level security only restricts users with the Viewer role in the workspace. It does not apply to Admins, Members, or Contributors. Those three roles carry edit permission on the semantic model, and a user who can edit the model can read all of it. Microsoft says it plainly in Row-level security (RLS) with Power BI: if you want RLS to apply to someone in a workspace, the only role you can give them is Viewer. Even a Viewer with Build permission still gets filtered. A Contributor does not.

Now connect that to the agent. The agent filters by the asking user’s identity. If the people allowed to ask the agent questions sit in the workspace as Contributors, because that is how the workspace was set up back when it was three developers and no governance, then RLS evaluates to “see everything” for every one of them. The agent is behaving correctly. It is honoring RLS. RLS just happens to be off for those identities.

The same rule applies to object-level security. OLS only binds Viewers too, as the object-level security documentation states in its limitations. The Admin, Member, and Contributor roles see the secured tables and columns regardless.

So the report side and the model side diverge exactly here. A Contributor browsing a report might still look safe, because the report only contains the visuals someone built. Point that same Contributor’s identity at the model through an agent that can write arbitrary DAX, and there is no report layer left to hide behind.

Hole two: service principals sit outside RLS entirely

The second hole is worse, because it fails silently in both directions.

Service principals cannot be added to an RLS role. The consequence, in Microsoft’s own words, is that RLS is not applied for apps using a service principal as the final effective identity. That is point three in the RLS considerations and limitations. The XMLA endpoint documentation says the same thing from the other side: service principals do not work with RLS and OLS and cannot be added as model role members.

This matters the moment any part of the chain runs as a service principal rather than a signed-in user. Automation, an embedded app, a fixed-identity Direct Lake model, a pipeline that calls the model. If the effective identity is a service principal, static RLS is simply not applied, and dynamic RLS is worse than not applied: USERPRINCIPALNAME() and USERNAME() return the application ID or an empty string for a service principal, so a filter written as [Region] = USERPRINCIPALNAME() matches nothing, or matches the wrong thing, depending on how the model was built. You either leak everything or you hand back silently incorrect slices, and neither failure announces itself.

So before connecting anything, I now need to know exactly what identity the thing on the other end runs as. “It uses RLS” is not an answer. “It queries as the signed-in user, who is a Viewer, whose role filters to their region” is an answer.

Hiding a column is not securing it

This one is subtle and I want to be careful with it.

When you set a column to hidden in the model, you remove it from the field list. You have not secured it. A hidden column is still queryable. Microsoft makes the equivalent point about perspectives, which are also just a curated view: a user can still query a table or column even when it is not visible to them, so a perspective is a convenience, not a security feature. That note lives in the report consumer security planning guidance. The hidden flag is the same kind of convenience. The agent generating DAX is not limited to your field list. It reads the model metadata, and hidden objects are still part of the model.

The actual controls are object-level security and column-level security. OLS removes the table or column for a role so completely that, to that role, it does not exist, and unlike RLS it also hides the object name and metadata, so the field cannot even be discovered. Column-level security does the same at the column grain by setting the metadata permission to none, as shown in the Analysis Services object-level security reference. You cannot author OLS in Power BI Desktop today. You set it in Tabular Editor or another external tool by setting the table or column permission to None for the role. And, to close the loop, OLS binds Viewers only, same as RLS.

So if there is a salary column, or a cost column, or anything that should not reach the agent, hiding it does nothing. Securing it with OLS or CLS does, for Viewer identities, and for no one else.

Report-level safe is not model-level safe

If there is one sentence to take from this, it is that one. Reports hide visuals. The model still answers DAX. An agent talks to the model, not to the report, so every assumption that lived in the report layer needs to be re-checked at the model layer before the agent goes live.

The check I run now

Before I connect a semantic model to a data agent, I work through this. It is short on purpose.

  1. List every RLS role and write down, in one line each, exactly what it filters. If you cannot describe a role’s filter in a sentence, you do not yet understand what the agent will return for its members.
  2. List the members of each role, and then list everyone with Admin, Member, or Contributor on the workspace. That second list is the bypass list. For those identities RLS and OLS do not apply at all. Decide whether every name on it should genuinely see all rows.
  3. Identify the agent’s effective identity. If it is a service principal or a fixed identity, RLS is not being applied, and you cannot fix that by adding the principal to a role, because you cannot add it to a role. Reach for a different pattern, such as a model whose fixed identity is itself scoped to safe data, or per-user identity passthrough.
  4. Check sensitive columns for OLS or CLS, not for the hidden flag. The hidden flag is decoration. Confirm the salary-shaped columns are actually secured, and remember that the security only binds Viewers.
  5. Test as the agent’s identity, not as yourself. The Test as role feature evaluates dynamic RLS using your own UPN, so it cannot show you what a service principal or a guest would see. Where it matters, sign in as the real identity, or run the same DAX the agent would run, as that identity, and look at the rows that come back.
  6. Default everyone who only needs to ask questions to Read on the model, nothing more. The agent needs Read, not Build, not a workspace role. Least privilege here is not a nicety. It is the only configuration where RLS actually does its job.

Where I am still unsure

I have not fully mapped how a Fabric data agent resolves identity across every sharing mode it offers. The sharing documentation is clear that the agent honors RLS and CLS and that consumers need Read on the model, and that lines up with the per-user story. What I want to test next is the boundary cases: an agent shared with a consumer who has no workspace role at all, a model on a fixed identity, and an agent invoked from automation rather than a chat window. I expect the identity that reaches the model to differ across those, and the difference is exactly what decides whether RLS holds. When I have run it, that is the next post.

Until then, the rule I am keeping is the unglamorous one. Before the agent, list the roles, list who bypasses them, find out what identity actually hits the model, and test as that identity. Not as me.

Sources

Power BI prep for AI routine

Posted by Jens Vestergaard on in Community 0 Comment

In the Fabric Ontology post from a couple of months ago, the third item on the pre-agent checklist was “build the semantic foundation first.” I moved on from that line pretty quickly. It deserved more than a sentence.

Most Power BI developers who have been in the field for a few years have inherited at least one model they did not design. It has measures with names like Amt_Adj_Net2 and FlagActCust, columns that were added for a specific report in 2021 and never cleaned up, and business logic that someone once explained verbally in a meeting. No descriptions. No documentation. The people who built it have either moved on or are too busy to explain it.

Now you are being asked to connect that model to a Fabric data agent. Or Power BI Copilot. And suddenly the question “what do these fields actually mean” has a hard deadline.

This post is a checklist. It is what I would do before connecting any semantic model to an AI tool, and it works backward from how those tools actually read your model.

How AI agents read your semantic model

When a Fabric data agent gets a question, it does not look at your data. It reads your metadata. The DAX generation tool builds queries from the model schema, descriptions, synonyms, relationships, data types, and the configuration you set in Prep data for AI, then validates and executes the query against the model. Microsoft documents this query flow in the semantic model best practices for data agent guide.

Copilot in Power BI works the same way. The principle generalises: a natural-language-to-query tool sitting on top of a model reads the metadata, not the data.

If your metadata is clear, the AI generates accurate queries. If it is ambiguous or missing, the AI guesses. And when an AI guesses about business logic, the answers look correct but frequently are not.

The audit checklist

1. Explicit measures over implicit aggregation

An implicit measure is what you get when a report visual or Q&A aggregates a numeric column on the fly. The number is calculated by the consuming layer, not defined as a calculation in the model. An explicit measure is a DAX expression you write and store in the model.

Microsoft’s own guidance for the Fabric data agent is direct on this: relying on implicit measures leads to unpredictable results, so you should create explicit DAX measures for the calculations you want users to query, and set the correct default summarization on numeric columns to prevent unintended aggregations. The optimize your semantic model for Copilot page makes the same point: include a set of predefined measures that users are most likely to request.

Go through every numeric column and ask: is there an explicit DAX measure for the business calculation it feeds? If not, and it matters for the questions the AI will receive, create one. This is the highest-priority item on the list. A model that leans on implicit aggregation will produce inconsistent answers from any AI tool regardless of how well everything else is documented.

2. Names that mean something

Go through every table name, column name, and measure name. For each one, ask: could a new developer understand what this is from the name alone, with no other context?

SalesAmount is fine. Amt is not. IsActiveCustomer is fine. Flag1 is not. Date on a calendar table is fine. D is not.

Microsoft lists non-descriptive naming as a common pitfall for the data agent, using exactly these kinds of examples: object names like TR_AMT, F_SLS, or DIM_GEO_01 give the DAX generation tool no context. Use clear, business-friendly names such as Total Revenue, Sales, or Customer Geography.

Rename anything that requires decoding. Yes, this will break existing reports if you rename columns, so do it carefully and test. If you genuinely cannot rename an object, make sure its description and synonyms carry the context instead. But an AI that can’t interpret TrxTypCd will give you wrong answers, and changing the name now is cheaper than debugging wrong outputs later.

3. Descriptions on measures

The description field on a measure is passed directly to the AI as part of its schema context. A measure without a description forces the agent to infer meaning from the DAX expression alone.

For a measure that calculates CALCULATE(SUM(Sales[Amount]), DATEADD(Calendar[Date], -1, YEAR)), the DAX is interpretable. For a measure that calls four other measures and has a name like Adj_Rev_Cy, it is not.

Use the TMDL triple-slash syntax if you are working in PBIP format:

/// Net revenue adjusted for returns and currency differences, current year
measure 'Adj Rev CY' = ...

If you are working in Power BI Desktop, the description field is in the properties pane.

Keep the description tight. In the DAX query view grounding path, Microsoft documents that measure descriptions are truncated after the first 200 characters when passed to Copilot as context. Front-load the meaning in the first sentence rather than burying it under a paragraph of caveats.

For a model with fifty or more measures and no descriptions, this is a significant amount of work. Power BI has a built-in Copilot button to generate measure descriptions from the DAX formula, which is a reasonable shortcut for a first pass. In a follow-up post I will cover that workflow and its limitations in more detail.

4. Hide what should not be queryable

Surrogate keys, flag columns, internal technical fields. If a column has no business meaning and is not needed for navigation, hide it. Hidden columns are invisible to AI tools. Fewer columns in the schema means less noise for the AI and fewer opportunities for a wrong-turn query.

A model with 180 visible columns where 120 of them are source system artefacts is harder for an AI to navigate than a model with 60 columns where everything is intentionally exposed.

5. Verify the star schema structure

Fact tables should contain measurements and foreign keys. Dimension tables should describe entities. Microsoft is explicit that DAX is optimised for star schema with clear fact and dimension tables, and lists flat or denormalised structures as a pitfall for the data agent. If you have many-to-many relationships, bidirectional filter behaviour, circular references, or deeply nested snowflake structures, the AI will not interpret them the way you intend.

This does not mean every unusual pattern is wrong. It means you need to know where they are, understand what they do, and either clean them up or document them in the AI instructions (see item 6 below).

Role-playing dimensions deserve specific attention. If your Date table participates in three relationships (order date, ship date, due date) and only one is active, the AI will use the active one by default for every date-related question unless you tell it otherwise. Microsoft’s guidance for ambiguous date fields is to use verified answers and AI instructions to specify which date field to use for each type of question.

6. Add AI instructions to the model

Power BI has a feature called AI instructions. It is free-text guidance that Copilot interprets, authored through the Prep data for AI experience in Power BI Desktop or in the semantic model settings in the Power BI service. The model needs Power BI Q&A turned on before the Prep data for AI tabs become available. Use AI instructions to:

  • Define domain-specific acronyms and abbreviations that appear in your field names
  • Direct the AI toward the correct measure for each type of question
  • Note any calculation quirks around fiscal year, currency conversion, or time intelligence
  • Explain role-playing dimension behaviour

From a product perspective this is metadata you configure, not a file you hand-edit. PBIP and TMDL projects may surface it as a project artifact on disk, but Microsoft documents the feature as AI instructions and the supported authoring path is the Prep data for AI surface. One caveat worth knowing for the data agent specifically: the DAX generation tool only reads the AI instructions you set in Prep data for AI, not instructions added at the data agent level, so keep model-specific guidance in Prep data for AI.

7. Scope the AI data schema

Alongside AI instructions, the Prep data for AI experience has a Simplify data schema tab where you define a focused subset of fields for Copilot to prioritise. If your model has 200 fields but only 50 are relevant for the business questions the AI will receive, scope the schema to those 50. When you include a measure, include its dependent objects too, or the generated DAX can fail.

Synonyms are a separate mechanism. If your users ask about “revenue” but the measure is called “Net Sales Amount”, you map those terms through Q&A linguistic modelling, not through the AI data schema. Both feed the same goal of reducing ambiguity, but they are configured in different places.

8. Mark the model as Approved for Copilot

Once the above steps are complete, mark the semantic model as Approved for Copilot in the Power BI service settings. The setting was previously called “prepped for AI”. Marking it removes the friction warnings that standalone Copilot shows on answers from models that have not been approved, and reports built on that model inherit the approval.

It is also a signal to your organisation that this model has been intentionally prepared for AI use.

What to do with the findings

This audit will produce a list. Some items are quick (renaming a column, hiding a key). Some are not (creating 40 explicit measures for a model that has been running on implicit ones for three years without anyone noticing).

Prioritise by impact. Start with the tables and measures the AI will actually touch. You do not need to fix the entire model before connecting anything to an AI tool. You need to fix the parts that will be queried.

The second thing this audit tends to reveal: there is no one who knows what most of these measures actually calculate. The descriptions you need to write require conversations with people who may no longer be at the company. That is a different problem to solve. But knowing you have it is better than finding out after the AI agent gives the board a wrong number.

When you work out how to describe a measure you did not write, and no one who wrote it is available, the answer is in the next post.

Sources

Microsoft Certified: Fabric Data Engineer Associate

Posted by Jens Vestergaard on in Community 0 Comment

I have renewed my Microsoft Certified: Fabric Data Engineer Associate certification for another year, and since this one actually required me to sit down and think rather than just click through some formalities, I thought it worth a quick note here.

The renewal is free. It is an online assessment on Microsoft Learn, not a full proctored exam. That might sound easier than the original DP-700, and in some ways it is. But the questions are updated, the skill areas evolve alongside the platform, and Microsoft has made a point of keeping the content aligned with what Fabric actually looks like today. The skills were last updated on April 20, 2026, with minor changes across most sections.

What the certification covers

The exam breaks down into three equally weighted areas, each around 30 to 35 percent of the total:

Implement and manage an analytics solution covers workspace configuration, lifecycle management with deployment pipelines and version control, security and governance at row, column, object, and folder level, and orchestration choices between Dataflows Gen2, pipelines, and notebooks.

Ingest and transform data is where most of the practical depth lives. Full and incremental load patterns, OneLake shortcuts and mirroring, PySpark, SQL, and KQL transformations, handling duplicates and late-arriving data, and streaming via Eventstreams and Spark structured streaming.

Monitor and optimize an analytics solution covers monitoring ingestion and transformation, resolving errors across pipelines, Dataflows, notebooks, Eventhouses, and Eventstreams, and performance optimization for Lakehouse tables, pipelines, data warehouses, and Spark.

Why bother renewing

Fabric moves fast. I have been working with Fabric in various capacities for a while now, and the platform I certified against originally does not look identical to what I use today. Real-Time Intelligence has matured, OneLake security has grown considerably, and the orchestration story has gotten richer. Going through the renewal assessment is a reasonable way to check whether there are gaps between what I knew and what the platform has become.

It is not a substitute for hands-on work. But it is a decent forcing function to at least read the change log and spend an hour or two with areas that have moved the most.

If you are already certified and have not renewed yet, the renewal assessment is here and the updated study guide for DP-700 is worth a read before you go in.

Semantic Sonar

Posted by Jens Vestergaard on in Community 1 Comment

Canary monitoring for Power BI semantic models

Quarterly sales season always brings the same ritual: a business user opens Power BI, clicks into the sales dashboard, and waits for the numbers to load. It has been a while since anyone looked at this particular report. The numbers look off. Someone hits refresh. The spinner turns, but the data does not budge. The last refresh was weeks ago.

That is when the emails start. “Can you check why the sales model hasn’t updated?” The support thread grows. IT investigates. The answer is usually not in the data. The refresh failed at 3 AM. Or the data source changed its connection string last Tuesday. Or someone revoked the service principal without telling anyone.

By the time the root cause is found, the report has been open in meetings for hours, and the numbers have been wrong for days.

I built Semantic Sonar because I wanted that conversation to happen at 3:01 AM when the refresh failed, not at 10:30 AM when the quarterly review is already underway. The idea is a canary: a lightweight DAX query fired at each semantic model on a schedule. If it comes back clean, the model is reachable and queryable. If it does not, something is wrong right now, not three days from now.

Registering the first model

The first page most people land on after setting up the system is the workspace browser. The alternative is typing workspace GUIDs and dataset IDs by hand, which I did exactly once before deciding to build the browser.

You pick a tenant from the dropdown, the browser calls the Power BI REST API with the tenant’s service principal, and it returns a list of workspaces. Expand a workspace and you see every semantic model in it with its display name. Click Register and a form pre-fills with the workspace ID, dataset ID, and a default DAX query:

EVALUATE ROW("Ping", 1)

That default is intentionally trivial. You want the canary to be cheap. Its job is not to test the data. Its job is to confirm that the model is reachable and that the service principal can still authenticate. A single-row EVALUATE ROW does that in under a second on any model that is working.

For most models I change the query to something slightly more meaningful: a measure that actually touches the fact table, so a permission problem on the underlying source would surface. But the principle stays the same: keep it light, keep it fast, run it often.

The interval sits at 60 minutes by default. For critical models I set it lower, 15 or 30 minutes. The minimum the system allows is 1 minute, though for most models that is more noise than signal.

The dashboard: what you see first

Once a few models are registered and the scheduler has had time to run, the dashboard starts telling its story.

Four numbers across the top: total models, active, failing, disabled. In a healthy fleet they read something like 24, 22, 1, 1. The single failing model is the one you care about. The disabled one was auto-disabled after 30 consecutive failures, which usually means it has been broken for a long time and nobody has got around to it.

Below the stat cards, there are two lists. The first is models at risk: anything with 10 or more consecutive failures. These are on a countdown. At 30 consecutive failures the system disables them automatically. The at-risk list is early warning. You have a window to investigate before the model drops off monitoring entirely.

The second list is recent failures: the last handful of failed checks across all models, with timestamps and the raw error message. That error message is usually enough to know immediately whether something needs attention now:

Power BI API returned 401: {"error":{"code":"TokenExpired",...}}

versus

Power BI API returned 404: {"error":{"code":"ItemNotFound",...}}

TokenExpired means a credential needs rotating. ItemNotFound means someone deleted or moved the model. Different problems, different urgency, different owner to call.

The dashboard auto-refreshes every 60 seconds. I leave it open on a side monitor. When the amber tile count goes up, I look.

Drilling into a model

From the dashboard or the models list, clicking through to a single model shows the full result history.

The top section repeats the key facts: current status badge (Active, Failing, Disabled), consecutive failure count, last run timestamp, last run success or failure. Below that is a latency chart. This is the view that catches the subtler problems.

A model that is technically succeeding but whose latency has climbed from 200ms to 4,000ms over a week is not a healthy model. Something upstream changed. Maybe a gateway is under new load. Maybe a DirectQuery source started doing full scans. The trend is visible here because every result gets recorded with its latency in milliseconds.

The full result history below the chart shows each check: timestamp, success or failure, latency, row count, and the first row returned as JSON. That last one is useful. If your canary query is:

EVALUATE ROW("SalesTotal", [Sum of Sales Value])

then the first row shows the actual value that came back. A value of 0 that held steady for months and then jumped to null is interesting. It does not mean the model is broken in the way the system defines it, but it means something changed.

The models list: the fleet view

The models list page is the operational view. It has search, filter by tenant, filter by status (all/active/disabled/failing), and filter by tag. Tags are free-form strings you can add to any model, so you can group by environment (production, staging), by team, or by data domain.

Sort by health score ascending and the most degraded models float to the top. The health score is a composite across consecutive failure count, recent latency relative to the model’s own baseline, and uptime across the rolling windows. It is not a formal SLA calculation; it is a signal for where to look first.

The card for each model shows the health grade (A through F), the last run time, the consecutive failure count, and a quick-run button. That last one is useful during an investigation: rather than waiting for the next scheduled run, you can trigger a check immediately and watch the result appear in the history a few seconds later.

Uptime: putting numbers on it

The Uptime page is for the conversation with stakeholders who want to know how reliable their models are in terms they can put in a report.

Each model gets three rolling windows: last 24 hours, last 7 days, last 30 days. Each window shows total checks, successes, and uptime percentage with a colour-coded bar:

  • 99.5% and above: green
  • 95 to 99.5%: yellow
  • 90 to 95%: orange
  • Below 90%: red

Across the top of the table, the aggregate uptime across all models for each window gives a fleet-level headline. “Fleet availability last 7 days: 97.4%” is a concrete number to bring to a monthly review.

The same page also shows p95 latency trends per model. Uptime tells you whether the checks are succeeding. Latency tells you whether the models that are succeeding are doing so at acceptable speed.

Radar: the visual overview

The radar page is the one I use when I want to scan the whole fleet at a glance. It renders each model as a dot on a canvas, positioned by p95 latency on one axis and health score on the other, with dot size reflecting which models have the most weight (based on how frequently they run). Colour is the health grade.

In a healthy deployment most dots cluster in the top-left: low latency, high health score, green or lime. The outliers stand out immediately. A large red dot in the bottom-right means a high-frequency model with degraded health and elevated latency. That is the one to investigate first.

Hovering a dot shows a tooltip with model name, tenant, score, grade, and current p95 in milliseconds.

Maintenance windows: the expected failures

One thing that caused false alarms early on was scheduled maintenance. The underlying SQL Server gets taken offline every Saturday between 2 and 4 AM. The canary runs every 30 minutes. The result: a string of failures, the at-risk counter climbing, an alert firing at 2:15 AM for something everyone already knew was happening.

Maintenance windows solve that. Each window has a start time, end time, and day-of-week pattern. Two flags control what happens during the window. suppressAlerts keeps running checks and recording results but does not fire webhooks. skipCanary stops running checks entirely, which is what you want when the data source is deliberately taken offline.

The scheduler checks for an active window before it enqueues a job. If skipCanary is set, it advances the next run time and skips the queue. The model’s consecutive failure counter does not increment. When the window ends, monitoring resumes from a clean state.

Webhooks: getting alerted

The dashboard is useful when you are looking at it. For everything else there are webhooks.

A webhook fires on three events: a model transitioning from healthy to failing, a model recovering after failures, and a model being auto-disabled. You configure one or more webhook URLs on a model, each with an optional shared secret. The payload is a JSON object with the event type, model name, tenant, and a details string.

Payloads are signed with HMAC-SHA256 and the signature goes into an X-Signature header, so the receiving system can verify the sender before acting on the message. Delivery is best-effort: a failed webhook delivery is logged in Application Insights but does not affect the canary result or retry behaviour.

In practice I point webhooks at a Power Automate flow that posts to Teams. A failure notification lands in the right channel within a minute of the first failed check.

The multi-tenant model

The system was built from the start to monitor multiple customers’ models from a single deployment. That shapes the credential design more than anything else.

Each customer is a tenant record in Cosmos DB: their display name, their Entra ID (the GUID of their Azure AD tenant), and the client ID of an app registration created in their tenant. The client secret for that app registration goes into Key Vault with the name tenant-{tenantId}-client-secret.

When the worker runs a check, it reads the right secret, acquires an OAuth 2.0 token scoped to the customer’s Entra tenant, and calls the Power BI executeQueries endpoint on their behalf using their own service principal. The token is cached in memory with a 5-minute buffer before expiry.

var clientSecret = await _keyVault.GetTenantClientSecretAsync(tenant.EntraId, ct);
var credential = new ClientSecretCredential(tenant.EntraId, tenant.ClientId, clientSecret);

All other service-to-service calls (the Function App talking to Cosmos DB, Key Vault, and the Storage Queue) use a system-assigned managed identity. No connection strings in configuration. The Power BI API is the one exception: it requires a credential that the customer’s tenant has consented to, and a managed identity from the hosting tenant cannot satisfy that requirement. So that leg stays on client credentials.

One edge case: live connections and SSAS

Most models work fine with the executeQueries endpoint. The ones that do not are models live-connected to Analysis Services or on-premises SSAS through a gateway, where the gateway blocks the query execution endpoint but still allows the metadata API.

For those, setting queryMode to rest on the model switches the worker to a lighter ping that calls the dataset metadata endpoint instead. It does not execute a DAX query, but it confirms that the service principal can reach the model and that the Power BI API returns a 200. If the gateway is down, that returns a 400 or 503. The rest of the pipeline, the result recording, the consecutive failure counter, the dashboard, is identical.

Audit log

The audit page records configuration changes: who added a model, who changed an interval, who disabled a tenant. It is the answer to “what changed right before the failures started.” Filter by tenant or by entity ID. The last 100 entries load by default.

I added this after the third time someone changed a model’s DAX query to something that always returns zero rows, and nobody could remember who had done it or when.

Deployment

The infrastructure is Bicep, deployed via the Azure Developer CLI:

azd auth login
azd up

That provisions everything: the Static Web App, the Function App, Cosmos DB, Key Vault, Storage, and Application Insights. The Bicep modules are in infra/modules/, and the Azure Developer CLI service definitions in azure.yaml handle the build and deploy sequence for both the .NET Functions project and the Next.js frontend.

After deployment, you add the first tenant and model through the UI, store the client secret in Key Vault, and wait for the first scheduled run. The README has the full post-deployment checklist.

What it does not do

Semantic Sonar tells you whether a model is reachable and queryable. It does not tell you whether the numbers coming out of it are correct. A model that returns [Sum of Sales Value] = 0 because the ETL truncated the fact table will pass every canary check.

The “failing” status also does not distinguish between the model being down, the underlying data source being down, and the service principal’s permissions having expired. The error message in the result history usually makes it clear, but the system does not attempt to classify the cause.

The 90-day TTL on results caps the uptime windows at 30 days. If you want longer retention you can change the TTL in CanaryResult.cs and the Cosmos DB container configuration, but the uptime calculation logic would need updating to match.

Getting the code

The repository is at github.com/vestergaardj/semantic-sonar. The README covers prerequisites, local development, and the full post-deployment walkthrough.