Microsoft Fabric

In two previous posts I went down the path of getting a semantic model ready for AI: descriptions on every measure, an instructions file, the schema tidied up enough that a Fabric data agent has something real to read. That work has a satisfying endpoint. The model looks ready.

Ready is not the same as right.

A data agent will answer almost any question you put to it. It answers the ones it understands and it answers the ones it does not understand, in the same calm tone and with the same neatly formatted number. The demo question works in the meeting, everyone nods, and the agent goes into circulation. Three weeks later someone asks it for “sales outside the US last year” and it returns a confident figure that quietly excludes returns. Nobody checks, because checking is the exact thing the agent was supposed to remove.

I have done enough BI to distrust a number I have not reconciled. So before I let an agent answer questions on behalf of other people, I want to do to it what I would do to any new measure: feed it inputs where I already know the output, and count how many it gets right.

The instinct is borrowed from unit testing

Earlier this year I wrote about unit testing DAX with Semantic Link: a known input, a known expected value, an assertion that passes or fails. The value of that pattern is not the assertion itself. It is the discipline of writing down what you expect before you run anything, so the tool cannot talk you into accepting whatever it happens to produce.

A data agent is harder to test than a measure, because the input is a sentence and the output is a sentence. But the shape is identical. A question, an expected answer, a verdict. If I can compute the expected answer myself, I can score the agent against it.

Computing the ground truth, not guessing it

The part that matters, and the part it is tempting to skip, is that the expected answers have to be correct independently of the agent. If I let the agent propose its own expected answers, I am grading its homework with its own answer key.

So the expected column comes from somewhere I trust. For each question I write the DAX myself and evaluate it against the model with Semantic Link, the same way I built the assertions in the unit testing post. The result of that DAX becomes the expected answer. It is the same instinct as the lakehouse validation post, where I ran DAX against the source to confirm the model agreed with it. Here the DAX is the source of truth and the agent is the thing under test.

A handful of questions the business actually asks is enough to start. Total sales last year. Sales outside the US. The top product in a category. The questions people ask in Teams and then argue about.

The evaluation set

The Fabric data agent SDK has an evaluation module that does exactly this, which saved me from building a harness by hand. It is in preview, and it needs an F2 or higher capacity (or P1 or higher), so it is not something you run on a trial tenant. You install it in a notebook:

%pip install -U fabric-data-agent-sdk

The ground truth is a dataframe with two columns, question and expected_answer:

import pandas as pd

df = pd.DataFrame(
    columns=["question", "expected_answer"],
    data=[
        ["Total sales for 2013", "19,968,887.95"],
        ["Top product by sales for Canada in 2013", "Mountain-200 Black, 42"],
        ["Which category had the highest sales for Canada in 2013", "Bikes (Total Sales: 938,654.76)"],
    ],
)

I keep this in a CSV in the lakehouse so it lives in source control next to the model, rather than in a notebook cell that gets overwritten the next time someone opens it.

Running it and reading the score

from fabric.dataagent.evaluation import evaluate_data_agent

evaluation_id = evaluate_data_agent(
    df,
    "RetailSalesAgent",
    table_name="sales_eval",
    data_agent_stage="production",
)

The function asks the agent each question, compares the answer to the expected value, and writes the verdicts to a Delta table. Two tables, actually: the summary, and a sales_eval_steps table with the reasoning the agent used to get there. The steps table is where the interesting failures live, because it shows you the DAX the agent generated, not just the number it landed on.

The summary is the figure I look at first:

from fabric.dataagent.evaluation import get_evaluation_summary

get_evaluation_summary("sales_eval")

It comes back with the total number of questions and an accuracy figure, split into three buckets: true, false, and unclear. The third bucket is the honest one. It means the judge could not decide whether the agent was right, which is a result in itself and not something to round away.

Then the row level, filtered down to the ones that did not pass:

from fabric.dataagent.evaluation import get_evaluation_details

get_evaluation_details(evaluation_id, "sales_eval", get_all_rows=False, verbose=True)

That returns the question, the expected answer, the actual answer and the verdict per row, plus a link to the full thread so I can read what the agent actually did to get there.

What the failures looked like

I ran twenty questions against the retail model from the documentation posts. Most passed. The ones that did not were more useful than the ones that did, and they fell into a few recognisable shapes.

• The ambiguous column, picked wrong. Two columns could plausibly answer a question about “region”, and the agent chose the one carried over from the source system rather than the reporting one. The number was real. It was the wrong number.
• Wrong grain. A question about monthly sales came back aggregated to the year. Correct total, wrong shape, presented as if it were the answer asked for.
• Wrong filter context. “Sales outside the US” came back including the US, because the agent built the filter as a positive selection it did not fully invert. This is the one that worried me most, because the figure looks reasonable and only the reconciliation catches it.
• Plausible but false. A measure that wraps four others got interpreted from its name, and the agent confidently returned a figure that no DAX in the model actually produces.

None of these are exotic. They are the same mistakes a new analyst makes in their first week, which is the right way to think about an agent: capable, fast, and in need of supervision until it has earned otherwise.

The part I have not solved

The judge in the middle is itself a language model. By default the SDK uses a built-in critic prompt to decide whether the actual answer matches the expected one, and you can replace it with your own through the critic_prompt parameter and the {query}, {expected_answer} and {actual_answer} placeholders. That is genuinely useful when the agent says 19,968,887.95 and my expected value reads 19.97M. They are the same answer in different clothes, and a strict string comparison would fail it for no good reason.

But it cuts both ways. A lenient critic will pass answers that are close enough in wording and wrong in substance. A strict one will fail answers that are right. And because the critic is itself non-deterministic, the same evaluation can land in slightly different places on different runs. The “unclear” bucket is where that uncertainty collects, and there is no getting around reading those rows yourself.

So I do not have a clean, fully automated grade I would trust without looking. What I have is a repeatable harness that turns a vague feeling of “the agent seems fine” into a number, a list of the specific questions it got wrong, and the agent’s own reasoning for each one. That is enough to stop a bad answer reaching a stakeholder, and enough to re-run after every model change so that a description edit or a new measure does not silently break something that used to work.

Ready was never the finish line. The evaluation set is.