On the UX Heroes Podcast: Structured Data, Metrics, and AI-Moderated Interviews

I joined Markus Pirker on UX Heroes, the podcast by Userbrain, to talk through how I actually run research: structured versus unstructured methods, the metrics that survive contact with a business decision, and an eight-month side project where I built my own AI interviewer. This is a written recap of that conversation, organized by theme.

The tension every moderated interview lives with

After more than a thousand moderated tests and interviews, my biggest personal learning is not a technique. It is a tension I have learned to live with rather than solve.

I love a precise plan. Nothing feels better than a study that is fully specified in advance. But the whole point, and the fun, of a moderated interview is that you can follow the person, react to what they say, and drift from the plan when it matters. That runs directly against how I work: methodically, always optimizing for data aggregation and structure. There is no clean trade-off here. The unstructured moments inside a structured session are sometimes worth the most, and you decide afterward how to handle those deviations in the analysis. It used to stress me. Now I can live with it.

What structure really means

People immediately understand structure when I compare it to a survey. In a survey you define the questions people receive. A test or an interview is the same idea: everything you ask, everything you want to observe, every criterion (for example, whether someone actually completes an e-commerce checkout) has to be decided in advance. In what structure, in what order, with what kind of help along the way.

This is independent of who or what runs the session, moderated, unmoderated, or AI-moderated. The roots are in psychology and experimental behavioral science: an experiment needs a fixed setup because you are measuring people's reactions, and the input has to be the same for everyone. Even the real-life moments can be planned. Probing is the classic one: where do you follow up, where do you help, where do you give a hint. That applies to unmoderated tests too, because you still have to decide the instructions people get, and whether spoilers or hints appear after a certain point.

Applied UX research is more open than a purely academic qualitative interview, but the goal is different. You are supplying a basis for decisions and solving a problem, not generating knowledge for its own sake. There is no standard script. You start from templates, but what carries the work is whether stakeholders trust the data: did you build rapport, pick the right participants, ask questions that trace back to the real business case. Rigor and method matter enormously, but in a stakeholder meeting they take a back seat. The central discussion is the design problem or the monetization problem, not your confidence intervals.

Why I am wary of time on task

Time on task is a metric I collect when it is asked for, but I am honest that I do not love it, for several reasons.

Pure quantitative data gives me few hooks for interpretation. It is hard to deliver a genuinely good insight from a chart alone. The distributions are awkward: many people finish very fast with little variance, and a few take very long, so the tail stretches out and you spend ages hunting for artifacts (did that person abandon the test, or were they just slow?). Then there is the golden question I still cannot answer well: what actually is a good time on task? Benchmarks exist at large companies, but fewer than people think. And defining when a task starts and ends sounds trivial until you try it on a real digital product. What is allowed to happen in between? Can people take a different valid path? If so, the numbers are no longer comparable.

The same tension shows up between thinking aloud and time on task. Ask everyone to narrate their thoughts and some will, some will not, half go quiet and half tell you their life story when something on screen reminds them of it. That is human, and it should be allowed. But it contradicts a strict timed plan. You cannot tell someone that everything they say is welcome and also ask them to only talk about this one flow because you are timing it. So you accept the messiness and interpret accordingly. The same caution applies when a sample of five to ten people shows a 20% time-on-task improvement and it gets celebrated as significant. The small-sample statistics debates are fun for the research community and for podcasts like this one, but they have little impact on the daily reality of small and mid-size companies.

The two metrics I run in every test

If you ask me which metric has the largest impact on practice, my 2026 answer is one I would not have predicted ten years ago, and it is still the good old System Usability Scale.

The SUS is over thirty years old and fairly antiquated, with plenty of known issues. But among UX metrics it has everything. Once you move away from time on task into the psychometric direction (aesthetics, flow, perceived usability), the SUS is short, it is a single factor, so you never have to explain how you get from a set of questions to a score, and it produces one number from 0 to 100. It carries the largest benchmark behind it, with adjective anchors like "excellent" mapped to the points, it is easy to cite, there are large accessible databases to benchmark against, and many people know the critical value (68) by heart. It even has a well-known visualization. I use it in every single test.

The second is the Net Promoter Score, and this one I argue about more, because a lot of UX people hate it. The methodological criticisms are partly justified: it is over-claimed, it is not diagnostic for usability problems. But the argument for it mirrors the SUS. People know it. Business decision-makers know it off the top of their head. SUS and NPS together are short, almost everyone in the field recognizes them or can look them up in seconds, and that wide adoption gives you credibility and comparability across services, industries, and competitors. That sheer market power is what makes them my go-to setup, and I do not see them being replaced any time soon.

Order matters: SUS first, then NPS. The reason is psychological. The usability-related variable should come before the business-oriented one, and the likelihood-to-recommend question belongs at the very end. Methodologically, what matters most is that the order stays the same across all participants. I no longer engage the old meme about not recommending an operating system at a dinner party. The point of these scores is a lowest common denominator that is comparable, not a prediction of the future, and my expectations of any single metric are modest, because no tool predicts the future as precisely as people wish it would.

Structured data is the foundation

Markus and I first talked about structured data in an earlier Userbrain webinar, and it is the throughline of everything above. Unmoderated user tests are, in a sense, drilled specifically to produce structured data.

The problem is that people without research training naively assume a structure that, in the age of AI, leans heavily on transcripts and summaries. The middle is missing: it is not defined when which metric or which question is captured. You end up with a summary, and people intuitively like summaries. Summaries are fine, but they are only a summary. The information that lets you tell a stakeholder "this is not based on vibes" is exactly what gets lost.

The concept I keep coming back to is tidy data: normal spreadsheets, with columns, rows, and cells. It sounds unbelievably trivial, and I can hardly believe I say it out loud in lectures. But the craft is deciding precisely which variable, observation, or question lives in a column, that the person is a row, and what defines a cell, where it starts and where it ends. Because in the end (and this is painful for my academic heart, since it applies even to qualitative work) things have to be counted, and every statement has to be traceable back to where it came from. Only a structure that dictates this tells you which indicator you can aggregate, and how often a given theme or issue actually appeared.

For unmoderated tests this matters even more. The whole appeal is that you should not have to watch every video. I am paranoid enough to watch them anyway, that is the researcher in me wanting to know whether it really happened that way. And I will admit I enjoy features like clip highlights and pre-structured moments that let me jump to the right point. Experienced people can watch maybe 20% of the videos and leave the rest to the system, but structure is what makes that safe: when I am unsure about a specific question, I only need to review the sections about that issue. Without structure, moderated or not, do not bother running the test. In the end, nobody will believe you.

Building an AI interviewer, and why it can be worse than a survey

Some clients asked me which AI-moderated interview platform was the best. I am always skeptical of anything handed to me as a finished black box, so I researched the tools, tried them, and eventually decided to build my own. We live in an era that promises anyone can build anything through prompts, so I thought: fine, I will.

The twist is that a real developer eventually had to come in and clean up my entire vibe-coded mess to make it work at all, and I was banned from pure vibe-prompting. It was still fun. Over about eight months (it started as a one-month idea last summer and escalated into a nights-and-weekends project) we interviewed 60 people across different groups and experimented with several approaches.

My conclusion: for now, AI-moderated interviews are still worse than a survey, because the simplest form of an AI-moderated interview is an interactive survey. You say something, you get a question, you say something, you get a question, exactly like logging into your favorite chat tool. A real qualitative interview is different, because it adds the ability to probe, follow up, and go deep on a problem. On an abstract level, a branching survey that goes deeper at certain points moves in that direction. We also explored what large language models add: judging whether a topic is exhausted, checking for saturation, and a diary-style variant where people answered by voice message and got follow-ups by text. The hardest part by far is real-time conversation: detecting when someone is finished, natural turn-taking, handling interruptions. The more models and checks you put in between to raise quality, the more latency the participant feels.

The recurring problem across all three approaches was traceability. When I only get an AI-generated summary and cannot trace the conversation in detail, I cannot authentically tell a client to invest serious money in changing things. We UX people often think only up to a certain point. We pour love into the study setup and forget that the research budget is nothing compared to the budget that has to be released if everyone believes the recommendations. Honestly, I focused so hard on the methodology and the technology that I could not present the outcomes as convincingly as research where I had at least watched some videos or had real contact with people. In a survey it is trivial to count how often a tag or theme appeared. With probing-magic and LLM-magic doing things I can no longer fully trace, I cannot convincingly justify how I reached a conclusion.

Yes, people tell me I am missing the point of a qualitative interview, that it is not about tracing and counting. They are partly right. But as the researcher I am still always required to make my recommendations crystal-clear and traceable when people invest budget, time, and effort, and the developers who implement them have every right to ask whether I am sure. An AI finds it easy to say "you should change the whole business model." What that means for the business is irrelevant to the AI, and my role is not just to read out a summary. The LinkedIn post that started this was written provocatively. There are real differences from a pure survey, and this will change as the technology improves and trust builds. I am still on the topic, still figuring out how to integrate it.

The hard rules that made AI probing usable

The most useful lessons were the constraints, not the cleverness.

We had runs where the AI meant well and probed a question into the ground. I spent two weeks trying to find a safe way to detect saturation, and the answer that worked was a hard rule: probe a question at most twice, that is it. A second rule followed: the probes have to be especially similar to each other, so the answers stay aggregatable.

The funniest learning shows the direction all of this is heading. We ran the study in batches rather than all 60 people at once, because I knew the early runs would be rough. The more test interviews we did with the guide the AI eventually used, and the more real answers I heard, the better I understood where the probes should go, and the less I had to leave to LLM-magic. It ended as essentially a decision tree, with the AI providing the conversational fluff so it felt conversational, while making no real decisions, choosing between two or three options at most. And that is exactly when I felt better as a researcher, because I could control it. Control is a good, and necessary, word for AI right now.

Key takeaways

1. Structure and flow are a tension, not a trade-off. Plan precisely, then decide afterward how to handle the deviations that made the session valuable.
2. Probing is designed, not improvised. Decide in advance where you follow up and where you help, whether the session is moderated, unmoderated, or AI-run.
3. Keep the interaction clean. Put the post-experience interview in one block after the task, since intervening mid-flow pulls people out of natural behavior.
4. Be honest about time on task. Collect it when required, but treat maybe 80% of it as signal and the rest as noise.
5. SUS then NPS, every time. Not because they are perfect, but because their reach buys credibility and comparability.
6. Tidy data is the precondition for being believed. Columns, rows, cells, and every statement traceable to its source.
7. AI probing needs hard limits. At most two probes per question, kept similar enough to aggregate, with the model doing conversational fluff, not decisions.

Go deeper

The AI-interviewer story continues in AI-Moderated Interviews: The Rag Rug Data Problem, which explains why inconsistent adaptive probing destroys aggregatability. For the metrics side, see UX Measurement Instruments on the SUS and other validated scales. And for the human skills that structure cannot replace, see The Art of Moderation.