The AI Test Manager Is Becoming the Control Layer for GenAI Delivery

From the way AI testing roles are now being described, I think we are moving past niche experimentation and into a serious branch of QA leadership.

Not because every company suddenly understands AI quality. Many do not. That is exactly the point.

A Test Manager for AI, GenAI, and agentic systems is not there to admire model outputs, collect vendor dashboards, or approve a prompt because it worked in a demo. The role exists because AI-enabled products introduce a different kind of delivery risk: probabilistic behaviour, unstable outputs, opaque decision paths, data dependency, prompt sensitivity, tool-use errors, and a very persuasive tendency to sound correct when it is not.

That last part matters.

Traditional QA has spent decades learning not to confuse a clean UI with a working system, not to confuse passed regression with customer confidence, and not to confuse a dashboard with evidence. Those instincts are not outdated in the age of AI. They are more useful now.

The mistake would be treating AI test management as a replacement for established QA practice. It is not. It is an extension of it into a more volatile delivery surface.

AI quality is not only a model problem

A lot of AI testing discussion still gets stuck at model evaluation.

Accuracy. Relevance. Groundedness. Toxicity. Hallucination rate. Bias. Context precision. Context recall.

These are useful measures, but they do not describe the whole product risk.

Most enterprise AI products are not just “a model”. They are systems. A GenAI feature may include a product workflow, a user interface, a prompt chain, a retrieval layer, a vector database, access controls, document ingestion, orchestration logic, moderation rules, audit logging, fallback behaviour, and production monitoring. An agentic workflow adds another layer again: tool selection, tool argument extraction, multi-step planning, memory, handoffs, and the ability to take action outside the chat window.

That means the Test Manager still needs the usual QA muscles: test strategy, traceability, risk assessment, defect triage, release readiness, stakeholder reporting, UAT coordination, and governance. The difference is that the evidence now has to cover both deterministic software behaviour and probabilistic AI behaviour.

OpenAI's evaluation guidance makes the same point: generative AI is variable, the same input can produce different outputs, and traditional software testing is not enough on its own for AI architectures. It also frames evals as evidence to combine with human judgement, not as self-contained truth.

That is the seam where the AI Test Manager role becomes valuable.

Not as “the person who knows promptfoo”. Not as “the Selenium person, but with AI”. As the person who can design an evidence system that product, engineering, risk, security, compliance, and business stakeholders can actually use.

The market is already describing this role, even if the wording is still messy

Recent AI testing and QA leadership job descriptions are still inconsistent, but the pattern is becoming easier to recognise. They combine AI/ML testing, GenAI validation, responsible AI, privacy, security, automation, performance, UAT, quality gates, and release readiness.

That mix is not accidental. It shows that AI delivery risk does not sit neatly inside one team. The quality problem crosses product behaviour, model behaviour, data, security, compliance, release governance, and production support.

NIST's AI Risk Management Framework positions AI risk management as something organisations need across the design, development, deployment, and use of AI systems, with the goal of promoting trustworthy and responsible AI. Its Generative AI profile builds on that for GenAI-specific risks and lifecycle considerations.

The EU AI Act also makes the direction of travel hard to ignore. It defines a risk-based approach, includes strict obligations for high-risk AI systems, and lists items such as risk assessment, dataset quality, logging, documentation, clear information to deployers, human oversight, robustness, cybersecurity, and accuracy.

For QA people, this should sound familiar.

Not because AI governance is “just testing”. It is not. But because the discipline of turning vague risk into testable evidence is already part of serious QA work. We have been doing versions of this in regulated delivery, payments, finance, healthcare, enterprise SaaS, and safety-sensitive systems for years.

AI changes the object under test. It does not remove the need for professional scepticism.

The useful tooling landscape is layered, not linear

There is no single 'AI Test Manager tool', because the tooling landscape is a stack. Some tools manage delivery evidence. Some exercise the product. Some evaluate model behaviour. Some observe production traces. Some test data quality. Some support bias and fairness analysis. Some red-team the system. Some enforce guardrails. Some help with performance and reliability.

A mature AI testing setup will usually need several of these layers, selected according to the product risk, autonomy level, domain, and release obligations

This is why the role needs senior QA judgment. Tool selection is not the hard part. Knowing what evidence each tool can and cannot provide is the hard part.

A prompt evaluation tool can show that a prompt version performs better against a curated dataset. It cannot prove that the product is ready for a regulated release.

A red-team report can expose prompt injection risks. It cannot tell you whether business users understand when an answer is uncertain.

A trace can show that an agent called the wrong tool. It cannot decide whether that failure blocks release unless someone has already defined severity, customer impact, recovery, and acceptable residual risk.

A fairness metric can indicate disparity. It cannot tell an organisation what tradeoff is acceptable without policy, domain context, and human accountability.

The tooling helps, but it does not make the release decision.

Not every AI-enabled product needs the same level of assurance. A low-risk summarisation feature, an internal assistant, a regulated customer-facing recommendation engine, and an agent that can trigger downstream actions should not have the same evidence burden. The Test Manager’s job is to scale the assurance model to the actual risk.

Traditional test management becomes the evidence backbone

It is tempting to see AI testing as something that lives in notebooks, experiment dashboards, or model monitoring tools.

Some of it does.

But enterprise delivery still needs a control plane: requirements, risks, test scope, execution evidence, defect status, release notes, approvals, and audit trail. That is where Jira, Azure DevOps, Xray, Zephyr, TestRail, qTest, and similar tools remain relevant.

The AI-specific work should not sit in a disconnected experiment folder that only the ML team understands. If a GenAI feature is going into production, its AI risks need to be visible in the same delivery conversation as functional defects, security findings, performance constraints, unresolved dependencies, UAT feedback, and release readiness.

This does not mean forcing every eval result into a test case management tool as if it were a manual regression step. That would only create process theatre.

It means the Test Manager should be able to link the evidence chain:

• What user or business outcome are we validating?

• What AI-specific risks are in scope?

• Which datasets, prompts, scenarios, logs, or traces support the assessment?

• Which checks are automated, which are human-reviewed, and which are sampled?

• What thresholds or review criteria are being used?

• What defects or risks were raised from the evidence?

• What is still unknown at release decision time?

For example, an internal RAG assistant should not only be tested on whether it gives fluent answers. The evidence chain should show which document set was used, whether permission boundaries were respected, how stale or conflicting sources are handled, what happens when the answer is uncertain, and how users can challenge or escalate a poor response.

That is recognisable QA work. The difference is that the test artefacts now include prompt suites, model comparisons, RAG evaluation datasets, trace analysis, red-team findings, bias reports, and production monitoring plans.

Functional automation still matters, but it is not the whole story

There is a strange habit in AI conversations of acting as if normal software testing has disappeared.

It has not.

The AI feature still needs to load. Buttons still need to work. API contracts still need to behave. Permissions still need to be enforced. The user still needs the right result in the right place. Regression still matters. Accessibility still matters. Compatibility still matters. Performance still matters.

Selenium still matters because browser automation remains part of the product evidence. Playwright is also moving into a wider role: not only test automation, but scripting and AI-agent workflows, including structured accessibility snapshots through its MCP server.

That is interesting because the boundary between test automation and agent automation is becoming less clean. The same product surfaces that QA teams used to automate for regression may now also be used by AI agents to navigate workflows.

For an AI Test Manager, this creates two responsibilities: keep normal automation honest, and know where deterministic checks stop. Do not let the AI label distract from the boring failures: broken flows, missing validation, bad error handling, unstable environments, weak API contracts, inconsistent permissions, or untested edge cases.

Second, recognise where deterministic checks stop. A Playwright test can verify that a user can submit a prompt and receive a response. It cannot, by itself, tell you whether the response is grounded, safe, compliant, complete, or appropriate for the user's context.

That is where evals come in.

Evals are becoming the new regression suite, but they need QA discipline

LLM evals are becoming one of the most important regression practices for AI-enabled delivery.

They let teams compare prompts, models, retrieval strategies, tool-use logic, and workflow changes against known scenarios. They can help detect regressions when a prompt changes, when a model is upgraded, when documents are re-indexed, or when an agent is given access to a new tool.

Tools like promptfoo are designed for evaluating and red-teaming LLM applications, with support for prompts, models, RAG pipelines, metrics, CI/CD usage, and side-by-side comparisons.

LangSmith supports offline evaluation before shipping and online evaluation against production interactions, with evaluators that can include human review, code rules, LLM-as-judge, and pairwise comparison.

Arize Phoenix focuses on AI observability and evaluation, including traces, datasets, experiments, prompt iteration, and scoring outputs to identify failures and regressions.

These tools are genuinely useful. They also need careful handling.

An eval suite is only as good as its scenarios, labels, thresholds, and review logic. A polished scorecard can still be weak evidence if the dataset is too small, too generic, too clean, too synthetic, or too far removed from production usage.

This is where traditional QA experience becomes a serious advantage.

A good Test Manager will ask awkward but necessary questions.

Is the eval dataset representative of actual user behaviour, or only of happy-path examples invented during development?

Do we have adversarial cases, ambiguity, missing context, multilingual input, low-quality documents, permission boundaries, and business-critical edge cases?

Are we testing generated text only, or also retrieval quality, tool selection, action execution, fallback behaviour, and refusal behaviour?

Do the scores correlate with human review, or are we using an LLM judge because it is convenient?

What is the release threshold, and who agreed it?

What happens when the model improves one metric and worsens another?

This is not anti-AI. It is basic test design applied to a probabilistic system.

Agentic AI needs behaviour testing, not just answer testing

Agentic systems raise the risk level because they do more than generate a response.

They choose. They route. They call tools. They retrieve data. They may update records, create tickets, send messages, trigger workflows, or make recommendations that users treat as decisions.

OpenAI's evaluation guidance separates simpler interactions from workflows, single-agent architectures, and multi-agent architectures, and identifies new evaluation points such as tool selection and whether the agent calls a tool with the correct arguments.

For QA, that moves the test focus from answer quality to behaviour quality.

For an agent, the test question is not only “was the final answer good?” It is also:

• Did it understand the user's intent?

• Did it ask for missing information instead of guessing?

• Did it select the right tool?

• Did it pass the right arguments?

• Did it respect permissions?

• Did it stop when it should stop?

• Did it recover from tool failure?

• Did it expose uncertainty clearly?

• Did it avoid taking irreversible action without confirmation?

• Did it leave an audit trail?

This is where agent testing starts to look closer to workflow testing, integration testing, security testing, and release governance than to prompt tweaking.

The Test Manager's job is to make those behaviours visible before production, then monitor them after production.

Responsible AI testing is not a checkbox category

Most AI job descriptions now mention responsible AI: fairness, bias, explainability, privacy, and security.

That is useful language, but it becomes meaningless quickly if nobody can explain what was assessed, against which population, using which evidence, and with what limitation.

A team can run a bias tool and still not understand whether the test population reflects the affected users. A team can generate an explainability chart and still not know whether the business can act on it. A team can apply a guardrail and still leak sensitive information through a retrieval path. A team can write “human in the loop” and still leave the human with no usable evidence, no decision rights, and no time to intervene.

ISTQB's AI Testing certification explicitly covers testing AI-based systems, including machine learning and generative AI systems, and calls out characteristics such as probabilistic behaviour, non-determinism, and reliance on data. Its Generative AI testing certification covers use of LLMs across the test process and includes risks such as hallucinations, bias, security, and privacy.

That framing is useful because it keeps AI testing connected to both engineering reality and professional testing practice.

The responsible AI tooling layer matters. Fairlearn and IBM AIF360 can support fairness assessment. SHAP and LIME can help with explainability for certain model types. Great Expectations, Soda, and Evidently can support data quality and drift checks. OWASP's LLM guidance, Microsoft PyRIT, garak, Giskard, NeMo Guardrails, and promptfoo red teaming can support security and adversarial testing.

But a tool finding is not a governance decision.

For AI Test Managers, the stronger position is not “we tested for bias”. It is: “Here is the fairness risk we identified, here is the population and data slice we assessed, here is the metric we used, here is the limitation of that metric, here is what changed as a result, and here is what remains a release risk.”

That is the level of evidence senior stakeholders need.

Security testing has moved closer to product behaviour

With GenAI, security is not only infrastructure, authentication, dependency scanning, or API exposure.

It is also behaviour.

Prompt injection, jailbreaks, insecure output handling, sensitive information disclosure, overreliance, vector database leakage, tool misuse, and excessive agency are product risks. They may show up through normal user interaction, not just through classic penetration testing routes.

OWASP's Top 10 for LLM Applications is useful because it translates LLM-specific risks into a security language teams can work with. It includes categories such as prompt injection, sensitive information disclosure, supply chain issues, improper output handling, excessive agency, and vector and embedding weaknesses.

This is another area where QA and security need to work closely rather than sequentially.

The Test Manager does not need to replace AppSec. But the role needs enough security awareness to make LLM security part of the test strategy, not a late-stage specialist activity that happens after the product behaviour is already shaped.

For example, if an internal knowledge assistant can retrieve documents, QA should be asking about permission boundaries and retrieval leakage early. If an agent can create Jira tickets, send emails, or update customer records, QA should be asking about confirmation, auditability, rollback, and abuse scenarios. If generated output is displayed in a browser or passed downstream to another system, QA should be asking about output handling and injection paths.

This is not separate from quality. It is part of whether the system can be trusted in use.

Observability is part of the test strategy now

AI systems need post-release evidence because pre-release testing cannot cover all production behaviour.

Shift-right testing, monitoring, incident review, and production validation have been part of mature delivery for years. What changes with GenAI is the type of production evidence the team needs.

For GenAI and agentic systems, a useful production trace may need to show prompt version, model version, retrieved context, tool calls, tool arguments, latency, token usage, safety filters, fallback paths, user feedback, and final output.

Without that, defect triage becomes guesswork.

A user says, “The assistant gave me the wrong answer.” That is not enough. The team needs to know whether the issue came from the prompt, model behaviour, missing context, stale retrieval content, access control, embedding quality, document parsing, tool failure, user ambiguity, or a genuine gap in expected behaviour.

This is why tools like LangSmith, Arize Phoenix, OpenTelemetry-based tracing, MLflow, Weights & Biases, and cloud-native AI monitoring matter. They help teams move from anecdote to evidence.

But observability should not be owned only by engineering. QA needs to define which production signals matter for quality.

For an AI feature, that may include:

• unresolved or low-confidence responses

• repeated fallback usage

• tool-call failure rate

• retrieval misses

• high-risk prompt categories

• user corrections

• human override frequency

• latency and timeout patterns

• safety filter triggers

• incidents caused by generated output

The useful question is not 'do we have monitoring?' It is: 'which production behaviours would change our release confidence, our support response, or our next regression suite?'

The AI Test Manager should own the evidence model

This is where the role becomes genuinely interesting: someone has to connect delivery reality with AI risk.

The core responsibility is not to own every AI tool or replace every specialist. It is to define the evidence model: what risks matter, what evidence is credible, what remains unknown, and what decision the business is being asked to make.

The AI Test Manager should be able to define an evidence model that looks something like this:

This is not a template for bureaucracy. It is a way to avoid over-confidence.

Because AI products can fail politely. They can fail fluently. They can fail in ways that look acceptable until a domain expert checks the detail. They can pass a generic benchmark and still be unsafe in the business context that matters.

That is why QA leadership belongs in the room.

The career opportunity for senior QA is real, but it is not automatic

Senior QA professionals have a credible path into AI test management because the discipline already rewards scepticism, risk thinking, evidence design, stakeholder communication, defect analysis, and release judgement.

But that transition will not happen by simply adding ‘AI testing’ to a CV. The credibility has to come from understanding how AI-enabled systems fail, how evidence is built, and how release risk is explained.

The useful skill set is more specific:

• understanding how LLM applications are built, including prompts, RAG, embeddings, context windows, model selection, tools, agents, and orchestration

• knowing how evals work, including datasets, metrics, human labels, LLM-as-judge, thresholds, regression comparison, and CI/CD integration

• being able to test agentic behaviour, not only generated answers

• understanding responsible AI risks well enough to coordinate with data, legal, security, compliance, and product teams

• keeping traditional QA disciplines visible: test planning, traceability, automation, defect governance, UAT, release readiness, and production validation

• translating technical AI risk into stakeholder decisions without hiding behind jargon

That last skill will matter more than people think.

A model engineer may understand the architecture better. A security specialist may understand adversarial testing better. A data scientist may understand model metrics better. A product owner may understand the user problem better.

The AI Test Manager earns her place by connecting those views into a practical quality strategy and making the release decision less vague.

The real risk is performative assurance

The weakest version of AI test management will look very familiar.

A dashboard with many scores and no decision criteria.

A prompt evaluation suite full of happy paths.

A fairness check nobody can explain.

A human approval step where the human has no evidence and no authority.

A sign-off that says “AI validation completed” without saying what was validated, against which risks, using which data, by whom, and with what known limitations.

This is the AI version of old-school QA theatre. Different vocabulary, same smell.

The better version is more demanding. It treats AI testing as a layered evidence problem. It uses tools, but does not worship them. It automates where the checks are repeatable, samples where full review is not practical, escalates where judgment is required, and keeps release risk visible.

That is where senior QA professionals can build real authority in this field.

Not by pretending AI can be tested exactly like traditional software.

By knowing which parts still can, which parts cannot, and how to build a governance model around the gap.

If I were assessing an AI-enabled feature for release, the first thing I would want to see is not the model name or the prompt document. I would want to see the evidence chain: the risks, the scenarios, the evals, the human review points, the defects, the monitoring plan, and the decision criteria. If those are missing, the team is not ready for a confident release. They are ready for a better demo.

Disclaimer: The perspectives expressed herein are personal interpretations intended to foster professional dialogue; they do not represent any official stance of current or former employers.