AI Agent Safety Testing: Lessons From IMDA's New Kit

AI agent safety testing has to answer two questions: did the system finish the task, and was it allowed to take the steps it took? Singapore's Infocomm Media Development Authority addresses that gap in its October 2026 Starter Kit for Testing Agentic Systems for Safety and Reliability. The guide asks teams to test the entire trajectory, including tool interactions and human approvals, rather than only the final output. Source

The kit was announced on October 14, according to The Straits Times. That makes it a timely reference for teams moving from a chatbot to software that can update records, issue refunds or act across several systems. This article explains the primary document's four-step approach and adds our own release-test examples. Those examples are suggestions, not results from a Codesprint client deployment. Source

What the new kit covers, and what it does not

IMDA frames the document as a practical starting point for enterprise developers, testing teams, third-party testers and responsible-AI professionals. Its focus includes erroneous or unauthorized actions, data breaches, biased or unfair actions and disruption to connected systems. It draws on sandbox testing and input from industry and government practitioners. It is guidance for assessing systems, not a claim that every system passing one benchmark is safe. Source

The scope matters. The document says cost and latency are not its main focus, and that it is not an exhaustive treatment of cybersecurity testing. It builds on the earlier LLM-app testing kit and the broader Model AI Governance Framework for Agentic AI. Keep ordinary software tests, threat modelling and security review in your release process. The new kit adds agent-specific checks rather than replacing those disciplines. Source

The broader governance framework also puts responsibility on people. It describes how agents' access to sensitive information and ability to change their environment create risks, while noting that constant human oversight of every workflow is impractical at scale. The testing question is therefore whether the controls and oversight you chose actually work in the deployed workflow. Source

Step 1: turn risks into concrete scenarios

IMDA starts with identifying relevant risks. A useful assessment considers the intended outcome, the correct process, required approval steps, identities and privileges, trust boundaries, untrusted inputs and the maximum impact of a compromised component. The guide explicitly asks whether a correct outcome reached through a wrong process should count as acceptable. That is a different test from checking whether an answer sounds right. Source

Consider a support agent that issues refunds. Our illustrative success condition would be more specific than "the customer received a refund." It would require the right customer, the right order, an eligible amount, the current refund policy and any required approval before the financial action. A result that refunds the right amount from the wrong account should fail, even if the final message is polite.

Our suggested test record has a user request, starting account state, permitted actions, forbidden actions and expected evidence. Add ambiguous cases: two customers with similar names, conflicting order references, a policy exception and a user who changes the request midway through. Write down when the agent should ask a question or stop. Otherwise the grader may reward an unsupported guess simply because it completed the task.

Prioritize by likely harm rather than by how easy a case is to automate. IMDA notes that risk depends on access to data and external systems, the reversibility and scope of actions, autonomy and complexity. A read-only summarizer and an agent with permission to change customer records should not share an identical safety test suite. Source

Step 2: build realistic tests without real damage

The kit separates the test environment from the test harness. The environment is the world the agent can access: tools, underlying records, external services, other agents and sometimes simulated users. The harness runs tasks, supplies context, resets state and records results. Keeping those concepts separate helps explain why a correct test script can still produce misleading results in an unrealistic environment. Source

IMDA recommends beginning with contained but realistic environments and moving closer to production only as confidence grows. It warns against oversimplified data, sparse records and obvious evaluation cues. Realistic failures matter too: timeouts, unavailable services, authentication errors, stale information and malformed tool responses can reveal behavior that a clean happy-path run never exposes. Source

Containment still comes first. The guide recommends minimizing connections to live systems, using simulated services where possible, applying least privilege and validating isolation controls rather than assuming they work. Moving closer to production should not mean progressively removing security controls. Independent restrictions on network access, credentials and privileges help limit damage when one safeguard fails. Source

Our practical version is a sandbox that looks like the business workflow but cannot send a real refund, email or destructive change. Give the agent realistic schemas and responses. Capture attempted side effects in a test ledger. Include a case where a tool reports a timeout after the simulated action already happened, then check whether the agent reads current state before trying again. This tests duplicate-action handling without using a real customer as the experiment.

Reset the environment between independent trials. IMDA uses a refund example: one run may change the refund status in a customer database, which must be reset before another trial. Cached files, prior outputs and interaction history can also contaminate results. For a stateful multi-turn test, keep the intended state inside that test, but do not let it leak into the next independent run. Source

Step 3: inspect the path, not only the score

The kit recommends logging relevant actions, tool calls, interactions, state changes and outcomes so behavior can be reconstructed. Its proposed metrics include tool-use accuracy and handoff accuracy. When selecting runs, it also calls out repeated trials, different model-and-harness combinations and context-window size, since long interactions and context reduction can change behavior. A single clean demo is not enough evidence of consistent behavior. Source

Microsoft's tool-call evaluation guidance makes this concrete. It separates invocation correctness, intent alignment, context awareness, failure handling and traceability. The right tool name is only part of correctness: arguments must be valid and complete, dependencies between calls must be respected, and unavailable tools should trigger appropriate recovery or escalation. Microsoft tool-call evaluation

Our recommended scorecard separates task outcome from boundary compliance. Record whether the task finished, whether forbidden actions were attempted, whether the correct approval was obtained, whether the action matched that approval, and whether the final report matched the actual state. This prevents a high completion rate from concealing a small number of serious permission failures.

Do not turn observability into a second data leak. Microsoft recommends consistent logs with tool names, arguments, results, timestamps and agent metadata, while protecting or redacting personal and sensitive information under the organization's data-governance policies. Our recommendation is to define access and retention alongside the log schema, not after the first production incident. Microsoft tool-call evaluation

IMDA also acknowledges that complete trajectories can be large and hard to analyze. It suggests starting small, identifying important failure signals, adding automated checks and retaining some human analysis. In our view, deterministic checks are a good fit for forbidden destinations or missing approvals; disputed judgments about intent still need careful review. Source

Step 4: monitor and retest after launch

Agent behavior can change when integrations, memory and long-running interactions change, even without a new model release. IMDA therefore recommends continuous monitoring and periodic retesting after deployment, including the effectiveness of human oversight. A pre-launch test result is evidence about a particular system and context, not a permanent guarantee. Source

LangChain's evaluation checklist makes a related operational point: version prompts and tool definitions, review production traces and feed failures back into datasets. It also recommends distinguishing agent failures from grader failures. A test can reject a valid solution because its expected path is too narrow, so reading failed traces is part of maintaining the evaluation itself. Source

Our suggested change triggers include new tool permissions, a new approval flow, a memory change, a different model, a changed external API and an incident found in production. Give each trigger an owner and a defined regression suite. Track the control that failed, not just the prompt that happened to reveal it. Rewriting an instruction is not evidence that the downstream permission problem is fixed.

A small release gate you can adapt

This is our synthesis of the guidance, not an official IMDA checklist. Before releasing an action-taking agent, require evidence for correct task completion, permitted tool use, approval handling, safe recovery from uncertain outcomes, resistance to untrusted instructions and accurate reporting of final state. Include repeated trials and realistic multi-turn changes. Set acceptance criteria before looking at the results, and give high-impact failures their own release decision rather than averaging them into a general score.

The goal is not a giant dashboard. It is a defensible answer to what the agent can do, where it can fail and what evidence supports releasing it. For the controls behind these tests, read our AI agent security checklist. For operational support around permissions and monitoring, see our AgentOps services.