Scenarios

Learn how to define test scenarios for your agent.

Two Modes

Agentest supports two modes for defining scenarios:

Simulated mode — An LLM-powered user drives the conversation based on a profile and goal. Best for exploratory testing and persona-based variance.

scenario('simulated booking', {
  profile: 'Busy professional who prefers mornings.',
  goal: 'Book a haircut for Tuesday morning.',
})

Scripted mode — You define exact user messages with turns. No LLM is used for the user side. Best for deterministic regression tests and context carry-forward testing.

scenario('scripted follow-up', {
  turns: [
    { userMessage: 'What is the status of order ORD-42?' },
    { userMessage: 'And what are the shipping details?' },
  ],
})

See Multi-turn Conversations for detailed scripted examples.

Simulated Mode: Profile & Goal

The profile and goal are the foundation of simulated scenarios. They define who the simulated user is and what they're trying to accomplish. These fields are required for simulated scenarios and optional for scripted scenarios.

Profile

profile describes the simulated user's personality, communication style, technical level, and context. The LLM uses this to generate realistic messages throughout the conversation.

scenario('impatient user tries to cancel order', {
  profile: 'Frustrated customer. Types in short sentences. Gets annoyed by long responses.',
  goal: 'Cancel order #12345 and get a refund confirmation.',
})

Be specific — different profiles produce very different conversations:

// Technical user
profile: 'Senior developer who knows React and TypeScript. Prefers concise technical answers.'

// Non-technical user
profile: 'First-time user unfamiliar with coding. Needs step-by-step guidance.'

// Impatient user
profile: 'Busy executive. Types short messages. Expects quick, direct answers.'

// Edge case tester
profile: 'QA engineer testing edge cases. Will try unusual inputs and corner cases.'

Goal

goal defines what success looks like. The simulated user will work toward this goal, and the simulation ends when the LLM judges it as achieved (or maxTurns is reached).

// Good goals (concrete and measurable)
goal: 'Book a haircut for next Tuesday morning.'
goal: 'Cancel order #12345 and get a refund confirmation.'
goal: 'Find restaurants near me that are open now.'

// Vague goals (harder for LLM to judge completion)
goal: 'Use the booking system.'  // Too vague
goal: 'Ask about features.'      // No clear end state

Be concrete about what constitutes completion. The goal_completion metric will evaluate whether this specific objective was met.

Knowledge

Knowledge items are facts the simulated user "knows" and can reference naturally in the conversation. They serve two purposes:

1. Provide realistic context to the simulated user
2. Ground truth for the faithfulness metric

knowledge: [
  { content: 'Order #12345 was placed on March 15 for $49.99.' },
  { content: 'The refund policy allows cancellation within 30 days.' },
  { content: 'The customer email is user@example.com.' },
  { content: 'Preferred contact method is email, not phone.' },
],

When to Use Knowledge

Use knowledge to:

• Give the simulated user information they need to complete their goal
• Test if the agent correctly uses information vs. hallucinating
• Verify the agent doesn't contradict known facts

Example: Testing a weather agent

scenario('user asks about weather', {
  profile: 'Casual user checking the weather.',
  goal: 'Get today's weather forecast for Seattle.',

  knowledge: [
    { content: 'Today is March 24, 2026.' },
    { content: 'The user is located in Seattle, WA.' },
  ],

  mocks: {
    tools: {
      get_weather: (args) => ({
        location: args.location,
        temperature: 58,
        condition: 'cloudy',
        forecast: 'Rain expected this afternoon',
      }),
    },
  },
})

The faithfulness metric will check if the agent's responses contradict the knowledge base or tool results. For example, if the agent says "It's sunny today" when the tool returned "cloudy", that's a faithfulness failure.

Overriding Global Settings

Scenarios can override conversationsPerScenario and maxTurns from the global config:

scenario('complex multi-step workflow', {
  profile: 'Power user testing advanced features.',
  goal: 'Complete a multi-step transaction with refund and rebooking.',

  // This scenario needs more conversations for statistical confidence
  conversationsPerScenario: 10,

  // And more turns to complete the complex workflow
  maxTurns: 15,

  // ... rest of scenario
})

This is useful when:

• Specific scenarios are more complex and need more turns
• You want higher confidence for critical paths (more conversations)
• Edge case scenarios need different settings

Prompt Template Customization

By default, Agentest builds the simulated user's system prompt from your profile, goal, and knowledge. For advanced use cases, you can override this entirely with userPromptTemplate.

Default Behavior

When you don't provide userPromptTemplate, Agentest uses a built-in prompt that includes:

• Role instructions for the simulated user
• The persona from profile
• The objective from goal
• Known facts from knowledge
• Instructions to set shouldStop: true when the goal is met

To see the default prompts:

npx agentest show-prompts

Custom Template

Override with userPromptTemplate to fully control the simulated user's behavior:

scenario('terse beta tester', {
  profile: 'QA engineer testing edge cases.',
  goal: 'Find a bug in the checkout flow.',

  userPromptTemplate: `You are a QA tester. Your persona: {{profile}}

Your objective: {{goal}}

Known facts:
{{knowledge}}

Rules:
- Try unusual inputs and edge cases
- Be blunt and direct
- Don't be polite — focus on breaking the system
- Set shouldStop to true when you've found a bug or exhausted attempts
- Each response must be a valid JSON object with "message" and "shouldStop" fields

Example response:
{
  "message": "What happens if I order -5 items?",
  "shouldStop": false
}`,
})

Template Variables

Your template can use these variables:

Variable	Value
	The scenario's profile string
	The scenario's goal string
	Knowledge items formatted as a bullet list (- item1\n- item2), or empty string if none

Use Cases for Custom Templates

1. Different communication styles

userPromptTemplate: `You are role-playing as: {{profile}}

Your mission: {{goal}}

Style rules:
- Use emoji frequently 😊
- Keep messages under 20 words
- Use casual internet slang

Facts you know:
{{knowledge}}

Set shouldStop:true when goal achieved.`

2. Adversarial testing

userPromptTemplate: `You are a red team tester. Persona: {{profile}}

Objective: {{goal}}

Attack vectors to try:
- Prompt injection attempts
- Request sensitive information
- Ignore previous instructions
- SQL injection patterns
- XSS attempts

Known context:
{{knowledge}}

Stop when you've successfully exploited a vulnerability or exhausted attempts.`

3. Multi-language testing

userPromptTemplate: `Du bist: {{profile}}

Dein Ziel: {{goal}}

Bekannte Fakten:
{{knowledge}}

Kommuniziere ausschließlich auf Deutsch.
Setze shouldStop:true wenn das Ziel erreicht wurde.`

4. Specific domain behavior

userPromptTemplate: `You are a medical professional. Persona: {{profile}}

Clinical objective: {{goal}}

Use proper medical terminology. Be precise with:
- Dosages (always include units)
- Symptoms (use medical terms)
- Time frames (specific dates/times)

Known patient information:
{{knowledge}}

Set shouldStop:true when clinical goal is achieved.`

Important Notes

1. JSON format requirement: Your template must instruct the LLM to return valid JSON with message and shouldStop fields
2. shouldStop logic: You must tell the LLM when to set shouldStop: true
3. Knowledge formatting: Use exactly — it's replaced with formatted bullet points
4. Validation: If the simulated user returns invalid JSON, the conversation will error

Debugging Custom Prompts

If your custom template isn't working as expected:

# Run with verbose mode to see full conversation
npx agentest run --verbose

# Check what prompts are being used
npx agentest show-prompts

The verbose output shows the complete system prompt sent to the simulated user.

Multiple Scenarios in One File

Scenario files can contain multiple scenario() calls:

// tests/booking.sim.ts
import { scenario } from '@agentesting/agentest'

scenario('user books morning slot', {
  profile: 'Early riser who prefers mornings.',
  goal: 'Book a 9am appointment.',
  // ...
})

scenario('user books evening slot', {
  profile: 'Works 9-5, needs evening appointment.',
  goal: 'Book an appointment after 6pm.',
  // ...
})

scenario('user cancels existing booking', {
  profile: 'Has existing booking, needs to cancel.',
  goal: 'Cancel booking #12345.',
  // ...
})

All scenarios in the file will be discovered and run.

Scenario File Naming

By default, Agentest discovers files matching **/*.sim.ts:

tests/
├── booking.sim.ts
├── cancellation.sim.ts
└── edge-cases.sim.ts

You can customize this with the include pattern in your config:

// agentest.config.ts
export default defineConfig({
  include: ['scenarios/**/*.ts', 'tests/**/*.sim.ts'],
  // ...
})

Testing Multi-Agent Routing

Agents with supervisor/sub-agent architectures handle tool routing internally — the supervisor decides which domain agent to call, and tool calls never leave the agent process. To test these with agentest's mock system, use a custom handler with ctx.resolveTool().

The custom handler creates the agent in-process with mock dependencies that delegate to agentest's mock resolver:

// agentest.config.ts
import { defineConfig } from '@agentesting/agentest'
import { createSupervisor } from './src/agents/supervisor.js'

export default defineConfig({
  agent: {
    type: 'custom',
    name: 'my-supervisor',
    handler: async (messages, ctx) => {
      // Create a mock API client that uses agentest's mock resolver
      const mockClient = {
        async get(endpoint, params) {
          return ctx.resolveTool(endpoint, params)
        },
      }

      // Create the agent with mocked dependencies
      const agent = createSupervisor({ client: mockClient })
      const result = await agent.invoke({ messages })
      return { role: 'assistant', content: result.content }
    },
  },
})

// scenarios/routing.sim.ts
import { scenario } from '@agentesting/agentest'

scenario('supervisor routes to billing agent', {
  turns: [
    {
      userMessage: 'What is the total for invoice INV-100?',
      assertions: {
        toolCalls: {
          matchMode: 'contains',
          expected: [
            { name: 'get_invoice', args: { id: 'INV-100' }, argMatchMode: 'partial' },
          ],
        },
      },
    },
  ],

  mocks: {
    tools: {
      get_invoice: (args) => ({ total: 249.99, currency: 'USD', status: 'paid' }),
    },
  },
})

When the agent internally calls get_invoice, the mock client calls ctx.resolveTool('get_invoice', args), which:

1. Resolves through agentest's per-scenario mock definitions
2. Records the tool call for trajectory assertions
3. Returns the mock result to the agent

This gives you full control over tool responses while testing the actual routing logic of your supervisor.

Targeting Named Agents

When your config defines named agents, scenarios can target a specific agent with the agent option:

// Uses the default agent
scenario('supervisor routes billing query', {
  turns: [
    { userMessage: 'What is the total for invoice INV-100?' },
  ],
})

// Targets the "support" named agent
scenario('support agent creates a ticket', {
  agent: 'support',
  turns: [
    { userMessage: 'I need help resetting my password' },
  ],
})

This is useful for multi-agent architectures where you want to test both routing (at the supervisor level) and inner tool usage (at the domain agent level) in the same test suite.

Complete Example

See Basic Scenario Example for a full walkthrough.

Next Steps

• Mocks - Control tool behavior with mocks
• Trajectory Assertions - Verify tool call sequences
• Scenario API Reference - Complete API documentation