Test agents

Agents are non-deterministic, so testing them starts with exercising behavior, not writing a perfect assertion. The fastest loop is often to have your coding agent prompt the Luno OS agent through the CLI, try realistic scenarios, inspect the resulting sessions, and iterate on the system instructions and tools. Test agents at the boundary you care about:

Test style	Use when
Qualitative CLI runs	You want to see how the agent behaves across realistic prompts, follow-ups, and tool-use cases
Definition tests	You want to verify tools, model, prompt, or metadata without calling an LLM
Prompt smoke tests	You want to verify the agent can run against a real model
Tool-use tests	You need evidence that the agent called a required action or subagent
End-to-end tests	You want to run the project server and test API behavior
Evals	You want to score reply quality (relevancy, rubric, keywords) against a threshold

Start qualitatively, then turn the stable contracts you discover into tests. Automated tests are best for definition shape, required tool calls, and a small number of critical prompt paths. Run tests from the project root:

luno test --project unit          # src/**/*.test.ts
luno test --project integration   # src/**/*.int.test.ts
pnpm test                              # same — calls luno test
luno evals run                    # src/**/*.eval.ts (real models, on demand)

Vitest ships with @meetluno/cli. No project vitest.config.ts is required.

Qualitative tests

Before writing test files, run the agent the way you expect people to use it. Ask your coding agent to call the Luno OS CLI with a batch of prompts, inspect the sessions, and report where the agent misunderstood instructions, skipped tools, or used tools incorrectly.

luno agents prompt support --message "A customer asks whether they can get a refund after 45 days." --wait
luno agents prompt support --message "Look up order ORD-123 and decide whether it is refundable." --wait
luno agents prompt support --message "Draft a concise Slack reply for that customer." --wait

--wait returns each completed session for review. For long prompts, choose a bounded wait with --timeout <seconds> and continue with agents sessions get <agent> <session-id> --wait if the session is still running. Good qualitative prompts cover:

Normal requests the agent should handle cleanly.
Tool-required requests where the agent must call an action, workflow, MCP tool, or subagent.
Missing-information cases where the agent should ask a clarifying question instead of guessing.
Follow-up messages in the same session.
Edge cases that should be refused, escalated, or handled cautiously.

Then inspect the session — start with a summary, then request richer detail if needed:

luno agents sessions get support <session-id> --summary
luno agents sessions get support <session-id> --include messages,events,trace

Use this loop to tune the system prompt, tools, skills, files, and model choice. Once the behavior feels right, write focused tests for the parts that should not regress.

Definition tests

Definition tests are fast and do not need provider keys.

import { describe, expect, it } from "vitest";
import support from "./support";

describe("support agent", () => {
  it("uses the expected model and support files", () => {
    expect(support.slug).toBe("support");
    expect(support.model).toBe("openai/gpt-5.6-sol");
    expect(support.systemPrompt).toContain("/workspace");
  });
});

Use this style to catch accidental model changes, missing tools, or prompt edits that remove required instructions.

Smoke-tests

The init template includes an agent integration test shaped like this:

import { describe, expect, it } from "vitest";
import hello from "./hello";

describe("hello agent", () => {
  it.skipIf(!process.env.AI_GATEWAY_API_KEY && !process.env.OPENAI_API_KEY)(
    "responds to a prompt",
    async () => {
      const result = await hello.prompt({ message: "Say hi in one word." });

      expect(result.messages.some((message) => message.role === "assistant")).toBe(true);
    },
  );
});

The provider-key guard keeps local and CI runs from failing when real model credentials are not available. Match the guard to the agent’s model (or use AI_GATEWAY_API_KEY, which works for any catalog id). Run integration tests from the project root:

luno test --project integration
# or
pnpm test -- --project integration

Integration tests load .env when present and skip when required keys (like AI_GATEWAY_API_KEY) are unset. Vitest and its config ship with @meetluno/cli — no project vitest.config.ts.

Tool use tests

When a tool call is the contract, assert on the recorded messages rather than only the final answer.

import { describe, expect, it } from "vitest";
import orchestrator from "./orchestrator";

function usedTool(messages: Array<{ role: string; toolName?: string }>, toolName: string) {
  return messages.some((message) => message.role === "toolResult" && message.toolName === toolName);
}

describe("orchestrator agent", () => {
  it.skipIf(!process.env.AI_GATEWAY_API_KEY && !process.env.OPENAI_API_KEY)(
    "delegates research",
    async () => {
      const result = await orchestrator.prompt({
        message: "Research whether the sky is blue. You must use ask_researcher.",
      });

      expect(result.error).toBeNull();
      expect(usedTool(result.messages, "ask_researcher")).toBe(true);
    },
  );
});

Keep the prompt narrow. Tests that ask for broad natural-language behavior are more likely to be flaky than tests that assert a specific tool contract.

Sessions and memory

Prompt tests create sessions. If a test should be repeatable, use a fresh session or disable memory on the agent under test:

export default defineAgent({
  slug: "classifier",
  name: "Classifier",
  description: "Classifies a message and returns only the label.",
  systemPrompt: "Classify the message and return only the label.",
  model: "openai/gpt-5.6-sol",
  memory: false,
});

If you need a multi-turn test, keep the returned sessionId and pass it to the next prompt:

const first = await support.prompt({ message: "Remember the word orchid." });
const second = await support.prompt({
  sessionId: first.sessionId,
  message: "What word did I ask you to remember?",
});

Evals

Evals score real agent replies instead of asserting exact text. Put them in src/**/*.eval.ts and describe each suite with describeEval from @meetluno/luno/evals: every case prompts the agent with input, scores the reply with each scorer, and fails when any score (0–1) is under threshold (default 0.7). Scores and the judge’s reasons are printed per case.

src/agents/support.eval.ts

import {
  createAnswerRelevancyScorer,
  createKeywordCoverageScorer,
  createRubricScorer,
  describeEval,
  gatewayModel,
} from "@meetluno/luno/evals";
import support from "./support";

const judge = gatewayModel("openai/gpt-5.6-luna");

describeEval("support answers refund questions", {
  agent: support,
  data: [
    { input: "Can I get a refund after 45 days?" },
    { name: "order lookup", input: "Is order ORD-123 refundable?" },
  ],
  scorers: [
    createAnswerRelevancyScorer({ model: judge }),
    createRubricScorer({
      model: judge,
      criteria: [{ description: "States the refund policy window" }],
    }),
    createKeywordCoverageScorer(),
  ],
  threshold: 0.7,
});

LLM-judged scorers (createAnswerRelevancyScorer, createRubricScorer, createToxicityScorer, …) take a judge model. Use gatewayModel("vendor/model") so the judge runs through the Luno OS gateway.
Code scorers such as createKeywordCoverageScorer() (how many of the prompt’s keywords the reply covers) and createToneScorer() need no model. createRubricScorer is binary: 1 only when every required criterion holds.
A case can carry groundTruth for comparison scorers (similarity, completeness), and createScorer builds a custom one. timeout sets the per-case limit (default 180 s).

Tool-call scorers

Scorers also see the tools the agent called (with their arguments), so you can check behavior, not only wording:

createToolCalledScorer({ tool }) — 1 when the agent called tool, 0 otherwise.
createToolArgsScorer({ tool, args }) — the share of the expected args the call matched (deep equality per key, extra args ignored); 0 when the tool was not called.
Mastra’s createToolCallAccuracyScorerCode({ expectedTool, strictMode?, expectedToolOrder? }), createToolCallAccuracyScorerLLM and createTrajectoryAccuracyScorerCode are re-exported too.

Expected args usually differ per input, so a case can add its own scorers:

describeEval("support looks up orders", {
  agent: support,
  scorers: [createToolCalledScorer({ tool: "lookup_order" })],
  data: [
    {
      input: "Where is order ORD-123?",
      scorers: [createToolArgsScorer({ tool: "lookup_order", args: { orderId: "ORD-123" } })],
    },
  ],
});

Run and review

Run them with luno evals run, optionally filtered by file name (luno evals run support); extra arguments go to Vitest. Evals call real models (the agent and the judge) and cost money, so luno test never runs them. When the project is linked and you are logged in, luno evals run uploads every case (input, reply, tools called, each scorer’s score and reason) as one eval run; --name nightly labels it and --no-upload keeps results local. A failed upload only warns, and the exit code is still the tests’ one. Review the history with luno evals list [--agent <slug>] and luno evals get <run-id>, or in the dashboard’s project Evals tab: the mean score per scorer over time, the runs table, and each run’s cases with the judge’s reasons.

Failure inspection

When a prompt test fails, inspect the session before changing code:

luno agents sessions list support --status failed
luno agents sessions get support <session-id> --summary
luno agents sessions get support <session-id> --include messages,events,trace

For deployed agents, use Runs (/history) in the web app and filter to agent runs. The detail panel shows messages, tool calls, metadata, and trace data.

Next steps

Run agents

Prompt agents locally and inspect sessions from the CLI.

Agent runs

Debug failed sessions in the web app.

Actions as tools

Build deterministic tool contracts that are easier to test.

Deploy a project

Run tests before deploying changed agents.

Get started

Learn

Build with AI

Resources

Qualitative tests

Definition tests

Smoke-tests

Tool use tests

Sessions and memory

Evals

Tool-call scorers

Run and review

Failure inspection

Next steps

Run agents

Agent runs

Actions as tools

Deploy a project

​Qualitative tests

​Definition tests

​Smoke-tests

​Tool use tests

​Sessions and memory

​Evals

​Tool-call scorers

​Run and review

​Failure inspection

​Next steps

Run agents

Agent runs

Actions as tools

Deploy a project

Qualitative tests

Definition tests

Smoke-tests

Tool use tests

Sessions and memory

Evals

Tool-call scorers

Run and review

Failure inspection

Next steps