> ## Documentation Index
> Fetch the complete documentation index at: https://docs.os.meetluno.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Test agents

> Test agents qualitatively and with automated checks.

Agents are non-deterministic, so testing them starts with exercising behavior, not writing a perfect assertion. The fastest loop is often to have your coding agent prompt the Luno OS agent through the CLI, try realistic scenarios, inspect the resulting sessions, and iterate on the system instructions and tools.

Test agents at the boundary you care about:

| Test style | Use when |
| - | - |
| Qualitative CLI runs | You want to see how the agent behaves across realistic prompts, follow-ups, and tool-use cases |
| Definition tests | You want to verify tools, model, prompt, or metadata without calling an LLM |
| Prompt smoke tests | You want to verify the agent can run against a real model |
| Tool-use tests | You need evidence that the agent called a required action or subagent |
| End-to-end tests | You want to run the project server and test API behavior |
| [Evals](#evals) | You want to score reply quality (relevancy, rubric, keywords) against a threshold |

Start qualitatively, then turn the stable contracts you discover into tests. Automated tests are best for definition shape, required tool calls, and a small number of critical prompt paths.

Run tests from the project root:

```bash theme={null}
luno test --project unit          # src/**/*.test.ts
luno test --project integration   # src/**/*.int.test.ts
pnpm test                              # same — calls luno test
luno evals run                    # src/**/*.eval.ts (real models, on demand)
```

Vitest ships with `@meetluno/cli`. No project `vitest.config.ts` is required.

## Qualitative tests

Before writing test files, run the agent the way you expect people to use it. Ask your coding agent to call the Luno OS CLI with a batch of prompts, inspect the sessions, and report where the agent misunderstood instructions, skipped tools, or used tools incorrectly.

```bash theme={null}
luno agents prompt support --message "A customer asks whether they can get a refund after 45 days." --wait
luno agents prompt support --message "Look up order ORD-123 and decide whether it is refundable." --wait
luno agents prompt support --message "Draft a concise Slack reply for that customer." --wait
```

`--wait` returns each completed session for review. For long prompts, choose a bounded wait with `--timeout <seconds>` and continue with `agents sessions get <agent> <session-id> --wait` if the session is still running.

Good qualitative prompts cover:

* Normal requests the agent should handle cleanly.
* Tool-required requests where the agent must call an action, workflow, MCP tool, or subagent.
* Missing-information cases where the agent should ask a clarifying question instead of guessing.
* Follow-up messages in the same session.
* Edge cases that should be refused, escalated, or handled cautiously.

Then inspect the session — start with a summary, then request richer detail if needed:

```bash theme={null}
luno agents sessions get support <session-id> --summary
luno agents sessions get support <session-id> --include messages,events,trace
```

Use this loop to tune the system prompt, tools, skills, files, and model choice. Once the behavior feels right, write focused tests for the parts that should not regress.

## Definition tests

Definition tests are fast and do not need provider keys.

```ts theme={null}
import { describe, expect, it } from "vitest";
import support from "./support";

describe("support agent", () => {
  it("uses the expected model and support files", () => {
    expect(support.slug).toBe("support");
    expect(support.model).toBe("openai/gpt-5.6-sol");
    expect(support.systemPrompt).toContain("/workspace");
  });
});
```

Use this style to catch accidental model changes, missing tools, or prompt edits that remove required instructions.

## Smoke-tests

The init template includes an agent integration test shaped like this:

```ts theme={null}
import { describe, expect, it } from "vitest";
import hello from "./hello";

describe("hello agent", () => {
  it.skipIf(!process.env.AI_GATEWAY_API_KEY && !process.env.OPENAI_API_KEY)(
    "responds to a prompt",
    async () => {
      const result = await hello.prompt({ message: "Say hi in one word." });

      expect(result.messages.some((message) => message.role === "assistant")).toBe(true);
    },
  );
});
```

The provider-key guard keeps local and CI runs from failing when real model credentials are not available. Match the guard to the agent’s `model` (or use `AI_GATEWAY_API_KEY`, which works for any catalog id).

Run integration tests from the project root:

```bash theme={null}
luno test --project integration
# or
pnpm test -- --project integration
```

Integration tests load `.env` when present and skip when required keys (like `AI_GATEWAY_API_KEY`) are unset. Vitest and its config ship with `@meetluno/cli` — no project `vitest.config.ts`.

## Tool use tests

When a tool call is the contract, assert on the recorded messages rather than only the final answer.

```ts theme={null}
import { describe, expect, it } from "vitest";
import orchestrator from "./orchestrator";

function usedTool(messages: Array<{ role: string; toolName?: string }>, toolName: string) {
  return messages.some((message) => message.role === "toolResult" && message.toolName === toolName);
}

describe("orchestrator agent", () => {
  it.skipIf(!process.env.AI_GATEWAY_API_KEY && !process.env.OPENAI_API_KEY)(
    "delegates research",
    async () => {
      const result = await orchestrator.prompt({
        message: "Research whether the sky is blue. You must use ask_researcher.",
      });

      expect(result.error).toBeNull();
      expect(usedTool(result.messages, "ask_researcher")).toBe(true);
    },
  );
});
```

Keep the prompt narrow. Tests that ask for broad natural-language behavior are more likely to be flaky than tests that assert a specific tool contract.

## Sessions and memory

Prompt tests create sessions. If a test should be repeatable, use a fresh session or disable memory on the agent under test:

```ts theme={null}
export default defineAgent({
  slug: "classifier",
  name: "Classifier",
  description: "Classifies a message and returns only the label.",
  systemPrompt: "Classify the message and return only the label.",
  model: "openai/gpt-5.6-sol",
  memory: false,
});
```

If you need a multi-turn test, keep the returned `sessionId` and pass it to the next prompt:

```ts theme={null}
const first = await support.prompt({ message: "Remember the word orchid." });
const second = await support.prompt({
  sessionId: first.sessionId,
  message: "What word did I ask you to remember?",
});
```

## Evals

Evals score real agent replies instead of asserting exact text. Put them in `src/**/*.eval.ts` and describe each suite with `describeEval` from `@meetluno/luno/evals`: every case prompts the agent with `input`, scores the reply with each scorer, and fails when any score (0–1) is under `threshold` (default `0.7`). Scores and the judge's reasons are printed per case.

```ts src/agents/support.eval.ts theme={null}
import {
  createAnswerRelevancyScorer,
  createKeywordCoverageScorer,
  createRubricScorer,
  describeEval,
  gatewayModel,
} from "@meetluno/luno/evals";
import support from "./support";

const judge = gatewayModel("openai/gpt-5.6-luna");

describeEval("support answers refund questions", {
  agent: support,
  data: [
    { input: "Can I get a refund after 45 days?" },
    { name: "order lookup", input: "Is order ORD-123 refundable?" },
  ],
  scorers: [
    createAnswerRelevancyScorer({ model: judge }),
    createRubricScorer({
      model: judge,
      criteria: [{ description: "States the refund policy window" }],
    }),
    createKeywordCoverageScorer(),
  ],
  threshold: 0.7,
});
```

* **LLM-judged scorers** (`createAnswerRelevancyScorer`, `createRubricScorer`, `createToxicityScorer`, …) take a judge `model`. Use `gatewayModel("vendor/model")` so the judge runs through the Luno OS gateway.
* **Code scorers** such as `createKeywordCoverageScorer()` (how many of the prompt's keywords the reply covers) and `createToneScorer()` need no model. `createRubricScorer` is binary: `1` only when every required criterion holds.
* A case can carry `groundTruth` for comparison scorers (similarity, completeness), and `createScorer` builds a custom one. `timeout` sets the per-case limit (default 180 s).

### Tool-call scorers

Scorers also see the tools the agent called (with their arguments), so you can check behavior, not only wording:

* `createToolCalledScorer({ tool })` — `1` when the agent called `tool`, `0` otherwise.
* `createToolArgsScorer({ tool, args })` — the share of the expected `args` the call matched (deep equality per key, extra args ignored); `0` when the tool was not called.
* Mastra's `createToolCallAccuracyScorerCode({ expectedTool, strictMode?, expectedToolOrder? })`, `createToolCallAccuracyScorerLLM` and `createTrajectoryAccuracyScorerCode` are re-exported too.

Expected args usually differ per input, so a case can add its own `scorers`:

```ts theme={null}
describeEval("support looks up orders", {
  agent: support,
  scorers: [createToolCalledScorer({ tool: "lookup_order" })],
  data: [
    {
      input: "Where is order ORD-123?",
      scorers: [createToolArgsScorer({ tool: "lookup_order", args: { orderId: "ORD-123" } })],
    },
  ],
});
```

### Run and review

Run them with `luno evals run`, optionally filtered by file name (`luno evals run support`); extra arguments go to Vitest. Evals call real models (the agent and the judge) and cost money, so `luno test` never runs them.

When the project is linked and you are logged in, `luno evals run` uploads every case (input, reply, tools called, each scorer's score and reason) as one eval run; `--name nightly` labels it and `--no-upload` keeps results local. A failed upload only warns, and the exit code is still the tests' one. Review the history with `luno evals list [--agent <slug>]` and `luno evals get <run-id>`, or in the dashboard's project **Evals** tab: the mean score per scorer over time, the runs table, and each run's cases with the judge's reasons.

## Failure inspection

When a prompt test fails, inspect the session before changing code:

```bash theme={null}
luno agents sessions list support --status failed
luno agents sessions get support <session-id> --summary
luno agents sessions get support <session-id> --include messages,events,trace
```

For deployed agents, use **Runs** (`/history`) in the web app and filter to agent runs. The detail panel shows messages, tool calls, metadata, and trace data.

## Next steps

> **[Run agents](https://docs.os.meetluno.ai/learn/agents/run-agents.md)**
>
> Prompt agents locally and inspect sessions from the CLI.

> **[Agent runs](https://docs.os.meetluno.ai/learn/logs/agent-runs.md)**
>
> Debug failed sessions in the web app.

> **[Actions as tools](https://docs.os.meetluno.ai/learn/actions/agent-tools.md)**
>
> Build deterministic tool contracts that are easier to test.

> **[Deploy a project](https://docs.os.meetluno.ai/learn/projects/deploy-a-project.md)**
>
> Run tests before deploying changed agents.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.