Prove a connected service works

You connect a storefront, a chat channel or a sign-in provider to your assistant. Then comes the uncomfortable question: does it actually work?

Until now the honest answer needed a person. Open the site, sign in as a real customer, ask the assistant something, read the reply, decide whether it looked right. That is slow, it is not repeatable, and nobody does it before every change.

Diagnostics replace that errand. You declare, once, what a working service answers. After that your mate can prove it — on demand, nightly, and right after you change a connector.

The idea in one line

A test account is the same thing an app store asks for at review: one operator-managed identity, per connected service, that checks can sign in as without a human clicking anything.

A fixture is you writing down the answer a working service gives: "order NW-1042 is shipped — ask about it and you should hear so."

What a run actually does

For each private connector your test account covers, six ordered steps. A run names the first step that failed, never a later one:

Then, for whatever else your fixtures declare: identity_subject (your own provider mints for the test subject), sign_in_widget (a visitor asked to sign in sees a form), channel_echo (a sandbox message comes back).

smoke_call proves a tool answers. fixture_call proves it answers correctly. That difference is the whole point: a storefront that cheerfully returns the wrong order passes the first and fails the second.

Declaring fixtures

In Console → Connections → Test account, pick the service, name its id, and declare what a check should prove:

{
  "calls": [
    {
      "tool": "get_order_status",
      "label": "NW-1042 is shipped",
      "args": { "order_number": "NW-1042" },
      "expect": { "contains": "shipped" }
    },
    { "tool": "list_my_orders", "label": "the customer has orders", "expect": { "min_items": 1 } }
  ]
}

The whole expectation vocabulary is four words:

Leave expect off and the call still asserts something real: that it succeeds.

You can also declare identity (a subject your provider can mint for), channel (a sandbox chat id and the echo you expect), and widgets.sign_in_form.

Never put a credential in a fixture

Fixtures are echoed into run evidence, the Console, and anything you export. An argument named like a credential — token, api_key, password, session and friends — is refused, not masked. Order ids and expected substrings only.

The test account's own password is different: it is generated, encrypted, shown to you once, and has no read-back path on any surface.

Running it

• Console → Connections → Test account → Run full check. Progress appears live; nothing polls.
• Ask your mate. "Check whether you can reach our private tools" runs the same diagnostics and reports the verdict — it will not tell you to go test it in a browser.
• Automatically, nightly and right after you change a connector, when Auto-check nightly is on.
• MCP: verify_tenant_private_tools.

Reading a result

Every step carries its evidence: the tool that was called, how long it took, and a redacted excerpt of what came back. Tokens, JWTs and email addresses are scrubbed on the way out, because the payload is your data, not ours.

A failing run names the step and gives the fix — including, for a missing consent, the exact URL to open once.

What a green result does not mean

Three things are deliberately not green:

• A step the run could not perform. If a fixture is declared and the runtime cannot run it, that is a failure, not a skip. "The checker could not run" and "your tools work" must never look the same.
• A min_items check against a result with no countable list. The run says it could not count rather than guessing that there were enough.
• A run that checked nothing — no connector, no fixtures. A green must never be able to mean "there was nothing to test".

A test account that declares no fixtures still checks reachability, and the verdict says exactly that rather than implying it proved more.

Sharing the proof

A finished run renders as a report you can hand to a customer, a partner or a store reviewer — the steps, the timings and the evidence, already redacted.

MCP tools

Each one has a Console twin, and both drive the same code — there is no second control plane.