Sentinel — Eval-as-MCP-Server — Case Study

Projects›Sentinel — Eval-as-MCP-Server

The problem & requirements

Functional

Score an agent's response against configurable checks — groundedness, prompt injection — callable as an MCP tool from any agent runtime
Deterministic pattern matching resolves clear-cut cases with no model call; ambiguous cases escalate to a judge, bounded by a per-metric latency budget
Every tenant explicitly configures fail-open vs. fail-closed behavior on judge timeout — no system-wide default
The same business logic runs behind two transports: stdio for local agents, stateless HTTP for remote deploy

Non-functional

Zero-config local dev: full test suite and both tools run offline against stub judges, no API key required
Fixed cost at idle: every layer of the stack scales to zero — Vercel functions, planned Neon/R2 persistence
Reproducibility: every verdict pins its metric version and judge model, re-derivable later
No unauthenticated tool calls: tenant ID comes only from a validated bearer token, never a request argument

Scale & constraints

A bootstrap build: single developer, low fixed cost, nothing requiring a company cloud account. The numbers below are the budget the architecture had to hold under from day one, not traffic figures yet.

250ms

Per-metric judge timeout, enforced server-side — a slow eval is a broken eval in the hot path

Automated tests across both metrics and both tool handlers, all passing with zero network calls

Fixed infra cost at idle — every layer of the stack scales to zero, no standing bill

API design

flag_injection(response, context?) → EvaluationResult

MCP tool call. Validated by the same zod schema on both the stdio and HTTP transports.

evaluateInjection(request, {judge, policy}) → Verdict

Deterministic tier resolves override + compliance pattern matches directly; only the ambiguous middle escalates.

check_groundedness(response, context) → EvaluationResult

Requires at least one context item. Nothing to ground against is a caller error, rejected before the metric runs.

resolveCitations(citations, context) → per-citation status

Negation-window check: a citation only appearing inside a denial ("no section 9.2 exists") counts as unresolved.

buildEvaluationResult(verdicts, policy) → EvaluationResult

Rolls up N verdicts into one action (strictest wins) and a 0–100 trust score.

withMcpAuth(handler, verifyToken) → authed HTTP route

Bearer-token verification on the Vercel deploy target. Tenant ID comes only from the validated token.

Data model

Entity	Key fields
ScoringRequest	response, context[] (source, content, id), taskDescription?, correlationId?, policyOverride?
Verdict	metric, outcome (pass/warn/block/timeout), score, threshold, reason, citedContextIds[], decidedBy (heuristic/judge/cache), judge?, metricVersion, latencyMs
EvaluationResult	requestId, verdicts[], action, trustScore, totalLatencyMs
MetricPolicy	metric, enabled, threshold, onTimeout (fail_open/fail_closed), maxJudgeLatencyMs
TenantPolicy	tenantId, metrics[], aggregation (strictest/weighted), archivePayloads (opt-in, default false)

decidedBy on every Verdict is what makes the cost story auditable — it's the field that proves a judge model wasn't called when it didn't need to be, not just a claim in a case study.

Architecture

Scoring flow

Both tools converge on the same escalation ladder: deterministic checks first, a judge model only for what they can't resolve.

📤

flag_injection

tool call

📤

check_groundedness

tool call

↓

⊘

Handler layer

validate · resolve tenant policy

↓

📐

Policy resolution

threshold · fail-open/closed

↓

Deterministic tier · regex / citation matching

abc

Pattern matcher

injection metric

🔍

Citation resolver

groundedness metric

Escalates only if ambiguous

↓

✓

Confident verdict

no judge call

⚖

Judge model call

Anthropic API

↓

📤

EvaluationResult returned

Fig. 2a — Most calls resolve at the deterministic tier. The judge is an escalation path, not the default route.

Deploy topology

Two transports, one shared handler core. Neither entrypoint imports anything the other depends on.

💻

Claude Desktop / Cursor

stdio transport

🌐

Vercel HTTP route

mcp-handler, bearer token

↓

Shared handler core

Tool handlers

🛡

flagInjection

📎

checkGroundedness

Supporting layers

📋

Policy registry

🧪

Judge stubs

node:test suitetsx runtimeVercel FunctionsClaude Desktop config

Fig. 2b — Adding the HTTP deploy target was one new route file plus a thin auth module, not a parallel implementation.

Key decisions & trade-offs

Trap. The obvious groundedness heuristic is dangerous. Checking whether a cited string merely appears in the source context looks right and is wrong — a document denying a claim ("no section 9.2 exists") contains the fabricated citation as a literal substring. A negation-window check around every match closes the gap a naive presence check would rubber-stamp.

Trade-off. Heuristic-first, judge as escalation, not default. Every metric runs deterministic pattern or citation matching first and only calls a judge model for the genuinely ambiguous middle, bounded by a per-metric timeout independent of the whole request's budget. Most traffic never reaches the judge.

Invariant. No default for fail-open vs. fail-closed. A tenant sets this explicitly per metric; there's no system-wide fallback. A finance tenant and a marketing tenant disagree about what "the judge timed out" should mean, and guessing wrong for either is worse than forcing the choice.

Constraint. Two MCP server packages, one shared handler layer. The stdio transport and the HTTP transport have different registerTool signatures and different zod version requirements. Keeping handlers as plain functions with zero transport imports meant the HTTP deploy target was additive, not a rewrite.

Bootstrap-first. Vendor names quarantined to two folders. Cloudflare Workers, Neon, R2, and Anthropic-direct over Bedrock/AgentCore, chosen because every piece scales to zero at idle. No vendor name appears outside judges/ and persistence/.

Verification. Proved the HTTP route in-process, not just by type-check. A background dev server can't survive between tool calls in every environment. Pointing a real MCP client's fetch option directly at the exported route handler proved the auth and scoring path end to end without depending on a live socket.

Lessons learned

💡 What actually held up

Writing the demo's exact fabricated-clause scenario as a named regression test, before it ever caught anything for real, is what proved the negation-window fix worked — not the fact that it compiled.
Re-validating input independently in the handler layer caught nothing in testing, but it's the reason a second transport couldn't silently skip a check the first one enforced.
Keeping handlers completely transport-agnostic meant the Vercel deploy target was one new file plus a thin auth module, not a parallel implementation.

↺What I'd do differently

I upgraded zod from v3 to v4 reactively, once the HTTP package's type requirements forced it, rather than checking both packages' peer dependencies before writing schema code.
A background dev server can't survive between tool calls in every environment. I'd default to testing route handlers as plain functions in-process from the start, not assume a live server would be easy to stand up.
The trust-score rollup shipped as an honest placeholder, but I didn't flag it as loudly in the code as the fail-open/fail-closed choice — a formula that looks precise gets trusted more than it should.

🔁If I started this over: write the negation-trap test before the heuristic that needs it to pass, not after — and settle the zod version across both MCP packages on day one instead of discovering the conflict at build time.

System

Sentinel — Eval-as-MCP-Server

Primary services

MCP · Vercel · Anthropic API

Status

Live demo, deployed on Vercel

Type

AI eval infrastructure

Interested in the architecture behind this or another project?

← Back to projects ✉ Get in touch