Case Study · AI infrastructure at AWS
Contact Center Agent & Test/Eval Pipeline
A production Connect + Lex + Bedrock contact center agent, backed by a git- and upload-driven eval pipeline that gates every prompt, knowledge base, or test-case change before it ships.
Amazon ConnectLexBedrockAppSyncDeepEvalReactS3DynamoDBSQSCloudWatchCloudFormationGitLab-CI / GitHub Actions / CodeBuildBedrock GuardrailsLambdaCognito / IAMGlue ETL + AthenaContact Lens transcriptsPythonTypeScript
01
The problem & requirements
Functional
- Handle voice/chat contacts via Amazon Connect, routed through Lex for intent, escalating to a Bedrock agent for complex or RAG-based answers
- Engineers add or edit test cases two ways: upload a file (.xlsx/.json/.jsonl/.csv) through the React/AppSync frontend, or edit in the codebase via a git diff on test-case files
- Either path triggers an automated eval run. DeepEval metrics, a RAG groundedness checker, and a KB retrieval check, per agent the test case is associated with
- Results (scores, turns, ground truth, Bedrock Guardrails info) are queryable in CI logs, S3, DynamoDB, and the frontend dashboard
Non-functional
- Traceability: every test run ties to a specific git commit/diff or uploaded file version
- CI/CD gate: pipeline flags or blocks deployment on score regression
- CI-vendor agnostic: works across GitLab-CI, GitHub Actions, and CodeBuild, not locked to one platform
- Isolation: CI eval traffic against Bedrock must not compete with production contact center traffic
- Auditability: CloudWatch logs and CloudFormation-provisioned infra, reproducible end to end
02
Scale & constraints
A typical PR touches ~200 test cases, each running 3 checks against an average of 2 associated agents.
1200
evaluation calls per pipeline run (200 cases × 3 checks × 2 agents)
~4 min
CI gate time at 10 concurrent calls, ~2s avg latency, a reasonable merge-time cost
5,000
concurrent production Connect contacts at peak. The number that drives Bedrock traffic isolation
03
API design
uploadTestFile(file) → presigned S3 URL
AppSync mutation; client PUTs the file directly to S3.
git diff → CI artifact → S3
CI computes the diff on test-case files (GitHub, GitLab, or CodeCommit) and pushes it to the same S3 bucket.
S3 event → Lambda → createTestRun(s3_key)
Writes TestRun {test_run_id, status: PENDING} to DynamoDB, publishes to SQS.
runTests(test_run_id)
Per associated agent: DeepEval, RAG claims checker, KB retrieval check.
writeResults(test_run_id, results[]) → S3 + DynamoDB
Flips TestRun.status to COMPLETED.
getTestRun(test_run_id)
AppSync query/subscription powering the frontend's live status view.
04
Data model
| Entity | Key fields |
|---|---|
| TestRun | test_run_id, source (upload/git_diff), source_ref, status, created_at, triggered_by |
| TestCase | test_case_id, test_run_id, agent_id, input, ground_truth, conversation_turns[] |
| TestResult | test_case_id, test_run_id, agent_id, deepeval_scores{}, rag_claims_score, kb_retrieval_score, guardrails_info{}, latency_ms, pass_fail |
| Agent config | agent_id, bedrock_model_id, kb_id, prompt_version, connect_flow_id |
source_ref on TestRun is what makes a score regression traceable back to the exact commit or upload that caused it. Without it, "the eval score dropped" has no owner.
05
Architecture
Eval pipeline
A test case either gets uploaded through the React/AppSync frontend or added via a git commit; both paths land in S3, trigger a Lambda that creates the test run record, and hand off through SQS to a CI/CD test stage running three checks per associated agent.
⇪
React upload
JSON / CSV / upload
⎇
Git commit / diff
GitLab · GitHub · CodeCommit
↓
▤
S3 test inputs
Raw uploaded/diffed files
→
ƒ
Lambda
create test run
↓
☰
SQS
decouples create from run
↓
CI/CD test stage · GitLab-CI · GitHub Actions · CodeBuild
✓
DeepEval
judge metrics
◎
Claims checker
groundedness
🔎
KB retrieval
recall / precision
↓
▤
S3 raw results
full, pass/fail per case
▦
DynamoDB scores
aggregated, queryable
↓
⇄
AppSync API + React dashboard
IAM + Cognito auth, CloudWatch + CloudFormation
Fig. 3a: Test entry (upload or git diff) converges on S3, runs through a CI test stage, and surfaces back in the same frontend.
Production runtime
Connect handles the channel and routing. Lex resolves simple intents directly, and complex or knowledge-dependent queries escalate to the Bedrock agent, which draws on both a knowledge base and Lambda-backed tools.
👤
Customer
voice / chat
→
☎
Amazon Connect
contact flow + routing
→
🔒
Cognito / IAM
auth
↓
Conversational AI core
Intent & fulfillment
🗣
Lex bot
intent classification
ƒ
Lambda tools
business logic
Reasoning & knowledge
🧠
Bedrock agent
LLM + guardrails
📚
Knowledge base
RAG retrieval
Contact Lens transcriptsGlue ETL + Athena analyticsCloudWatch dashboardsCloudFormation IaC
Fig. 3b: Lex resolves intent directly; complex queries escalate to the Bedrock agent for RAG-backed reasoning.
06
Key decisions & trade-offs
Bottleneck. Per-agent fan-out in the test stage. A test case associated with multiple agents multiplies the call count (200 cases × 2 agents × 3 checks = 1,200 calls). Agents run in parallel within the CI stage, with a per-PR cap on how many cases need the full 3-check suite vs. a lighter smoke subset.
Decoupling via SQS. Lambda publishes a message rather than invoking the CI pipeline directly, a CI outage doesn't lose the test run request, and multiple CI backends (GitLab-CI, GitHub Actions, CodeBuild) can all consume from the same queue depending on which repo triggered it.
Trade-off. Blocking vs. non-blocking CI gate. Blocking the merge on eval regression is safer but costs iteration speed (~4 min); a non-blocking informational run with required manual sign-off is faster but relies on someone reading the report before it talks to real customers, blocking is the right default.
Reproducibility.
source_ref (commit SHA or upload version) plus a pinned bedrock_model_id and prompt_version on the agent config means a score regression is always attributable to a specific change, not a moving target.Guardrails as data, not just a gate. Bedrock Guardrails info is stored per test result, not just pass/fail, so the claims checker can distinguish "failed because ungrounded" from "failed because guardrails blocked it." These need different fixes.
Production isolation. CI eval calls against Bedrock hit a separate rate-limit bucket from live contact center traffic, so a large eval run never competes with a customer waiting on a live response.
07
Lessons learned
💡 What actually held up
- Making blocking the default CI gate, not an option, held up under real pressure to ship faster. Every time someone pushed for a non-blocking "just this once," the ~4 minute cost looked small next to what a bad prompt change could do to a live customer conversation.
- Storing Guardrails info as structured data, not just a pass/fail flag, turned out to be the single most useful debugging decision. Separating "failed because ungrounded" from "failed because guardrails blocked it" meant fixes went to the right place instead of a shared bucket of "eval failures."
- Decoupling test-run creation from execution via SQS paid off during a real CI outage. The queue just held the backlog instead of silently dropping requests, and everything caught up once the runners came back.
↺What I'd do differently
- I didn't plan for per-agent fan-out until it was already a problem. As more agents got added to the platform, the 1,200-call PR cost crept up without anyone deciding it should. A smoke-subset vs. full-suite split should've been part of the design from the first agent, not a cap added after CI got slow.
- Bedrock traffic isolation came after a scare, not before it. A large eval run competing with live contact volume during a peak window was the thing that forced separate rate-limit buckets. I'd provision that isolation on day one for anything touching a model shared with production.
- Agent config (model ID, KB, prompt version) started as loose parameters, not a versioned entity. Retrofitting
prompt_versiononto existing test results after the fact was more painful than just treating agent config as a first-class, versioned object from the start.
🔁If I started this over: provision Bedrock rate-limit isolation and a versioned agent-config entity before the first agent ships, and design the smoke-vs-full test split into the CI stage from day one rather than adding it once fan-out cost becomes visible.
System
Contact Center Agent & Eval Pipeline
Primary services
Connect · Bedrock · AppSync
Status
Shipped in production at AWS
Type
Contact center + eval CI/CD