Case Study · AI infrastructure at AWS
Contact Center Agent & Test/Eval Pipeline
A production Connect + Lex + Bedrock contact center agent, backed by a git- and upload-driven eval pipeline that gates every prompt, knowledge base, or test-case change before it ships.
Amazon ConnectLexBedrockAppSyncDeepEval
01
The problem & requirements
Functional
- Handle voice/chat contacts via Amazon Connect, routed through Lex for intent, escalating to a Bedrock agent for complex or RAG-based answers
- Engineers add or edit test cases two ways: upload a file (.xlsx/.json/.jsonl/.csv) through the React/AppSync frontend, or edit in the codebase via a git diff on test-case files
- Either path triggers an automated eval run — DeepEval metrics, a RAG groundedness checker, and a KB retrieval check — per agent the test case is associated with
- Results (scores, turns, ground truth, Bedrock Guardrails info) are queryable in CI logs, S3, DynamoDB, and the frontend dashboard
Non-functional
- Traceability: every test run ties to a specific git commit/diff or uploaded file version
- CI/CD gate: pipeline flags or blocks deployment on score regression
- CI-vendor agnostic: works across GitLab-CI, GitHub Actions, and CodeBuild — not locked to one platform
- Isolation: CI eval traffic against Bedrock must not compete with production contact center traffic
- Auditability: CloudWatch logs and CloudFormation-provisioned infra, reproducible end to end
02
Scale & constraints
A typical PR touches ~200 test cases, each running 3 checks against an average of 2 associated agents.
1200
evaluation calls per pipeline run (200 cases × 3 checks × 2 agents)
~4 min
CI gate time at 10 concurrent calls, ~2s avg latency — a reasonable merge-time cost
5,000
concurrent production Connect contacts at peak — the number that drives Bedrock traffic isolation
03
API design
uploadTestFile(file) → presigned S3 URL
AppSync mutation; client PUTs the file directly to S3.
git diff → CI artifact → S3
CI computes the diff on test-case files (GitHub, GitLab, or CodeCommit) and pushes it to the same S3 bucket.
S3 event → Lambda → createTestRun(s3_key)
Writes TestRun {test_run_id, status: PENDING} to DynamoDB, publishes to SQS.
runTests(test_run_id)
Per associated agent: DeepEval, RAG claims checker, KB retrieval check.
writeResults(test_run_id, results[]) → S3 + DynamoDB
Flips TestRun.status to COMPLETED.
getTestRun(test_run_id)
AppSync query/subscription powering the frontend's live status view.
04
Data model
| Entity | Key fields |
|---|---|
| TestRun | test_run_id, source (upload/git_diff), source_ref, status, created_at, triggered_by |
| TestCase | test_case_id, test_run_id, agent_id, input, ground_truth, conversation_turns[] |
| TestResult | test_case_id, test_run_id, agent_id, deepeval_scores{}, rag_claims_score, kb_retrieval_score, guardrails_info{}, latency_ms, pass_fail |
| Agent config | agent_id, bedrock_model_id, kb_id, prompt_version, connect_flow_id |
source_ref on TestRun is what makes a score regression traceable back to the exact commit or upload that caused it — without it, "the eval score dropped" has no owner.
05
Architecture
Eval pipeline
A test case either gets uploaded through the React/AppSync frontend or added via a git commit; both paths land in S3, trigger a Lambda that creates the test run record, and hand off through SQS to a CI/CD test stage running three checks per associated agent.
⇪
React upload
JSON / CSV / upload
⎇
Git commit / diff
GitLab · GitHub · CodeCommit
↓
▤
S3 — test inputs
Raw uploaded/diffed files
→
ƒ
Lambda
create test run
↓
☰
SQS
decouples create from run
↓
CI/CD test stage · GitLab-CI · GitHub Actions · CodeBuild
✓
DeepEval
judge metrics
◎
Claims checker
groundedness
🔎
KB retrieval
recall / precision
↓
▤
S3 — raw results
full, pass/fail per case
▦
DynamoDB — scores
aggregated, queryable
↓
⇄
AppSync API + React dashboard
IAM + Cognito auth, CloudWatch + CloudFormation
Fig. 3a — Test entry (upload or git diff) converges on S3, runs through a CI test stage, and surfaces back in the same frontend.
Production runtime
Connect handles the channel and routing. Lex resolves simple intents directly, and complex or knowledge-dependent queries escalate to the Bedrock agent, which draws on both a knowledge base and Lambda-backed tools.
👤
Customer
voice / chat
→
☎
Amazon Connect
contact flow + routing
→
🔒
Cognito / IAM
auth
↓
Conversational AI core
Intent & fulfillment
🗣
Lex bot
intent classification
ƒ
Lambda tools
business logic
Reasoning & knowledge
🧠
Bedrock agent
LLM + guardrails
📚
Knowledge base
RAG retrieval
Contact Lens transcriptsGlue ETL + Athena analyticsCloudWatch dashboardsCloudFormation IaC
Fig. 3b — Lex resolves intent directly; complex queries escalate to the Bedrock agent for RAG-backed reasoning.
06
Key decisions & trade-offs
Bottleneck — per-agent fan-out in the test stage. A test case associated with multiple agents multiplies the call count (200 cases × 2 agents × 3 checks = 1,200 calls). Agents run in parallel within the CI stage, with a per-PR cap on how many cases need the full 3-check suite vs. a lighter smoke subset.
Decoupling via SQS. Lambda publishes a message rather than invoking the CI pipeline directly — a CI outage doesn't lose the test run request, and multiple CI backends (GitLab-CI, GitHub Actions, CodeBuild) can all consume from the same queue depending on which repo triggered it.
Trade-off — blocking vs. non-blocking CI gate. Blocking the merge on eval regression is safer but costs iteration speed (~4 min); a non-blocking informational run with required manual sign-off is faster but relies on someone reading the report before it talks to real customers — blocking is the right default.
Reproducibility.
source_ref (commit SHA or upload version) plus a pinned bedrock_model_id and prompt_version on the agent config means a score regression is always attributable to a specific change, not a moving target.Guardrails as data, not just a gate. Bedrock Guardrails info is stored per test result — not just pass/fail — so the claims checker can distinguish "failed because ungrounded" from "failed because guardrails blocked it." These need different fixes.
Production isolation. CI eval calls against Bedrock hit a separate rate-limit bucket from live contact center traffic, so a large eval run never competes with a customer waiting on a live response.
System
Contact Center Agent & Eval Pipeline
Primary services
Connect · Bedrock · AppSync
Status
Shipped — in production at AWS
Type
Contact center + eval CI/CD