Case Study · AI infrastructure at AWS
Contact Center Agent & Test/Eval Pipeline
A production Connect + Lex + Bedrock contact center agent, backed by a git- and upload-driven eval pipeline that gates every prompt, knowledge base, or test-case change before it ships.
Amazon ConnectLexBedrockAppSyncDeepEvalReactS3DynamoDBSQSCloudWatchCloudFormationGitLab-CI / GitHub Actions / CodeBuildBedrock GuardrailsLambdaCognito / IAMGlue ETL + AthenaContact Lens transcriptsNova SonicStrands SDKAgentCore Gateway + MCPKinesis Firehose + KMSComprehend + QuickSightCloudFrontOpenSearch Serverless / Aurora PostgreSQL / S3 VectorsPythonTypeScript
01
The problem & requirements
Functional
- Handle voice/chat contacts via Amazon Connect, routed through Lex for intent, escalating to a Bedrock agent for complex or RAG-based answers
- Engineers add or edit test cases two ways: upload a file (.xlsx/.json/.jsonl/.csv) through the React/AppSync frontend, or edit in the codebase via a git diff on test-case files
- Either path triggers an automated eval run. DeepEval metrics, a RAG groundedness checker, and a KB retrieval check, per agent the test case is associated with
- Results (scores, turns, ground truth, Bedrock Guardrails info) are queryable in CI logs, S3, DynamoDB, and the frontend dashboard
- Support speech-to-speech voice interactions (Nova Sonic) alongside text, and let agents call customer and third-party APIs, data sources, and other agents through AgentCore Gateway using Model Context Protocol
- System admins manage model selections, prompts, parameters, and feature flags from a config UI without a redeploy
- Instrumented conversation logs feed a conversation analytics pipeline for product owners, plus an asynchronous auditor that flags policy violations
Non-functional
- Traceability: every test run ties to a specific git commit/diff or uploaded file version
- CI/CD gate: pipeline flags or blocks deployment on score regression
- CI-vendor agnostic: works across GitLab-CI, GitHub Actions, and CodeBuild, not locked to one platform
- Isolation: CI eval traffic against Bedrock must not compete with production contact center traffic
- Configurability: model, prompt, parameter, and feature-flag changes are made by admins at runtime, and every change is versioned against the agent config
- Data protection: conversation logs are encrypted with a customer-managed KMS key end to end
- Auditability: CloudWatch logs and CloudFormation-provisioned infra, reproducible end to end
02
Scale & constraints
A typical PR touches ~200 test cases, each running 3 checks against an average of 2 associated agents.
1200
evaluation calls per pipeline run (200 cases × 3 checks × 2 agents)
~4 min
CI gate time at 10 concurrent calls, ~2s avg latency, a reasonable merge-time cost
5,000
concurrent production Connect contacts at peak. The number that drives Bedrock traffic isolation
03
API design
uploadTestFile(file) → presigned S3 URL
AppSync mutation; client PUTs the file directly to S3.
git diff → CI artifact → S3
CI computes the diff on test-case files (GitHub, GitLab, or CodeCommit) and pushes it to the same S3 bucket.
S3 event → Lambda → createTestRun(s3_key)
Writes TestRun {test_run_id, status: PENDING} to DynamoDB, publishes to SQS.
runTests(test_run_id)
Per associated agent: DeepEval, RAG claims checker, KB retrieval check.
writeResults(test_run_id, results[]) → S3 + DynamoDB
Flips TestRun.status to COMPLETED.
getTestRun(test_run_id)
AppSync query/subscription powering the frontend's live status view.
04
Data model
| Entity | Key fields |
|---|---|
| TestRun | test_run_id, source (upload/git_diff), source_ref, status, created_at, triggered_by |
| TestCase | test_case_id, test_run_id, agent_id, input, ground_truth, conversation_turns[] |
| TestResult | test_case_id, test_run_id, agent_id, deepeval_scores{}, rag_claims_score, kb_retrieval_score, guardrails_info{}, latency_ms, pass_fail |
| Agent config | agent_id, bedrock_model_id, kb_id, prompt_version, connect_flow_id |
source_ref on TestRun is what makes a score regression traceable back to the exact commit or upload that caused it. Without it, "the eval score dropped" has no owner.
05
Architecture
Eval pipeline
A test case either gets uploaded through the React/AppSync frontend or added via a git commit; both paths land in S3, trigger a Lambda that creates the test run record, and hand off through SQS to a CI/CD test stage running three checks per associated agent.
⇪
React upload
JSON / CSV / upload
⎇
Git commit / diff
GitLab · GitHub · CodeCommit
↓
▤
S3 test inputs
Raw uploaded/diffed files
→
ƒ
Lambda
create test run
↓
☰
SQS
decouples create from run
↓
CI/CD test stage · GitLab-CI · GitHub Actions · CodeBuild
✓
DeepEval
judge metrics
◎
Claims checker
groundedness
🔎
KB retrieval
recall / precision
↓
▤
S3 raw results
full, pass/fail per case
▦
DynamoDB scores
aggregated, queryable
↓
⇄
AppSync API + React dashboard
IAM + Cognito auth, CloudWatch + CloudFormation
🌐
CloudFront + S3 web app bucket
hosts the tester UI
▤
CloudWatch log group
run logs, feeds conversation analytics
↓
LLM-as-a-Judge / LLM-as-a-Tester · DeepEval agent evaluation (Harmonix)
Fig. 3a: Test entry (upload or git diff) converges on S3, runs through a CI test stage, and surfaces back in the same frontend.
Production runtime
Connect handles the channel and routing. Lex resolves simple intents directly, and complex or knowledge-dependent queries escalate to Lambda and the Strands SDK, which draw on Bedrock models, knowledge bases with vector stores, and MCP tools through AgentCore Gateway.
👤
Customer
voice / chat
→
☎
Amazon Connect
contact flow + routing
→
🔒
Cognito / IAM
auth
↓
Conversational AI core
Intent & fulfillment
🗣
Lex bot
intent + contact flows
ƒ
Lambda + Strands SDK
orchestration, semantic search, LLM access
🎙
Nova Sonic
speech-to-speech voice
Reasoning & knowledge
🧠
Bedrock models
Nova, Claude, Llama, Pixtral, Jamba
📚
Knowledge bases
RAG retrieval
∷
Embeddings
Nova, Cohere, Marengo
↓
Vector stores
🔎
OpenSearch Serverless
▦
Aurora PostgreSQL
pgvector
▤
S3 Vectors
Agentic tool calling
⇄
AgentCore Gateway
Model Context Protocol
⚙
Customer + 3rd-party APIs
🗄
Data sources + system interfaces
🤖
Third-party agents
Contact Lens transcriptsCloudWatch dashboardsCloudFormation IaC
Fig. 3b: Lex resolves intent directly; Lambda orchestrates Bedrock models, knowledge bases, and gateway-brokered tools, with Nova Sonic for voice.
Conversation analytics
Instrumented conversation logs, Contact Lens data, and Contact Trace Records stream through Firehose into a data lake that product owners explore in QuickSight.
☰
CloudWatch logs
instrumented conversation data
→
⇉
Data Firehose
encrypted with KMS CMK
→
ƒ
Lambda
transform
→
▤
S3
data lake
↓
🧠
Comprehend
sentiment / entities
⚙
Glue
catalog + ETL
🔎
Athena
SQL
📊
QuickSight (Q)
dashboards for product owners
Fig. 3c: Conversation and Contact Lens data flow into a KMS-encrypted lake and out to product-owner dashboards.
Auditing & monitoring
Application logs and audit requests queue asynchronously, an auditor function scores each conversation with a Bedrock LLM, and alarms notify product owners of violations.
ƒ
Lambda
app logs + audit requests
→
☰
SQS queue
→
ƒ
Auditor function
→
🧠
Bedrock LLM
judge
→
🔔
CloudWatch alarms
violations, Lambda + pipeline errors
Fig. 3d: Asynchronous conversation audit keeps judging off the live response path.
Configuration management
System admins tune the accelerator through a React UI backed by AppSync and DynamoDB, without touching the deployment.
👤
System admins
→
🌐
CloudFront + S3
React web app
→
🔒
Cognito
auth
→
⇄
AppSync
→
▦
DynamoDB
models, prompts, params, flags
Fig. 3e: Runtime configuration is data, editable by admins and read by the agent Lambda.
06
Key decisions & trade-offs
Tool access through a gateway, not per-agent glue. Agents built with the Strands SDK reach customer APIs, data sources, and third-party agents through AgentCore Gateway over MCP, so tool auth and routing live in one place instead of in every Lambda.
Config as data. Model selections, prompts, parameters, and feature flags live in DynamoDB behind an AppSync admin UI. Runtime changes never need a redeploy, and each one is versioned so eval results stay attributable.
Bottleneck. Per-agent fan-out in the test stage. A test case associated with multiple agents multiplies the call count (200 cases × 2 agents × 3 checks = 1,200 calls). Agents run in parallel within the CI stage, with a per-PR cap on how many cases need the full 3-check suite vs. a lighter smoke subset.
Decoupling via SQS. Lambda publishes a message rather than invoking the CI pipeline directly, a CI outage doesn't lose the test run request, and multiple CI backends (GitLab-CI, GitHub Actions, CodeBuild) can all consume from the same queue depending on which repo triggered it.
Trade-off. Blocking vs. non-blocking CI gate. Blocking the merge on eval regression is safer but costs iteration speed (~4 min); a non-blocking informational run with required manual sign-off is faster but relies on someone reading the report before it talks to real customers, blocking is the right default.
Reproducibility.
source_ref (commit SHA or upload version) plus a pinned bedrock_model_id and prompt_version on the agent config means a score regression is always attributable to a specific change, not a moving target.Guardrails as data, not just a gate. Bedrock Guardrails info is stored per test result, not just pass/fail, so the claims checker can distinguish "failed because ungrounded" from "failed because guardrails blocked it." These need different fixes.
Production isolation. CI eval calls against Bedrock hit a separate rate-limit bucket from live contact center traffic, so a large eval run never competes with a customer waiting on a live response.
07
Lessons learned
💡 What actually held up
- Making blocking the default CI gate, not an option, held up under real pressure to ship faster. Every time someone pushed for a non-blocking "just this once," the ~4 minute cost looked small next to what a bad prompt change could do to a live customer conversation.
- Storing Guardrails info as structured data, not just a pass/fail flag, turned out to be the single most useful debugging decision. Separating "failed because ungrounded" from "failed because guardrails blocked it" meant fixes went to the right place instead of a shared bucket of "eval failures."
- Decoupling test-run creation from execution via SQS paid off during a real CI outage. The queue just held the backlog instead of silently dropping requests, and everything caught up once the runners came back.
↺What I'd do differently
- I didn't plan for per-agent fan-out until it was already a problem. As more agents got added to the platform, the 1,200-call PR cost crept up without anyone deciding it should. A smoke-subset vs. full-suite split should've been part of the design from the first agent, not a cap added after CI got slow.
- Bedrock traffic isolation came after a scare, not before it. A large eval run competing with live contact volume during a peak window was the thing that forced separate rate-limit buckets. I'd provision that isolation on day one for anything touching a model shared with production.
- Agent config (model ID, KB, prompt version) started as loose parameters, not a versioned entity. Retrofitting
prompt_versiononto existing test results after the fact was more painful than just treating agent config as a first-class, versioned object from the start.
🔁If I started this over: provision Bedrock rate-limit isolation and a versioned agent-config entity before the first agent ships, and design the smoke-vs-full test split into the CI stage from day one rather than adding it once fan-out cost becomes visible.
System
Contact Center Agent & Eval Pipeline
Primary services
Connect · Bedrock · AppSync
Status
Shipped in production at AWS
Type
Contact center + eval CI/CD