CASE STUDY · 2026
How I Engineered a Prompt That Reviews GenAI Architecture Like a Senior Engineer
Getting senior architect feedback on GenAI system designs is slow and typically generic. Most “review my architecture” outputs produce platitudes: “make sure you handle scaling,” “add monitoring,” “consider security.” These are useless to an engineer who already knows those exist.
GenAI systems have a distinct failure mode taxonomy that general-purpose reviewers miss — API quota exhaustion across tenants, context-window cost economics, embedding staleness, reranker latency tradeoffs. I built a single-prompt system with 13 explicit constraints to eliminate these failure modes by name.
THE CORE INSIGHT
“A prompt is a wasted week.” Build order: prompt quality first, frontend last. Reviews fail in specific, nameable ways. Each failure mode needs an explicit rule — not a general instruction to ‘be accurate’ — but ‘this specific pattern is a failed output.’”
KEY DECISIONS
DECISION 1
AWS Bedrock over direct Anthropic/OpenAI
CHOSEN
AWS Bedrock (Claude model via boto3)
REJECTED
Direct Anthropic API, OpenAI
Portfolio differentiation — demonstrates AWS depth, not just 'call OpenAI.' Bedrock abstracts model provider; multi-model support falls out naturally.
DECISION 2
Single model call, no chains
CHOSEN
One prompt in, one structured JSON out
REJECTED
Multi-step chains, LangGraph, retries-with-improve-quality
Chains add latency, cost, and failure surfaces without proportional quality gain for this task. If output is bad, fix the prompt — not the architecture.
DECISION 3
Strict output schema with named fields
CHOSEN
JSON schema: critical_risks, killer_insight, review_confidence, key_unresolved_decisions, architecture_diagram
REJECTED
Free-form markdown review, prose output
Schema forces complete structured output; schema fields encode domain knowledge. review_confidence with missing-fields list surfaces 'this review is 55% confident because you didn't specify your auth model.'
DECISION 4
Tool use for structured output
CHOSEN
Bedrock tool_use with tool_choice: {type: tool} + full JSON Schema
REJECTED
Instructing the model to 'output valid JSON', string parsing
Instruction-based JSON compliance fails under token pressure — model truncates mid-structure. Tool use forces the API to validate schema compliance before returning.
DECISION 5
Access gate: open demo, verified custom input
CHOSEN
Demo runs canonical example freely; custom input verified server-side via X-Access-Code header
REJECTED
Fully open (cost exposure), fully closed (no demo value)
Each review call costs real money. Original REACT_APP_ACCESS_CODE approach was a security failure — code in JS bundle, extractable from DevTools in under 60s. Fixed: code moved to server env var.
TECHNICAL ARCHITECTURE
LAYER
CHOICE
WHY
Frontend
React (CRA), plain CSS
No complexity; CSS variables for theming; no component library to fight
Backend
FastAPI (Python) on Vercel Serverless
Thin API layer; Python natural for Bedrock boto3
Model access
AWS Bedrock Runtime via boto3
Vendor abstraction; Claude primary, DeepSeek/Llama/Nova supported
Structured output
Claude tool_use (JSON Schema)
Forces schema compliance at API level, not instruction level
Diagrams
Mermaid.js (dynamic import)
Model outputs Mermaid syntax; client renders; zero server cost
Auth
Server-side env var, X-Access-Code header
Secret never in JS bundle
Stateless — no DB, no session, no user accounts. Split endpoints: /api/review/demo (open, IP rate-limited 5 req/60s) vs /api/review (code required).
PROMPT ENGINEERING: 13 FAILURE MODE RULES
Each rule was added in response to a specific, observed failure. The naive prompt produces the same generic output for every system — these rules are what make the output discriminating.
Rule 1: CONCRETENESS
Failure: Generic risks not tied to stated components
Every critical_risks item must name at least one specific component the user mentioned. 'Retrieval can be slow' = failed output. 'Synchronous Pinecone retrieval will compound tail latency as corpus grows' = correct.
Rule 2: NO INVENTED NUMBERS
Failure: Fabricated benchmarks presented as fact
Never output invented benchmarks. Reference real numbers the user provided, or reason directionally: 'as rows grow, Y becomes bottleneck because Z' — not 'Y fails at 250 QPS.'
Rule 6b: STRENGTHS RULE
Failure: Praising tool capabilities instead of user decisions
'GPT-4 provides high-quality generation' is not a strength. Only list a strength if it reflects an explicit architectural decision the user stated. 2 honest strengths beat 3 invented ones.
Rule 8: CONSERVATIVE SCORING
Failure: Optimistic scoring on underspecified dimensions
Never award above 3/5 when critical information is missing. Missing info must reduce the score, not be silently assumed away. Security can't exceed 3 without auth strategy + tenant isolation + data residency + audit logging.
Rule 12: NEVER INVENT TOPOLOGY
Failure: Inventing pod/shard counts and criticizing them
Explicit prohibition on pod count, shard count, replica count, namespace count, region configuration — any infrastructure-level specifics not provided by the user.
Rule 13: KILLER INSIGHT QUALITY BAR
Failure: Killer insight duplicating top critical_risk or stating the obvious
Counterintuitive means contradicts what a competent non-expert would naturally focus on. 'The sub-2s SLA is more likely broken by GPT-4 generation latency than by retrieval latency' passes. 'You need monitoring' fails.
THE DISCRIMINATING TEST
On the canonical input (1M users, OpenAI embeddings + Pinecone + GPT-4, 10K DAU, sub-2s SLA), the output must surface:
“The sub-2-second SLA is more likely broken by GPT-4 generation latency than by retrieval latency.”
This is what a mid-level engineer focused on Pinecone optimization would miss. Swap the system prompt with “please review my GenAI architecture” and the output degrades immediately to generic. The 13 rules are doing real work.
DOMAIN KNOWLEDGE ENCODED IN THE PROMPT
- Vector DB behavior: API quota exhaustion hits before infrastructure limits at DAU scale
- Retrieval contamination: shared index namespaces leak tenant data in multi-tenant deployments
- Cost structure: generation token cost dominates infrastructure cost; chunking strategy controls unit economics more than retrieval tuning
- Embedding drift: corpus changes make index stale; retrieval quality degrades invisibly
- Reranker tradeoff: synchronous reranking adds tail latency under load, often not worth the precision gain
- Compliance: customer data sent to external model providers may violate DPA/data residency requirements
EXPLICITLY DEFERRED TO V2
— Streaming response (currently 15s blank loading state)
— Automated evals / regression harness
— Persistent review history
— Multi-turn refinement
— Diagram upload / IaC parsing
— Mobile-optimized layout
— Error telemetry
— Adversarial input handling
12-day portfolio/demo project. Scope was deliberate.