czay.dev
0/5
Roadmaps

LLM vs. Karar Modeli (Decision Model) & Shadow Mode Geçiş Rehberi

This interactive architectural guide is based on the real-world engineering case study by AWS engineers Carr Onstott and Mehul Jaiswal, who replaced Claude Sonnet 4.6 with the Gemma 4-based Strands Decider E4B decision model in one of the world's largest internal ticketing systems—cutting the cost per 1 million tickets from $22,200 to $500 (a 98% reduction, 44x cheaper).

The most expensive mistake modern software teams make is wiring a massive generative LLM to a closed-catalog routing or classification task and paying it to generate hundreds of tokens of JSON and reasoning on every call. Use the 4-stage checklist below to audit your own AI endpoints and safely migrate classification workloads using a 3-way Shadow Mode switch.

  • CoreKnow this well before you apply
  • Nice to haveShows up a lot in job posts, gives you an edge
  • LaterOnce you're on the job, when you need it

Click a topic to see what you need to know and where to learn it.

  1. 01

    Stage 1: Diagnosis — Why Are We Generating 170 Output Tokens?

    Identify hidden bottlenecks in your system that require probability distribution (Decision Model) rather than text generation (Generative AI).

  2. 02

    Stage 2: Prompt Compression & The JSON Key Order Trap

    Compress category definitions to fit the 4,096-token window and prevent the JSON key ordering quirk that shifts probabilities by up to 0.19.

  3. 03

    Stage 3: 3-Way Shadow Mode Switch (BEDROCK -> DUAL -> DECIDER)

    Never flip a switch and hope: run both models in parallel on real traffic with a 45-second rollback guarantee.

  4. 04

    Stage 4: Isolated GPU Hosting & 98% Cost Verification

    Run the model on a network-isolated single A10G GPU instance and verify the $22,200 -> $500 cost drop.

Topic by topic

Every topic on the map with its description, what to know and resources — in one list.

  1. 01Stage 1: Diagnosis — Why Are We Generating 170 Output Tokens?

    • Auditing Closed-Catalog Workloads· Core

      In Amazon's internal ticketing system, customer escalations, capacity requests, alarms, and security findings mapped to 20 fixed categories. Claude Sonnet 4.6 read 6,554 input tokens per ticket and generated 170 output tokens of JSON and reasoning.

      What you should know

      • Audit your LLM calls: Is the generated JSON text actually shown to users, or does your code merely parse a 'category_id' or 'route' field for an if/switch branch?
      • If your option set is fixed (e.g., 5 to 50 labels + 1 'none' option) and no explanation text is needed, this is a Decision Model job, not a generative LLM job.
      • Output tokens are typically 5x more expensive than input tokens ($15/1M vs $3/1M) and dominate latency. Dropping output tokens from 170 to 1 immediately cuts latency by 78%.

      Resources

  2. 02Stage 2: Prompt Compression & The JSON Key Order Trap

    • Shrinking Input Tokens from 6,554 to 2,009· Core

      Amazon's production label descriptions for Sonnet 4.6 were longer than Strands Decider's entire 4,096-token window. The team rewrote each of the 20 labels into a crisp 1-2 sentence criterion.

      What you should know

      • Choose the Gemma 4-based E4B build over the 2B build; in Amazon's tests, E4B beat both 2B and production Claude Sonnet 4.6 in accuracy.
      • Rewrite each option as a 1-2 sentence criterion and always include a 'none' option so the model isn't forced into a bad guess.
      • On long tickets or documents, drop the oldest comments first so the payload stays cleanly inside the 4,096-token context window.
    • Critical Trap: JSON Key Ordering (0.19 Probability Shift)· Core

      The most surprising finding from Amazon's engineers: sending the exact same ticket content with JSON keys in a different order shifted output probabilities by up to 0.19 (19%)!

      What you should know

      • Before sending a JSON payload to a decision model, deterministically serialize all objects with alphabetically sorted keys (sort_keys=True).
      • Tune a strict confidence threshold before applying a label; if the score falls below threshold, abstain or route to fallback rather than applying a low-confidence label.

  3. 03Stage 3: 3-Way Shadow Mode Switch (BEDROCK -> DUAL -> DECIDER)

    • 3-Mode Execution via Feature Flag / AppConfig· Core

      Triggered by SQS, the labeling Lambda reads a 3-way setting from AWS AppConfig on every message: BEDROCK (Sonnet only), DUAL (both run in parallel; Sonnet applies, disagreements are logged), and DECIDER (Strands Decider only).

      What you should know

      • Step 1 (BEDROCK / Legacy LLM): Your baseline production path; the new decision model is not called yet.
      • Step 2 (DUAL / Shadow Mode): Both Sonnet 4.6 and Strands Decider are called in parallel on live traffic. Sonnet's label is applied, while every disagreement is logged and judged.
      • Step 3 (DECIDER / Full Cutover): Once disagreement analysis proves the decision model beats the LLM, flip the switch to DECIDER. Rolling back is a 45-second config change.

  4. 04Stage 4: Isolated GPU Hosting & 98% Cost Verification

    • SageMaker ml.g5.xlarge (1x A10G 24GB) & Network Isolation· Core

      Strands Decider E4B uses about 19 GiB of VRAM on a single 24 GB NVIDIA A10G GPU (ml.g5.xlarge at $1.408/hr) and completes inference in 640 ms (897 ms end-to-end).

      What you should know

      • Copy model weights from S3 prior to startup and completely isolate the container network with zero outbound internet access.
      • One ml.g5.xlarge instance processes ~5,600 tickets/hour: $250 per 1M tickets at full utilization, or ~$500 in practice with burst headroom (44x cheaper than Sonnet 4.6's $22,200!).
      • Bonus win: Because call latency drops from 4,215 ms to 897 ms, average AWS Lambda execution duration and compute billing also drop by 74%!

      Resources