AI Solutions Architect Bootcamp — Day 1, Lesson 4: LLM Inference and AWS Bedrock Architecture
You've now covered LLM architecture, tokenization, context windows, and persistent memory. This lesson connects those foundations to production engineering.
Lesson 4 of 7 · 35–45 minutes · AWS Bedrock · AI Solutions Architect level
Our central question: when an application sends a request to an LLM, how does the model generate its response, and how do we design the infrastructure to deliver that response quickly, reliably, and affordably? By the end, you'll understand inference, decoding parameters, streaming, latency, throughput, and how to make AWS Bedrock architecture decisions.
1. What is LLM inference?
Inference is the process of using an already-trained model to produce an output from a given input. Training and inference are different workloads.
| Training | Inference |
|---|---|
| Updates model weights | Normally uses fixed model weights |
| Learns patterns from training data | Applies learned patterns to new input |
| Usually computationally intensive | Runs whenever an application requests a prediction |
| Produces a trained or fine-tuned model | Produces generated content or predictions |
For example, a user asks your enterprise assistant: "Summarize this insurance claim in three sentences." The application constructs a prompt, sends it to a model through Amazon Bedrock, and receives generated text. That's inference.
What happens inside the model?
| 1. Prompt arrives | Instructions + claim data + user request |
| 2. Tokenization | Text is converted into token IDs |
| 3. Prefill | The model processes the input context |
| 4. Decode | Generate output tokens autoregressively |
| 5. Output processing | Detokenization, validation, response delivery |
Two phases worth remembering:
Prefill — the model processes the supplied input tokens. Longer prompts generally require more computation and can increase the time before generation begins.
Decode — the model generates subsequent tokens, usually one at a time for an autoregressive text model. Implementations use techniques such as KV caching to avoid recomputing all prior attention work from scratch.
This distinction explains why input length, output length, and streaming affect performance differently.
2. How does the model choose the next token?
Suppose the model is generating: "The insurance claim has been..." It computes scores for possible next tokens, converts them into a probability distribution, and applies a decoding strategy.
| Candidate token | Illustrative probability |
|---|---|
| approved | 55% |
| submitted | 25% |
| reviewed | 12% |
| rejected | 8% |
Invented probabilities for teaching. Actual model probabilities depend on the entire context and are not necessarily exposed by Bedrock.
The decoding strategy determines whether the model consistently favors the most likely token or samples among multiple candidates. This leads to temperature and top-p — try both in the explorer below.
3 & 4. Temperature and top-p: the decoding parameter explorer
Temperature reshapes the model's token probability distribution before sampling. Lower temperatures sharpen the distribution, making high-probability tokens more dominant. Higher temperatures flatten it, increasing the chance of sampling less probable tokens.
Top-p (nucleus sampling) controls which candidate tokens are eligible for sampling at all: the model considers the smallest set of highest-probability tokens whose cumulative probability reaches or exceeds the configured threshold.
Decoding Parameter Explorer
Starting from the same four illustrative candidate tokens, adjust temperature and top-p to see how sampling changes.
This simulator applies temperature scaling and nucleus sampling to four invented logits. It demonstrates the mechanics, not the exact behavior of a particular model or API.
How an AI Solutions Architect uses temperature
| Use case | Typical approach |
|---|---|
| Insurance claim field extraction | Low temperature |
| Policy document Q&A | Low temperature |
| Structured JSON generation | Low temperature, schema validation |
| Marketing ideas | Moderate or higher temperature |
| Creative storytelling | Higher temperature may be useful |
These are design starting points, not universal numeric rules. Supported temperature ranges and parameter behavior vary by model.
Critical: low temperature does not guarantee factual accuracy or perfectly deterministic responses. It reduces sampling randomness; it does not fix missing knowledge, hallucinations, or unreliable tools. For regulated workflows, pair low-randomness generation with validation, authoritative data sources, and business rules.
Temperature vs top-p
| Temperature | Top-p |
|---|---|
| Reshapes token probabilities | Restricts the candidate set |
| Influences randomness | Influences sampling diversity |
| Does not inherently remove candidates | Excludes candidates outside the selected probability mass |
| Often tuned for response consistency | Often tuned for diversity and variation |
Model providers may apply these controls differently. For experiments, change one parameter at a time so you can identify its effect.
Other decoding controls
Maximum output tokens (maxTokens) places an upper bound on generated tokens. It helps control response length and cost, but a response may be truncated if the limit is too small. Stop sequences tell the model to stop generation when certain sequences appear, subject to model support. Top-k, when supported, limits consideration to a fixed number of high-scoring tokens — it is not a universal Converse parameter. AWS exposes common controls such as maxTokens, temperature, topP, and stopSequences through Bedrock's inference configuration; support and valid ranges depend on the model.
5. Latency vs throughput
These are two different performance characteristics. Latency measures how long a request takes. Throughput measures how much work the system processes over time. A system can have good latency for one user but poor throughput under heavy load. Conversely, a system may process many requests concurrently while individual users still experience long response times.
Three performance metrics you must know
| Metric | Meaning | Why it matters |
|---|---|---|
| TTFT | Time to first token | Perceived responsiveness, especially for streaming |
| End-to-end latency | Time until the complete answer arrives | Total user wait time |
| Output tokens/sec | Speed of output generation | Response completion time |
For example, suppose time to first token = 0.8 seconds, output tokens = 200, and the average generation rate after the first token ≈ 50 tokens/second. A rough estimate for full response time is: 0.8 + (199 ÷ 50) = 4.78 seconds. This is an approximation — actual timing also includes transport, scheduling, validation, and other application overhead.
What affects inference latency?
The most important factors include input token count, generated output length, model architecture and size, available compute, concurrency, queueing, and tool or network delays. For agentic AI, the number of sequential LLM calls can become especially important: an agent that makes four sequential model calls may have much higher end-to-end latency than a single-call chatbot, even if each individual call is reasonably fast.
6. Streaming: why ChatGPT-style responses feel faster
Without streaming, the application typically waits until the response is complete before displaying it. With streaming, it can display generated text incrementally — "The claim..." → "The claim was..." → "The claim was received..." Streaming can improve perceived responsiveness without necessarily reducing total model computation time.
Amazon Bedrock supports ConverseStream for compatible models. The streaming operation requires the appropriate model invocation permissions.
AWS Python example: streaming inference
import boto3
client = boto3.client("bedrock-runtime", region_name="us-east-1")
# Replace with an available model ID or inference profile.
model_id = "amazon.nova-lite-v1:0"
response = client.converse_stream(
modelId=model_id,
messages=[
{"role": "user", "content": [{"text": "Explain how an insurance claim is processed."}]}
],
inferenceConfig={"maxTokens": 300, "temperature": 0.2, "topP": 0.9}
)
for event in response["stream"]:
if "contentBlockDelta" in event:
delta = event["contentBlockDelta"]["delta"]
if "text" in delta:
print(delta["text"], end="", flush=True)This is a demonstration, not a production-ready service. A real implementation must handle stream errors, timeouts, cancellations, non-text events, and partial responses. The selected model must support the requested parameters and streaming.
Security consideration: if the response requires validation before release, streaming unvalidated text directly to the user may be inappropriate. In those cases, buffer or validate the relevant output before presenting it.
7. AWS Bedrock production inference architecture
Let's design an enterprise AI assistant that must handle high request volumes:
| Web / mobile / enterprise client |
| ↓ API Gateway + authentication |
| ↓ AI orchestration service — validation · model routing · context assembly |
| ↓ Inference control layer — concurrency limits · retry/backoff · timeouts · budgets |
| ↓ Amazon Bedrock — on-demand / inference profile / provisioned throughput |
| ↓ Output validation + response |
The three capacity options to understand:
| Option | Architectural use |
|---|---|
| On-demand inference | Flexible workloads with usage-based pricing |
| Cross-Region inference profiles | Route supported requests across eligible Regions to improve capacity and resilience |
| Provisioned Throughput | Purchase dedicated inference capacity for supported models and workloads |
Cross-Region inference profiles can route requests across multiple Regions, while Provisioned Throughput is a separate capacity mechanism. Inference profiles do not currently support Provisioned Throughput.
Important: data residency
If a regulated customer requires inference processing within the United States, a geographic US inference profile may be appropriate, subject to verifying its destination Regions and applicable policies. A global profile can route processing across supported commercial Regions worldwide, so it may not meet that customer's requirements. This is a classic AI Solutions Architect interview question: performance and availability improvements must not violate compliance constraints.
8. Capacity planning: requests per minute vs tokens per minute
Imagine your application receives 60 requests per second, each using an average of 2,000 input tokens and 500 output tokens. That gives 60 × 60 = 3,600 requests/minute, and 3,600 × 2,500 = 9,000,000 tokens/minute.
Interactive Inference Capacity Planner
Inference Capacity Planner
Estimate request volume, token throughput, and concurrency from basic workload assumptions.
Uses average workload assumptions and Little's Law (concurrency ≈ arrival rate × average duration) for estimated concurrency under steady conditions. AWS quota accounting may differ due to model-specific token burndown rates, caching, and reserved output accounting.
Why this matters: Amazon Bedrock can enforce model-specific requests-per-minute and tokens-per-minute quotas. Some models have different quota consumption rates for output tokens. You must verify the exact quota for the model, Region, and inference profile before deployment. A production design also needs traffic burst analysis, rate limiting, exponential backoff with jitter, bounded retries, and graceful degradation.
9. Hands-on exercise: build an inference benchmark
Your practical assignment is to compare three inference configurations using the same prompt:
import boto3
import time
client = boto3.client("bedrock-runtime", region_name="us-east-1")
model_id = "amazon.nova-lite-v1:0"
prompt = (
"Extract the claim type, damaged item, "
"and incident location from this message: "
"'My car bumper was damaged near the airport.' "
"Return a concise response."
)
configs = [
{"temperature": 0.1, "topP": 0.9},
{"temperature": 0.5, "topP": 0.9},
{"temperature": 0.9, "topP": 0.9},
]
for config in configs:
start = time.perf_counter()
response = client.converse(
modelId=model_id,
messages=[{"role": "user", "content": [{"text": prompt}]}],
inferenceConfig={"maxTokens": 150, **config}
)
elapsed = time.perf_counter() - start
print("\nConfiguration:", config)
print("Elapsed seconds:", round(elapsed, 3))
print("Usage:", response.get("usage"))
print("Metrics:", response.get("metrics"))
print("Output:", response["output"]["message"]["content"])Run this only when you have AWS credentials, model access, and billing controls configured. For a meaningful experiment, repeat each configuration across multiple requests, collect p50/p95 latency, compare token usage, and score the output against expected structured fields. A single request per configuration is not enough to draw reliable conclusions. Also remember that temperature changes randomness, not necessarily quality — the best setting depends on the workload and model.
Your architect's decision checklist
For each production inference workload, you should be able to answer:
- What is the required response latency, including p95?
- What are the expected average and peak request rates?
- How many input and output tokens are expected?
- Which model offers acceptable accuracy at the required cost?
- Is streaming appropriate, or must output be validated before release?
- What are the AWS quotas and regional restrictions?
- How will throttling, failures, retries, and fallback behavior work?
Lesson 4 — Architect Knowledge Check
Lesson 4 Assessment
You need at least 4 out of 5 to proceed. The questions test architectural reasoning, not just definitions.
The key takeaway: production AI architecture is not simply choosing a powerful model. It's designing a system that meets accuracy, latency, throughput, security, and cost requirements under real operating conditions.
Next in this series: Lesson 5 — Embeddings and Vector Search, the mathematical and architectural foundation of semantic search, AI memory retrieval, and RAG.
No comments:
Post a Comment