AI Solutions Architect Bootcamp — Day 1, Lesson 3: Context Windows and AI Memory
You passed Lesson 2 by working through tokenization and token cost math. Now we move to one of the most important topics in enterprise AI architecture: how an AI system handles information over time.
Lesson 3 of 7 · 30–40 minutes · AWS Bedrock · AI Architect level
By the end of this lesson, you'll understand the difference between a model's context window and application memory, how to design persistent AI memory, how LangGraph handles state, and how AWS services fit into the architecture.
1. What is a context window?
A context window is the maximum amount of tokenized information a model can process within a single inference request, subject to that model's input and output limits. Think of it as the model's working space.
Suppose you're designing an AI insurance assistant. A user asks: "What happened with the claim I submitted three months ago?" The model cannot automatically search your company's historical claim records — the application must retrieve the relevant information and supply it to the model.
Example: a 128,000-token context window
| Allocation | Tokens |
|---|---|
| System instructions | 12,800 |
| Conversation history | 19,200 |
| Retrieved documents | 44,800 |
| Reserved output | 12,800 |
| Remaining capacity | 38,400 |
Illustrative allocation, not a particular Bedrock model's specifications. Actual model limits and accounting rules vary.
For this simplified example: 12,800 + 19,200 + 44,800 + 12,800 + 38,400 = 128,000 tokens. The model's input, generated output, and applicable internal token allocations must fit within its supported limits.
Important distinction
A model advertising a 128K-token context window does not mean it can always accept 128K input tokens and then generate unlimited output. Architects must check both the total context capacity and the model's separate maximum output allowance.
2. Context window is not memory
This is the most important distinction in today's lesson. Context is the information supplied for the current inference. Memory is information an application preserves and makes available across interactions. A model can respond to previous conversations only when the relevant information is supplied again, recovered through a supported stateful service, or otherwise made available through the application.
| Context window | Persistent memory |
|---|---|
| Temporary working information used during inference. Bounded by model token limits. | Stored information retrieved or loaded across sessions. Managed by application storage and retrieval. |
Practical example: a memory-enabled insurance assistant
Imagine a user tells your application on Monday: "My vehicle insurance expires on December 15." On Friday, the user asks: "When does my insurance expire?"
If the Friday request does not contain Monday's information, the model generally cannot answer reliably from that previous conversation alone. But if your application saved the insurance expiry date in a database and retrieves it on Friday, the model can answer using that information.
| Monday — user provides information | "Insurance expiry: December 15" → validated and saved to a user-scoped database record |
| Friday — user asks a question | "When does my insurance expire?" → application retrieves authorized memory (expiry date, source, year) → LLM receives the retrieved context and generates a grounded response |
The key is that the database stores the information. The LLM interprets the question and helps formulate the answer. For a straightforward date lookup, you may not even need an LLM — a deterministic database query could answer faster, more cheaply, and more reliably. That's an important AI Solutions Architect decision: know when not to use AI.
3. The four kinds of memory an AI architect should understand
In practice, AI memory is not one database or one feature. Different information needs different storage and retrieval strategies.
| Memory type | Purpose | Example |
|---|---|---|
| Working context | Information used in the current model call | Current question and retrieved documents |
| Conversation memory | Preserve the state of an ongoing conversation | Earlier messages in the same support case |
| Long-term memory | Remember information across conversations | Insurance expiry date or user preference |
| Procedural memory | Preserve instructions and reusable processes | Rules for handling claim approvals |
The first three describe where and how information is available. Procedural memory describes the kind of information being retained, so these categories can overlap.
How LangGraph implements memory
LangGraph offers two important persistence mechanisms:
- Checkpointer — saves graph state for a conversation thread, including messages and workflow progress.
- Store — saves application-defined information that can be accessed across different conversation threads.
These are distinct but complementary. Consider an enterprise claims workflow: a user submits a claim, the agent extracts information, and the workflow pauses for human approval. The next day, an approver returns. A LangGraph checkpointer can preserve the workflow state so execution can resume instead of restarting the entire process. Separately, a long-term store can retain authorized user preferences or relevant facts for future conversations.
This maps closely to concepts you already know from enterprise case management: a persisted case state is different from reusable customer data.
4. Designing enterprise AI memory on AWS
Here is a reference architecture combining Bedrock, LangGraph, and persistent storage:
| Web / mobile / enterprise client |
| ↓ API Gateway + authentication |
| ↓ LangGraph orchestration service — state management · tool routing · context assembly |
| ↓ Checkpoint DB (conversation and workflow state) · Long-term store (facts and preferences) · Retrieval index (semantic search) |
| ↓ Context builder — select, authorize, rank, trim, and format information |
| ↓ Amazon Bedrock — foundation model inference |
| ↓ Validate → respond → persist updates |
Conceptual architecture. Database and model calls are orchestrated by the application; the vertical arrangement illustrates the logical flow rather than mandatory sequential execution.
AWS service choices
| Requirement | Possible AWS service |
|---|---|
| Model inference | Amazon Bedrock |
| API access | Amazon API Gateway |
| Agent execution | ECS/Fargate or Lambda, depending on workload |
| Relational memory and checkpoints | Amazon RDS / Aurora PostgreSQL |
| Key-value memory | Amazon DynamoDB |
| Document storage | Amazon S3 |
| Vector search | OpenSearch or a supported vector-enabled database |
| Secrets and encryption | Secrets Manager and KMS |
| Observability | CloudWatch and tracing tools |
AWS also documents DynamoDB-backed LangGraph long-term memory with namespaced records and optional semantic search.
Architecture decision: don't automatically introduce a vector database for every memory record. Exact fields such as policy numbers, expiration dates, and account identifiers are often better served by structured database queries. Semantic search is useful when a question requires finding conceptually related information.
5. How Amazon Bedrock handles conversation context
The Bedrock Converse API supports multi-turn conversations, but your application ordinarily maintains the message history and includes relevant messages in subsequent requests. A standard Converse request is not, by itself, a persistent memory system. AWS also offers other API patterns with different state-management capabilities, so the distinction is specific to the API you choose.
Python example: managing conversation history
import boto3
client = boto3.client("bedrock-runtime", region_name="us-east-1")
model_id = "amazon.nova-lite-v1:0"
messages = [
{"role": "user", "content": [{"text": "My insurance expires December 15, 2026."}]}
]
first = client.converse(
modelId=model_id,
messages=messages,
inferenceConfig={"maxTokens": 200}
)
messages.append(first["output"]["message"])
messages.append({"role": "user", "content": [{"text": "When does my insurance expire?"}]})
second = client.converse(
modelId=model_id,
messages=messages,
inferenceConfig={"maxTokens": 200}
)
print(second["output"]["message"]["content"])The second request includes the first user message and the first assistant response. That's how the model can answer the follow-up. If you omit the earlier information, the model has no reliable basis for recalling that expiry date.
The messages list above lives only in the Python process. To preserve it across restarts, you need persistent storage or a state-management service.
6. Context management: why more information isn't always better
Imagine an AI assistant has accumulated 200,000 tokens of conversation history. Even if the selected model supports a sufficiently large context window, passing the entire history may be a poor design. Long contexts can increase latency and cost, and relevant information may be harder for the model to use consistently.
An architect typically considers four strategies:
- Trimming — keep the most recent messages while removing older content that is no longer needed. Preserve required system instructions and valid conversation structure.
- Summarization — convert older conversations into shorter summaries, while retaining critical facts separately. Summaries can lose detail, so don't use them as the only source of truth.
- Selective retrieval — retrieve relevant, authorized facts and documents based on the current question. This becomes central to RAG, which we'll study in Lesson 6.
- Structured memory — store important information as explicit fields, with provenance, timestamps, and update controls. This is especially valuable for dates, policy numbers, and user-confirmed facts.
7. Hands-on exercise: build a context budget
Suppose your AWS Bedrock model supports a hypothetical total context budget for this exercise. You need to reserve some capacity for generated output and some for system instructions. How much capacity remains for conversation history and retrieved documents?
Interactive Context Budget Planner
Context Budget Planner
Adjust the sliders to see how the remaining budget for retrieved documents changes.
This is the remaining capacity before accounting for message-format overhead and other applicable tokens.
The goal isn't to fill every available token. It's to select the smallest context that reliably supports the user's task.
8. Enterprise architecture challenge
Before the assessment, consider this scenario. You're designing an AI assistant for 100,000 enterprise users. It must remember previous conversations, retrieve documents, respect tenant boundaries, and support deletion of user data.
A sound architecture needs more than a large context window. It needs authenticated user identity, tenant-scoped queries, access checks before retrieval, durable memory, encryption, retention policies, and deletion across primary records, derived indexes, and other retained copies according to policy.
For example, when a user edits an incorrectly captured fact, the system should update the authoritative record and refresh or invalidate any derived summaries or embeddings. Otherwise, the AI may continue retrieving outdated information. This is the difference between a chatbot prototype and an enterprise AI platform.
Lesson 3 — Knowledge Check
Lesson 3 Assessment
Covers context budgeting, memory design, LangGraph, and AWS Bedrock. Target: 4/5 or better.
Today's core principle: a large context window helps an LLM process more information in one request. A well-designed memory architecture determines what information should be preserved, retrieved, updated, protected, and supplied to the model over time. That's a foundational distinction for an AI Solutions Architect.
Next in this series: Lesson 4 — Inference, Embeddings, and the path toward RAG and Agentic AI.
No comments:
Post a Comment