AI Solutions Architect Bootcamp — Day 1, Lesson 2: Tokens and Tokenization
You completed Lesson 1 by correctly separating LLM interpretation from authoritative business decisions. Now we'll study something that directly affects AI architecture, performance, cost, scalability, and model selection: tokens.
Lesson 2 of 7 · 25–30 minutes · AWS Bedrock · AI Architect level
By the end of this lesson, you should be able to explain how tokenization works, calculate LLM request costs, understand input and output token budgets, and optimize an enterprise AI application.
1. What exactly is a token?
A token is a unit of information that a language model processes. LLMs do not directly process text as humans read it — text is converted into numerical token IDs before the model processes it. A token might represent an entire word, part of a word, punctuation, a space combined with a word, or another sequence of characters.
For example, consider the sentence "Understanding tokenization is important." It might illustratively break down into six tokens:
Understand | ing | tokenization | is | important | .
Six hypothetical tokens. The actual segmentation depends on the model's tokenizer; this is not a verified tokenizer output.
A common misconception is: one word = one token. That is not true. Different models can tokenize the same text differently — a common word may be represented by one token, while an uncommon technical term may be split into several tokens. This matters because LLM providers commonly measure usage and billing in tokens, not words.
2. How tokenization works internally
Let's follow the processing pipeline:
| Raw text "Approve the insurance claim" |
→ | Tokenizer Splits text by vocabulary and rules |
→ | Token IDs e.g. [1042, 872, 5291, 318] |
→ | Embedding layer IDs → vectors |
→ | Transformer Processes in context |
The token IDs shown are fictional examples, not actual IDs from a specific model.
What is a tokenizer vocabulary?
Every tokenizer has a vocabulary mapping token IDs to token representations. A simplified example:
| Token ID | Token representation |
|---|---|
| 1042 | Approve |
| 872 | the |
| 5291 | insurance |
| 318 | claim |
These are illustrative mappings. The model receives the token IDs, which are converted into learned vector representations before being processed by Transformer layers.
Why do we split words into smaller units?
Imagine your AI application receives technical terms like LangGraph, Kubernetes, Supabase, or hyperautomation. A tokenizer may recognize some as single tokens and split others into multiple pieces. Subword tokenization allows models to represent unfamiliar or rare words without needing a separate vocabulary entry for every possible word.
The important architectural lesson: technical terminology, code, JSON, and multilingual content can have different token densities.
3. Input tokens vs. output tokens
This distinction is critical for AWS AI applications. Every model request has two major usage components: input tokens (instructions, user request, retrieved context) and output tokens (the model-generated response). Input and output tokens are often priced differently — output tokens can be substantially more expensive than input tokens, depending on the model.
Example: AI insurance claims assistant
| Component | Tokens |
|---|---|
| System instructions | 300 |
| User's question | 100 |
| Retrieved policy text | 1,800 |
| Previous conversation context | 300 |
| Total input | 2,500 |
| Generated response (output) | 500 |
For one request: 2,500 input + 500 output = 3,000 tokens total. For 10,000 requests at the same usage per request: 10,000 × 3,000 = 30,000,000 tokens — 30 million. As an AI Solutions Architect, you must understand how this scales into operational costs.
4. AWS Bedrock: token pricing and cost architecture
Amazon Bedrock provides access to different foundation models. Pricing varies by model, AWS Region, inference tier, and other features. Let's use a hypothetical pricing scenario — input tokens at $0.003 per 1,000 and output tokens at $0.015 per 1,000 — to understand the math. These are teaching assumptions, not current AWS prices.
Interactive AWS token cost calculator
Token Cost Simulator
Illustrative rates — adjust the numbers to see how cost scales.
Excludes additional charges such as retrieval, storage, provisioned capacity, networking, and application infrastructure. Actual billing may also account for caching and other model-specific features.
Architect insight
Suppose an agent performs five LLM calls to complete one user request. If every call has the same token usage as the example above, one user request could consume five times as many model tokens as a single-call application. That's why agentic AI architecture requires careful attention to the number of model calls, accumulated context, and output lengths.
5. How to count tokens in Amazon Bedrock
Amazon Bedrock supports token counting for supported models through its CountTokens API. This helps estimate input size before invoking the model. Here is a Python example using Boto3:
import boto3
bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")
model_id = "amazon.nova-lite-v1:0"
response = bedrock.count_tokens(
modelId=model_id,
input={
"converse": {
"messages": [
{"role": "user", "content": [{"text": "Explain tokenization in AI."}]}
]
}
}
)
print("Input tokens:", response["inputTokens"])This requires AWS credentials, permission to call the API, and a supported model in the selected Region. Model access and availability must be checked in your AWS account.
For actual inference using Bedrock's Converse API, usage information can be obtained from the response:
response = bedrock.converse(
modelId=model_id,
messages=[{"role": "user", "content": [{"text": "Explain tokenization in AI."}]}],
inferenceConfig={"maxTokens": 300}
)
usage = response["usage"]
print("Input:", usage["inputTokens"])
print("Output:", usage["outputTokens"])
print("Total:", usage["totalTokens"])The important distinction: CountTokens estimates the tokenized input, while actual inference usage reports the tokens consumed by the completed request.
Official reference: Amazon Bedrock CountTokens documentation.
6. Five ways AI architects optimize token usage
Consider a memory-enabled AI assistant. A user asks: "What did I discuss about my insurance last month?" A naive architecture might send the user's entire stored history to the LLM — that's inefficient and can expose unnecessary information. A better architecture uses targeted retrieval.
| Inefficient design | Optimized design |
|---|---|
| User question → send entire history (100,000 tokens) → LLM | User question → relevant retrieval (3,000 tokens) → LLM |
Illustrative input sizes. The optimized approach retrieves only authorized, relevant context, but retrieval quality must also be evaluated.
Five common optimization techniques:
- Selective retrieval — fetch only the relevant document chunks or memory records.
- Prompt optimization — remove unnecessary repeated instructions while retaining essential safeguards.
- Conversation summarization — compress older conversational history when appropriate, preserving critical facts.
- Model routing — use smaller, less expensive models for straightforward tasks and stronger models for complex reasoning.
- Caching and output limits — reuse eligible repeated content and constrain response lengths where appropriate.
Reducing tokens isn't the only goal. You must balance cost, accuracy, latency, security, and reliability.
7. Architecture scenario: 1 million requests per month
Imagine you're designing an enterprise claims assistant:
| Monthly requests | 1,000,000 |
| Average input | 4,000 tokens / request |
| Average output | 800 tokens / request |
| Average total | 4,800 tokens / request |
At this scale, the system processes approximately 4.8 billion tokens monthly. But an architect must look beyond monthly totals — you also need to estimate peak requests per second, concurrent inference calls, tokens per minute, latency requirements, model quotas, and retry behavior. A solution that looks affordable on a monthly spreadsheet can still fail under peak load.
Lesson 2 — Architect Knowledge Check
Lesson 2 Assessment
Target: 4/5 or better.
One final architect-level thought: token optimization is not just about saving money. It is also about controlling the information that reaches the model, protecting sensitive data, maintaining response quality, and keeping AI systems predictable under production workloads.
Next in this series: Lesson 3 — Context Windows and Memory. We'll learn why even a model with a very large context window does not automatically have long-term memory, and how to architect that memory correctly.
No comments:
Post a Comment