Effective Token Usage Guide · V1.0

1.1 If You Do Not Know What Is Expensive, You Do Not Know What to Save

1.2 Token Profile of a Single Request

To understand where the money goes, you must first understand the token composition of a single request.

The developer looks only at total cost, not its composition:

- Output has a high share, so they assume the model is too verbose;
- In reality, the Prompt does not constrain the output format, so the model responds freely;
- The optimization effort targets the wrong problem.

After every critical task, the developer routinely checks `/cost`:

- They notice that the proportion of cache-read tokens remains below 20% for a certain task type;
- They check whether the prefix changes frequently or a Skill is reloaded every time;
- After adjustment, the Cache hit rate remains above 70%;
- Input cost for the same task falls to one-third of its previous level.

1.3 Key Metrics and Target Values

Based on the five fields above, we define a set of SLIs that the team must monitor continuously.

The team looks only at total monthly cost and does not track the metrics above:

- Cache hit rate remains around 30%;
- The Output/Input ratio reaches 5;
- Nobody notices because total cost is still within budget;
- By the time the budget becomes tight, substantial structural waste has accumulated.

The team reviews three metrics in every weekly meeting:

- Cache hit rate: 78%
- Output/Input ratio: 1.2
- Average tokens per task: p50 = 45K, p95 = 180K

Outliers, such as a Skill that suddenly raises p95, receive focused remediation.
Within six months, token cost falls by 55% for the same workload.

1.4 Team-Level Billing Attribution

Aggregate totals are nowhere near enough. Costs must be attributed across multiple dimensions.

2.1 Your Request Is Usually the Cheapest Part

A one-line request: "Build a login feature." "Fix bug."

Result:

- The model misunderstands and enters multiple rounds of clarification;
- The number of tool calls doubles;
- Output grows and retries increase;
- Total cost rises instead.

**Root cause**: saving money in the wrong place.

"Please fix this bug. The reproduction steps are..."

- Compress 20K of Skill instructions to 8K by removing filler and merging duplicate rules;
- Limit the number of lines returned by tools (`head -n 100`);
- Reference the exact file with `@path/to/file` instead of making the model search blindly.

2.2 "It Remembers" Is an Illusion

LLMs have no memory.

When it appears that the model remembers, what actually happened is that all previous content was sent again.

User request
   ↓
Model decision
   ↓
Tool call
   ↓
Tool result (large!)
   ↓
Model processing
   ↓
Next tool call...

2.3 Five Types of Cost

In addition to Input and Output, three other types of cost are often underestimated.

2.4 Prompt Cache: The Foundation of Optimization

Prompt Cache caches the computation of a prefix, not the answer.

2.5 Quality Degradation in Long Contexts — Not Just More Expensive, but Less Effective

An excessively long context imposes three penalties:

Has the current session exceeded 20 turns?
├─ No  → Is Context usage above 50%?
│        ├─ No  → Continue
│        └─ Yes → Run /compact
└─ Yes → Can you still locate the critical information clearly?
         ├─ Yes → Run /compact
         └─ No  → Start a new session

2.6 Five-Layer Cost-Governance Model

3.1 Session Isolation

[Required] Each session must serve one clear objective. Do not mix fundamentally different tasks in the same session.

Session length grows with conversation history, so every subsequent request becomes more expensive. Create a separate session for each type of task.

Correct Example 1 — Open Multiple CodeBuddy Sessions

# Session 1: Fix a nil-pointer dereference in the order module
# Session 2: Refactor the authentication logic in the user service
# Session 3: Add unit tests for the auth module

Correct Example 2 — Run /clear After Completing a Task

User: Fix bug: http://github.com/project/issues/xxx
AI:   [Bug fix completed]
User: /clear
User: Create an implementation plan for the new requirement at http://github.com/project/issues/xxx

Incorrect Example

# In the same session:
# Fix a nil-pointer dereference in the order module → review an auth-module PR → write a technical
# plan for a new requirement → add unit tests
# Task types are mixed and the history context keeps growing

3.2 Compress Long Histories

[Required] When a session exceeds 20 turns, or when the history context has grown significantly and Context usage exceeds 50%, run /compact or create a new session.

The model does not need the complete trial-and-error process. It actually needs only the current objective, completed work, paths already ruled out, the current blocker, and the next steps.

Correct Example

AI:   [Completes one phase]
User: /compact
User: Continue with the second phase

Incorrect Example

# Dozens of failed attempts remain in the history during debugging
# After the bug is fixed, unrelated work continues in the same session
# Every subsequent request carries the irrelevant debugging history

3.3 Externalize Information

[Recommended] Persist long-lived information in the file system rather than relying on session memory.

Store long-lived information in the appropriate location:

• Project documentation such as README and CONTRIBUTING
• Memory files such as CODEBUDDY.md / CLAUDE.md
• Summary files / decision records
• Task lists / Issues

A session should carry only the current working state, not the complete history of the project.

Correct Example

# Record the database selection decision in docs/decisions/001-database-selection.md
# Record the team's API naming conventions in CODEBUDDY.md / CLAUDE.md
# Discuss only the interface currently being changed in the session

Incorrect Example

# Discuss database selection, API conventions, and exception-handling strategy
# in detail within the session
# Depend on the Agent to reference these discussions automatically later
# The session keeps growing, and other team members cannot easily find the decisions

3.4 Skill Management

[Required] Follow the principle "keep always-loaded Skills to a minimum; load infrequently used Skills at the project level." Do not install a large number of Skills indiscriminately at the user level.

Every Skill includes a Description, Instructions, Examples, and trigger logic. If this information remains resident in the context, it contributes to token usage on every request.

Correct Example

# User-level resident Skills: frequently used general-purpose Skills such as
# commit, code-review, and unit-test
# Project-level Skills: framework-specific workflows such as gRPC code generation,
# or tool-specific workflows such as database migration scripts
# Periodic cleanup: remove Skills that have not been used for more than two weeks

Incorrect Example

# Install 30+ user-level Skills at once, covering Python, Go, Rust, frontend,
# operations, and other technology stacks
# Only commit, code-review, and unit-test are used in day-to-day work
# Definitions for all other Skills remain in the context and continuously consume tokens

3.5 MCP Management

[Required] Keep the MCP toolset to the minimum necessary. Every additional MCP server adds another Tool Definition and increases tool-selection overhead.

Too many tools do more than increase the volume of definition text. They also:

• Expand the choice space and increase decision latency
• Increase the probability of incorrect tool calls
• Add to the definition payload carried by every request

Correct Example

# User-level resident tool: Headroom context compression
# Project level: no more than five MCP servers for the current project's core business
# Disable or remove: infrequently used MCP servers unrelated to the current project

Incorrect Example

# Install GitHub, Notion, TAPD, Gongfeng, Browser, Kubernetes, Docker, and multiple
# internal-system MCP servers at the user level
# The nominal feature set is comprehensive, but only Filesystem and Git are used frequently

3.6 Choosing Between CLI and MCP

[Required] Use MCP by default. Fall back to a CLI only when that CLI appears extensively in AI training data, as is the case for mainstream tools such as git, gh, kubectl, docker, npm, and pip.

The central tradeoff is this: a CLI avoids the cost of loading a Tool Definition, reducing tokens per turn. However, when the AI lacks reliable prior knowledge of a CLI, it can easily misspell a flag, use the wrong subcommand, or invent an argument, triggering a retry. As described in Chapter 2, one retry costs approximately one complete request (⭐⭐⭐⭐⭐), usually far more than the Tool Definition that was saved. MCP spends a small number of definition tokens on each turn in exchange for standardized parameter schemas, clearer descriptions, and fewer incorrect calls and retries.

Selection Guidelines

How to Evaluate Whether an AI Knows a CLI

"Mainstream and familiar to AI" is not an exact standard, but the criteria below can help. If uncertain, use MCP by default. The cost of a mistake is asymmetric: treating an obscure CLI as mainstream can trigger retries with ⭐⭐⭐⭐⭐ cost, while treating a mainstream CLI as obscure and using MCP adds only a few kilobytes of Tool Definition.

Supporting indicators such as Stars and download counts are references only and should not determine the choice by themselves. A direct test has the highest priority and takes only one message.

Correct Example — CLI Wins

# Inspect Git changes; the AI is highly familiar with git
git diff --stat
git log --oneline -10

# Create a PR; gh is extensively represented in training data
gh pr create --title "feat: add batch query for UserService" --body "..."

Correct Example — MCP Wins

# The AI has no prior knowledge of a proprietary internal release-system CLI
# and can easily misspell a flag and trigger a retry
# Wrap it as an MCP Tool with a standardized schema describing its parameters and behavior

# The AI must make a structured decision based on the previous result
# Example: query inventory → decide whether to scale based on the structured result
# → call the scaling API
# MCP's structured return value avoids cascading retries caused by CLI-output parsing failures

Incorrect Example

# Mature CLIs such as gh and kubectl are already highly familiar to the AI,
# yet GitHub and Kubernetes MCP servers are installed as well
# The context gains several kilobytes of Tool Definitions without improving call accuracy

# A proprietary internal CLI is called directly without an MCP wrapper
# Every call becomes a gambling loop: guess arguments → receive an error
# → interpret the error → guess again; retry cost becomes uncontrolled

3.7 Reference Files with Full Paths (@path)

[Required] When referencing a file, use @ followed by its full or relative path. Do not provide only a filename and make the Agent search for it.

A filename alone triggers a "search → possibly confirm multiple times → read" workflow. A full path allows the file to be located and read directly. The larger the project, the more expensive the search.

Correct Example

Review the transaction-handling logic in CreateOrder at @src/order/service.go
Modify the token-validation logic in @internal/auth/middleware.go

Incorrect Example

Review the transaction-handling logic in service.go
Modify the token-validation logic in middleware.go
# The Agent must traverse the project tree, find files with the same name,
# confirm the intended target, and then read it

3.8 State the Complete Intent

[Required] State the complete task objective, context, and acceptance criteria in a single message. Avoid piecemeal, back-and-forth requests.

Every interaction consumes tokens by reassembling the context, reloading history, and reconfirming state. A complete request in one message is significantly more efficient than multiple rounds of incremental confirmation.

Correct Example

Review the CreateOrder function in @src/order/service.go, identify potential defects,
fix them, and write unit tests for the corrected function.

Incorrect Example

User: Take a look at the CreateOrder function
AI:   What aspects should I check?
User: See whether it has any bugs
AI:   Inventory deduction is not protected by a transaction. Should I fix it?
User: Fix it
AI:   The fix is complete. Should I add tests?
User: Yes, add them

3.9 Prefer Tools to LLM Reasoning

[Required] Select the execution method in this order: use an existing specialized tool > write a script > rely on LLM reasoning.

The core value of an LLM is handling uncertainty — tasks that require understanding context, reasoning, judgment, and tradeoffs. For mechanical operations with deterministic workflows and explicit rules, prefer specialized tools or scripts.

Correct Example

# Existing specialized tool: code linting
golangci-lint run ./...

# Existing specialized tool: code formatting
gofmt -w ./src

# Existing specialized tool: database migration
alembic upgrade head

Incorrect Example

User: Check whether this Go code has formatting problems
AI:   Reads every file → analyzes indentation, naming, and import order line by line
      → outputs modification suggestions
# golangci-lint can do this with one command; Agent reasoning is both inefficient
# and vulnerable to misjudgment

User: I need the number of paid machines over the past 30 days, every day
AI:   Re-reads the table schema and sample data every day → estimates the count
# Instead, have the AI generate a fixed script and execute it to obtain the result

3.10 Lazy-Load Tools

[Recommended] Configure tools that are not used frequently for deferred loading with Defer, so their definitions do not remain resident in the context.

By default, the definitions of all registered tools are injected into the context of every request. The more tools there are, the greater the resident token cost. CodeBuddy supports the Defer(...) modifier for marking nonessential tools for lazy loading. Their definitions do not appear in the model's tool list and are loaded and called only when the model actively discovers them through ToolSearch.

Correct Example — Agent

# .codebuddy/agents/code-reviewer.md
---
name: code-reviewer
description: Code review agent
tools:
  - Read
  - Grep
  - Bash
  - Defer(Glob)          # Lazy-loaded: discovered through ToolSearch only when file search is needed
  - Defer(WebFetch)      # Lazy-loaded: triggered only when online documentation is needed
---

Correct Example — CodeBuddy

# Specify lazy-loaded tools when starting the CLI
codebuddy --tools "Read,Edit,Bash,Defer(Glob),Defer(Grep),Defer(WebFetch)"

Correct Example — MCP

{
  "mcpServers": {
    "mcp-server-tapd": {
      "command": "uvx",
      "args": ["mcp-server-tapd"],
      "defer_loading": true
    }
  }
}

Incorrect Example

---
name: research-agent
description: Conduct research through multiple web search engines
# Omitting a tools declaration enables many built-in CodeBuddy tools by default
---
XXX content

4.1 Match the Task First; Do Not Simply Choose the Cheapest Model

Model routing does not mean "always use the cheapest model." It means classifying the task correctly first:

• Complex task → powerful model
• Simple task → inexpensive model
• Repetitive task → stable model

Architecture design requires multi-step reasoning, so use an expensive model when needed. An inexpensive model is entirely sufficient for writing unit tests, adding comments, or generating commit messages. Large-scale classification and summarization are better suited to low-cost models or offline batch processing.

4.2 Task-to-Model Mapping

[Required] Select an appropriate model tier based on task complexity. Do not use the strongest model by default for every task.

Decision rule: If correctness can be verified immediately after the work is produced, use a lightweight model. If the result requires repeated analysis and tradeoffs, use a high-performance model.

4.3 Bind Models to Skills / Agents / Commands

Model routing should not exist only as a mental rule in the main chat. It should be implemented in execution units.

Bind models to Skills: Explicitly bind inexpensive models to Skills for writing unit tests, generating commits, fixing formatting, and producing boilerplate code.

In CodeBuddy, for example, the frontmatter of a SKILL.md file can declare the model and context strategy directly:

---
name: XXX
description: XXXX
context: fork
model: deepseek-v4-pro
---

The model field specifies the model used by the Skill. context: fork runs this fixed workflow in an isolated execution context, which is appropriate for a Skill with clear boundaries that can complete its task independently. This allows low-complexity Skills to use inexpensive models automatically without carrying the complete history of the main session.

Bind models to Agents: Agents also support model binding through the model parameter. Set the model in the Agent definition so that everyone on the team uses the intended tier by default instead of having to remember to switch.

Planner  → Mid-to-high tier
Coder    → Mid tier
UnitTest → Inexpensive
Reviewer → Powerful

Note: In CodeBuddy, the lite keyword represents an inexpensive or lightweight model. It automatically selects the lower-cost counterpart of the model selected by the user. For example, if the user selects deepseek-v4-pro, the lite model is deepseek-v4-flash. Users can also set the CODEBUDDY_SMALL_FAST_MODEL="hy3-ioa" environment variable to specify the concrete model represented by lite.

The following frontmatter defines an Agent that uses a lightweight model:

---
name: AgentName
description: XXXX
model: lite
tools: Read, Grep, Glob, Bash
---

Bind models to Commands: Many Commands, such as /commit and /check-ci-status, follow fixed templates and are well suited to a preset inexpensive model.

Once this design is encoded, cost governance changes from "remember to switch models" to "the system is cost-efficient by default." The following is an example of Command frontmatter:

---
description: "XXX"
argument-hint: "[message]"
allowed-tools: Bash(git:*)
model: lite
---

5.1 Karpathy's CLAUDE.md: Automated Trimming and Anti-Hallucination Anchors

[Recommended] If the team has not established an alternative convention, consider adopting Andrej Karpathy's CLAUDE.md where appropriate for the project.

Positioning: A high-quality CLAUDE.md template developed by the community. Through a predefined rule set, it requires the model to trim context, reduce filler, suppress hallucinations, and enforce engineering conventions.

Andrej Karpathy's CLAUDE.md addresses how the model should think. It embeds an efficient reasoning protocol at the System Prompt layer to reduce unnecessary token generation at the source.

---
# Global response strategy
- Minimalism: Unless an explanation is requested, output only code blocks. Do not
  include pleasantries such as "Okay" or "Let me help you."
- Determinism: If the Read tool returns empty or no results, stop and report an
  error immediately. Do not guess the file contents.
- Tools first: When a task concerns file content, code definitions, or Git status,
  call a tool. Never guess based on memory.

# Context management
- Search before reading: In an unfamiliar codebase, use Grep/Glob before deciding
  what to Read.
- Line limits: When reading a file, specify `offset` and `limit`; read no more than
  200 lines by default.
- Focus changes: After modifying code, show only the relevant diff. Do not repeat
  the entire file.

# Output format
- Present information with Markdown lists and tables.
- Include necessary comments in code snippets, but remove redundant sample output.
---

# Project-specific rules: insert the project's architecture, technology stack,
# and special conventions here

5.2 RTK — Rust Token Killer

[Recommended] Quickly reduce tool-output volume and token usage.

Positioning: Intercepts shell command output and compresses it before the Agent sees it. It compresses only CLI output, not prompts or code.

# 1. Install the internal fork; Rust must already be installed
git clone https://git.woa.com/godwinchen/rtk.git
cd rtk && cargo install --path .

rtk init --global --agent codebuddy   # Install the hook to intercept all commands transparently
rtk gain                              # Confirm that savings data is available

Enumerating objects: 15, done.
Counting objects: 100% (15/15), done.
Delta compression using up to 10 threads
Compressing objects: 100% (8/8), done.
Writing objects: 100% (8/8), 2.34 KiB | 2.34 MiB/s, done.
Total 8 (delta 5), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (5/5), completed with 3 local objects.
To github.com:org/repo.git
   abc1234..def5678  main -> main

ok main

PASS src/utils.test.ts
PASS src/config.test.ts
...
PASS src/chain.test.ts
FAIL src/order.test.ts > should handle empty cart
  AssertionError: expected 400 got 500
    at Object.<anonymous> (src/order.test.ts:42:17)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Promise (src/order.test.ts:38:3)
    at async Context.<anonymous> (src/order.test.ts:35:5)
FAIL src/auth.test.ts > expired token
  Error: timeout of 2000ms exceeded
    at Object.<anonymous> (src/auth.test.ts:18:12)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Promise (src/auth.test.ts:14:3)

Test Suites: 2 failed, 43 passed, 45 total
Tests:       2 failed, 178 passed, 180 total
Snapshots:   0 total, 0 failed
Time:        12.345 s
Ran all test suites.

FAIL 2/45 tests
  src/order.test.ts > should handle empty cart → expected 400 got 500
  src/auth.test.ts > expired token → timeout

5.3 Caveman

Caveman is an output-compression plugin for AI coding Agents. Prompt rules make the model respond like a "smart caveman": remove filler and social niceties while preserving technical facts.

# macOS · Linux · WSL · Git Bash
curl -fsSL https://raw.githubusercontent.com/JuliusBrussee/caveman/main/install.sh | bash

# Windows · PowerShell 5.1+
irm https://raw.githubusercontent.com/JuliusBrussee/caveman/main/install.ps1 | iex

/plugin marketplace add https://github.com/studyzy/caveman

# Select the caveman plugin after running the command. The system returns:
#  ⎿ ✓ Added marketplace: caveman
#  ⎿ ✓ Installed caveman. Restart CodeBuddy Code to load new plugins.

Hook SessionStart
  CAVEMAN MODE ACTIVE — level: full
  Respond terse like smart caveman. All technical substance stay. Only fluff die.
……

5.4 Ponytail

The core of Ponytail is not "how to write code," but what to consider before writing code. It defines a strict decision sequence:

Level 1: Does this really need to be built? (YAGNI)
Level 2: Does the codebase already contain it? Reuse it
Level 3: Can the standard library do it? Use it
Level 4: Does the browser / operating system provide it natively? Use it
Level 5: Is it available in an already-installed dependency? Use it
Level 6: Can it be done in one line? Write one line
Level 7: Only if none of the above works, write the minimum amount of new code

Each level is a gate. Reaching Level 7 means genuinely starting to write code, but most requirements are stopped at earlier levels. An experienced engineer does not create a new file immediately — they first grep the codebase for an existing solution.

/plugin marketplace add https://github.com/studyzy/ponytail

# After adding the marketplace, select the ponytail plugin.
# If you did not select it, run:
/plugin install ponytail@ponytail

5.5 Comparison of the Four Tools

In one sentence each:

• Karpathy's CLAUDE.md makes the Agent think clearly before acting
• RTK compresses what the Agent sees
• Caveman compresses what the Agent says
• Ponytail compresses the code the Agent writes

The four tools cover different dimensions of compression and can be combined into a complete pipeline:

Agent calls a command
    ↓
RTK (first stage: compress CLI command output)
    ↓
LLM receives clean input
    ↓
Ponytail (controls what to write: do not write unnecessary code)
    +
Caveman (controls how to say it: remove unnecessary prose)
    ↓
Compressed output is returned to the user

6.1 Why a Single Agent Becomes More Expensive over Time

Core reason: Transformers have no memory.

The Transformer architecture used by LLMs has no memory across turns. On every turn, the complete "System Prompt + conversation history + current input" must be sent to the model again. This means:

Turn 1: Context = System Prompt
Turn 2: Context = System Prompt + turn 1
Turn 3: Context = System Prompt + turn 1 + turn 2
...
Turn N: Context = System Prompt + all previous N-1 turns

Adding too much irrelevant information also reduces response quality. Research has repeatedly demonstrated this through the "Lost in the Middle" phenomenon: models pay the least attention to information in the middle of the context. Good Context Engineering is not about inserting as much as possible; it is about carefully choosing the information needed right now.

6.2 Subagents: The Lowest-Cost Form of Task Isolation

Why do Subagents save tokens?

Core mechanism: A Subagent has a one-time, isolated context. Quadratic growth is contained within the Subagent, while the main Agent's history grows linearly.

LLM APIs are stateless, and every tool call resends the complete history. How history accumulates therefore determines whether token usage grows linearly or quadratically.

Main Agent: Poll N×K times → read entire large files → process intermediate artifacts
            Every message resends all preceding history
            ⇒ Main history is O(N×K), and every turn resends it in full
            ⇒ Total tokens ≈ O((N×K)²), quadratic growth

Subagent: Poll N×K times → read entire large files → process intermediate artifacts
          This remains in the Subagent's one-time context and does not enter main history

Main Agent: Receive only a concise result for each task, with no intermediate artifacts
            ⇒ Main history is O(N), adding only one result per task
            ⇒ Total tokens ≈ O(N), linear growth

7.1 The Ceiling of Individual Optimization

Token-optimization techniques available to an individual are already quite mature:

These produce linear gains. Each person independently explores the best Prompt, encounters the same pitfalls, and accumulates the same knowledge. On a team of ten, the same pitfall may be encountered ten times and the same best practice may be independently rediscovered ten times.

Leverage formula for team collaboration:

Team savings = one-time optimization investment × number of users × average daily usage frequency

If a Skill takes two hours to refine and ten people each use it three times per day, that two-hour investment delivers value across 30 uses every day.

7.2 Build and Share Team Assets

Team member A contributes a Skill
       ↓ push (creates a PR, merged after review)
   Team Repo (Git repository)
       ↓ pull (triggered automatically at CodeBuddy SessionStart)
AI tools used by team members B/C/D... automatically receive the Skill

8.1 Five Principles

1. Measure first, then optimize
2. Never solve again what can be captured as an asset
3. A stable prefix is essential to effective caching
4. Context quality matters more than context quantity
5. Team consistency matters more than individual perfection

8.2 Revised Core Formula

Lower cost =
    less repeated context
  + more appropriate model routing
  + more precise code retrieval
  + clearer Agent responsibilities
  + measurable, continuous optimization
  + organization-wide asset reuse

8.3 Three Long-Term Recommendations

• Make measurement part of the team's muscle memory
• Make asset capture the default action
• Base optimization decisions on data, not intuition