LLM 长对话分层记忆架构设计

 3|**URL:** https://mp.weixin.qq.com/s/3sNvtDuhvxoSAvPcmOUzsA

• Tags: ai-engineering, local-llm Date: 2025-05-24

Summary

An article discussing a system design interview question from Alibaba about building a long-conversation system with only 8K token window that can support 100+ rounds of dialogue.

Key Points

The Core Challenge: How to allocate 8K tokens wisely when you need to maintain context across 100+ conversation turns. The analogy: LLM = CPU, 8K window = RAM, conversation history = hard disk.

4-Layer Memory Architecture (Token Budget: 8000):

1. Short-term Memory (2500 tokens) — Last 3-5 turns of raw conversation. Highest accuracy but expensive. Older content gets dropped.

2. Mid-term Summary (1500 tokens) — Compressed summaries generated every 5 turns in background. Recursive rolling memory chain. Loses detail but preserves logical structure.

3. Long-term External Store (2000 tokens) — Vector DB + RAG. All 100 turns stored as embeddings. Semantic similarity search retrieves relevant historical fragments on demand.

4. State Management (500 tokens) — Structured user profile (name, preferences, task progress, confirmed facts). Always injected at top of system prompt. Prevents “context drift” where model forgets core requirements.

Context Orchestrator sits on top, deciding how to allocate budget across these 4 layers for each new query.

3 Expert-Level Insights:

• System-level thinking: Memory layer design matters more than just using bigger context windows
• Engineering trade-offs: The 2500/1500/2000/500 split comes from empirical testing, not theory
• Logic anchoring: Structured state prevents core identity/task intent from being diluted by window noise