AI agents, planning, and memory

AI agent systems in production need operational consideration for how they use compute and memory.

Agent capabilities:

• Reasoning — LLMs generate step-by-step solutions (Chain-of-Thought prompting)
• Planning — Decomposing complex tasks into sub-steps
• Tool use — Calling APIs, running code, querying databases

Memory types:

• In-context (working) memory — Information within the active context window; GPU VRAM holds the KV cache
• External memory — Vector databases (RAG), key-value stores; retrieval is CPU/network-bound
• Parametric memory — Knowledge encoded in model weights; requires retraining to update

RAG (Retrieval-Augmented Generation):

• Combines LLM with external knowledge retrieval
• Vector database stores embeddings (FAISS, Milvus, Qdrant)
• GPU-accelerated embedding generation
• CPU/network bound for retrieval — design I/O path carefully