LLMs and inference infrastructure
Large Language Models (LLMs) in production have different resource profiles than training:
| Aspect | Training | Inference |
|---|---|---|
| Compute pattern | Batch, long-running | Request-driven, latency-sensitive |
| Memory | Very high (model + optimizer states) | High (model weights only) |
| Parallelism | Data + model + pipeline parallel | Tensor parallel for large models |
| Optimization | Throughput | Latency + throughput |
Inference optimization techniques:
- Quantization (INT8, FP16, BF16) — reduces GPU memory footprint
- Batching strategies: static batching, dynamic batching, continuous batching
- KV cache management for transformer models
- Model compilation with TensorRT for optimized kernels
NVIDIA Triton Inference Server:
- Multi-framework serving (PyTorch, TensorFlow, ONNX, TensorRT)
- Dynamic batching and concurrent model execution
- Kubernetes-native — deploy as a pod with GPU resources
- gRPC and HTTP/REST APIs