AI storage tiers
AI storage requirements differ from traditional workloads: very large datasets (TB to PB), frequent large checkpoints, and high-throughput sequential reads.
Storage tiers:
| Tier | Technology | Use Case |
|---|---|---|
| Tier 1 — Local NVMe scratch | NVMe SSDs in compute nodes | Dataset caching, temp outputs (non-persistent) |
| Tier 2 — Parallel distributed FS | Lustre, IBM Spectrum Scale (GPFS) | Shared training datasets, checkpoints |
| Tier 3 — NFS | Standard NFS | Home directories, shared code, tools |
| Tier 4 — Object storage | S3-compatible | Model registry, archiving, long-term |
Capacity planning factors:
- Raw dataset size × 2–3× (copies, preprocessing, augmentation)
- Checkpoint storage (large language models = 100s of GB per checkpoint)
- Model registry (all versions)
- Staging area between tiers
- Data growth rate