HyperTier
Personal open-source project. A tiered-memory inference runtime to run large AI models on small GPUs. In early development; the hardware measurement layer is built.

- Role
- Author and maintainer
- Period
- Nov 2026 — Present
- Status
- In progress
- Stack
- Python
- PyTorch
- CUDA
- pytest
- GitHub Actions
HyperTier is my own project, and it starts from a real pain: the models worth running are far bigger than the GPU most people have. The design is a tiered-memory inference runtime that keeps in VRAM only what a pass needs and streams the rest from pinned RAM and NVMe, overlapping transfers with compute.
It is in early development. The repository contains the first roadmap phase, and the package on PyPI reserves the name until the first functional release.
The core idea
Bandwidth is the bottleneck, not capacity:
time_per_pass ≥ max(compute, non_resident_bytes / link_bandwidth)
Before writing a runtime, every term of that bound has to be measured on the target hardware. That is what the first phase does.
What is built (phase F0)
- Bandwidth probes. RAM to VRAM copies, pinned against pageable memory, from 1 MiB to 1 GiB. They are timed with CUDA events after a warm-up and reported as median and 90th percentile.
- Copy and compute overlap. A pinned copy on its own CUDA stream runs against matrix multiplications on the default stream, to measure how much of a transfer can hide behind compute.
- NVMe reads with
O_DIRECTinto page-aligned or pinned buffers, with a check that the page cache is not answering. - Pinned-memory limit probe. It grows allocations and backs off on failure while keeping a guard of free RAM.
- VRAM tracker. A context manager that records the allocator peak against the free memory the driver reports.
- Benchmark CLI that writes an anonymized JSON hardware profile.
Engineering practice
- GPU code is testable without a GPU: lazy imports, injectable allocators and a
cudatest marker that skips itself. - CI runs lint, tests and a package build on Python 3.10 to 3.13.
- Apache-2.0, with SPDX headers and a security policy.
What comes next
Layer streaming, KV-cache compression, Triton kernels, video diffusion support and the MoE expert cache are roadmap phases F1 to F4. There are no model-speed numbers yet; they will come with the first runtime release.