Local LLM VRAM & RAM Requirements 2026: What Fits in 8GB, 12GB, 16GB, 24GB, 32GB, 48GB and 128GB

By AiCybr Editorial · Published August 10, 2026

The number on a GPU box is not the amount of model you can safely load. Local-LLM memory use is the sum of model weights, KV cache, runtime/compute buffers, framework overhead and whatever else is already occupying the accelerator. Long context and concurrent requests can consume several additional gigabytes even when the model weights themselves fit.

That is why a 16 GB GPU does not automatically mean “a 32B model at 4-bit fits,” even though the simple arithmetic says 32 billion parameters × 4 bits is about 16 GB. Real quantized formats have metadata and scaling overhead, the runtime needs working memory, and the KV cache grows with context.

This guide gives planning ranges rather than false guarantees. Exact fit depends on the model architecture, quantization format, context length, inference engine, batch/concurrency settings and how much of the model is placed on GPU versus system memory.

Quick answer: practical model classes by memory tier

For a single-user local inference setup using a roughly 4-bit weight quantization, these are sensible planning targets rather than hard limits:

The key word is comfortable. If the goal is experimentation, CPU offload and lower-bit quantization can stretch each tier. If the goal is an always-on server with long context or multiple simultaneous users, choose more memory than the minimum.

1. Start with weight memory, then add everything else

A useful first approximation for dense models is:

weight memory ≈ parameter count × bits per stored weight ÷ 8

This gives a lower-bound planning number before quantization metadata, alignment, runtime buffers and cache are added.

These are not expected file sizes or guaranteed runtime footprints. Real 4-bit formats often exceed the simple 0.5 byte-per-parameter figure because scales, grouping metadata and some tensors may use higher precision.

Hugging Face documents that 8-bit loading roughly halves memory compared with 16-bit weights and that 4-bit loading compresses further. llama.cpp supports a wide range of integer quantizations from 1.5-bit through 8-bit and explicitly supports CPU+GPU hybrid inference when a model exceeds total VRAM.

2. Why a model that “fits on paper” can still run out of memory

Quantization overhead

“4-bit” is a family of formats, not one universal physical size. Quantization groups need scale information, metadata and sometimes higher-precision tensors. Different GGUF, GPTQ, AWQ and bitsandbytes configurations can therefore have different memory footprints for the same nominal model size.

For buying hardware, do not size a system so the theoretical weight payload equals 100% of available VRAM. That leaves no useful margin for the rest of inference.

Runtime and compute buffers

Inference engines reserve memory for kernels, temporary tensors, graph capture, allocator state and other working buffers. vLLM exposes gpu_memory_utilization specifically because the model executor and KV cache must share a finite device-memory budget rather than assuming all nominal VRAM is available.

Desktop memory is not fully free memory

On a display GPU, the operating system, browser, desktop compositor and other GPU applications also consume VRAM. An advertised 24 GB card may have materially less than 24 GB free when the inference server starts.

3. KV cache: the reason context length changes the answer

Once model weights fit, the KV cache often becomes the next major memory consumer. It stores attention keys and values for tokens already processed so the model does not recompute the entire sequence for each generated token.

KV-cache memory grows with factors including:

• sequence/context length;
• number of layers;
• key/value head count and dimensions;
• cache precision;
• number of concurrent sequences.

The exact formula is architecture-specific, especially because modern models often use grouped-query or multi-query attention. The important purchasing rule is simpler:

A model that fits at 4K or 8K context may not fit at 64K or 128K context with the same quantization and concurrency.

vLLM can automatically infer how much memory remains for KV cache after loading the model, and its current configuration also allows explicitly setting kv_cache_memory_bytes. It can choose a smaller max_model_len when the model’s advertised maximum context does not fit available GPU memory.

llama.cpp likewise allows the KV cache to use multiple precisions, including FP16/BF16 and quantized cache types, and can keep/offload cache data differently depending on configuration.

Long context is not free even when a model advertises it

A model card saying “128K context” describes a supported model capability, not a promise that your 12 GB or 16 GB GPU can serve 128K context at the same speed and concurrency as a data-center accelerator.

For local use, decide the context you actually need before buying hardware. Coding agents, repository analysis and document-heavy workflows can benefit from larger contexts. Short chat, summarization and lightweight assistants often do not need the maximum window every model advertises.

4. What 8 GB VRAM is good for

An 8 GB GPU remains useful for local inference, but it is best treated as a small-model tier.

A 7B–8B model at a sensible 4-bit quantization normally leaves much more workable headroom than trying to force a 14B model into the same card. That headroom can be spent on context, a better cache format, multimodal components or simply avoiding out-of-memory failures.

Use 8 GB when:

• you mainly run 7B–8B class models;
• you value speed over maximum model size;
• CPU offload is acceptable for occasional larger models;
• you do not need very long context or many concurrent users.

Do not buy 8 GB specifically for local AI in 2026 if you already know you want 30B+ dense models. It can still be a useful general-purpose GPU, but the memory ceiling becomes the recurring constraint.

5. What 12 GB VRAM is good for

Twelve gigabytes is a meaningful step up for 12B–14B-class models and gives more room for context than an 8 GB card at the same model size.

It is still not a comfortable 30B-class full-GPU tier. Lower-bit quantization or CPU offload can make larger models run, but “runs” and “runs entirely in GPU memory with useful context” are different goals.

For a buyer who expects local AI to become a major workload, 12 GB should be viewed as a moderate model tier, not a long-term high-capacity tier.

6. What 16 GB VRAM is good for

Sixteen gigabytes is a flexible middle tier because it can run smaller models at higher precision/quality or move into roughly 20B-class models at 4-bit-class quantization.

The common mistake is assuming a dense 32B model at exactly 4 bits must fit because the simple weight payload is ~16 GB. That leaves effectively no room for quantization metadata, KV cache or runtime buffers. Some 32B configurations can be made to work through tighter quantization or partial offload, but they should not be treated as a universal “16 GB full-GPU” target.

For reliable 32B-class use, more than 16 GB accelerator memory is preferable.

7. Why 24 GB is still a very useful local-AI tier

Twenty-four gigabytes is where 30B–32B-class local models become much easier to run without immediately spending every byte on weights.

Compared with 16 GB, the extra headroom can be used for:

• better quantization quality;
• longer context;
• multimodal projectors/encoders;
• larger runtime buffers;
• fewer CPU-offloaded layers.

A 24 GB card can also experiment with larger-than-32B models through lower-bit quantization or offload, but a conventional dense 70B model at Q4-class weight sizes is still too large to treat as a comfortable full-GPU workload: the theoretical 4-bit payload alone is ~35 GB before overhead.

8. What changes at 32 GB

Thirty-two gigabytes gives a 32B-class model substantial flexibility. Instead of merely asking “can it fit?”, you can often choose between:

• higher-quality weight quantization;
• more context;
• larger batch/concurrency settings;
• more free VRAM for other components.

Some ~70B models become possible only at aggressive sub-4-bit quantizations or with some memory placed outside the GPU. That can be useful, but the result should not be confused with a 70B model running comfortably at Q4 entirely inside 32 GB.

For buyers deciding between a faster lower-VRAM card and a somewhat slower card with 32 GB, local-LLM workloads are one of the clearest cases where capacity can matter more than peak compute once the preferred model otherwise does not fit.

9. 48 GB: the practical 70B Q4-class threshold

A dense 70B model has a theoretical 4-bit weight payload around 35 GB. A 48 GB accelerator therefore provides materially more realistic space for quantization overhead, cache and runtime memory.

That does not mean every 70B model at every context length fits every 48 GB setup. Architecture and serving configuration still matter. But 48 GB is a much more credible planning tier for people whose priority is running 70B-class dense models primarily on one accelerator.

This is also where workstation/datacenter GPUs can become attractive despite their price: their value is often the memory capacity and reliability/features, not simply raw gaming performance.

10. 64 GB, 96 GB and 128 GB: capacity stops being the only question

At 64 GB and above, many local-AI decisions shift from “can the weights fit?” to:

• how fast is the memory subsystem?
• does the backend support the accelerator well?
• how much context/concurrency is required?
• is the model dense or mixture-of-experts?
• is the system unified memory or discrete VRAM?
• what fraction of memory can the runtime actually use?

A 128 GB unified-memory system can theoretically hold far larger quantized weight sets than a 24 GB or 32 GB GPU. But unified capacity does not make it equivalent to 128 GB of high-bandwidth discrete GPU VRAM. Memory bandwidth, compute throughput, kernels and backend support determine generation speed.

This is why systems such as high-memory Apple Silicon machines and GB10-class personal AI workstations are interesting: they change the capacity boundary dramatically, but they do not make model size the only benchmark that matters.

11. System RAM requirements when you use CPU offload

llama.cpp supports hybrid CPU+GPU inference specifically so models larger than VRAM can run by keeping part of the workload in host memory.

A useful system-RAM planning rule is:

Have enough RAM for the full model file plus operating-system/application overhead and a safety margin, especially if you expect substantial CPU offload.

Practical tiers:

Again, RAM capacity alone does not imply good inference speed. DDR5 system memory has far less bandwidth than modern high-end GPU memory, so moving more layers to CPU memory generally trades speed for capacity.

12. Unified memory is not the same thing as VRAM, but it changes the sizing problem

On unified-memory systems, CPU and GPU can address a shared physical memory pool rather than copying the entire model between separate system RAM and VRAM pools.

This is valuable for local AI because a 64 GB, 96 GB or 128 GB unified-memory machine can devote a large fraction of that capacity to model weights. However:

• the operating system still needs memory;
• not all physical memory is necessarily available to the GPU/runtime;
• memory bandwidth differs dramatically between products;
• backend quality matters;
• a model fitting does not guarantee interactive token generation speed.

Compare unified-memory systems by usable model capacity + memory bandwidth + real runtime support + measured inference speed, not by capacity alone.

13. Dense models and MoE models need different thinking

Parameter count is especially misleading for mixture-of-experts (MoE) models.

An MoE model can have a very large total parameter count while activating only a subset of experts for each token. That may reduce compute per token relative to a dense model of the same total size, but the stored expert weights can still require large memory capacity unless the runtime offloads or streams them.

So do not compare a “100B MoE” and “100B dense” model by parameter count alone. Check:

• total parameters;
• active parameters per token;
• quantized checkpoint size;
• whether experts can be CPU-offloaded;
• the engine’s MoE implementation;
• measured memory use for your context/concurrency target.

llama.cpp, for example, exposes CPU-MoE options that can keep MoE weights in host memory, which is useful when capacity matters more than maximum speed.

14. Context length can matter more than moving from Q4 to Q5

For a single-user short-context workload, allocating more memory to better weight precision may make sense.

For an agent serving multiple large documents or a coding repository, the better use of memory may be a smaller quantization plus a much larger KV cache.

There is no universally best quantization because the optimization target differs:

15. Why serving multiple users changes memory requirements

A desktop chat with one active conversation and an API server handling many simultaneous requests are different workloads.

Concurrent sequences require more cache capacity. vLLM’s design deliberately manages available GPU memory between model weights and KV cache, and its serving parameters allow administrators to constrain cache memory, sequence counts and maximum model length.

If you are sizing a shared home-lab or small-team server, do not use the minimum memory figure from a single-user benchmark. Capacity headroom is part of reliability.

16. Do you need 128 GB RAM if your GPU has 24 GB?

Not necessarily.

If your preferred models fit fully in 24 GB VRAM and you are not doing heavy CPU offload, 32 GB or 64 GB of system RAM can be entirely reasonable depending on the rest of the workstation workload.

128 GB system RAM becomes useful when you:

• regularly load model files larger than VRAM;
• use CPU/GPU hybrid inference;
• run multiple models/services simultaneously;
• keep vector databases, containers, IDEs and datasets resident alongside inference;
• experiment with large MoE models or very large quantized checkpoints.

Buy system RAM for the workload you actually intend to run, not because “local AI requires 128 GB” as a universal rule.

17. Is more VRAM always better than a faster GPU?

No, but insufficient VRAM is a hard boundary more often than insufficient compute.

If Model A fits completely on a 24 GB GPU but Model B requires 32 GB, a much faster 24 GB GPU may still force Model B into CPU offload while a slower 32 GB accelerator keeps it entirely on-device. Depending on the workload, the latter can be faster in practice despite lower peak compute.

Once both GPUs have enough memory for the exact same model, context and batch settings, raw compute, memory bandwidth and software support become more important.

The right order for local-LLM buying decisions is usually:

1. choose the model class and context/concurrency target;
2. determine the memory capacity required with safe headroom;
3. eliminate accelerators that cannot meet that capacity target;
4. then compare speed, bandwidth, power, software support and price.

18. CPU-only local LLMs: RAM is the capacity, bandwidth is the speed limit

CPU-only inference can run surprisingly large quantized models if the machine has enough RAM. llama.cpp is designed for CPU execution as well as GPU acceleration and can memory-map model files.

The trade-off is throughput. Large desktop/server RAM pools are cheaper per gigabyte than high-end VRAM, but system-memory bandwidth and CPU matrix throughput are much lower than modern GPU accelerators for many LLM workloads.

CPU-only makes sense when:

• the model is used infrequently;
• capacity matters more than latency;
• power/space constraints favor an existing CPU server;
• a GPU is unavailable;
• privacy/locality matters more than interactive speed.

For an interactive assistant used all day, accelerator memory is usually the more important capacity target.

19. A safer buying checklist

Before choosing a GPU or unified-memory system, answer these questions in order:

1. Which model size/class do I actually want to run?
2. What quantization am I willing to use? Q4, Q5, Q6, Q8 and FP16 can produce very different memory requirements.
3. What context length do I actually need? Do not automatically size for the model’s maximum advertised context.
4. One user or many? Concurrency directly affects cache pressure.
5. Must the model stay entirely on the accelerator? If not, system RAM can extend capacity through offload.
6. Do I run multimodal models? Vision/audio encoders and projectors consume additional memory.
7. Do I need several models loaded simultaneously? Model switching is cheap in money but not always in latency; keeping several resident multiplies memory use.
8. What else uses the GPU? Displays, browsers, video tools and other AI applications reduce free VRAM.
9. What backend will I use? CUDA, ROCm, Metal, Vulkan and other backends have different support/performance characteristics.
10. Can I validate with the actual checkpoint file? The model’s real quantized file size plus engine-reported memory is more trustworthy than a generic parameter-count chart.

20. The simplest rule that avoids most mistakes

For local AI, do not buy to the theoretical minimum.

If your target model’s quantized weights already consume almost all advertised accelerator memory, move up a memory tier or accept that you will need a lower-bit quantization, shorter context or CPU offload.

A little unused memory is not wasted. It is what lets the runtime hold KV cache, handle longer prompts, support multimodal components and survive normal desktop/server overhead without constant out-of-memory tuning.

Bottom line

• 8 GB: excellent for small 7B–8B models; larger models quickly become compromise-heavy.
• 12 GB: a practical 12B–14B tier.
• 16 GB: strong midrange capacity, but not a universal 32B full-GPU tier.
• 24 GB: one of the most useful capacities for 30B–32B-class local models.
• 32 GB: gives 32B models much more quality/context headroom and opens aggressive larger-model experimentation.
• 48 GB: a credible single-accelerator target for dense 70B models around Q4-class quantization.
• 64 GB+: increasingly about quality, context, MoE architecture and serving workload rather than simple “does it fit?” arithmetic.
• 96–128 GB unified/system memory: enables very large local checkpoints, but capacity must be evaluated together with bandwidth, compute and backend support.

The durable sizing formula is not “parameters × four bits.” It is:

weights + quantization overhead + KV cache + runtime buffers + concurrency + safety margin.

That is the number your hardware actually has to support.

Primary sources and further reading

• llama.cpp README — quantization, CPU/GPU hybrid inference and supported backends
• llama.cpp server options — GPU layers, KV-cache types and offload controls
• vLLM model configuration — maximum context length and automatic fit behavior
• vLLM KV-cache configuration — GPU memory utilization and explicit cache sizing
• Hugging Face Transformers bitsandbytes guide — 8-bit and 4-bit quantization
• Hugging Face Transformers quantization configuration