First-Principles Guide to On-Premise LLM Inference Hardware

Generative AI is growing up. The experimental toys are gone, replaced by core enterprise infrastructure, and the financial cracks in cloud-hosted API models are finally breaking wide open. Renting external APIs is an easy way to build a proof of concept. But for healthcare providers, financial institutions, and defense contractors, those external pipelines eventually crash into a wall of strict data sovereignty rules, compliance audits, and unpredictable bills.

When your production apps demand continuous, high-throughput processing of sensitive data, sending millions of tokens through third-party APIs is a massive security liability and a financial drain. You need a better way. The solution is moving those workloads to dedicated, physical infrastructure you actually control.

Do not treat this like a standard virtualization project. Sourcing and configuring on-premise LLM inference hardware requires an entirely different playbook than scaling traditional virtual machines or standard relational databases. We wrote this guide to break down the hardware selection process from first principles, giving you a clear, mathematical model to calculate your memory needs, pick the right silicon, and evaluate the real total cost of ownership.

Why On-Premise Inference Decouples Cost and Compliance

Regulated enterprises face a structural dead end when they rely on external APIs from providers like OpenAI or Anthropic. Three critical friction points block scale.

First, the compliance bottleneck: passing customer data to third-party APIs triggers a mountain of regulatory paperwork under GDPR, HIPAA, or the EU AI Act. Even running virtual private clouds on hyperscalers like Microsoft Azure or AWS means sharing physical silicon under the hood. That shared footprint keeps your security team exposed to third-party breaches and complex compliance audits.

Second, the variable cost trap: token-metered pricing turns enterprise AI into an unpredictable budget sinkhole. As thousands of your developers ship code, your monthly API bills skyrocket without warning.

Finally, high-security environments like defense, industrial controls, or proprietary R&D demand absolute isolation. You cannot run a true air-gapped system if your application must ping a cloud-hosted model to think.

Local hardware solves this. By deploying open-weight models like Kimi K2.6, DeepSeek, or Qwen on your own single-server infrastructure, you own the entire data lifecycle. It is that simple. You erase the external dependencies, secure your perimeter, and trade volatile API bills for predictable, flat-rate infrastructure costs.

The Math Behind VRAM Sizing for On-Prem LLMs

Compute power is not your bottleneck. When you run large language models, the real fight is over memory bandwidth and capacity. These architectures are aggressively memory-bound. To generate a single token, the processor cores must pull every single model parameter out of memory.

It is a brutal physical constraint.

If your hardware lacks the Video RAM to hold both the weights and the active user context, the system crashes. You either get a hard out-of-memory error or watch performance drop off a cliff as data swaps to agonizingly slow system RAM.

Calculating Your Base Model Footprint

To find the baseline VRAM needed just to load a model into memory, you need two variables: the parameter count ($P$P) and the precision level ($B$B), which represents your bytes per parameter as determined by quantization.

Here is the formula for your baseline footprint:

$${\text{VRAM}}_{\text{weights}}=P\times B\times 1.2$$

We apply a 1.2 multiplier to reserve a 20% buffer for CUDA runtime overhead, activation memory, and basic system overhead.

There is a trap in P that catches a lot of teams. Today’s leading open-weight models are Mixture-of-Experts architectures, and P is the total parameter count, not the active one. Take Kimi K2.6, Moonshot AI’s flagship released in April 2026: it is a 1-trillion parameter MoE with only 32 billion parameters activated per token, spread across 384 experts with 8 routed plus 1 shared expert selected per token. That 32B active figure is what determines your compute and your token generation speed. It has nothing to do with your memory budget. The router can reach for any expert on any token, so every one of those 1T parameters has to be resident in VRAM before you serve a single request.

Let us see how quantization alters the hardware demands of Kimi K2.6 ($1\text{T}$ total parameters):

• FP16 (16-bit precision = 2 bytes per parameter):$$1\text{T}\times 2\text{ bytes}\times 1.2=\mathrm{2,400}\text{ GB of VRAM}$$This exceeds any single node on the market. You are into multi-node territory before you start.
• FP8 (8-bit precision = 1 byte per parameter):$$1\text{T}\times 1\text{ byte}\times 1.2=\mathrm{1,200}\text{ GB of VRAM}$$This fits an 8-way B200 node (1,536 GB) with headroom. An 8-way H200 node at 1,128 GB falls just short.
• INT4 (4-bit precision = 0.5 bytes per parameter):$$1\text{T}\times 0.5\text{ bytes}\times 1.2=600\text{ GB of VRAM}$$This opens up several single-node builds: 8x RTX PRO 6000 Blackwell (768 GB), 8x H200 (1,128 GB), or 4x B200 (768 GB). On Blackwell silicon you would reach for NVFP4 rather than INT4 at this same footprint, for reasons covered in the hardware section below.

The lesson is that quantization is not a nice-to-have optimization for frontier open-weight models. It is the difference between a single server and a cluster.

Sizing the KV Cache for Real User Traffic

Do not assume loading the weights means you are done. If you intend to serve multiple users simultaneously, you must carve out a massive chunk of VRAM for the Key-Value (KV) Cache.

Every time a user interacts with the model, the KV Cache stores the attention keys and values of previous tokens. This keeps the model from recalculating the entire history for every new word it spits out. The size of this cache expands and contracts based on how many people are using the system and how long their conversations get.

For a conventional transformer using Multi-Head or Grouped-Query Attention, you calculate the KV Cache size like this:

$${\text{Size}}_{\text{KVCache}}=2\times L\times H\times D\times C\times B$$

Where L is the layer count, H the number of key-value heads, D the head dimension, C the total active context in tokens across all in-flight requests, and B your precision in bytes. The leading 2 accounts for storing both a key and a value.

Kimi K2.6 does not work this way, and the difference is worth real money. It inherits the Multi-head Latent Attention (MLA) design from the DeepSeek-V2/V3 lineage. Instead of caching a full key and value tensor for every attention head, MLA projects them down into a single low-rank latent vector that all heads decompress from at read time. You cache the compressed representation, not the expanded one.

That collapses the formula to a per-layer constant:

$${\text{Size}}_{\text{KVCache}}=L\times (L_{\text{kv}}+R)\times C\times B$$

Where $L_{\text{kv}}$ is the KV latent rank and $R$ is the shared RoPE key dimension carried alongside it. For Kimi K2.6 those are 512 and 64 respectively, so every token costs a flat 576 values per layer regardless of how many attention heads the model runs.

Let us calculate the same realistic scenario: Kimi K2.6 with a 16-bit KV Cache, serving 32 concurrent users each running a 16,000-token context window (a total $C$ of $512,000$ tokens).

• Layers (LL): 61
• KV latent rank (LkvLkv​): 512
• RoPE key dimension (RR): 64
• Total Tokens (CC): 512,000
• Precision (BB): 2 bytes

$${\text{Size}}_{\text{KVCache}}=61\times 576\times 512,000\times 2\text{ bytes}\approx 35.98\text{ GB of VRAM}$$

Look closely at those numbers, because they invert the conventional wisdom.

On a dense GQA model, the KV cache is the thing that ambushes your capacity planning and can easily outgrow the weights themselves. On an MLA model like K2.6, the cache is almost a rounding error: roughly 36 GB against 600 GB of INT4 weights, about 6% of your footprint. Serving your 32 users at INT4 needs around 636 GB of VRAM in total, and the weights account for nearly all of it.

This changes what you shop for. With MLA the binding constraint is raw VRAM capacity to hold resident experts, so you optimize for total memory across the node and treat long context as cheap. Drop the same 512,000 tokens onto a dense 70B model with GQA and the cache alone would run several times larger, which is exactly why context length gets rationed on those deployments and not on this one. Verify which attention mechanism your target model uses before you size anything. The two architectures produce hardware bills that differ by an order of magnitude.

Comparing Hardware Tiers for Silicon Selection

Selecting the right hardware architecture requires balancing performance requirements against your budget. Different silicon classes target different phases of development and deployment:

Pricing reflects mid-2026 street and integrator quotes and moves fast. The RTX PRO 6000 Blackwell alone saw its list price rise 55% in roughly sixteen months as the DRAM shortage bit. Treat these as planning anchors, not quotes.

Unified Memory: Capacity Without Throughput

Datacenter GPUs cost too much for simple experimentation, and Apple’s unified memory architecture has long been the cheap way around that: the GPU shares the main memory pool, so a Mac Studio holds models that would otherwise demand a rack of enterprise cards. The reference configuration here is the M3 Ultra Mac Studio at 512GB running at 819 GB/s, an unusual amount of addressable memory for a machine that draws under 500W and sits on a desk.

Availability note. The 512GB configuration is not currently orderable from Apple. The global DRAM shortage driven by AI infrastructure spending forced two rounds of cuts in 2026: the 512GB option was withdrawn in early March, and the 256GB option followed in early May, leaving the M3 Ultra Mac Studio at a single 96GB configuration on the current price list. Existing 512GB machines remain in service and circulate on the secondary market, and an M5 Ultra refresh reportedly tested at up to 768GB is expected around late 2026, though supply may cut that ceiling before it ships. Plan procurement accordingly, and treat the specifications below as a capability ceiling rather than something you can order this quarter.

Even at 512GB, one box will not hold Kimi K2.6. INT4 weights alone need 600 GB, so a single Studio lands just short of the largest open-weight models. That is what pushes you toward a cluster.

Lashing several Studios together works, and it buys you exactly one thing: capacity. It does not buy you concurrency, and that distinction decides whether this is the right machine for your workload.

The tooling is real. MLX Distributed handles tensor parallelism over Thunderbolt 5, EXO orchestrates a mesh on top of the same MLX backend, and Apple’s addition of RDMA over Thunderbolt 5 in late 2025 cut cross-node latency to under 50 microseconds. Reported figures put Kimi K2 at roughly 5 tokens per second over conventional networking versus about 25 with RDMA on the same hardware.

At 512GB per node the arithmetic is kind. Two Studios pool 1,024 GB, which clears INT4 Kimi K2.6 with nearly 400 GB of headroom for longer contexts or a second model. Three Studios reach 1,536 GB and hold the model at full FP8 fidelity, double the memory capacity of the four-card B200 node priced above and at a small fraction of its cost. That comparison holds on capacity alone; the two machines are not substitutes.

Then you hit the interconnect, and the physics are unforgiving. Thunderbolt 5 with RDMA delivers somewhere around 50 to 60 Gbps in practice, call it 6 to 7 GB/s. Local unified memory on the same machine runs at 819 GB/s. You are asking a fabric roughly 120 times slower than local memory to carry traffic that a single server would keep on its internal bus. Tensor parallelism makes this hurt most: every transformer layer requires an all-reduce of activations across all nodes on every single token. Pipeline parallelism eases the pressure by cutting sync points, but idles most of the cluster unless you keep it packed. Either way, every concurrent request you add multiplies traffic across the fabric that was already your bottleneck.

There is one further asymmetry that matters. On a GPU node, concurrent users are close to free: continuous batching amortizes a single weight read across the entire batch, so per-user throughput degrades slowly as you add people. On a Mac cluster requests are effectively serialized, so per-user throughput falls closer to linearly. Three things drive it:

• Bandwidth is the wrong number to compare. Memory bandwidth binds only at batch size one, where you read each weight and use it once. There the M3 Ultra’s 819 GB/s against the Blackwell card’s 1,792 GB/s is a fair fight at roughly 2x, which is exactly why single-user Mac benchmarks look respectable. Prefill and batched serving are bound by matrix throughput instead, and there an 80-core M3 Ultra manages about 65 TFLOPS FP16 against roughly 500 TFLOPS dense on the RTX PRO 6000, or near 2 PFLOPS at 4-bit. Call it 8x like-for-like and 30x at the precision you would actually deploy.
• Quantization buys Apple memory but no compute. Every recent NVIDIA generation ships tensor cores with a native matmul path at successively lower precision, and throughput roughly doubles with each step down. Apple’s M3 GPU has no matrix unit at all. Per-core Neural Accelerators arrived with the M5, where Apple reports up to 4x faster time-to-first-token, but the 512GB machine is an M3 Ultra. The silicon with the matmul hardware does not yet come with the memory.
• Batching is what makes GPUs scale, and it needs headroom Apple does not have. Raising batch size reuses each weight read across more tokens, sliding the workload from bandwidth-bound to compute-bound. On a GPU that is nearly free, because the tensor cores sat idle at batch one, so aggregate throughput climbs as users arrive. On Apple silicon the compute roofline is already close, so additional users queue instead of amortizing. Better schedulers like vLLM-mlx will close part of the software gap; they cannot raise the ceiling.

None of that makes a Mac cluster useless. It makes it a specific tool. For a small team doing R&D against a frontier open-weight model, for offline batch evaluation, or for an air-gapped site that needs capability more than throughput, two or three Studios on a Thunderbolt mesh is a defensible and remarkably cheap answer. Past roughly four to eight concurrent users the curve bends hard, and an enterprise assistant with sub-second latency expectations wants HBM3e and a real serving stack instead.

Production Hardware for Enterprise Scaling

When you go live, system uptime and response times are all that matter, and your first decision is how much quantization you are willing to accept.

The NVIDIA RTX PRO 6000 Blackwell is the value floor for serious work. Its 96GB of GDDR7 runs at 1,792 GB/s across a 512-bit bus, and it ships in both Workstation and Server Edition variants on PCIe 5.0. That bandwidth is roughly double a Mac Studio’s, but the number that actually separates them is the 752 fifth-generation Tensor Cores sitting behind it. Eight of them pool 768 GB, which clears Kimi K2.6 at 4-bit with its cache and change to spare, and being Blackwell silicon they run NVFP4 natively. You give up HBM3e bandwidth and SXM interconnect, so this build suits steady production traffic rather than latency-critical real-time work.

If you need to support several hundreds of employees simultaneously with zero lag, H200 and B200 servers in the SXM form factor are the industry standard. The H200’s 141GB of HBM3e runs at 4.8 TB/s, and eight of them give you 1,128 GB, comfortable for INT4 but short of the 1,200 GB that FP8 weights demand.

The real argument for the B200 is not its 192GB per card, it is NVFP4. Blackwell’s fifth-generation Tensor Cores execute this 4-bit floating point format natively, using per-block FP8 scaling to preserve dynamic range that flat INT4 throws away. The practical difference is large: INT4 weights must be dequantized back to 16-bit before any math runs, while NVFP4 feeds the tensor cores directly, delivering roughly 4-bit memory economics with accuracy and throughput that track FP8 rather than INT4. So you buy Blackwell to run Kimi K2.6 in about 600 GB without paying the quality tax that INT4 normally charges. Worth noting that NVFP4 is a generational feature rather than a B200 exclusive, so the RTX PRO 6000 Blackwell supports it too; what the B200 adds is 8.0 TB/s of HBM3e and an NVLink domain to keep the node’s cards in lockstep under heavy tensor parallelism.

Cost Comparison: On-Premise Hardware vs. Cloud APIs

Let us look at the raw numbers. We need to compare buying physical hardware for your own racks against paying rent to public cloud giants like OpenAI, Anthropic, or AWS.

To keep things realistic, we will use a concrete example sized to the hardware. Imagine your team runs an enterprise mesh of agents serving a frontier model in production. It processes 200 million input tokens and generates 50 million output tokens per day. That is the kind of sustained volume that justifies a Blackwell node in the first place; at a tenth of it, the arithmetic below tilts back toward the API.

Option A: Proprietary Cloud APIs

• Average Cost: ~$5.00 per million input tokens and ~$15.00 per million output tokens.
• Daily Cost:$$(200\times \mathrm{\$}5.00)+(50\times \mathrm{\$}15.00)=\mathrm{\$}\mathrm{1,000.00}+\mathrm{\$}750.00=\mathrm{\$}\mathrm{1,750.00}\mathrm{/}\text{day}$$
• Annual Cost:$$\mathrm{\$}\mathrm{1,750.00}\times 365=\mathbf{\$}\mathrm{638,750.00}\mathbf{/}\text{year}$$
• Note: this scales linearly and without ceiling. Double your usage and you double the bill.

Option B: Cloud GPU Rental (AWS p6-b200.48xlarge, 8x B200)

• On-Demand Cost: ~$113.93 per hour for the full 8-GPU node.
• Annual Cost (on-demand, running continuously):$$\mathrm{\$}113.93\times 24\text{ hours}\times 365\text{ days}\approx \mathbf{\$}\mathrm{998,051.00}\mathbf{/}\text{year}$$
• Annual Cost (Capacity Blocks reservation, ~$12.36 per accelerator-hour x 4 GPUs):$$\mathrm{\$}49.42\times 24\times 365\approx \mathbf{\$}\mathrm{432,919.00}\mathbf{/}\text{year}$$
• Note: AWS does not sell a four-GPU B200 instance. The smallest P6-B200 shape is the full eight-accelerator node, so on-demand renting means paying for eight cards whether your model needs them or not. You also pay for every compute-hour regardless of whether the system is saturated or idle overnight.

Option C: Physical On-Premise Infrastructure (A Single-Server 4x B200 Node)

• Upfront Hardware CapEx: ~$235,000.00 (four B200 cards at street pricing, plus chassis, CPU, memory, NVMe storage, and networking).
• Power & Cooling OpEx: ~5.5 kW continuous draw at $0.12/kWh:$$5.5\text{ kW}\times 24\text{ hours}\times 365\text{ days}\times \mathrm{\$}0.12=\mathbf{\$}\mathrm{5,781.60}\mathbf{/}\text{year}$$
• Datacenter Rack Space & Network: ~$700.00/month for high-density, liquid-cooling-capable space = $8,400.00 / year.
• 3-Year Depreciation Amortization:$$\frac{\text{CapEx}}{\text{3 Years}}=\mathrm{\$}\mathrm{78,333.33}\mathrm{/}\text{year}$$
• Total Cost in Year 1: ~$249,181.60 (Including full hardware purchase).
• Average Amortized Cost Over 3 Years:$$\mathrm{\$}\mathrm{78,333.33}\text{ (Amortized CapEx)}+\mathrm{\$}\mathrm{5,781.60}\text{ (Power)}+\mathrm{\$}\mathrm{8,400.00}\text{ (Hosting)}=\mathbf{\$}\mathrm{92,514.93}\mathbf{/}\text{year}$$

Those 768 GB across four cards hold Kimi K2.6 at NVFP4 with its cache and room to spare, which is the whole reason a four-card node is viable here. Insisting on FP8 weights would push you to eight cards and roughly double the CapEx line.

Yearly Cost Comparison (At Steady 250M Tokens/Day Workload)
---------------------------------------------------------------------------------
Option A: Cloud APIs               | [████████████████████████] $638,750 (Scales linearly)
Option B: Rented B200s (AWS)       | [██████████████████] $432,919 (4-GPU reserved)
Option C: On-Premises 4x B200      | [████] $92,515 (Fixed cost)
---------------------------------------------------------------------------------

Buying your own silicon changes the financial game by capping your spending. Your teams can run 250 million tokens or scale up to 2.5 billion daily tokens, and your operational bill will not budge by a single cent.

This predictability is exactly why we built DiscreteStack. By running our flat-rate private AI operating system on a single local server, you completely bypass the volatile, metered pricing models of hyperscale clouds and proprietary APIs.

Training Interconnects vs. Inference Pipelines

A common and expensive mistake when planning on-premise deployments is designing an inference node using the same specifications as a model training cluster. The requirements of these two workloads are fundamentally different:

During training, GPUs must constantly share billions of gradient weights during every backward pass. This requires massive, multi-million dollar investments in high-speed interconnects like NVIDIA InfiniBand or specialized NVLink fabrics to prevent communication bottlenecks.

For inference, however, the model weights are static and the mathematical operations are purely forward-pass. If a model fits entirely within the VRAM of a single physical server, the internal system bus carries the traffic and you do not need expensive InfiniBand fabrics or multi-node clustering. This is precisely why the memory arithmetic earlier in this guide matters so much: getting Kimi K2.6 down to 600 GB at INT4, or 1,200 GB at FP8, is what keeps the entire deployment inside one chassis and off the cluster network.

One caveat worth respecting. Sharding a trillion-parameter model across 8 cards still means an all-reduce on every layer for every token, and PCIe 5.0 lanes are noticeably tighter for that traffic than an SXM node’s NVLink domain. A PCIe build like 8x RTX PRO 6000 Blackwell is entirely viable and cost-effective for steady production serving, but expect SXM H200 or B200 nodes to hold their latency better under heavy tensor parallelism. The point stands regardless: single-chassis inference avoids the multi-million dollar interconnect investment that training clusters demand.

Hardware-Native Optimization

Bare metal is only as good as the software driving it. If you run vanilla open-weight models using naive Python runners, you waste up to 70% of your GPU’s actual capacity.

To squeeze out every possible token per second, your software layer must be compile-optimized for your exact silicon. High-throughput production engines rely on three main techniques to keep the hardware saturated:

  +-------------------------------------------------------------+
  |                   Production LLM Engine                     |
  +-------------------------------------------------------------+
         |                       |                       |
         v                       v                       v
+-----------------+     +-----------------+     +-----------------+
| FlashAttention  |     | vLLM Paged-     |     | TensorRT-LLM    |
| v1/v2/v3        |     | Attention       |     | Compilation     |
| (Reduces memory |     | (Reduces VRAM   |     | (Fuses kernels  |
|  bottleneck)    |     |  fragmentation) |     | to specific GPU)|
+-----------------+     +-----------------+     +-----------------+

• PagedAttention (vLLM): Traditional inference forces the system to allocate KV cache memory contiguously. This fragments your VRAM, frequently wasting more than 60% of your capacity. PagedAttention treats KV cache memory like virtual memory pages in a standard operating system, wiping out fragmentation so you can host far more concurrent users on a single chip.
• FlashAttention (v1/v2/v3): These algorithms cut down the constant traffic between the GPU’s High Bandwidth Memory and its internal SRAM. By streamlining these memory access paths, FlashAttention lets you run massive context windows without hitting a performance wall.
• Kernel Fusion & Model Compilation (TensorRT-LLM): Compiling a model welds multiple mathematical steps into single, highly optimized operations, which slashes latency.

These optimizations demand deep, frustrating system-level integration. You can spend months forcing your infrastructure team to build and patch these delicate pipelines from scratch, or you can deploy a pre-built private AI operating system designed to run on local hardware from day one.

Operational Realities of Air-Gapped and Local AI Deployments

Running bare metal in your own data center means you inherit the operational grind of security, patching, and compliance. This gets significantly harder when you pull the plug on the internet.

1. Air-Gapped Compliance and Security: Classified environments cannot risk leaks. When you cut the external network, cloud-based identity checks and remote API calls instantly break. You need an entirely self-contained stack that handles local user authentication, writes immutable audit logs, and enforces strict access controls right on the physical machine.
2. Model Lifecycle and Local Management: No internet means no simple download scripts. You have to handle model updates, security patches, and weights distribution entirely by hand. This demands a tight, predictable offline deployment pipeline so you can validate and ship updates without exposing the system.
3. Hardware-Native Standardization: Infrastructure engineers waste weeks fighting CUDA dependencies, broken PyTorch versions, and orchestration tooling. It is a massive time sink. By shipping a unified, pre-configured runtime that talks directly to the silicon, you bypass that entire compatibility headache.

Moving Forward with DiscreteStack

That is a long list of problems to own: compilation pipelines, offline model lifecycle, air-gapped identity, and a serving stack that holds up under real concurrency. We built DiscreteStack so you do not have to build them.

It is a complete private AI operating system on a single server. Hardware-native compilation, predictive admission control, and model-hardware co-optimization sit on top of the prefix caching, decode scheduling, and batching described earlier, and the result is 13x more throughput from the same silicon: 90% cache hit rates against a 20% baseline, 4.3x faster decode, and 3.2x the concurrent requests. That last figure is the one that decides whether a node serves a team or an individual.

If you have not bought hardware yet, we will spec the node with you and help you obtain it. If you already have it, we compile for the topology you own.

There is no token metering. You pay a flat license fee per execution node per year, which means the hours your GPUs spend on overnight RAG indexing and batch jobs cost nothing at the margin. We build in the EU, deploy air-gapped when your environment demands it, and hold ISO 27001, 9001, and 42001 – worth more than usual with EU AI Act enforcement beginning August 2026. As an EU-incorporated company, we carry no US CLOUD Act exposure.

Run it against your current spend with our cost comparison, or try DiscreteStack and be live in 24 hours.