Why Local LLMs Don't Need C++ or Python: Building a 15MB Native AOT Inference Engine in .NET 10

Bare-Metal LLM Inference in Pure C#: Bypassing CUDA Toolkits and Native C++ DLLs The conventional consensus across AI engineering is simple: high-performance local LLM execution belongs exclusively to C++ runtimes, multi-gigabyte CUDA toolkits, and bindings over llama.cpp or vLLM. When orchestrating local models from managed languages like C#, typical implementations rely on interop wrappers over unmanaged native binaries (cudart64.dll, cublas64.dll, or libllama). This introduces DLL distribution overhead, host-to-device PCIe bandwidth bottlenecks during token sampling, and severe desktop instability when running compute on display-bound integrated GPUs. By stripping out runtime layers and interacting directly with driver interfaces, managed runtimes can match and outpace conventional native daemons. Glacier.Inference runs direct memory-mapped GGUF models in pure C# .NET 10 across NVIDIA, AMD, and Intel silicon without external C++ binaries. ┌────────────────────────────────────────────────────────────────────────┐ │ Glacier.Inference Core │ │ ├─ MemoryMappedFile Zero-Copy GGUF Reader (Sub-30ms cold mapping) │ │ ├─ Pure C# Bare-Metal SASS Engine (Direct P/Invoke nvcuda.dll) │ │ ├─ Bare-Metal Direct3D 12 Compute (HLSL Wave32 via Vortice.D3D12) │ │ ├─ SpeculativeEngine (N-gram Prompt Lookup & Batched Verification) │ │ ├─ Fused In-VRAM GPU Argmax Reduction (Warp-shuffle, 4-byte transfer) │ │ └─ Adaptive Unmanaged KV-Cache (FP16 / FP8 Dynamic Ring Buffer) │ └────────────────────────────────────────────────────────────────────────┘ Typical CUDA execution routes instructions through cudart64.dll and cublas64.dll. Glacier completely circumvents the CUDA runtime layer. It communicates directly with the base Windows GPU driver (nvcuda.dll) via low-level P/Invoke, dispatching precompiled fatbinaries straight to the Streaming Multiprocessors (SMs). // Direct driver context and module dispatch without cudart64.dll [DllImport("nvcuda.dll", EntryPoint = "cuLaunchKernel")] public static unsafe extern CUresult LaunchKernel( CUfunction f, uint gridDimX, uint gridDimY, uint gridDimZ, uint blockDimX, uint blockDimY, uint blockDimZ, uint sharedMemBytes, CUstream hStream, void** kernelParams, void** extra); By compiling modular .cuh routines into an embedded universal cubin containing dedicated slices (sm_75, sm_80, sm_86, sm_89, sm_90), the engine verifies zero DRAM stack spills (STACK: 0 via cuobjdump). Registers house all dequantization multipliers and accumulators without hitting DRAM stack frames. The result is a self-contained ~15 MB single-file Native AOT executable replacing a 4.5 GB toolchain. In standard architectures, evaluating greedy token selection involves transferring logit tensors back over PCIe to the host CPU: Logit Transfer Per Step = 152K vocab × 4 bytes ≈ 608 KB At 45 tokens per second, transferring 608 KB back and forth over the host interface introduces micro-stalls and PCIe latency. Glacier fuses the final linear projection and reduction directly on the device with a 512-thread warp-shuffle kernel: // GPU-side warp reduction eliminating CPU transfer overhead [numthreads(512, 1, 1)] void ArgmaxReduction(uint3 tid : SV_DispatchThreadID, uint3 lid : SV_GroupThreadID) { // 512-thread tree reduction within registers across 152k logits // Emits exactly 1 int32 winning token index into device memory } Instead of copying 608 KB across the bus every token, Glacier transfers exactly 4 bytes (one int32 token ID), dropping reduction latency to ~3.2 µs. Deploying local inference on consumer hardware often fails on laptops where AMD RDNA or Intel Arc graphics double as the primary display adapter. Long compute dispatches trigger Windows Timeout Detection and Recovery (TDR), resetting the display driver. Glacier implements a pure Direct3D 12 compute engine using Vortice.D3D12 and HLSL Wave32 compute shaders: Fine-Grained Command Dispatches: Eliminates TDR resets by cooperating cleanly with Desktop Window Manager (DWM). Unified Memory (UMA) Saturation: On architectures like the AMD Radeon 890M (16 CUs, RDNA 3.5), weights map straight into unified LPDDR5X memory without secondary PCIe staging copies. 32-Token GEMM Tiling: Prefill GEMV loops unroll into 4-tile register chunks, keeping dequantized vectors inside fast SIMD32 wave registers. Autoregressive transformer generation is inherently memory-bandwidth bound. Every generated token requires streaming the entire model weight footprint through the compute core. On a 128-bit GDDR6 memory bus running at 256 GB/s, reading a 4.68 GB model sets a strict theoretical wall clock limit: Max Theoretical Serial Throughput = 256 GB/s ÷ 4.68 GB ≈ 54.7 tokens/sec Glacier integrates batched speculative verification (VerifyBatch) using zero-cost prompt suffix lookup (PromptLookupDraftProvider): using var target = new InferenceSession("models/Qwen2.5-7B-Instruct-Q4_K_M.gguf"); using var engine = new SpeculativeEngine(target); var options = new SpeculativeOptions { MaxDraftTokens = 4, // Propose 4 candidate tokens in <1 µs MaxTokens = 256 }; var result = await engine.GenerateAsync("Explain quicksort in C#", options); Instead of reading 4.68 GB from VRAM $K$ times for $K$ tokens, the candidate sequence is verified against the transformer in a single batch pass. The weights are pulled through the memory bus only once, accelerating generation rates to 72–104+ tokens/sec on an RTX 4060 laptop GPU. The following runs compare Glacier against an Ollama local daemon on identical hardware. Model: DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf (4.68 GB) Metric Glacier.Inference (.NET 10) Local Daemon (Go + C++ CUDA) Margin Runtime Architecture Pure C# (Native AOT) C++ / cuBLAS / libllama Zero external DLLs Binary Footprint ~15 MB Single File ~4.5 GB Toolkit + Engine 300x smaller Cold Start to First Token 1.50 s 3.50+ s 2.3x faster Serial Generation 41.92 tok/s (208 GB/s) 43.20 tok/s (216 GB/s) Within 3% of cuBLAS Speculative Generation 72.5 – 104.8 tok/s N/A (Serial decode) Up to 2.4x faster Sampling Overhead ~3.2 µs (In-VRAM) ~800 µs (DtoH transfer) 250x reduction Model: Qwen3-30B-A3B-Instruct-Q3_K_L.gguf (13.58 GB MoE, 3B Active) Execution Pipeline Memory Model Generation Rate Turnaround Time Direct3D 12 Compute (HLSL Wave32) Unified LPDDR5X (Direct) 21.68 tok/s 1.54 s Host CPU SIMD (24T AVX-512) System Memory 0.89 tok/s 18.20 s Running a 30B parameter Mixture-of-Experts architecture in pure C# directly on an integrated APU delivers 21.68 tok/s, beating multi-threaded AVX-512 CPU execution by 24.4x. Managed languages don't have to surrender low-level compute workloads to external runtime stacks. By combining MemoryMappedFile zero-allocation weight access, Direct3D 12 compute pipelines, direct driver P/Invokes, and in-VRAM warp reductions, pure .NET 10 delivers bare-metal throughput while keeping deployment to a single, portable binary. The complete code, benchmarks, and standalone CLI binaries are available on GitHub: Glacier.Inference.