Articles Tagged “LLM Inference”
10 articles found
Nvidia KV Cache Transfer Skips 7-Second Re-Prefills
Nvidia researchers moved a 32,768-token KV cache between model sizes in 278 milliseconds, replacing a 7-second re-prefill with closed-form linear math.
LFM2.5-DSpark Speeds Local AI Inference Up to 3.2x
Liquid AI released ~300M-parameter draft models for the LFM2.5 family, delivering up to 3.18x GPU throughput and 57% lower function-calling latency.
Sparse Mixture of Experts Explained for 2026 Models
Why a 2.4T-parameter model can be cheaper than a 70B one: what active parameters mean, how routing works, and what MoE really costs to self-host in 2026.
Local LLM Servers Compared: Ollama vs vLLM vs llama.cpp
Ollama, llama.cpp, vLLM and LM Studio all serve local models. Here's which one fits your hardware, from a 16GB laptop to a four-GPU workstation.
OpenAI Cuts Luna Prices 80% After AI Rewrote Its Kernels
OpenAI dropped GPT-5.6 Luna to $0.20 per million input tokens after Sol rewrote its own GPU kernels, cutting end-to-end serving costs by 20%.
Liquid AI Encoders Hit 8K Context on CPU 3.7x Faster
Liquid AI's new LFM2.5-Encoders run 8,192-token inputs on a plain CPU roughly 3.7x faster than ModernBERT-base, from just 230M parameters and open weights.
LLM Quantization Guide: GGUF vs AWQ vs MLX in 2026
A practical guide to LLM quantization formats — GGUF, AWQ, GPTQ and MLX — with VRAM math, quality trade-offs and picks for every kind of machine.
OpenAI and Broadcom's "Jalapeño" Chip: Custom Silicon for Inference
OpenAI's first custom chip, Jalapeño, is an inference-focused ASIC co-built with Broadcom, promising better performance-per-watt for large language models.
NVIDIA Releases Nemotron Diffusion Language Models — A Single Checkpoint That Generates Text Up to 6.4x Faster
NVIDIA Nemotron Labs released a family of diffusion language models on May 23, 2026 — 3B, 8B, and 14B text models plus an 8B VLM that generate tokens in parallel and refine them, hitting 6.4x speedups via self-speculation.
New Self-Distillation Technique Triples LLM Inference Speed With a Single Model
Researchers achieve 3x faster LLM inference by baking multi-token prediction directly into model weights — no draft model or extra hardware required.