Kog Inference Engine Squeezes 30x From Existing GPUs

The prevailing answer to slow AI inference has been to buy different silicon. Kog, an 11-person startup out of Paris, is making the less fashionable argument that the hardware enterprises already own is leaving most of its performance on the table — and that the gap is a software problem.

• Demonstrated throughput: 3,000 tokens per second on single-request decoding in the company's May tech preview
• Hardware used: Standard AMD MI300X and NVIDIA H200 datacenter GPUs, not custom accelerators
• Model tested: Laneformer 2B, a 2-billion-parameter model Kog has since open-sourced
• Team: 11 people, of whom 10 are engineers and researchers including five PhDs

What Kog Is Optimising

The Kog Inference Engine works below the frameworks most teams touch, targeting the low-level GPU code paths that determine how efficiently a decoding step actually executes. Founder and CEO Gael Delalleau's background spans cybersecurity research and high-performance GPU work, which is a reasonable combination for the kind of person willing to read a lot of assembly.

The headline claim is a path to 30x faster inference. The demonstrated result is narrower and worth stating precisely: 3,000 tokens per second on single-request decoding, using a 2-billion-parameter model. Small models are the friendliest case for this kind of optimisation, so the number should be read as a proof of the technique rather than a guarantee of what a frontier-scale model will do.

Delalleau told TechCrunch the company plans to demonstrate its first major model at 10x speed by September before raising a Series A. That is the milestone to watch — the one that tells you whether the approach scales past the demo.

Why Does Single-Request Speed Matter Most?

Most inference optimisation targets throughput across many concurrent users, which is what a model provider cares about. Single-request decoding latency is what an individual developer feels when an agent is working through a multi-step task and every token arrives in series.

That distinction is why the software-engineering use case surfaced first for Kog. Coding agents are latency-sensitive in a way that batch workloads are not. It is the same pressure driving hardware-side work like NVIDIA's Nemotron 3.5 Lightning for agents on RTX PCs — the industry has broadly concluded that agentic workloads need a different performance profile than chat.

The funding reflects a European deep-tech pattern: a seed round co-led by Varsity VC, founded by Delalleau's former cofounder Kamel Zeroual, with backing from Scaleway, Bpifrance and the French Tech 2030 programme. Kog says the May preview generated 200 tangible business leads, which is a lot of inbound for a tech demo.

What It Means If the Approach Holds

An organisation that has already bought MI300X or H200 capacity and can multiply its effective throughput in software has changed its cost structure without a procurement cycle. That is a meaningfully different proposition from swapping accelerators.

It also fits a broader pattern we keep seeing in AI infrastructure coverage: squeezing more from constrained hardware is where a surprising amount of the real progress happens. The extreme end of that is running a 28.9-million-parameter LLM on an ESP32-S3 at 9 tokens per second — different scale entirely, same instinct.

Open-sourcing Laneformer 2B was a smart move for a small team making large claims. It lets others check the work, which is the fastest route to being believed.

Sources: TechCrunch — August 14, 2026; Kog Labs engineering blog — 2026.