French Startup Kog Aims to Accelerate AI Inference on Datacenter GPUs
French artificial intelligence startup Kog is taking an alternative path in the push for faster AI inference by focusing on software optimization for standard enterprise GPUs rather than relying on custom-built silicon.
What Happened
As the tech sector increasingly focuses on lowering the latency and cost of AI inference, Kog revealed a technical preview demonstrating high-speed single-request decoding on widely used datacenter processors, specifically the Nvidia H200 and AMD MI300X. While market attention has recently highlighted specialized hardware such as Cerebras systems following its May public debut, Kog asserts that standard graphic processing units still possess substantial untapped potential due to expanding memory bandwidth.
During its initial demonstration, Kog achieved an output of 3,000 per-request tokens per second (TPS). This performance benchmark was achieved using Laneformer 2B, an open-sourced 2-billion parameter model designed specifically for the test. Following the showcase, which gained traction on Hacker News in May, founder and Chief Executive Officer Gaël Delalleau reported receiving roughly 200 commercial business leads.
The company observed that prospective business clients are primarily seeking optimizations for larger models rather than fine-tuning small parameter options. Consequently, the team is working to adapt its methodology to large language models (LLMs) with the broader objective of achieving up to 30 times faster inference.
Key Highlights
- Target Hardware: The software is engineered for standard datacenter accelerators, including the AMD MI300X and Nvidia H200, and does not extend to consumer laptop processors.
- Demonstration Performance: Reached 3,000 tokens per second on a purpose-built 2-billion parameter model dubbed Laneformer 2B.
- Primary Use Cases: Initial commercial traction focuses on software engineering workflows where latency causes significant delays, as well as prompt-based game and app generation.
- Engineering Approach: Rather than using hardware-agnostic abstraction layers that bypass Nvidia CUDA, Kog relies on low-level reverse engineering down to binary and assembly code.
- Team and Backing: The 11-person startup is supported by Scaleway, backed by France’s Bpifrance and the French Tech 2030 initiative, with seed funding co-led by Varsity VC.
Why This Matters
Inference speed and running costs represent substantial operational hurdles for companies deploying generative AI. In professional software development environments, developers using code-generation assistants can face extended waiting periods for outputs, prompting model providers like Anthropic to charge higher pricing tiers for faster execution modes.
By squeezing higher throughput directly out of enterprise hardware already installed in datacenters, companies could bypass the need to procure rare or specialized custom chips. However, Kog’s low-level optimization method requires weeks or months of hands-on reverse engineering for each specific chip model, creating a technical bottleneck for its 11-person team as it works across various architectures.
What to Watch Next
Kog is working to implement its acceleration technology on a major large language model with a target speed increase of 10 times, an achievement Delalleau aims to reach by September. Reaching that performance milestone is expected to anchor customer validation and support the company’s upcoming Series A fundraising efforts.
Over the longer term, the company intends to integrate its engineering workflows into automated agent-based pipelines, which could enable support for a broader range of hardware chips and model architectures.
Frequently Asked Questions
What type of processors does Kog’s software support?
Kog focuses exclusively on high-end enterprise datacenter GPUs, having conducted demonstrations on AMD MI300X and Nvidia H200 chips. The software is not built for consumer hardware or laptop GPUs.
How does Kog’s acceleration methodology work?
Founder Gaël Delalleau applies methods from solid-state physics and offensive cybersecurity, focusing on reverse-engineering GPU architecture at the binary and assembly levels to unlock memory bandwidth that is underutilized during decoding tasks.
What is Laneformer 2B?
Laneformer 2B is an open-sourced, 2-billion parameter model created by Kog to demonstrate its technical capability of reaching 3,000 tokens per second on datacenter hardware.
Source: Based on reporting published by TechCrunch.