Hetzner Trials LLM Inference, OpenAI-Compatible API

German cloud infrastructure provider Hetzner has begun experimenting with a large language model (LLM) inference service via an OpenAI-compatible API. The experiment is still in early stages, free, without an SLA, and currently supports only a single model. Yet it draws attention because Hetzner's track record in operational cost efficiency could reshape the increasingly crowded inference market.

What is Hetzner Inference?

The service, named Hetzner Inference, is accessible via the Experiments dashboard in a Hetzner account. Users simply create an API token and point their OpenAI client to https://inference.hetzner.com/api/v1. Currently, the only available model is Qwen/Qwen3.6-35B-A3B-FP8, a Mixture-of-Experts model with 35 billion total parameters and 3 billion active parameters. The model accepts text and image inputs, has a context window of 262,000 tokens, and uses FP8 quantization.

Hetzner also provides a short tutorial on connecting the OpenCode code editor to this API, so testing can be done without writing code.

How Fast is Performance?

Jonas Scholz, co-founder of Sliplane who tried the service on July 23, 2026, recorded impressive numbers. In his tests, the median time to first token was 153 milliseconds for short requests on an already-open connection, and output speed reached 224 tokens per second for longer text generation.

“That is fast!” he wrote, though quickly adding that this was just one data point without performance guarantees under multi-user load.

However, the model's capabilities remain limited. Scholz described it as “a small, slightly shitty LLM”, able to follow formatting instructions and understand images, but failing at simple arithmetic.

Why Hetzner Enters Inference?

For Scholz, the more interesting angle is not the model itself, but the business rationale behind the experiment. Open-weight model inference is becoming a commodity with thin margins because anyone can download the same weights and run similar software. An edge can only be gained if a provider can operate GPUs at very low cost or utilize idle GPU capacity.

Hetzner, with decades of experience efficiently building and managing its own data centers, is seen as a plausible candidate to turn inference into a low-cost infrastructure product. Additionally, a shared inference service could improve utilization of GPUs that were previously only rented dedicated.

“If anyone can turn inference into a thin-margin infrastructure product, Hetzner is at least a trustworthy candidate,” Scholz speculated in his report.

The biggest challenge, however, lies in hardware. Currently, Hetzner's publicly available dedicated GPU lineup includes only the NVIDIA RTX 4000 SFF Ada (20 GB VRAM) and RTX PRO 6000 Blackwell Max-Q (96 GB). The Qwen model used in the experiment takes about 38 GB in FP8 format and fits on a single RTX PRO 6000. But for large models like GLM-5 with 754 billion parameters, a multi-GPU cluster with hundreds of gigabytes of memory and fast interconnects is needed, hardware like B200/B300 that is not yet in Hetzner's public catalog. The big question: is this experiment just the first step toward a more serious hardware investment?

What It Means for Developers

For developers and startups in cost-sensitive markets like Indonesia, where affordable inference solutions are often sought, this experiment could be a positive development in the long run. Open-source models like Qwen are already popular among local developers, and an OpenAI-compatible API makes integration easy without significant code changes. Currently, anyone can try the service for free, though it is not production-ready. If Hetzner later expands model support and maintains its cost efficiency, competition in the inference market could drive down prices, benefiting users in Indonesia and beyond. However, everything remains highly speculative; for now, only a single test model with limited capacity is available.