TheModelDesk

Hugging Face has updated the transformers modeling backend for vLLM, achieving performance that now matches or exceeds custom vLLM native implementations for various LLM architectures. This update allows model authors to utilize their existing transformers implementations to get high-speed vLLM inference without needing to write a separate, hand-optimized port.

Benchmarks using Qwen3 models—including a 4B dense model, a 32B dense model with tensor parallelism, and a 235B FP8 Mixture-of-Experts (MoE) model with data and expert parallelism—demonstrate that the transformers backend meets or beats native throughput across these configurations. Users can activate this functionality by using the `--model-impl transformers` flag during serving.

Technically, the backend now employs torch.fx for static graph analysis to identify optimizable patterns, combined with abstract syntax tree (AST) manipulation to rewrite operations. This process enables dynamic, runtime application of inference-specific layer fusions. Specifically, it maps operations to optimized vLLM kernels, including MergedColumnParallelLinear, QKVParallelLinear, and those required for Expert Parallelization in MoE models.

To use these improvements, developers should upgrade their vllm package via `uv pip install --upgrade vllm --torch-backend auto`. While most Hugging Face models are supported, those using linear attention are not currently compatible, and custom models hosted on Hub repos may fail if they do not comply with standard implementation patterns.