# Custom Kernels Are King: How QwenASR Makes CPU Inference Blazing Fast AI-readable version of this post. Path: /systems/custom-kernels-are-king Category: Systems Date: 2026-07-29 Author: Shubham Singh Reading time: 4 min Description: A human-friendly dive into how QwenASR squeezes maximum performance out of CPUs by optimizing memory, kernels, and scheduling. ## The Line I Couldn't Ignore I recently came across a project that really impressed me. The team built a speech recognition engine in pure Rust that runs entirely on the CPU. In their benchmarks, it was even faster than several GPU-based implementations. At the time, I had just finished optimizing my own Parakeet 0.6B CPU project. Most of my work focused on making the user wait less by processing audio while it was still being recorded. Then I started reading the QwenASR documentation and source code. What impressed me wasn't just the benchmark numbers. It was how much engineering went into making it fast. They didn't just quantize the model or switch runtimes. They optimized almost every part of the system—from loading weights and memory management to custom CPU kernels, scheduling, and benchmarking. That's what this blog is about. Not "CPU beats GPU," but how careful engineering at every layer can add up to huge performance gains. ![Pipeline speed versus kernel speed](/qwen-custom-kernels/pipeline-vs-kernel.svg) --- ## Why This Engineering Matters Most articles about faster AI focus on one simple trick: converting a model to a smaller format or picking a different runtime. Useful, but shallow. QwenASR is interesting because its performance comes from solving dozens of small efficiency problems together. The team systematically attacked bottlenecks in data movement, memory allocation, thread scheduling, and kernel execution. What stands out most in their research logs is discipline. They tried many performance ideas, tested each one against both speed and transcription accuracy, and discarded anything that didn't deliver real improvements. When you see a fast tool, it's easy to summarize it as "written in fast Rust." But that hides the actual effort. Someone had to write custom SIMD assembly kernels, build a disk cache for instant startup, and fine-tune multi-core scheduling so CPU cores don't stall waiting for each other. ```mermaid flowchart LR A[Safetensors loader] --> B[BF16 conversion] B --> C[INT8 sidecar] C --> D[Encoder path] D --> E[Decoder path] E --> F[Custom kernels] F --> G[Dynamic scheduler] G --> H[Accuracy checks] ``` --- ## What the Numbers Show According to the project's benchmarks, QwenASR processed a **28.2-second audio file in just 613 milliseconds** on a CPU. That means it transcribes speech **46 times faster than real-time**. Here is how it compared to other implementations on the same test audio: | Input Audio | Implementation | Hardware | Processing Time | Real-Time Factor | |---|---|---|---:|---:| | 28.2 sec | **qwen-asr (latest)** | **CPU (Rust)** | **613 ms** | **46.00x** | | 28.2 sec | mlx-audio (Python MLX) | GPU | 688 ms | 40.94x | | 28.2 sec | second-state MLX | GPU | 1,414 ms | 19.91x | | 28.2 sec | upstream C | CPU | 1,660 ms | 16.96x | | 28.2 sec | qwen-asr (first Rust port) | CPU | 1,698 ms | 16.61x | Chewing through nearly 30 seconds of speech in roughly half a second on a standard CPU shows what's possible when software is written specifically for the hardware it runs on. ![Latency comparison](/qwen-custom-kernels/latency-comparison.svg) --- ## Two Different Engineering Strategies Comparing QwenASR to my own Parakeet CPU project highlights two distinct ways to make AI feel fast: 1. **My Parakeet Project (Pipeline Optimization):** I focused on user experience. The app captures microphone audio and processes it in overlapping 10-second chunks while the user is still speaking. The model runs behind ONNX, and latency disappears because most of the transcription finishes before the user hits stop. 2. **QwenASR (Compute Engine Optimization):** QwenASR is an engine-first project. Instead of wrapping an existing runtime, the team wrote the entire inference stack—from weight loading to matrix math. They focused on making every single calculation and memory access as cheap as possible. Put simply: **My project optimized *when* the work happens. QwenASR optimized *how expensive* the work is.** ![Who owns which layer](/qwen-custom-kernels/ownership-stack.svg) --- ## The Key Performance Upgrades ### 1. Attacking Memory Bottlenecks First In modern AI inference, moving data from RAM to the processor often takes more time than doing the math. QwenASR quantizes heavy decoder weights to INT8 format and saves them as a pre-formatted sidecar file on disk. On future starts, it maps this file directly into memory (`mmap`), avoiding repeated weight conversions and slow startup copying. ```mermaid flowchart LR A[Original weights] --> B[Quantize decoder paths] B --> C[INT8 format] C --> D[Save sidecar file] D --> E[mmap on startup] E --> F[Instant loading] ``` ### 2. Fusing Operations Together Normally, neural networks run one operation, write intermediate results to memory, and read them back for the next operation. QwenASR fuses adjacent steps into single loops. By keeping data inside CPU caches instead of bouncing back and forth to main memory, it eliminates unnecessary data passes. ### 3. Native SIMD Hardware Kernels Instead of relying entirely on standard compiler optimizations, the team wrote custom SIMD kernels (NEON for ARM and Apple Silicon, AVX for x86 CPUs). Where platform-native libraries like Apple's Accelerate framework were faster, they integrated those directly. ```mermaid flowchart TD A[Kernel Dispatch] --> B[Apple Accelerate] A --> C[ARM NEON] A --> D[x86 AVX2] A --> E[Generic Fallback] ``` ### 4. Smarter Thread Scheduling Modern CPUs have a mix of high-performance and high-efficiency cores. Assigning work evenly can cause fast cores to sit idle waiting for slow cores to finish. QwenASR uses dynamic chunking and work-stealing: faster cores automatically process more work, keeping all cores productive. ### 5. Reusing Temporary Memory Allocating new memory while generating text causes unpredictable performance spikes. QwenASR pre-allocates reusable buffers for layers, tokens, and workspaces. Removing repeated memory allocations keeps execution smooth and consistent. ### 6. Disciplined Accuracy Guardrails Speed improvements are useless if transcription accuracy drops. Every optimization was tested against Word Error Rate (WER) metrics. When an experimental INT4 quantization path improved speed but caused minor accuracy errors, the team deleted it. ```mermaid flowchart LR A[New Idea] --> B[Speed Test] B --> C[Accuracy Test] C --> D{Better & Accurate?} D -->|Yes| E[Keep] D -->|No| F[Discard] ``` --- ## Summary Writing custom kernels and building a custom engine yields incredible performance, but only when paired with careful profiling and strict accuracy checks. While pipeline tricks like live chunking make applications feel instant to users, QwenASR shows how much speed is left on the table when you optimize the underlying compute engine itself. Combining both approaches is the ultimate blueprint for fast, local AI.