Many people look at systems such as Google's AI Overviews and assume that current AI capabilities represent the limits of what large language models can achieve. In reality, both hardware and software are advancing rapidly, enabling models to run faster, scale to larger parameter counts, and operate on hardware that would have been impractical only a short time ago.
Recent advances include Mixture of Experts (MoE) architectures, low-bit quantization, and improved memory management. Together, these techniques reduce computational requirements, shrink storage footprints, and allow increasingly large models to execute efficiently using less hardware.
Although Google operates some of the world's largest AI data centers, computational efficiency remains critical because AI Overviews must evaluate an enormous and constantly changing web. Every generated answer requires retrieving relevant information, processing it, and producing a response within tight latency and cost constraints.
Key Technologies
MoE Routing: Routes each token to only the required expert sub-networks rather than the entire model, reducing the computational workload for each token.
Bit Quantization: Reduces the storage size and memory bandwidth required for model weights, allowing larger models to fit into available storage and memory with only a modest reduction in numerical precision.
Memory Mapping: Allows the operating system to load portions of the model's weights into RAM only as they are needed, enabling models much larger than physical memory to execute efficiently.
TurboQuant / TurboVec algorithms: Compresses vector embeddings and key-value (KV) caches, reducing the memory and computational cost of retrieval and inference. This can make it more practical for AI systems to incorporate freshly retrieved information into generated answers.
Running Large Mixture of Experts Models Locally
Running a 744-billion-parameter large language model (LLM) locally on a laptop using only a CPU and 25 gigabytes of RAM, with no connection to a data center, may seem impossible. In fact, it has been demonstrated, although the resulting performance is far below what would be considered practical for everyday use.
Cloud Code's video, Run the GLM-5.2 on Your Laptop Locally (No GPU Needed), demonstrates one such implementation. Although the laptop configuration is primarily a proof of concept, the same general approach can be applied to more capable hardware, allowing increasingly larger models to run outside of traditional GPU-based deployments.
Youtube Demonstrations
Key Techniques
The first technique is to use a Mixture of Experts (MoE) architecture. Rather than activating all 744 billion parameters for every token, the model activates only the small subset of expert networks needed for that portion of the input. This reduces the computational workload during inference but does not reduce the amount of memory required to store the model.
The second technique is to quantize the model's 1.4 terabytes of weights into a dynamic 2-bit representation, reducing their storage footprint to approximately 239 gigabytes. The quantized weights are stored on an SSD.
The third technique is memory management. The portion of the model's weights needed for every inference request—approximately 10 gigabytes—is kept resident in system RAM. The remaining weights stay on the SSD and are memory-mapped, allowing the operating system to load only the portions required during inference rather than the entire model into RAM.
Together, these three techniques reduce computation, shrink the model's storage footprint, and allow the remaining weights to be loaded into memory only as needed. Individually, none of the techniques is sufficient. Combined, they make it possible to run extremely large language models on hardware that would otherwise appear incapable of supporting them.
Performance
Performance depends on the available hardware; different people have tested GLM-5.2 on different systems and reported differences in performance. On a CPU-only desktop, inference may achieve approximately one token per second. A system equipped with a single NVIDIA RTX 5090 GPU may generate around 11 tokens per second, while systems with four GPUs may achieve between 10 and 20 tokens per second, depending on the hardware configuration and workload.
For comparison, the average adult reads at approximately seven tokens per second. Although the laptop implementation is significantly slower than human reading speed, it demonstrates that a 744-billion-parameter model can execute locally without relying on cloud-based GPU infrastructure.
Chat Performance vs. Coding Performance
The reported performance figures primarily reflect conversational use, where responses are relatively short and generated interactively. GLM-5.2, however, is optimized for coding tasks, which often require generating thousands of tokens for a single response.
On a CPU-only system producing approximately one token per second, generating a large software revision may take many hours or even days. More capable hardware can reduce that time significantly, but long-form code generation remains substantially more demanding than interactive chat.
This highlights an important consideration when evaluating AI systems: the computational cost depends not only on the size of the model but also on the length and complexity of the output. Short conversational responses can be practical on modest hardware, whereas large coding or document-generation tasks generally require considerably more computing resources.
Published
by Wayne Smith – Raising the Standards
Wayne Smith has worked in online marketing, search, and web development for several decades. His work includes building document retrieval systems, search engine simulations, and AI-assisted information systems. Drawing from software testing, information retrieval, and reverse engineering, he studies how AI systems discover, interpret, and synthesize information into answers.
This article is part of the State of AI series, which examines how advances in AI are changing search and information retrieval. Recent innovations such as Mixture of Experts (MoE) architectures, low-bit quantization, memory mapping, and more efficient retrieval techniques are making increasingly large language models practical on both consumer hardware and enterprise-scale systems. Understanding these advances helps explain how AI systems are becoming faster, more scalable, and better able to retrieve and synthesize current information while reducing the computational cost of inference.