LPU

Groq's Language Processing Unit (LPU) is a custom-designed inference accelerator that uses static RAM (SRAM) instead of traditional DRAM to achieve deterministic, low-latency performance for large language models. Founded in 2016 by former Google TPU engineers, Groq targets developers and enterprises running real-time AI applications—such as conversational voice assistants, multi-step agentic workflows, and high-frequency text generation—where sub-millisecond latency and predictable throughput are critical. The LPU is available through GroqCloud, a cloud inference platform with Free, Developer, and Enterprise tiers, and can also be deployed on-premises via the GroqRack for regulated or air-gapped environments. Over 1.9 million developers have used GroqCloud, with enterprise deployments at Dropbox, Volkswagen, and Riot Games, and Meta partnered with Groq in April 2025 to deliver fast inference for the official Llama API.

The LPU's architecture is fundamentally different from GPUs: it uses a programmable assembly line design that processes tokens in a deterministic, single-pass manner without batching. This eliminates the variable latency and memory bottlenecks common in GPU inference. The LPU relies entirely on on-chip SRAM for data access, which is orders of magnitude faster than the HBM (High Bandwidth Memory) used in NVIDIA H100 clusters. For example, the LPU delivers Llama 2 70B at 300 tokens per second—10 times faster than H100 clusters running the same model. On GroqCloud, Llama 3.1 8B Instant achieves 840 tokens per second, while Llama 3.3 70B Versatile runs at 394 tokens per second. The architecture also supports popular models including Llama, Qwen, Mistral, Whisper, and OpenAI open-weights, as well as text-to-speech models like Canopy Labs Orpheus and ASR models like Whisper V3 Large at 217x real-time speed.

In the inference market, Groq competes directly with NVIDIA's H100 and B200 GPUs and Google's TPUs, but with a narrower focus. While NVIDIA GPUs dominate training and general-purpose AI workloads, the LPU is purpose-built for inference, offering deterministic execution and consistent performance that GPUs cannot match due to their variable batching and memory architecture. Groq's pricing model—charging per token with no idle infrastructure fees—contrasts with GPU cloud providers that bill by compute hour, making Groq more cost-effective for bursty or unpredictable inference loads. However, under equivalent throughput conditions, the LPU's hardware cost per chip is higher, and its SRAM-based design limits the size of models that can be run without sharding. Groq has deployed data centers in four global regions, with SOC 2, GDPR, and HIPAA compliance.

The honest trade-offs: The LPU is not a general-purpose processor—it cannot train models or run non-inference workloads, limiting its flexibility compared to GPUs. Its reliance on SRAM means that very large models (e.g., 200B+ parameters) require significant chip-level parallelism, increasing system complexity and cost. Additionally, the LPU's ecosystem of supported frameworks and integrations is smaller than NVIDIA's CUDA ecosystem, which may require additional engineering effort for custom model deployment. Finally, while per-token pricing avoids idle costs, heavy sustained throughput can be more expensive per token than a reserved GPU instance, making Groq less economical for constant high-volume inference without careful workload planning.

Per-token pricing with no idle fees. GroqCloud tiers: Free (community support, zero-data retention), Developer (higher limits, chat support, batch processing, prompt caching), Enterprise (custom models, regional endpoints, dedicated support, LoRA fine-tunes). Example: Llama 3.1 8B Instant at $0.05/M input tokens and $0.08/M output tokens; Llama 3.3 70B at $0.59/M input and $0.79/M output.