RDU
The Reconfigurable Dataflow Unit (RDU) is a custom AI inference accelerator developed by SambaNova Systems, designed specifically for agentic AI workloads that require rapid, sequential token generation and low-latency model switching.
Publisher review
The Reconfigurable Dataflow Unit (RDU) is a custom AI inference accelerator developed by SambaNova Systems, designed specifically for agentic AI workloads that require rapid, sequential token generation and low-latency model switching. Unlike conventional GPUs that rely on a fixed von Neumann architecture, the RDU employs a dataflow execution model combined with a unique three-tier memory hierarchy, enabling it to dynamically reconfigure its data paths on the fly. This makes the RDU particularly well-suited for applications such as real-time function calling, structured output generation, and multi-turn conversational agents, where inference speed and energy efficiency are critical. SambaNova targets enterprises and cloud providers that need to deploy large language models at scale without the prohibitive power and latency costs of GPU-based infrastructure.
The fifth-generation SN50 chip delivers 5X more compute and 4X more network bandwidth than its predecessor, the SN40, and supports hot-swapping between models in milliseconds. In benchmark tests, the RDU achieves over 250 tokens per second per user and exceeds 4,500 tokens per second when handling 256 concurrent users. This throughput translates to a 3.7X overall speedup over an NVIDIA DGX H100 and a 6.6X speedup over a DGX A100. The architecture also reduces machine footprint by up to 19X compared to GPU clusters, lowering total cost of ownership for inference-heavy deployments. SambaNova's infrastructure advantages are validated by independent benchmarks: on the Berkeley Function Calling Leaderboard, DeepSeek-V3 running on SambaNova achieved 98% accuracy in simple function calls and 95% in multiple function scenarios, outperforming Fireworks and Together AI by up to 6 percentage points.
SambaNova competes directly with Cerebras Systems, Groq, and NVIDIA's H200 and B200 GPUs. While Cerebras focuses on wafer-scale training and Groq's LPU targets low-latency inference for smaller models, the RDU differentiates itself through reconfigurability and superior performance on agentic tasks. SambaNova has raised over $1.1 billion in funding and collaborates with Intel to deliver high-performance, cost-efficient AI inference solutions. The SN50 chip claims a 5X speed advantage over competitive chips and runs agentic AI at 3X lower cost than GPUs, positioning it as a strong alternative for enterprises seeking to escape NVIDIA's ecosystem.
Despite its strengths, the RDU has notable trade-offs. Performance varies significantly by inference provider even when running the same model, meaning users cannot assume uniform results across platforms. There are inherent trade-offs between inference speed and throughput, and the architecture currently offers limited support for arbitrary kernel code, restricting flexibility for custom model modifications. Additionally, SambaNova's relatively closed ecosystem and smaller developer community compared to NVIDIA's CUDA may pose integration challenges for teams accustomed to GPU-based workflows.
How it works
-
Reconfigurable dataflow execution
The RDU dynamically reconfigures data paths at runtime, enabling efficient execution of diverse model architectures without hardware redesign.
-
Three-tier memory architecture
Combines on-chip SRAM, high-bandwidth memory, and host memory to minimize data movement and reduce latency during inference.
-
Millisecond model switching
Hot-swaps between models in milliseconds, allowing rapid deployment of different AI agents without downtime.
-
High throughput per user
Delivers over 250 tokens per second per user and exceeds 4,500 tokens per second with 256 concurrent users.
-
Energy-efficient inference
Achieves best tokens per watt ratio, reducing power consumption by up to 19X compared to GPU-based clusters.
-
Agentic AI optimization
Purpose-built for agentic inference, excelling in function calling and multi-turn conversations with up to 31 percentage points higher accuracy.
-
Scalable rack-level integration
From chip to rack, the SN50 integrates into dense systems that deliver 5X more compute and 4X more network bandwidth than previous generation.
Strengths and trade-offs
Strengths
- Achieves 98% accuracy in simple function calling with DeepSeek-V3, outperforming Fireworks (94%) and Together AI (96%) on the Berkeley Function Calling Leaderboard.
- Delivers 3.7X overall speedup over an NVIDIA DGX H100 and 6.6X speedup over a DGX A100 in inference benchmarks.
- Reduces machine footprint by up to 19X compared to GPU clusters, lowering data center space and cooling requirements.
- Supports hot-swapping between models in milliseconds, enabling real-time multi-agent workflows without service interruption.
Trade-offs
- Performance of the same model varies significantly across inference providers, meaning SambaNova's results are not universally replicable on other platforms.
- Inference speed and throughput involve inherent trade-offs; optimizing for one metric can degrade the other in production deployments.
- Limited support for arbitrary kernel code through SambaNova Composer restricts flexibility for custom model modifications.
- Smaller developer ecosystem compared to NVIDIA's CUDA may increase integration effort for teams reliant on GPU-optimized libraries.
Pricing context
Per-token API billing with no hourly rental; specific per-token rates are not publicly disclosed and vary by model and deployment scale.
Getting started with RDU
-
Sign up for SambaNova
Go to the SambaNova Systems website and create an account. Provide your email, organization details, and billing information. After approval, you will receive API credentials to access the RDU inference platform.
-
Connect your API credentials
Retrieve your API key from the SambaNova dashboard. Set it as an environment variable in your development environment or include it in your application's configuration file to authenticate requests.
-
Select and configure a model
Choose a supported large language model from the SambaNova model catalog, such as DeepSeek-V3. Configure inference parameters like temperature, max tokens, and top-p in your API call to match your agentic workload requirements.
-
Run a function calling test
Send a sample API request with a function calling prompt, including a list of available functions and a user query. Verify that the RDU returns accurate function call outputs with low latency, confirming the setup works.
-
Deploy for multi-user inference
Scale your application to handle multiple concurrent users by sending batched API requests. Monitor throughput and latency using SambaNova's dashboard, adjusting model switching and concurrency settings to optimize performance.
Frequently Asked Questions
What is the SambaNova RDU and how does it work?
The Reconfigurable Dataflow Unit (RDU) is a custom AI inference accelerator from SambaNova Systems. It uses a dataflow execution model with a three-tier memory hierarchy to dynamically reconfigure data paths at runtime, enabling efficient execution of diverse model architectures without hardware redesign.
How does the RDU compare to NVIDIA GPUs for AI inference?
In benchmarks, the RDU achieves a 3.7X overall speedup over an NVIDIA DGX H100 and a 6.6X speedup over a DGX A100. It also reduces machine footprint by up to 19X compared to GPU clusters, lowering total cost of ownership for inference-heavy deployments.
What makes the RDU suitable for agentic AI workloads?
The RDU is purpose-built for agentic inference, excelling in function calling and multi-turn conversations. It supports millisecond model switching and delivers over 250 tokens per second per user, enabling rapid, sequential token generation and low-latency responses for real-time AI agents.
What are the key features of the SambaNova SN50 chip?
The fifth-generation SN50 chip delivers 5X more compute and 4X more network bandwidth than its predecessor. It features reconfigurable dataflow execution, a three-tier memory architecture, millisecond model switching, and high throughput of over 4,500 tokens per second with 256 concurrent users.
What are the weaknesses of the SambaNova RDU?
Performance varies across inference providers, and there are trade-offs between inference speed and throughput. The architecture offers limited support for arbitrary kernel code, and its smaller developer ecosystem compared to NVIDIA's CUDA may pose integration challenges for teams used to GPU workflows.
How is the SambaNova RDU priced and deployed?
SambaNova uses per-token API billing with no hourly rental; specific rates are not publicly disclosed and vary by model and deployment scale. The RDU targets enterprises and cloud providers, integrating from chip to rack with scalable rack-level systems for dense, energy-efficient inference.
Alternatives
How RDU compares
Direct head-to-head against 2 competitors. Picked by 7wData.
RDU
- Pricing
- Per-token API billing with no hourly rental; specific per-token rates are not publicly disclosed and vary by model and deployment scale.
- Target
- The Reconfigurable Dataflow Unit (RDU) is a custom AI inference accelerator developed by SambaNova Systems, designed specifically for agentic AI workloads that require rapid, sequential
- Strength
- Achieves 98% accuracy in simple function calling with DeepSeek-V3, outperforming Fireworks (94%) and Together AI (96%) on the Berkeley Function Calling Leaderboard.
- Watch for
- Performance of the same model varies significantly across inference providers, meaning SambaNova's results are not universally replicable on other platforms.
ParkMobile
- Pricing
- $0.99-$2.50 per transaction + parking fees
- Target
- Urban parking management
- Deployment
- Mobile app, web
- Strength
- Widest US coverage (500+ cities)
- Watch for
- Service fees vary by location
SpotHero
- Pricing
- Dynamic pricing based on location/demand
- Target
- Airport/event parking reservations
- Deployment
- Mobile app, API
- Strength
- Guaranteed spot reservations
- Watch for
- Non-refundable bookings common
User reviews
No user reviews yet. Be the first to write one.
Sources
Reporting on this tool draws on these publicly available sources.