MemKV

MemKV is a purpose-built context memory store for AI inference workloads, developed by MinIO.

Reviewed by 7wData

On this page

Publisher review

MemKV is a purpose-built context memory store for AI inference workloads, developed by MinIO. It is designed for organizations running large-scale agentic AI systems that require persistent, shared context across GPU clusters at petabyte scale. The product targets the specific problem of GPU underutilization in inference pipelines, where GPUs often spend more time recomputing key-value (KV) state than generating tokens, leading to poor time-to-first-token (TTFT). By offloading context storage to a flash-based, RDMA-accessible store, MemKV aims to keep GPUs busy generating tokens rather than recomputing context. Its primary audience includes AI infrastructure teams deploying large language models (LLMs) and multi-agent systems that need low-latency context retrieval without the overhead of traditional in-memory caches.

MemKV delivers microsecond context retrieval at petabyte scale by using native flash storage accessed end-to-end over 800 GbE RDMA. This architecture increases GPU utilization from approximately 50% to over 90%, according to MinIO, by eliminating the recompute tax for AI inference workloads. The store is available on the NVIDIA BlueField-4 STX architecture and supports native integration with NVIDIA Dynamo and NVIDIA NIXL, enabling direct data paths between GPUs and the context store. MemKV provides persistent, shared context across GPU clusters, meaning that if a GPU fails or a request is routed to a different node, the context is not lost and does not need to be recomputed. This breaks the traditional speed-scale tradeoff, allowing both high throughput and low latency at production concurrency levels.

MemKV competes directly with in-memory key-value stores like Redis, KeyDB, Dragonfly, and Valkey, which are often used for caching AI inference context. However, those tools are general-purpose and not optimized for the specific demands of petabyte-scale AI inference. Redis, for example, is limited by DRAM capacity and single-threaded architecture, whereas MemKV is designed to scale horizontally across flash storage with RDMA networking. KeyDB and Dragonfly offer multi-threaded improvements but still rely on DRAM, making them cost-prohibitive at petabyte scale. Valkey, a Redis fork, shares similar limitations. MemKV's advantage is its ability to provide persistent, shared context at a fraction of the cost of DRAM-based solutions, though it introduces a dependency on flash storage and RDMA infrastructure.

The primary trade-off with MemKV is its dependency on specialized hardware: it requires NVIDIA BlueField-4 STX architecture and 800 GbE RDMA networking, which may not be available in existing data centers. Additionally, as a new product, its ecosystem is immature compared to Redis, with fewer third-party integrations and community support. The pricing is not publicly disclosed, which makes cost comparison difficult for potential buyers. Finally, while MemKV excels at context storage for inference, it is not a general-purpose cache and cannot replace Redis for other workloads like session management or rate limiting without significant architectural changes.

Get the AI & data signal, daily.

335k+ subscribers read this every morning. One email, both newsletters. Unsubscribe anytime.

How it works

  1. Microsecond context retrieval

    Delivers context retrieval in microseconds at petabyte scale using flash storage and 800 GbE RDMA, reducing latency for AI inference.

  2. GPU utilization boost

    Increases GPU utilization from approximately 50% to over 90% by eliminating the recompute tax for KV state in inference workloads.

  3. Petabyte-scale shared context

    Provides persistent, shared context memory across GPU clusters at petabyte scale, eliminating context loss during failures or routing changes.

  4. NVIDIA BlueField-4 STX support

    Available on NVIDIA BlueField-4 STX architecture, enabling direct data paths between GPUs and the context store via RDMA.

  5. NVIDIA Dynamo and NIXL integration

    Supports native integration with NVIDIA Dynamo and NVIDIA NIXL, allowing direct communication between GPUs and MemKV without CPU involvement.

  6. End-to-end 800 GbE RDMA

    Accesses context memory over 800 GbE RDMA, providing high-bandwidth, low-latency data transfer between storage and compute nodes.

  7. Eliminates recompute tax

    Reduces the recompute tax for AI inference workloads by storing KV state persistently, so GPUs do not need to recompute context on each request.

Strengths and trade-offs

Strengths

  • MemKV improves time-to-first-token (TTFT) at production concurrency by delivering microsecond context retrieval at petabyte scale.
  • It increases GPU utilization from approximately 50% to over 90%, reducing idle time spent on recomputing KV state.
  • It eliminates context loss across GPU clusters by providing persistent, shared context memory, so failures or routing changes do not require recomputation.
  • It breaks the speed-scale tradeoff by combining high throughput and low latency, enabling agentic AI workloads that were previously impractical.

Trade-offs

  • MemKV requires specialized hardware, including NVIDIA BlueField-4 STX architecture and 800 GbE RDMA networking, which may not be available in existing infrastructure.
  • Pricing is not publicly disclosed, making it difficult for organizations to compare total cost of ownership against alternatives like Redis or Dragonfly.
  • As a new product, MemKV has a limited ecosystem of third-party integrations and community support compared to mature tools like Redis.
  • MemKV is purpose-built for AI inference context storage and cannot serve as a general-purpose cache for other workloads such as session management or rate limiting.

Pricing context

Not publicly disclosed; interested parties must contact MinIO for a quote.

Getting started with MemKV

  1. Contact MinIO for access

    Reach out to MinIO's sales team to request a quote and obtain access to MemKV. Since pricing is not public, you will need to discuss your infrastructure requirements and receive deployment credentials and documentation.

  2. Deploy on BlueField-4 STX

    Set up MemKV on NVIDIA BlueField-4 STX architecture with 800 GbE RDMA networking. Follow MinIO's deployment guide to install the software on your flash storage nodes and configure RDMA interfaces for low-latency communication.

  3. Integrate with NVIDIA Dynamo

    Connect MemKV to your inference pipeline using native NVIDIA Dynamo and NIXL integration. Configure the data paths so that GPUs can directly access the context store over RDMA without CPU involvement, enabling microsecond retrieval.

  4. Store KV context from inference

    Modify your AI inference application to write key-value context data to MemKV during token generation. Use the provided client library to persist KV state, ensuring that context is shared across GPU clusters and survives failures.

  5. Monitor GPU utilization gains

    After deployment, monitor GPU utilization metrics to verify the reduction in recompute overhead. Compare time-to-first-token and throughput against your previous setup to confirm that MemKV is delivering the expected performance improvements.

Frequently Asked Questions

What is MemKV and what is it used for?

MemKV is a context memory store for AI inference workloads, developed by MinIO. It stores key-value state persistently across GPU clusters at petabyte scale, reducing the need for GPUs to recompute context and improving time-to-first-token for large language models.

How does MemKV improve GPU utilization for AI inference?

MemKV offloads context storage to flash-based RDMA-accessible storage, eliminating the recompute tax. According to MinIO, this increases GPU utilization from about 50% to over 90% by keeping GPUs busy generating tokens instead of recomputing key-value state for each request.

How does MemKV compare to Redis for AI inference caching?

MemKV is purpose-built for petabyte-scale AI inference context storage, using flash and RDMA for microsecond retrieval. Redis relies on DRAM and is limited by capacity and single-threaded architecture, making it cost-prohibitive at scale. MemKV offers persistent shared context at lower cost.

What hardware does MemKV require to run?

MemKV requires NVIDIA BlueField-4 STX architecture and 800 GbE RDMA networking. This specialized hardware enables direct data paths between GPUs and the context store, but may not be available in existing data centers, limiting deployment options for some organizations.

Can MemKV replace a general-purpose cache like Redis for other workloads?

No, MemKV is purpose-built for AI inference context storage and cannot replace Redis for workloads like session management or rate limiting without significant architectural changes. It is optimized for persistent shared context in agentic AI systems, not general caching.

What is the pricing for MemKV and how can I get it?

Pricing for MemKV is not publicly disclosed. Interested organizations must contact MinIO directly for a quote. This makes it difficult to compare total cost of ownership against alternatives like Redis or Dragonfly without engaging with the vendor.

Alternatives

How MemKV compares

Direct head-to-head against 3 competitors. Picked by 7wData.

This tool

MemKV

Pricing
Not publicly disclosed; interested parties must contact MinIO for a quote.
Target
MemKV is a purpose-built context memory store for AI inference workloads, developed by MinIO.
Strength
MemKV improves time-to-first-token (TTFT) at production concurrency by delivering microsecond context retrieval at petabyte scale.
Watch for
MemKV requires specialized hardware, including NVIDIA BlueField-4 STX architecture and 800 GbE RDMA networking, which may not be available in existing infrastructure.

Valkey

Pricing
Free open-source (BSD-3)
Target
Redis drop-in replacement
Deployment
Ubuntu/ARM64
Strength
Multithreaded Redis fork
Watch for
New project, smaller ecosystem

KeyDB

Pricing
Free open-source
Target
High-throughput Redis workloads
Deployment
Docker/K8s
Strength
6x Redis throughput per node
Watch for
Requires tuning for scale

MinIO MemKV

Pricing
Contact sales
Target
GPU cluster context memory
Deployment
NVIDIA BlueField-4
Strength
Petabyte-scale RDMA access
Watch for
Early adoption phase

User reviews

No user reviews yet. Be the first to write one.

Sources

Reporting on this tool draws on these publicly available sources.

  1. www.storagereview.com
  2. www.min.io
  3. www.hpcwire.com
  4. www.linkedin.com