Toshi Search
By Toshi Search
Toshi is an open-source, Rust-based full-text search engine derived from Tantivy, designed for developers needing high-performance indexing and query capabilities.
Publisher review
Toshi is an open-source, Rust-based full-text search engine derived from Tantivy, designed for developers needing high-performance indexing and query capabilities. It targets technical users who require a memory-safe, native-speed search solution for large datasets, with benchmarks showing it can ingest and index 5 million documents in 80-90 seconds on an Intel 8700K processor with 32GB DDR4 RAM (15GB allocated to writers). The engine supports HTTP POST queries, returns scored results with document details, and offers experimental clustering via the `-e` flag. However, its maintainers explicitly warn against production use due to its experimental status and incomplete feature set.
Configuration occurs through command-line flags, including host/port specification (`-h`, `-p`), data path selection (`-d`), log level control (`-l`), and Consul integration for cluster coordination. The Rust foundation provides memory safety guarantees and native performance, while Tantivy's core indexing logic enables efficient full-text operations. Toshi builds via `cargo build --release` and includes an experimental mode (`-x`) for testing unreleased features. Its HTTP API returns JSON-formatted results with relevance scores, document IDs, and stored field values.
Compared to alternatives like Elasticsearch or Solr, Toshi occupies a niche as a lightweight, single-binary option with Rust's compile-time safety but lacks enterprise features like distributed sharding, replication, or a query DSL. The project competes more directly with other Tantivy-based tools than commercial search platforms, prioritizing raw speed over ecosystem maturity. Community reports indicate the codebase is actively developed but lacks production-grade stability guarantees or documentation breadth.
Significant trade-offs include the absence of authentication, limited query syntax, and no built-in high availability—features standard in established competitors. Performance comes at the cost of operational complexity: clustering requires manual Consul setup, and index management lacks GUI tools. The experimental label reflects missing functionality like schema migrations, though Rust's thread safety prevents whole classes of concurrency bugs common in C++ search engines.
How it works
-
Rust-based engine
Built in Rust for native-speed indexing/querying and memory safety, inheriting Tantivy's core search algorithms.
-
Bulk ingest performance
Indexes 5M documents in 80-90 sec on an 8700K CPU with 32GB RAM (15GB to writers).
-
HTTP query API
Accepts POST requests returning JSON with scores, doc IDs, and stored fields.
-
Experimental clustering
Enables distributed mode via `-e` flag with Consul coordination (manual setup required).
-
Configurable logging
Supports log level control (`-l`) and output customization for debugging.
-
CLI configuration
Sets host/port (`-h`/`-p`), data path (`-d`), and cluster name via command line.
-
Experimental features
Unstable capabilities accessible via `-x` flag for early testing.
Strengths and trade-offs
Strengths
- Processes 5 million documents in under 90 seconds on mid-range hardware, demonstrating strong ingest performance.
- Rust implementation eliminates memory safety vulnerabilities common in C/C++ search engines.
- HTTP API returns detailed JSON results with relevance scoring and document metadata.
- Supports clustering for horizontal scaling when paired with Consul service discovery.
Trade-offs
- Maintainers explicitly warn against production use due to unstable experimental status.
- Lacks authentication, query DSL, and schema migration tools expected in mature search platforms.
- Clustering requires manual Consul configuration rather than built-in node management.
- No graphical interface for index inspection or query debugging compared to Kibana/Solr admin.
Pricing context
Free and open-source (MIT licensed); no commercial tiers or paid support options documented.
Getting started with Toshi Search
-
Install Rust
Install Rust using rustup to ensure cargo is available. This provides the toolchain required to build Toshi from source.
-
Clone repository
Clone the Toshi GitHub repository to your local machine. This provides access to the source code and build scripts.
-
Build release binary
Run `cargo build --release` in the repository root. This compiles an optimized binary with full search capabilities.
-
Configure data path
Launch Toshi with `-d` flag specifying a directory for index storage. This persists search data between sessions.
-
Query via HTTP
Send POST requests to the running instance's HTTP endpoint. Include search terms in the request body to retrieve scored JSON results.
Frequently Asked Questions
What is Toshi Search?
Toshi is an open-source, Rust-based full-text search engine derived from Tantivy. It offers high-performance indexing and query capabilities, targeting developers needing memory-safe, native-speed solutions for large datasets. However, it’s experimental and not recommended for production use due to incomplete features.
How fast is Toshi Search?
Toshi indexes 5 million documents in 80-90 seconds on an Intel 8700K CPU with 32GB RAM, allocating 15GB to writers. This benchmark highlights its strong ingest performance, making it suitable for developers prioritizing raw speed over enterprise-grade features.
What are Toshi Search’s key features?
Toshi offers Rust-based memory safety, HTTP POST query API with JSON results, experimental clustering via Consul, and CLI configuration for host/port, data path, and logging. It also includes experimental features accessible via the `-x` flag for testing unreleased capabilities.
How does Toshi compare to Elasticsearch?
Toshi is a lightweight, single-binary search engine prioritizing raw speed and memory safety but lacks enterprise features like distributed sharding, replication, or a query DSL. It competes more directly with Tantivy-based tools than mature platforms like Elasticsearch.
What are Toshi Search’s limitations?
Toshi lacks authentication, query DSL, schema migrations, and built-in high availability. Clustering requires manual Consul setup, and there’s no GUI for index management. Its maintainers explicitly warn against production use due to its experimental status and incomplete feature set.
How do you configure Toshi Search?
Toshi is configured via CLI flags, including host/port (`-h`, `-p`), data path (`-d`), log level (`-l`), and Consul integration for clustering. It builds using `cargo build --release` and supports experimental features with the `-x` flag for testing.
Alternatives
How Toshi Search compares
Direct head-to-head against 2 competitors. Picked by 7wData.
Toshi Search
- Pricing
- Free and open-source (MIT licensed); no commercial tiers or paid support options documented.
- Target
- Toshi is an open-source, Rust-based full-text search engine derived from Tantivy, designed for developers needing high-performance indexing and query capabilities.
- Strength
- Processes 5 million documents in under 90 seconds on mid-range hardware, demonstrating strong ingest performance.
- Watch for
- Maintainers explicitly warn against production use due to unstable experimental status.
Sonic
- Pricing
- Open-source (MIT)
- Target
- Lightweight search for developers
- Deployment
- Single binary
- Strength
- SQL-like protocol for queries
- Watch for
- Limited feature set vs Elasticsearch
Tantivy
- Pricing
- Open-source (MIT)
- Target
- Rust-native full-text search
- Deployment
- Embeddable library
- Strength
- Lucene-like indexing in Rust
- Watch for
- No REST API out of box
User reviews
No user reviews yet. Be the first to write one.
Sources
Reporting on this tool draws on these publicly available sources.