TileDB Core
TileDB Core is an open-source array database management system designed to unify all types of data and associated code into a single solution.
Publisher review
TileDB Core is an open-source array database management system designed to unify all types of data and associated code into a single solution. It is built for organizations that manage large-scale, heterogeneous datasets—such as genomics, biomedical imaging, single-cell data, and point clouds—that cannot be efficiently modeled as tables. The platform targets data engineers, scientists, and analysts in industries like life sciences, healthcare, and geospatial analysis who need to store, query, and analyze multi-dimensional dense and sparse arrays at petabyte scale. Founded in 2017 and headquartered in Cambridge, United States, TileDB emphasizes FAIR compliance and trusted research environments, making it suitable for regulated sectors where data governance is critical.
TileDB's core technology uses a multi-dimensional array storage format that tiles data into fixed-size chunks, enabling efficient compression and parallel access. It supports organizing data into folder hierarchies with metadata, structuring arrays for various modalities, and collaborating through federated queries between namespaces and organizations. For genomics, TileDB can reduce storage costs by up to 97% compared to file-based approaches (e.g., VCF or BAM files) and run complex variant queries that once took days in hours at biobank scale. The engine also supports vector search via TileDB-Vector-Search, treating vectors as 1D arrays, and integrates with major programming languages (Python, R, Java) and cloud providers (AWS, Azure, GCP). On-premise and cloud storage are both supported, with the platform designed to separate storage and compute for scalability.
In the market, TileDB competes with traditional database systems like SQL Server, MongoDB Atlas, and Oracle AI Database, but differentiates by handling non-tabular data natively. Unlike SQL Server, which excels at relational tables, or MongoDB, which focuses on document stores, TileDB arrays can represent images, genomics variants, and time-series data without forcing them into relational or document schemas. Its competitors often require additional preprocessing or specialized tools for such data, whereas TileDB provides a single engine. However, TileDB is less mature for transactional workloads and lacks the broad ecosystem of SQL-based tools, making it a niche choice for multimodal data rather than a general-purpose database.
The honest trade-offs: TileDB's setup and implementation are complex, requiring familiarity with array concepts and configuration of storage backends. Documentation and API examples are sparse, as noted in user reviews, which can slow onboarding. The usage-based pricing model, while cost-effective for large datasets, may become unpredictable for workloads with frequent small queries. Additionally, TileDB's community is smaller than those of established databases, meaning fewer third-party integrations and community support resources. For organizations already invested in SQL ecosystems, migrating to TileDB requires significant re-engineering of data pipelines and query logic.
How it works
-
Multi-dimensional array storage
Stores dense and sparse n-dimensional arrays, tiling data into fixed-size chunks for efficient compression and parallel access across cloud and on-premise storage.
-
Genomics variant analysis
Enables biobank-scale variant queries by combining efficient array storage with distributed cloud compute, reducing query times from days to hours.
-
Federated queries across namespaces
Supports queries across different organizations and namespaces within a trusted research environment, enabling secure multi-tenant data collaboration.
-
Vector search capability
Treats vectors as 1D arrays, leveraging TileDB-Vector-Search library for similarity search and integration with generative AI and LLM workflows.
-
Multimodal data support
Handles diverse data types including genomics, biomedical imaging, single-cell data, and point clouds within a single database engine.
-
FAIR compliance and TRE
Provides features for Findable, Accessible, Interoperable, and Reusable data management, along with trusted research environment controls for regulated data.
-
Cloud and on-premise deployment
Runs on AWS, Azure, and GCP object stores as well as on-premise storage, with storage and compute separated for scalability and cost control.
Strengths and trade-offs
Strengths
- Reduces genomics storage costs by up to 97% compared to file-based formats like VCF or BAM, as claimed in the solution brief.
- Supports federated queries between namespaces and organizations within a trusted research environment, enabling secure multi-party data analysis.
- Integrates vector search natively via TileDB-Vector-Search, treating vectors as 1D arrays for generative AI applications.
- Handles petabyte-scale datasets with efficient tiling and compression, allowing complex variant queries to complete in hours instead of days.
Trade-offs
- Implementation and setup require deep understanding of array concepts and cloud storage configuration, increasing initial complexity.
- API documentation and code examples are sparse, as acknowledged by the company in its own blog posts, slowing developer onboarding.
- Usage-based pricing can become unpredictable for workloads with many small, frequent queries, potentially increasing costs.
- Smaller community and fewer third-party integrations compared to established databases like SQL Server or MongoDB, limiting ecosystem support.
Pricing context
Usage-based pricing model with pre-paid packages and enterprise plans available; specific dollar figures are not publicly listed on the website.
Getting started with TileDB Core
-
Install TileDB Core
Download the TileDB Core library from the official GitHub repository or use a package manager like pip for Python. Follow the installation guide for your operating system and verify the installation by importing the library in your code.
-
Connect to storage backend
Configure your storage backend by setting environment variables or using a configuration file. Specify the cloud provider (AWS, Azure, GCP) or local filesystem path. Provide credentials such as access keys or service account tokens to enable read and write access.
-
Define array schema
Create an array schema by specifying the domain (dimensions and their extents), attributes (data types), and tile extents. Use the TileDB API in your chosen language (Python, R, Java) to define a dense or sparse array that matches your data structure.
-
Load data into array
Write your dataset into the array by opening it in write mode and submitting data as a NumPy array or similar structure. Set the coordinates for sparse arrays or use subarray ranges for dense arrays. Close the array after writing to ensure data is committed.
-
Query array data
Open the array in read mode and perform a query by specifying a subarray or coordinate range. Use the TileDB query API to retrieve slices of data, apply filters on attributes, and iterate over results. Close the array after the query to release resources.
Frequently Asked Questions
What is TileDB Core and what is it used for?
TileDB Core is an open-source array database management system that stores and queries multi-dimensional dense and sparse arrays at petabyte scale. It unifies diverse data types like genomics, biomedical imaging, and point clouds into a single engine for data engineers and scientists.
How does TileDB Core reduce genomics storage costs?
TileDB Core uses a multi-dimensional array storage format with tiling and compression, reducing genomics storage costs by up to 97% compared to file-based formats like VCF or BAM. It also enables complex variant queries to run in hours instead of days at biobank scale.
Can TileDB Core handle vector search for AI applications?
Yes, TileDB Core supports vector search via TileDB-Vector-Search, treating vectors as 1D arrays. This enables similarity search and integration with generative AI and large language model workflows, all within the same database engine used for other data types.
What are the main differences between TileDB Core and MongoDB?
TileDB Core natively handles non-tabular data like images and genomics arrays, while MongoDB focuses on document stores. TileDB excels at multi-dimensional array storage but lacks MongoDB's broad ecosystem and maturity for transactional workloads, making it a niche choice for multimodal data.
How does TileDB Core's pricing model work?
TileDB Core uses a usage-based pricing model with pre-paid packages and enterprise plans available, though specific dollar figures are not publicly listed. This can be cost-effective for large datasets but may become unpredictable for workloads with many small, frequent queries.
What are the main challenges of implementing TileDB Core?
TileDB Core's setup requires deep understanding of array concepts and cloud storage configuration, increasing initial complexity. API documentation and code examples are sparse, slowing onboarding. Its smaller community and fewer third-party integrations also limit ecosystem support compared to established databases.
Alternatives
How TileDB Core compares
Direct head-to-head against 2 competitors. Picked by 7wData.
TileDB Core
- Pricing
- Usage-based pricing model with pre-paid packages and enterprise plans available; specific dollar figures are not publicly listed on the website.
- Target
- TileDB Core is an open-source array database management system designed to unify all types of data and associated code into a single solution.
- Strength
- Reduces genomics storage costs by up to 97% compared to file-based formats like VCF or BAM, as claimed in the solution brief.
- Watch for
- Implementation and setup require deep understanding of array concepts and cloud storage configuration, increasing initial complexity.
SQL Server
- Pricing
- Custom/Contact sales
- Target
- Enterprise environments requiring stability and transactional integrity
- Deployment
- On-premise, Cloud
- Strength
- Strong tooling, predictable performance, mature administrative capabilities
- Watch for
- Limited scalability flexibility compared to cloud-native platforms
MongoDB Atlas
- Pricing
- $57/month for shared, $0.10/hour for dedicated
- Target
- Global, distributed transactional workloads
- Deployment
- Multi-cloud
- Strength
- Managed multi-cloud architecture, built-in sharding
- Watch for
- Networking flexibility and cost predictability issues at scale
User reviews
No user reviews yet. Be the first to write one.
Sources
Reporting on this tool draws on these publicly available sources.