Nessie
Nessie is an open-source, Git-inspired transactional catalog for data lakes, designed to bring version control principles to data management.
Publisher review
Nessie is an open-source, Git-inspired transactional catalog for data lakes, designed to bring version control principles to data management. It is built for data engineers and data teams who need to manage data evolution with the same rigor as code, enabling concurrent work on different data engineering tasks—such as feature development, data cleaning, and transformation—without interfering with each other. Nessie integrates with popular data processing tools like Apache Iceberg, Hive, Spark, Dremio, and Trino, and can be deployed as a Docker image or on Kubernetes, making it suitable for both small teams and large-scale production environments.
At its core, Nessie provides a Git-like interface for data catalogs, allowing users to create branches, make commits, and merge changes. Each commit represents a snapshot of the data at a specific point in time, enabling precise tracking of data evolution. Branches let multiple engineers work on separate tasks concurrently while maintaining a single source of truth on the main branch. Merges allow changes from one branch to be integrated into another, similar to code merges in Git. Nessie also supports cross-table transactions and visibility, meaning changes across multiple tables can be committed atomically, ensuring consistency. Rollback capabilities let users revert to any previous commit, ensuring data quality and consistency.
In the open-source data catalog space, Nessie competes with Apache Polaris and Unity Catalog (Databricks). Nessie differentiates itself by focusing on version control as a first-class feature, rather than just metadata management. Unlike Polaris, which emphasizes multi-engine support and governance, Nessie provides explicit branching and merging for data, which is particularly valuable for teams that need to test and deploy changes to production without risk. However, Nessie is tightly coupled with Apache Iceberg tables, which limits its use with other table formats like Delta Lake or Apache Hudi. The Reddit community notes that Nessie is often chosen for its versioning capabilities, while Polaris is preferred for its broader catalog features.
The primary trade-off with Nessie is its dependency on Apache Iceberg, meaning teams using Delta Lake or Hudi cannot adopt it without migrating their table format. Additionally, Nessie adds operational complexity—it requires running a separate catalog service (as a Docker container or on Kubernetes), which must be maintained alongside the data lake. While Nessie excels at versioning, it does not include built-in data quality or governance features beyond version tracking, so teams must layer those on separately. Performance can also be a concern: because Nessie stores commit history and metadata, large numbers of commits or branches may impact query planning time, especially compared to simpler catalogs that do not maintain full version history.
How it works
-
Git-like version control
Provides branches, commits, and merges for data catalogs, enabling concurrent work on data engineering tasks such as feature development and data cleaning.
-
Cross-table transactions
Supports atomic commits across multiple tables, ensuring consistency when changes span several datasets in a single operation.
-
Open data lake integration
Works with Hive, Spark, Dremio, Trino, and other engines, allowing teams to use their existing processing tools without vendor lock-in.
-
Apache Iceberg compatibility
Natively works with Apache Iceberg tables, leveraging Iceberg's snapshot isolation and schema evolution for versioned data management.
-
Docker and Kubernetes deployment
Can run as a Docker container or on Kubernetes, making it easy to deploy in containerized environments and scale horizontally.
-
Rollback capabilities
Allows reverting to any previous commit, enabling teams to recover from errors and maintain data quality without manual restoration.
-
Multi-branch collaboration
Enables multiple engineers to work on separate branches simultaneously, merging changes into a main branch for production deployment.
Strengths and trade-offs
Strengths
- Enables data engineers to work on different tasks concurrently using separate branches, reducing conflicts and accelerating development cycles.
- Allows committing changes to track data evolution over time, providing a full audit trail of who changed what and when.
- Facilitates branching and merging for testing and deploying changes, similar to Git workflows, which reduces risk when promoting changes to production.
- Supports a wide array of storage backends, including MinIO and other S3-compatible object stores, giving teams flexibility in infrastructure choices.
Trade-offs
- Requires Apache Iceberg tables, meaning teams using Delta Lake or Apache Hudi must migrate their table format before adopting Nessie.
- Adds operational overhead by requiring a separate catalog service to run as a Docker container or on Kubernetes, which must be maintained and monitored.
- Does not include built-in data quality or governance features beyond version tracking, so teams must integrate additional tools for those capabilities.
- Large numbers of commits or branches may impact query planning performance, as Nessie must traverse commit history to resolve table snapshots.
Pricing context
Nessie is open-source and free to use. Dremio offers a managed version as part of its Dremio Cloud service, with pricing based on compute and storage usage. No separate Nessie-specific pricing tiers are published.
Getting started with Nessie
-
Deploy Nessie as Docker container
Pull the Nessie Docker image from Docker Hub and run it as a container. Map port 19120 to access the REST API. Verify the service is running by checking the health endpoint at http://localhost:19120/api/v1/config.
-
Configure Iceberg catalog with Nessie
In your Spark or Trino session, set the catalog type to 'NESSIE' and provide the Nessie URI (e.g., http://localhost:19120/api/v1). Specify the warehouse location in your object store, such as an S3 bucket or MinIO path, to store Iceberg tables.
-
Create a branch for development
Use the Nessie CLI or REST API to create a new branch from the main branch. For example, run 'nessie branch dev-branch main'. This isolates your changes from the production data until you are ready to merge.
-
Commit changes to a table
Write data to an Iceberg table using Spark or another engine while on your development branch. After the write, commit the changes to Nessie by calling the commit API or using the CLI with 'nessie commit -m "description"'. This creates a snapshot of the table state.
-
Merge branch into main
Switch to the main branch and merge your development branch using the Nessie CLI: 'nessie merge dev-branch main'. Resolve any conflicts if they arise. Verify the merge by querying the table on the main branch to confirm the changes are visible.
Frequently Asked Questions
What is Nessie and how does it work for data lakes?
Nessie is an open-source, Git-inspired transactional catalog for data lakes. It brings version control to data management, letting users create branches, make commits, and merge changes. Each commit captures a data snapshot, enabling concurrent work and precise tracking of data evolution over time.
Does Nessie work with Apache Iceberg or other table formats?
Nessie is tightly coupled with Apache Iceberg tables, leveraging Iceberg's snapshot isolation and schema evolution. Teams using Delta Lake or Apache Hudi must migrate their table format before adopting Nessie, as it does not natively support those formats.
How does Nessie compare to Apache Polaris and Unity Catalog?
Nessie differentiates by focusing on version control as a first-class feature, offering explicit branching and merging for data. Polaris emphasizes multi-engine support and governance, while Unity Catalog is Databricks' proprietary solution. Nessie is often chosen for its versioning capabilities, while Polaris is preferred for broader catalog features.
What are the main weaknesses and limitations of Nessie?
Nessie requires Apache Iceberg tables, adding a migration hurdle for teams using other formats. It also adds operational complexity by needing a separate catalog service on Docker or Kubernetes. It lacks built-in data quality or governance features beyond version tracking, and large commit histories may slow query planning.
How do you deploy Nessie in a production environment?
Nessie can be deployed as a Docker container or on Kubernetes, making it suitable for containerized environments and horizontal scaling. It integrates with storage backends like MinIO and S3-compatible object stores, and works with processing tools such as Spark, Hive, Dremio, and Trino.
Can Nessie handle cross-table transactions and rollbacks?
Yes, Nessie supports atomic commits across multiple tables, ensuring consistency when changes span several datasets. It also allows rollback to any previous commit, enabling teams to recover from errors and maintain data quality without manual restoration.
Alternatives
How Nessie compares
Direct head-to-head against 3 competitors. Picked by 7wData.
Nessie
- Pricing
- Nessie is open-source and free to use. Dremio offers a managed version as part of its Dremio Cloud service, with pricing based on compute and storage usage. No separate Nessie-specific pricing tiers are published.
- Target
- Nessie is an open-source, Git-inspired transactional catalog for data lakes, designed to bring version control principles to data management.
- Strength
- Enables data engineers to work on different tasks concurrently using separate branches, reducing conflicts and accelerating development cycles.
- Watch for
- Requires Apache Iceberg tables, meaning teams using Delta Lake or Apache Hudi must migrate their table format before adopting Nessie.
Unity Catalog
- Pricing
- Custom/Contact sales
- Target
- Enterprise data lakehouse users
- Deployment
- Cloud
- Strength
- Deep integration with Databricks ecosystem
- Watch for
- Vendor lock-in with Databricks
Apache Polaris
- Pricing
- Open source
- Target
- Open source adopters
- Deployment
- On-prem, cloud
- Strength
- Community-driven, vendor-neutral
- Watch for
- Limited enterprise support
Project Nessie
- Pricing
- Open source
- Target
- Git-like versioning workflows
- Deployment
- On-prem, cloud
- Strength
- Branching and merging capabilities
- Watch for
- Emerging ecosystem, limited adoption
User reviews
No user reviews yet. Be the first to write one.
Sources
Reporting on this tool draws on these publicly available sources.