whylogs
whylogs is an open-source library that produces statistical summaries of tabular data, called whylogs profiles, without copying raw data out of its environment.
Publisher review
whylogs is an open-source library that produces statistical summaries of tabular data, called whylogs profiles, without copying raw data out of its environment. Designed for data engineers and ML teams, it runs inside existing pipelines or as a sidecar container, computing metrics on the fly. The tool supports pandas DataFrames and integrates with orchestrators like Airflow and ZenML for data validation and monitoring. After WhyLabs, Inc. discontinued its hosted platform, the company open-sourced the entire platform and committed to maintaining whylogs as a standalone project, ensuring continued development for privacy-preserving logging and LLM monitoring via the companion langkit toolkit.
whylogs profiles capture distributions, missing values, counts, cardinality, and schema using streaming algorithms that scale with the number of features, not rows. Benchmarks show that profiling 10 GB of data (34M rows × 43 columns) on two c5a.2xlarge instances costs roughly $0.026 and completes in 2.6 minutes per instance when run in parallel. For 100 GB (290M rows × 43 columns) on 16 instances, processing takes 2.7 minutes per instance at an estimated $0.221. This approach avoids data duplication, reducing infrastructure and security overhead compared to post-processing solutions.
In the open-source data quality space, whylogs competes with Great Expectations, Soda, and Deequ. Unlike those tools, which focus on declarative assertions and post-hoc validation, whylogs emphasizes lightweight, real-time profiling that can be embedded directly into streaming or batch pipelines. Its small library footprint makes it easy to set up, and it is complementary to Great Expectations for teams that want both profiling and expectation-based testing. The project is community-backed and continues to evolve as an open standard for data logging.
Trade-offs include documentation complexity that can slow initial adoption, and potential performance strain on very large datasets—though overhead typically stays under 1% for most use cases, it can reach ~5% at 10M+ queries per second with thousands of features. The discontinuation of the WhyLabs hosted platform means users now rely entirely on self-managed infrastructure or community support. Additionally, whylogs currently focuses on tabular data, limiting its applicability to unstructured or non-tabular data sources without custom extensions.
How it works
-
Data Profiling
Automatically generates statistical summaries (distributions, missing values, counts, cardinality, schema) from tabular data using streaming algorithms.
-
Data Validation
Validates data against predefined or custom expectations, enabling automated corrective actions in pipelines.
-
Data Docs
Generates human-readable HTML documentation of expectations and validation results, similar to Great Expectations Data Docs.
-
Cost-Effective
Open-source and free to use; profiling 10 GB costs ~$0.026 on AWS c5a.2xlarge instances, with estimated $1.60–$2.45 per TB.
-
Community-Backed
Supported by an active open-source community and maintained by the former WhyLabs team after the company's discontinuation.
-
Integration
Integrates with ETL tools like Airflow and ML platforms like ZenML for data validation and monitoring in production pipelines.
-
Privacy-Preserving Logging
Computes statistics in the customer environment without duplicating raw data, supporting privacy-preserving monitoring for AI and LLMs.
Strengths and trade-offs
Strengths
- Small library footprint makes it easy to set up and deploy in existing pipelines without significant overhead.
- Profiling 10 GB of data (34M rows × 43 columns) costs approximately $0.026 and completes in 2.6 minutes per instance on c5a.2xlarge instances.
- Supports parallelization and scales with the number of features rather than rows, enabling efficient processing of large datasets.
- Continues to support open-source development and privacy-preserving logging for AI systems after the WhyLabs platform discontinuation.
Trade-offs
- Documentation complexity can be a challenge for new users, making initial setup and advanced usage harder than some alternatives.
- May struggle with very large data volumes; overhead can reach ~5% at 10M+ queries per second with thousands of features.
- The discontinuation of the WhyLabs hosted platform means users must self-manage infrastructure or rely on community support for monitoring and visualization.
- Primarily focused on tabular data in pandas DataFrame format, limiting applicability to unstructured or non-tabular data sources.
Pricing context
Free and open-source under the Apache 2.0 license; no paid tiers or hosted platform available after WhyLabs discontinuation.
Getting started with whylogs
-
Install whylogs library
Run `pip install whylogs` in your Python environment. This installs the open-source library and its dependencies, enabling you to generate statistical profiles from tabular data without copying raw data.
-
Load data into DataFrame
Import pandas and load your tabular data into a DataFrame using `pd.read_csv()` or similar methods. whylogs works directly with pandas DataFrames, so ensure your data is in a compatible format.
-
Create a whylogs profile
Use `import whylogs as why` and call `profile = why.log(df)` to generate a statistical summary. This captures distributions, missing values, counts, and cardinality using streaming algorithms that scale with features.
-
Validate data with expectations
Define expectations using `why.expectation_builder()` or custom conditions, then run `profile.validate()` to check your data against these rules. This enables automated corrective actions in your pipeline.
-
Integrate into pipeline
Embed whylogs in your Airflow DAG or ZenML step by calling the profiling function on each batch. Schedule the validation to run after data ingestion, ensuring continuous monitoring without manual intervention.
Frequently Asked Questions
What is whylogs and what does it do?
whylogs is an open-source library that generates statistical summaries of tabular data, called profiles, without copying raw data. It captures distributions, missing values, counts, cardinality, and schema using streaming algorithms, and integrates with tools like Airflow for data validation.
How does whylogs compare to Great Expectations?
whylogs focuses on lightweight, real-time profiling embedded in pipelines, while Great Expectations emphasizes declarative assertions and post-hoc validation. whylogs is complementary to Great Expectations, allowing teams to combine profiling with expectation-based testing for comprehensive data quality monitoring.
What are the costs of using whylogs for data profiling?
whylogs is free and open-source under Apache 2.0. Profiling 10 GB of data on two c5a.2xlarge instances costs about $0.026 and takes 2.6 minutes per instance. For 100 GB on 16 instances, it costs roughly $0.221 with similar per-instance time.
Can whylogs be integrated into existing data pipelines?
Yes, whylogs integrates with ETL tools like Airflow and ML platforms like ZenML. It runs inside existing pipelines or as a sidecar container, computing metrics on the fly. It supports pandas DataFrames and works with streaming or batch processing for real-time monitoring.
What are the limitations of whylogs?
whylogs primarily focuses on tabular data, limiting its use for unstructured data. Documentation complexity can slow adoption, and overhead may reach 5% at high query volumes. The discontinuation of the WhyLabs hosted platform means users must self-manage infrastructure or rely on community support.
How does whylogs ensure privacy-preserving logging?
whylogs computes statistical summaries in the customer environment without duplicating raw data, supporting privacy-preserving monitoring for AI and LLMs. This approach reduces infrastructure and security overhead by avoiding data duplication and keeping sensitive data within its original environment.
Alternatives
- Evidently AI ↗
- Aporia ↗
- Fiddler AI ↗
How whylogs compares
Direct head-to-head against 3 competitors. Picked by 7wData.
whylogs
- Pricing
- Free and open-source under the Apache 2.0 license; no paid tiers or hosted platform available after WhyLabs discontinuation.
- Target
- whylogs is an open-source library that produces statistical summaries of tabular data, called whylogs profiles, without copying raw data out of its environment.
- Strength
- Small library footprint makes it easy to set up and deploy in existing pipelines without significant overhead.
- Watch for
- Documentation complexity can be a challenge for new users, making initial setup and advanced usage harder than some alternatives.
Evidently AI
- Pricing
- Open source core, paid cloud from $500/month
- Target
- ML teams needing drift detection
- Deployment
- Python, Docker
- Strength
- Interactive drift visualization
- Watch for
- Limited JVM support
Aporia
- Pricing
- Custom/Contact sales
- Target
- Enterprise ML monitoring
- Deployment
- SaaS, VPC
- Strength
- End-to-end model guardrails
- Watch for
- Requires data egress to cloud
Fiddler AI
- Pricing
- Custom/Contact sales
- Target
- Large-scale model monitoring
- Deployment
- SaaS, on-prem
- Strength
- Explainability integrations
- Watch for
- Minimum $50k annual commitment
User reviews
No user reviews yet. Be the first to write one.
Sources
Reporting on this tool draws on these publicly available sources.