NannyML Core
NannyML Core is an open-source Python library for evaluating, testing, and monitoring machine learning and large language model systems in production.
Publisher review
NannyML Core is an open-source Python library for evaluating, testing, and monitoring machine learning and large language model systems in production. It is designed for data scientists and ML engineers who need to detect data drift, target drift, and prediction drift across tabular, text, and multi-modal data. The library provides interactive dashboards and can estimate model performance even when ground truth is delayed or unavailable, making it suitable for batch monitoring workflows.
NannyML Core integrates with MLOps tools like MLflow, Grafana, Prometheus, and FastAPI, and it converts reports into pass/fail test suites for CI/CD pipelines. The project is maintained by NannyML Inc., which also offers a commercial cloud version with enhanced security controls.
How it works
-
Data drift detection
Identifies shifts in feature distributions between reference and current datasets using univariate and multivariate statistical tests.
-
Target drift monitoring
Tracks changes in target variables over time, even when ground truth is delayed or unavailable, to detect concept drift.
-
Prediction drift monitoring
Monitors shifts in model output distributions to catch performance degradation before it impacts business outcomes.
-
Multi-modal data support
Supports tabular, text, and multi-modal data types, enabling monitoring across diverse ML and LLM applications.
-
Performance estimation without ground truth
Estimates model accuracy and other metrics using confidence-based methods when true labels are not immediately available.
-
CI/CD test suites
Converts monitoring reports into pass/fail test suites that can be integrated into automated CI/CD pipelines for regression testing.
-
Root cause analysis
Provides root cause analysis for detected drifts, helping engineers trace issues back to specific features or data segments.
Strengths and trade-offs
Strengths
- NannyML Core can estimate model performance without ground truth, a capability that distinguishes it from many open-source monitoring tools that require true labels.
- It supports both univariate and multivariate drift detection, giving users flexibility to catch complex interactions between features.
- The library generates interactive dashboards and HTML reports that can be embedded in Jupyter notebooks or Streamlit apps for exploratory analysis.
- NannyML Core integrates with MLflow, Grafana, Prometheus, and FastAPI, allowing teams to plug it into existing MLOps stacks without major rework.
Trade-offs
- NannyML Core lacks built-in support for image or audio data types, limiting its use for computer vision or speech models.
- The open-source version does not include a managed UI or alerting system; users must build their own alerting layer on top of Prometheus or Grafana.
- Documentation for advanced features like root cause analysis and multi-modal monitoring is less detailed than for basic drift detection.
- The free tier of NannyML Cloud caps at 2 models and 10 million predictions per month, which may be restrictive for teams with many models or high traffic.
Pricing context
NannyML Core is open-source (free). NannyML Cloud offers three paid tiers: Starter at $399/month (2 models, 10M predictions, email support), Scale at $999/month (6 models, private Slack support), and Enterprise (unlimited models, 24/7 support, contact for pricing). All tiers include a 30-day free trial.
Getting started with NannyML Core
-
Install NannyML Core
Install the open-source Python library using pip by running `pip install nannyml` in your terminal. Ensure your Python environment is version 3.8 or higher to avoid compatibility issues.
-
Load your reference data
Load your reference dataset (e.g., training data) into a pandas DataFrame. This dataset serves as the baseline for drift detection. Use `nml.load()` or read from CSV files to prepare the data.
-
Configure drift detection
Set up a drift detector by choosing the analysis type, such as data drift or target drift. Use `nml.UnivariateDriftCalculator` or `nml.MultivariateDriftCalculator` and specify the features and thresholds for statistical tests.
-
Run monitoring on new data
Pass your current production data (as a pandas DataFrame) to the fitted calculator using the `.calculate()` method. This compares the new data against the reference and outputs drift metrics and alerts.
-
Export results for CI/CD
Convert the monitoring results into a pass/fail test suite using `nml.suites`. Export the suite as a JSON file or integrate it into your CI/CD pipeline to automatically flag regressions.
Frequently Asked Questions
What is NannyML Core and what does it do?
NannyML Core is an open-source Python library for evaluating, testing, and monitoring machine learning and large language model systems in production. It detects data drift, target drift, and prediction drift across tabular, text, and multi-modal data.
How does NannyML Core detect drift in production ML models?
NannyML Core identifies shifts in feature distributions using univariate and multivariate statistical tests. It also tracks changes in target variables and model output distributions, even when ground truth is delayed or unavailable, to catch performance degradation.
Can NannyML Core estimate model performance without ground truth?
Yes, NannyML Core estimates model accuracy and other metrics using confidence-based methods when true labels are not immediately available. This capability distinguishes it from many open-source monitoring tools that require true labels for performance evaluation.
What integrations does NannyML Core support for MLOps workflows?
NannyML Core integrates with MLflow, Grafana, Prometheus, and FastAPI, allowing teams to plug it into existing MLOps stacks. It also converts monitoring reports into pass/fail test suites for automated CI/CD pipelines, enabling regression testing.
What are the limitations of NannyML Core's open-source version?
NannyML Core lacks built-in support for image or audio data types, limiting use for computer vision or speech models. The open-source version does not include a managed UI or alerting system; users must build their own alerting layer on top of Prometheus or Grafana.
What is the pricing for NannyML Cloud and what does each tier include?
NannyML Cloud offers three paid tiers: Starter at $399/month for 2 models and 10M predictions, Scale at $999/month for 6 models with private Slack support, and Enterprise with unlimited models and 24/7 support. All tiers include a 30-day free trial.
Alternatives
How NannyML Core compares
Direct head-to-head against 2 competitors. Picked by 7wData.
NannyML Core
- Pricing
- NannyML Core is open-source (free). NannyML Cloud offers three paid tiers: Starter at $399/month (2 models, 10M predictions, email support), Scale at $999/month (6 models, private Slack support), and Enterprise (unlimited models, 24/7 support, contact for pricing). All tiers include a 30-day free trial.
- Target
- NannyML Core is an open-source Python library for evaluating, testing, and monitoring machine learning and large language model systems in production.
- Strength
- NannyML Core can estimate model performance without ground truth, a capability that distinguishes it from many open-source monitoring tools that require true labels.
- Watch for
- NannyML Core lacks built-in support for image or audio data types, limiting its use for computer vision or speech models.
Evidently AI
- Pricing
- Open-source (free), Pro $500/month, Enterprise custom
- Target
- Teams needing lightweight drift detection
- Deployment
- Python library
- Strength
- 100+ built-in metrics for tabular/text data
- Watch for
- Limited enterprise support in OSS version
Aporia
- Pricing
- Custom pricing (contact sales)
- Target
- Enterprise ML monitoring
- Deployment
- SaaS or on-prem
- Strength
- Real-time alerting and root cause analysis
- Watch for
- Higher cost for small teams
User reviews
No user reviews yet. Be the first to write one.
Sources
Reporting on this tool draws on these publicly available sources.