Dolma
Dolma is an open dataset of 3 trillion tokens and a companion toolkit for curating large-scale language model training corpora, created by the Allen Institute for AI (AI2) to train their OLMo autoregressive language model.
Publisher review
Dolma is an open dataset of 3 trillion tokens and a companion toolkit for curating large-scale language model training corpora, created by the Allen Institute for AI (AI2) to train their OLMo autoregressive language model. It is designed for researchers and engineers who need a transparent, reproducible foundation for pretraining or fine-tuning large language models, offering both a ready-to-use corpus and the means to build custom datasets from scratch. The dataset is available for download on the HuggingFace Hub under the ODC-BY license, which permits sharing and adaptation with attribution, and the toolkit is open-source on GitHub. Dolma's primary audience includes academic labs, independent researchers, and organizations that prioritize data provenance and control over proprietary, black-box alternatives.
The Dolma Toolkit enables high-performance curation of datasets for (pre)-training ML models, with features including fast deduplication, built-in taggers, extensibility via plugins, and cloud support for S3 and local paths. It processes data in Rust for speed, supports streaming from remote sources, and includes tokenizers that can handle custom fields and BOS/EOS tokens. The toolkit is portable across environments, from a single machine to distributed clusters, and is designed to handle the scale of trillions of tokens without requiring specialized infrastructure. The dataset itself is a mixture of Common Crawl, Refined Web, StarCoder, C4, Reddit, Semantic Scholar, arXiv, StackExchange, Flan, and CC News, providing a broad base of web content, academic publications, code, books, and encyclopedic materials.
In the landscape of open language model datasets, Dolma competes directly with The Pile (EleutherAI), C4 (Google), and RedPajama (Together Computer). At 3 trillion tokens, Dolma is roughly 3x larger than The Pile (825 GB) and 1.5x larger than RedPajama-v2, offering a more comprehensive mix of sources. Unlike C4, which is derived solely from Common Crawl with heuristic filtering, Dolma includes curated subsets like Semantic Scholar and arXiv for scientific content, and StarCoder for code. Its toolkit differentiates it from static datasets by allowing users to reproduce, modify, or extend the curation pipeline, a capability not offered by The Pile or C4. However, Dolma is English-only and does not include multilingual data, a gap that RedPajama partially addresses with its multilingual subsets.
Key trade-offs: Dolma's ODC-BY license requires attribution for derivatives, which may be a compliance burden for some commercial users compared to the public domain C4. The toolkit, while powerful, has a steep learning curve for non-developers, as it requires familiarity with Rust and command-line tools. The dataset's focus on English limits its applicability for multilingual models, and its reliance on Common Crawl introduces noise from low-quality web pages despite filtering. Additionally, Dolma is tied to the OLMo project, and its curation decisions (e.g., tokenization with OLMo's tokenizer) may not be optimal for other architectures without re-processing. Despite these limitations, Dolma's combination of scale, transparency, and tooling makes it a strong choice for researchers who need to understand and control their pretraining data.
How it works
-
3 trillion token corpus
Dolma provides a ready-to-use dataset of 3 trillion tokens from web, academic, code, books, and encyclopedic sources for LLM pretraining.
-
High-performance curation toolkit
The Dolma Toolkit is written in Rust for fast deduplication, tagging, and filtering, supporting streaming from S3 and local paths.
-
Built-in taggers and filters
Includes taggers for PII detection, quality scoring, and language identification, plus rule-based and statistical filters for content extraction.
-
Extensible plugin architecture
Users can add custom taggers, tokenizers, or data sources via plugins, enabling tailored curation pipelines without modifying core code.
-
Cloud and local support
Supports reading/writing data from S3, local filesystems, and HuggingFace Hub, with configurable parallelism for distributed processing.
-
Reproducible data pipelines
All curation steps are defined in configuration files, allowing exact reproduction of the Dolma dataset or creation of custom variants.
-
ODC-BY licensed dataset
Dolma is licensed under the Open Data Commons Attribution license, permitting sharing and adaptation with attribution to AI2.
Strengths and trade-offs
Strengths
- At 3 trillion tokens, Dolma is one of the largest open datasets for LLM pretraining, offering more than 3x the tokens of The Pile.
- The Dolma Toolkit provides a fully reproducible curation pipeline, enabling researchers to verify and modify every step of dataset creation.
- Dolma includes diverse sources like Semantic Scholar, arXiv, and StarCoder, covering scientific literature and code that are absent from C4.
- The dataset is freely available on HuggingFace Hub under the ODC-BY license, with no usage fees or access restrictions beyond attribution.
Trade-offs
- Dolma is English-only, limiting its use for multilingual model training compared to datasets like RedPajama that include non-English data.
- The toolkit requires Rust and command-line expertise, creating a barrier for researchers who prefer Python-only or GUI-based tools.
- The dataset's reliance on Common Crawl introduces noise from low-quality web pages, which may require additional filtering for downstream tasks.
- Dolma's curation decisions (e.g., tokenization with OLMo's tokenizer) may not be optimal for other model architectures without re-processing.
Pricing context
Free and open-source. The dataset is available under the ODC-BY license with no monetary cost. The toolkit is MIT-licensed on GitHub.
Getting started with Dolma
-
Install the Dolma Toolkit
Clone the Dolma repository from GitHub and install Rust if not already present. Run `cargo build --release` to compile the toolkit. Verify the installation by running `dolma --help` to see available commands.
-
Download the Dolma dataset
Access the Dolma dataset on HuggingFace Hub under the AI2 organization. Use `git lfs` or the HuggingFace CLI to download the dataset files to your local machine or cloud storage. Ensure you have sufficient disk space for 3 trillion tokens.
-
Configure a curation pipeline
Create a YAML configuration file specifying input sources, output paths, and processing steps. Define taggers for quality scoring, language identification, and deduplication. Set tokenizer parameters to match your model architecture, adjusting BOS/EOS tokens as needed.
-
Run deduplication on a sample
Execute `dolma dedup --config config.yaml` on a small subset of the dataset to test performance. Monitor memory and CPU usage. Adjust parallelism settings in the config to optimize throughput on your hardware.
-
Schedule regular dataset updates
Write a cron job or CI pipeline that periodically pulls the latest Dolma dataset from HuggingFace Hub and re-runs your curation pipeline. Store output in S3 or local paths with versioned directories to track changes over time.
Frequently Asked Questions
What is the Dolma dataset and who created it?
Dolma is an open dataset of 3 trillion tokens created by the Allen Institute for AI (AI2) to train their OLMo language model. It includes web content, academic papers, code, and books, designed for researchers needing transparent pretraining data.
How does Dolma compare to other open datasets like The Pile or C4?
At 3 trillion tokens, Dolma is about 3 times larger than The Pile and 1.5 times larger than RedPajama-v2. It includes curated scientific and code sources absent from C4, but is English-only, unlike RedPajama's multilingual subsets.
What is the Dolma Toolkit and what can it do?
The Dolma Toolkit is a high-performance curation tool written in Rust for fast deduplication, tagging, and filtering. It supports streaming from S3 and local paths, has a plugin architecture, and enables reproducible data pipelines for LLM pretraining.
Is the Dolma dataset free to use and what license does it have?
Yes, Dolma is free and open-source under the ODC-BY license, which allows sharing and adaptation with attribution to AI2. The toolkit is MIT-licensed on GitHub. No monetary cost is required for access or use.
What are the main limitations of the Dolma dataset?
Dolma is English-only, limiting its use for multilingual models. The toolkit requires Rust and command-line expertise, which may be a barrier. Its reliance on Common Crawl introduces noise, and tokenization is optimized for OLMo, not other architectures.
How can I download or access the Dolma dataset?
The Dolma dataset is available for download on the HuggingFace Hub under the ODC-BY license. The companion toolkit is open-source on GitHub. You can also use the toolkit to build custom datasets from scratch using the provided curation pipeline.
Alternatives
How Dolma compares
Direct head-to-head against 2 competitors. Picked by 7wData.
Dolma
- Pricing
- Free and open-source. The dataset is available under the ODC-BY license with no monetary cost. The toolkit is MIT-licensed on GitHub.
- Target
- Dolma is an open dataset of 3 trillion tokens and a companion toolkit for curating large-scale language model training corpora, created by the Allen Institute
- Strength
- At 3 trillion tokens, Dolma is one of the largest open datasets for LLM pretraining, offering more than 3x the tokens of The Pile.
- Watch for
- Dolma is English-only, limiting its use for multilingual model training compared to datasets like RedPajama that include non-English data.
Trader Joe's Dolmas
- Pricing
- $2.99 per can
- Target
- Budget-conscious consumers seeking shelf-stable dolmas
- Deployment
- Grocery retail
- Strength
- Gluten-free, long shelf life, affordable
- Watch for
- Limited to Trader Joe's stores
Mediterranean Specialty Brands
- Pricing
- $4-$8 per can
- Target
- Ethnic food markets, gourmet grocers
- Deployment
- Specialty retail
- Strength
- Authentic recipes, olive oil preservation
- Watch for
- Higher price point than mass-market options
User reviews
No user reviews yet. Be the first to write one.
Sources
Reporting on this tool draws on these publicly available sources.