BigScience’s AI language model is finally available
After more than a year of planning and training, a volunteer-led project has produced an open source language model that they claim is as powerful as OpenAI’s GPT-3, but free and open for anyone to use (if they have the computing power). Dubbed Bloom, the model is available in open source along with the code and datasets used to create it. Brooklyn-based AI startup Hugging Face has released a free web app that lets anyone try Bloom without having to download it.
Bloom is the brainchild of BigScience, an international, community-powered project with the goal of making large natural language models widely available for research. Large language models, or “LLMs” for short, can translate, summarize and write text with humanlike nuance — more or less. (See GPT-3.) But they’ve been historically costly to create, keeping them out of reach of researchers and firmly within the hands of Big Tech companies like Meta, Google and Microsoft.
That’s finally changing, thanks in part to the efforts of BigScience. The group’s more than 1,000 volunteer researchers — supported by ethicists, philosophers, legal scholars and engineers from startups and large tech companies alike — spent months working toward Bloom, which rivals in scale LLMs made by firms like OpenAI and Alphabet’s DeepMind. One of the largest open source models to work across multiple languages, Bloom is designed to be applied in a range of research applications, such as extracting information from historical texts.
“Bloom is able to generate text in 46 natural languages and dialects and 13 programming languages,” reads a blog post shared with TechCrunch ahead of the release. “Although it was never trained on any of those specific tasks, Bloom can be asked to produce summaries or translations of text, output code from instructions, and follow prompts to perform original tasks such as writing recipes, extracting information from a news article, or composing sentences using a newly-defined invented word … Bloom’s performance will continue to improve as the workshop continues to experiment and advance on top of Bloom.”
BigScience’s backers also hope that Bloom will spur new investigations into ways to combat the problems that plague all LLMs, including bias and toxicity. LLMs have a tendency to spout falsehoods and exhibit prejudices against religions, sexes, races and people with disabilities. They also struggle with the basic tenets of writing, often changing the subject of a conversation without a segue and endlessly repeating — or even contradicting — themselves.
BigScience’s origins lie in discussions years ago between Hugging Face chief science officer Thomas Wolf, GENCI’s Stéphane Requena and IDRIS‘ Pierre-François Lavallée. The founders envisioned creating software, datasets, LLMs and tools to explore the social impact of AI, which only in recent years has received increased attention from the research community.
Soon, steering committees were formed to give members of BigScience — who hailed from more than 60 countries and 250 institutions — scientific and general advice, design collaborative tasks and organize workshops, hackathons and public events. Different working groups were charged with tackling challenges like data governance, proving theorems in mathematics and archival strategies, as well as privacy and informed consent and other legal issues.
Bloom is the sum total of their work. It was trained using $7 million worth of publicly funded (through grants) compute time on the Jean Zay supercomputer located near Paris, France, which ranks among the most powerful machines in the world.
A robust discussion is ongoing in academic circles about the carbon impact of AI training; data centers aren’t particularly environmentally friendly. But BigScience says that Jean Zay, thanks to its unique cooling system and nuclear power source, was able to train Bloom with a carbon footprint equivalent to a Paris-to-New York flight.
