OctoML
OctoML, founded in 2019 by University of Washington professor Luis Ceze and former Microsoft engineer Jason Knight, originally built an automated machine-learning compiler to optimize models for diverse hardware.
Profile
OctoML provides a cloud platform for deploying and optimizing machine learning models, particularly large language models, for inference on various hardware.
OctoML, founded in 2019 by University of Washington professor Luis Ceze and former Microsoft engineer Jason Knight, originally built an automated machine-learning compiler to optimize models for diverse hardware. The company raised $85 million in total funding, including a $28 million Series B in 2021 led by Addition, and a $40 million Series C in 2022 led by Tiger Global. By 2023, OctoML shifted its business model from open-source Apache TVM consulting to a managed inference platform, OctoAI, competing with providers like Together AI and Fireworks AI.
The company has not disclosed recent revenue or headcount figures; its last publicly reported funding round was the 2022 Series C. OctoML's platform targets enterprises deploying large language models (LLMs) and generative AI, offering optimized inference on NVIDIA, AMD, and AWS Inferentia hardware. The company has faced increased competition from cloud-native inference services and open-source model optimization tools, and has not announced a new funding round since 2022.
In 2025, OctoML laid off an undisclosed number of employees as part of a restructuring to focus on its OctoAI product. The company's current market position is uncertain, as it has not publicly shared updated metrics or major customer wins since the shift to OctoAI.
Who buys this
- Enterprises deploying large language models and generative AI applications
- AI startups needing cost-efficient model inference
- Organizations using NVIDIA, AMD, or AWS Inferentia hardware
- Companies migrating from on-premise ML infrastructure to managed cloud services
Strengths and what to watch
Strengths
- Deep expertise in model optimization through Apache TVM, enabling efficient inference across multiple hardware types
- Multi-cloud and multi-hardware support, reducing vendor lock-in for customers
- Early mover in the managed inference space with a focus on cost and latency reduction
Watch for
- No disclosed funding since 2022 and no public revenue figures, raising questions about financial sustainability
- Intense competition from well-funded rivals (Together AI, Fireworks AI, Replicate) and cloud-native services from AWS, GCP, and Azure
- Layoffs in 2025 and pivot from open-source tools to a proprietary platform may have alienated parts of the developer community
Key Information
- Industry
- AI Developer Platforms
- Founded
- 2019
Frequently Asked Questions
What does OctoML do?
OctoML provides a cloud platform called OctoAI for deploying and optimizing machine learning models, especially large language models, for inference. It supports multiple hardware types including NVIDIA, AMD, and AWS Inferentia to reduce costs and latency.
How much funding has OctoML raised?
OctoML raised $85 million in total funding, including a $28 million Series B in 2021 led by Addition and a $40 million Series C in 2022 led by Tiger Global. The company has not disclosed any new funding rounds since 2022.
What is OctoAI and how does it work?
OctoAI is OctoML's managed inference platform that optimizes and runs large language models on cloud hardware. It uses expertise from Apache TVM to efficiently deploy models across NVIDIA, AMD, and AWS Inferentia, targeting enterprises and AI startups.
Who are OctoML's main competitors?
OctoML competes with Together AI, Fireworks AI, and Replicate in the managed inference space. It also faces competition from cloud-native services offered by AWS, Google Cloud, and Azure, as well as open-source model optimization tools.
Did OctoML have layoffs in 2025?
Yes, in 2025 OctoML laid off an undisclosed number of employees as part of a restructuring to focus on its OctoAI product. The company has not publicly shared updated metrics or major customer wins since shifting to the managed inference platform.
What hardware does OctoAI support for model inference?
OctoAI supports inference on NVIDIA, AMD, and AWS Inferentia hardware. This multi-hardware support helps reduce vendor lock-in and allows customers to choose cost-effective options for deploying large language models and generative AI applications.
Sources
- techcrunch.com — Context on AI startup funding landscape in 2025; OctoML not listed among $100M+ rounds, confirming no large recent raise
- octoml.ai — Company website; confirms current product focus on OctoAI platform and hardware support (NVIDIA, AMD, AWS)
- techcrunch.com — General TechCrunch coverage of AI industry trends; used for context on competitive landscape and cost management in AI inference
- dcthemedian.substack.com — Analysis of AI infrastructure spending and market dynamics affecting inference startups like OctoML