Revised Insights on OpenAI’s o3 Model
In December, OpenAI introduced its latest AI model, o3, with the cooperation of the ARC-AGI creators, who developed a benchmark for assessing advanced AI systems. However, recent updates have led to a reassessment of o3’s performance, revealing figures that aren’t as striking as initially perceived.
Updated Cost Assessments
Just last week, the Arc Prize Foundation, responsible for overseeing ARC-AGI, adjusted its estimates regarding the computational expenses associated with o3. Initially, they believed the top-performance variant, o3 high, incurred costs of around $3,000 for solving each ARC-AGI problem. Now, the estimate has skyrocketed to potentially $30,000 for every task executed.
This revision is significant, as it underscores the potential expense associated with cutting-edge AI systems when addressing specific tasks, particularly in the early stages. While OpenAI has not disclosed pricing for o3 or made it available to the public, the foundation suggests that the pricing of OpenAI’s o1-pro model may serve as a useful benchmark.
Benchmarking Against o1-pro
The o1-pro model is recognized as OpenAI’s most expensive offering thus far. According to Mike Knoop, a co-founder of the Arc Prize Foundation, “We consider o1-pro to be a more accurate representation of the true costs of o3, especially concerning the computational requirements during testing.” Knoop emphasized the inherent uncertainty surrounding o3’s pricing, which is why it remains labeled as a preview on their leaderboard until official figures are revealed.
Given the resources that o3 high reportedly consumes, a high price tag is not unexpected. The Arc Prize Foundation indicates that o3 high utilized 172 times more computational power than its lower-computing counterpart, o3 low, while engaging with ARC-AGI.
Speculation on Future Pricing Models
For some time, speculation has circulated regarding the costly pricing structures that OpenAI might introduce for its enterprise clientele. In early March, reports suggested that the company could charge as much as $20,000 monthly for specialized AI “agents,” such as those mimicking software development capabilities.
While proponents may argue that even the steepest fees for OpenAI’s models could still be more economical than hiring human contractors, AI researcher Toby Ord raised a cautionary point on social media. He noted that these models may not reach optimal efficiency levels. For instance, the o3 high model required 1,024 attempts on each task within ARC-AGI to secure its top score.
