Are these comparisons based on API price? If so, then the OpenAI/Anthropic subscriptions (which are a 15-40X discount on model pricing) turn the comparison into a bloodbath, even more in favor of the closed source models.
These general intelligence/capability benchmarks don't apply to your specific task. If you are coding, you want just benchmarks for coding. I don't see performance+price at this benchmark for coding. When you look at just coding, DeepSeek is Pareto optimal for a good portion of coding benchmarks.
For example here, although even this is very flawed because it only benchmarks max/xhigh effort: https://artificialanalysis.ai/agents/coding-agents
Closed models might be better generalists because they don't have that much constraint on size and come from well funded labs that can afford to better train for generalization. If a model's actual goal is to be open then smaller is better.
I see this take frequently; that coding does not require general intelligence. Perhaps I am in the minority, but rarely do I find myself in coding situations that are completely divorced from other subject areas. The code is modeling or interacting with the world, either through business processes, or (in my case) simulating the physical world, and I need the agent to have some working knowledge of not just the code, but also physics, materials, or social conventions. Are you all really able to just "write code" without dealing with all the externalities?
Are you approaching it like programming? I first design the program I want it to write, and then let the LLM implement it. All the important decisions it has to make are contained in the design. The LLM only has to know programming and a little bit of algorithms to implement it.
Yes, 100% agree. Without more details on how they got their data, it's very hard to understand why they don't value DS Flash 4.1 as much as pretty much every other analysis I've seen (and my own experience).
IMO then it is missing the forest for the trees regarding open weight models. Qwen 3.8 Flash Next or 27B, for example, are accessible enough to run at home/office at very usable speeds. Provided memory companies don't collude to keep margins at 80-90% they are at the moment, it seems to me this will become a lot more accessible for many use cases while models at this size still have room to improve.
I'm just going to paste my comment from another post since it is applicable here too, maybe even more so.
I've had really great results with cheap flash agents using a system that has a parent agent that spawns 2 sub agents. They remain active throughout the whole session and don't lose context.
The 2 sub agents get the exact same prompt and tools as each other and they independently answer but before responding back up to the parent they exchange their findings which must be backed up by a citation and then debate their findings and citations repeatedly until they can agree on correctness of all points. They are also heavily encouraged to never assume and always verify with the prompt and tools and guards where they can be realistically enforced.
It churns through tokens but tokens are crazy cheap with some of the plans out there. Could Opus one shot most of those same queries in like 1/4 time? Yep, but that's not the point for me.
DeepSeek Flash for parent then DeepSeek Flash and GLM 5.3 flash as the debating subagents gives best overall results in my experience.
For context I do equal parts sysadmin/devops and having them write software
I run with DSH, I've found it crazy customisable with the Cordis framework and I feel like I understand how the components in it work. So I'm actually able to articulate to the agent what I want for the customisations. I Tried pi and OMP but I couldn't get exactly what I wanted.
As for the session maintenance, I have it configured to trigger a compaction when it ends a 'debate' with >25% of context used but before the compaction it puts what it thinks are key points into an MD file. The approach has worked so far in terms of stopping the agent from getting silly with too much context and retaining information. These flash agents get too flakey for my liking past 25%
Even as the article points out, this fluctuates constantly. In a few months, we're guaranteed to get a new batch of open models, and the balance will shift again.
What's more important is that open models can be inspected, they can be customized, and you don't have to worry that a company will decide that you're not allowed to use them for stuff like reverse engineering things or do security work.
That's why open models, and especially ones you can run locally will always be important. When you're renting closed models, you're basically a serf who has no say in how the model can be used.
That's to be expected since all the proprietary models were just released. Give it six months and the Chinese labs will distill from the proprietary models like they did last time.
The article itself goes against this intuition, open models have taken up as much as 40-50% of frontier for capable/affordable. Right now is very much a low point. The fact there's a lot of churn as new models are released is not unexpected, I think.
If required, so far I'm not impressed with newest proprietary. i have a claude sub mostly rotting at this point, so i come back from time to time until the year long sub expires, but at this stage i will not renew and see no point in doing so.
Have you tried Opus 5.5 and Fable 5.1? They're still better than any of the available open-weights models. (And rumors are Fable 5.5 is coming out tomorrow or next week.)
I have a Claude $200 (Ultra these days) plan and use Opus 5.5 for work with ultracode workflows and all the rest every day. I also use DeepSeek V4.1 Flash 1107 through a separate subscription plan with Pi.
Using DeepSeek I've made the Wayland compositor I use every day. It's finished now in the sense that it doesn't crash or have any bugs, nor can I think of any features to add or anything to change.
When somebody else is paying, I'll absolutely just throw Opus 5.5 at it. Who cares. It gives the best chances that the output is what I want, at no extra cost to me. But for my personal projects, using DeepSeek in Pi with a workflow extension is competent.
Sorry for throwing lots of text at you. All of this to say there's more to it than just which model is more intelligent. There's also speed, cost, and "good enough". Perhaps the problems I'm working on right now are just not complicated enough for "Opus is better" to be a real selling point.
Have used 5.5, ran usual battery of tests, and the downhill death spiral in a science/chemistry context continues. Code wise, it is better than open weight in several ways, but you really have to work hard to fight it from creating such a heavily abstracted mess it becomes unmaintainable for humans. Requires too much handholding for me. It's scientific reasoning is more confident as well, which would be great if it were also more correct.
Open weight models are generally more humble when working at fronteir, seek human input more reliably on thi gs they have no buisness deciding on, and can be trusted to not fuck up codebase in some way. Also, the way claude talks by default and tends towards even with all the "use this standard or that" tricks, grinds my gears in a way that drive me postal.
So, all in all, the shit prose, the constant oversight robbing me of attention better spent elsewhere, and high cost, and seemingly correct and difficult to catch errors on first pass make avoidance/non-reliance a nobrainer.
When dumb models are right, they're right, because likely its a solved problem elsewhere, and when they're wrong its quite transparent. Claude and fable are extremely good at spinning resilliant bullshit that can take quite some mental labor to verify and disprove, usual problem is flawed or deeply i correct assumptions on top of which then relativly sound reasoning and evidence is layered, or otgerwise weird contradictory decisons made "because i thought it would be better this way" that waste 48 hours of gpu time on an invalid model because claude fucked the one small part entrusted to it, which was extremely well defined and reasoned/backed by literature and it just made an executive decision to deviate without consulting or warning about it.
True but let's think about whether this actually matters in practice going forward, given that today, Deepseek/any OS-27B variant can do all the things that most devs need it to.
Let's scale this out 3 years into the future:
- Opus 6.7 costs an arm and a leg to run for 2 prompts
- Astra 7.89 literally requires you to buy-now-pay-later because it's so expensive
- As a bootstrapped dev, your only option is serving local Qwen-4.04-27B that runs on your old RTX (that you bought in 2024)
Sure, it's worse performing. But if you can't afford the best ones in the first place, what's so bad about having a "pretty decent" model that you run essentially free (only paying electricity) if it does what you need it to?
... and if we're all in the permanent underclass anyway, what does it matter that your local model is dumber than the SOTAs? At least you have one! Some people who can't afford GPUs still have to type emails by hand.
Despite its simplicity, price-performance is not the only metric; this obsession is a typical consequence of extreme overextension of the Friedman doctrine, and it is why so much software and hardware has been qualitatively shitty even when it matters that it’s not. That something can be measured doesn’t automatically make it preferable as a decision-making basis over intangibles.
It’s pretty obvious that local AI has a higher TCO if you look only at money. But there are other things to be considered.
edit: and of course from first party labs as well, like xiaomi, z.ai, kimi etc
Closed models might be better generalists because they don't have that much constraint on size and come from well funded labs that can afford to better train for generalization. If a model's actual goal is to be open then smaller is better.
Let’s say I want to run qwen 27B or flash next on a machine I own, off solar panels that are also sunk cost.
How did they price the marginal cost of a token? Depreciation on the box + reverse billing opportunity cost with the power company?
I've had really great results with cheap flash agents using a system that has a parent agent that spawns 2 sub agents. They remain active throughout the whole session and don't lose context.
The 2 sub agents get the exact same prompt and tools as each other and they independently answer but before responding back up to the parent they exchange their findings which must be backed up by a citation and then debate their findings and citations repeatedly until they can agree on correctness of all points. They are also heavily encouraged to never assume and always verify with the prompt and tools and guards where they can be realistically enforced.
It churns through tokens but tokens are crazy cheap with some of the plans out there. Could Opus one shot most of those same queries in like 1/4 time? Yep, but that's not the point for me.
DeepSeek Flash for parent then DeepSeek Flash and GLM 5.3 flash as the debating subagents gives best overall results in my experience.
For context I do equal parts sysadmin/devops and having them write software
As for the session maintenance, I have it configured to trigger a compaction when it ends a 'debate' with >25% of context used but before the compaction it puts what it thinks are key points into an MD file. The approach has worked so far in terms of stopping the agent from getting silly with too much context and retaining information. These flash agents get too flakey for my liking past 25%
What's more important is that open models can be inspected, they can be customized, and you don't have to worry that a company will decide that you're not allowed to use them for stuff like reverse engineering things or do security work.
That's why open models, and especially ones you can run locally will always be important. When you're renting closed models, you're basically a serf who has no say in how the model can be used.
Using DeepSeek I've made the Wayland compositor I use every day. It's finished now in the sense that it doesn't crash or have any bugs, nor can I think of any features to add or anything to change.
When somebody else is paying, I'll absolutely just throw Opus 5.5 at it. Who cares. It gives the best chances that the output is what I want, at no extra cost to me. But for my personal projects, using DeepSeek in Pi with a workflow extension is competent.
Sorry for throwing lots of text at you. All of this to say there's more to it than just which model is more intelligent. There's also speed, cost, and "good enough". Perhaps the problems I'm working on right now are just not complicated enough for "Opus is better" to be a real selling point.
Open weight models are generally more humble when working at fronteir, seek human input more reliably on thi gs they have no buisness deciding on, and can be trusted to not fuck up codebase in some way. Also, the way claude talks by default and tends towards even with all the "use this standard or that" tricks, grinds my gears in a way that drive me postal.
So, all in all, the shit prose, the constant oversight robbing me of attention better spent elsewhere, and high cost, and seemingly correct and difficult to catch errors on first pass make avoidance/non-reliance a nobrainer.
When dumb models are right, they're right, because likely its a solved problem elsewhere, and when they're wrong its quite transparent. Claude and fable are extremely good at spinning resilliant bullshit that can take quite some mental labor to verify and disprove, usual problem is flawed or deeply i correct assumptions on top of which then relativly sound reasoning and evidence is layered, or otgerwise weird contradictory decisons made "because i thought it would be better this way" that waste 48 hours of gpu time on an invalid model because claude fucked the one small part entrusted to it, which was extremely well defined and reasoned/backed by literature and it just made an executive decision to deviate without consulting or warning about it.
so fuck claude.
Open-weights excel primarily at coding.
I'm running full swarming on a 90$ subscription lol
Let's scale this out 3 years into the future: - Opus 6.7 costs an arm and a leg to run for 2 prompts - Astra 7.89 literally requires you to buy-now-pay-later because it's so expensive - As a bootstrapped dev, your only option is serving local Qwen-4.04-27B that runs on your old RTX (that you bought in 2024)
Sure, it's worse performing. But if you can't afford the best ones in the first place, what's so bad about having a "pretty decent" model that you run essentially free (only paying electricity) if it does what you need it to?
... and if we're all in the permanent underclass anyway, what does it matter that your local model is dumber than the SOTAs? At least you have one! Some people who can't afford GPUs still have to type emails by hand.
(jk, this is actually a cool dataset though ^)
It’s pretty obvious that local AI has a higher TCO if you look only at money. But there are other things to be considered.