The risk is that USA and China will come together on a joint "AI non-proliferation" agreement in the same way that nuclear technology became restricted by the countries that already had the capability. This will lead to a de facto techno-colonization of the rest of the world just between themselves.
This was a topic of discussion at Trump and Xi's AI summit just last month. We're now starting to see the export controls coming into place.
"Second, we encourage open source, open collaboration, and resource sharing. The Chinese government is committed to making artificial intelligence a significant driving force for promoting common prosperity and safeguarding common security. To support the construction of an open source innovation ecosystem for artificial intelligence, convenient measures such as automatic licensing will be applied to technology exports involving open source AI technologies, Chinese companies building and operating their own computing centers overseas, conducting chip fabrication overseas, utilizing overseas resources for joint research and development, and participating in the formulation of international standards, basically achieving "immediate reporting and export." Technologies that have already entered the public domain through open source or published academic papers will not be included in the export management scope, creating a favorable environment for international exchange and cooperation in artificial intelligence technology."
their official stance suggests they remain in support of open-weights
Companies making closed-weight models typically don't export them at all, but sell access to them as a service. (Which doesn't count as an export of the model.) So it's primarily companies releasing open weights (if they pass the thresholds for the restricted category), or that host their models in datacenters outside China that will need to apply for an export license.
No, what they've been doing so far is changing nothing.
What they are saying is that they won't be changing anything.
That could of course change overnight.
There are multiple aspects to this: China is cottoning on to the meaning of soft power, and besides that being a major supplier of AI related models, services and hardware further cements their place in the world economy. They may not be able to compete directly with the USA in this domain but they are able to undercut them.
Of course there are risks as well. Models produced and released into the wild for which you have not seen the training and conditioning data could have payloads that you'd rather not have. But the same thing goes for service providers in other countries as well. You can't really trust hosted inference, not to tell you the truth in so far as is was present in the training data (censorship, not inference errors) and not to leak your information.
The proposed export controls do add some categories where export is completely prohibited. For example, sandbox detection. While the provisions specifically for AI models are generous enough, it's still possible that they trigger some other prohibition by being too capable. The statement about encouraging open source only applies to cases where export is merely restricted, not prohibited, and even then companies need to report their intention to export in advance, in principle giving the government the opportunity to deny them an export license.
Care to substantiate that? Not that I disagree, just curious.
My (not thorougly informed) take is that they are are taking measures to have the capability of restriction but are also adding provisions so that this capability does not add friction to the current practice of enabling export of open weight models.
So to me it's like they are keeping their options open rather than leaning on one versus the other.
I understand why China wants to restrict exports of their models.
But I feel the only reason Chinese models have been able to gain such popularity is BECAUSE they are open source.
I wouldn’t trust models from China enough to use them in important work unless they were open.
Restricting any powerful Chinese model [that ranks in the global top 10] is going to make them very difficult to advocate for, and very hard for global users to adopt.
Feels like this could be the beginning of the end of the democratisation of powerful AI.
And yes, I appreciate the irony that Chinese models have been the ones enabling that democratisation!
They gained popularity because THEY ARE GOOD, not because they are open weight. There are so many other models that are not popular. Do you know about arcee and their models? every used inkling? how about laguna? these are American open weights, they didn't take off because they are not as good as the rest. Do you know about the command-x series from COhere? probably not, this is a Canadian model, folks into local were really into it 2 years ago when they were keeping. do you know about dbrx? Yeah, databricks an American company also gave us a model. Anyways, my point is the competition is strong and giving away for free doesn't matter if it can't keep up. It's expensive, to send bandwidth, storage, gpu compute to run a model when there are better options.
> But I feel the only reason Chinese models have been able to gain such popularity is BECAUSE they are open source.
Why do people keep calling these "open source" models? They're not that different from NVIDIA's binary blobs. Sure, they work, and you can run them and get results, but the source code, training data, and training process are largely hidden and proprietary. Publishing weights is not the same as publishing the source. They're open-weight models, not open-source models.
There are risks in both cases, but they are very different.
If you're using a closed model, the model provider gets all your data, and can if they want to send back actively hostile tool calls and rely on your harness to catch them without any effort on their side to train that behaviour into their models. The only things you can do about that is to place careful restrictions
If you're using an open model against your own system or a third party provider, the origin of that model could have trained in "triggers" that'd cause the model to produce actively hostile tool calls as well, but they'd have no direct control and would need to train in that behaviour "blind" to how you'd actually use the model.
It doesn't mean you can be complacent about trusting the model, but it does change the likelyhood they'd be able to do any damage, not least because you'd be able to run evals against the model without any risk that the model or provider API could just randomly change on you after a while.
So it takes far more trust to run a closed model than to run an open model even on a hosted third party provider, and certainly far more than to run an open model on your own hardware.
You can run Chinese models locally without Internet, while with US models you need to have Internet connection. Guess where it is easier to hide a backdoor.
Why do they care about global usage though? This seems primarily so that they can control it for their billion plus population, they really don't need global users at all, much to our chagrin.
Latest and greatest China is able to make is reserved for the use of businesses and entities in the Chinese market.
Everyone else gets access to models of more limited capabilities.
It's basically the US strategy. Obvious difference being only that the Chinese allow on prem hosting of the lesser models. So plus one to them I guess.
If we want to move China or the US on from the whole, "super intelligence models for me, more limited intelligence models for everyone else" strategy, then "everyone else" will have to come up with some models competitive with the latest and greatest from the US and China. Absent that, there's just no pressure to bring the US and China to the "share the wealth" table.
Unfortunately, "everyone else" coming up with a competitive model is not gonna happen anytime soon.
The pressure for them to export models is that doing so super-charges the growth of the domestic industry. Same reason why they export solar panels, instead of just hoarding all of them for domestic deployment.
People pay money for the solar panels, though. It's not obvious to me that giving away expensive LLMs for free is meaningfully helping grow the domestic economy, as opposed to foreign inference providers who get their GPUs from Nvidia.
Giving away open weight models, if you dismiss altruism as a motivation from the outset, is a way to salt the earth while at the same time building mindshare.
It makes it harder to extract decent margins from models at or below the quality of the open models, and so makes it harder both for new entrants and for any existing providers that are depending on cheaper models for revenue. At the same time you make people comfortable with your product.
It's the IP version of dumping products below cost if you're capital rich and want to winnow down the competitive field while "buying" customers.
Let's just hope that both US and Chinese models remain usable enough by the masses for long enough for someone in Europe or Japan or some other actual free country to make a truly open AI :')
Yes, this happened weeks ago and was never discussed on HN.
Part of the problem is that there are too many major stories happen back-to-back nowadays, and important stories like this one never get discussed. HN only shows 30 stories on the front page, which limits discussion.
Correct, this news just hasn't gotten traction until now on HN it seems, as I've already seen this news on r/LocalLlama and thought it was another piece of newer news.
slightly ot, but this seems like a good place to ask, since I don't think I have the strategic-business mind to understand it all myself: What's the play here with China having supported publishing open weight models this far? I know the official line is for the good of mankind and all that, but presumably it's more strategic than that. Soft power? Commoditise your competition?
So far, there have not been any obvious drawbacks to open-weight models from the government perspective, so when a company wanted to release their weights, they weren't prohibited from doing so. Similarly, there have not been any obvious drawbacks to closed-weight models, so when a company wanted to keep their weights to themselves, they weren't prohibited from doing so.
So the real question you should be asking yourself is why you thought that China supported publishing open-weight models, even though they merely didn't prohibit it.
I'm pretty daft with this but my opinion is that they are open source so they can help Russia war efforts (and maybe Iran) unofficially and indirectly.
The US economy highly depends on the valuation of the tech giants whereas the Chinese economy doesn't.
Additionally, the financial and technological equilibrium which makes OpenAI or Anthropic possible is rather precarious so they are poking on it, hoping to make it fall.
there's alot of reasons, but one of the more commonly cited ones is that China facing chip restrictions means they lack the compute to provide the models for global usage, and making them open-weight is one of the ways to circumvent these restrictions.
somebody wake me up when we have "SLSA, but for model provenance".
I don't (just) want "open weight + verified distribution". I want a dang audit trail.
If/when one of these models is trusted enough to make real decisions with teeth, and if/when one of those decisions screws up badly enough to hurt somebody, I want to be able to play back the tape and pinpoint the source dataset + training method responsible for the failure. At LEAST at a coarse level, on par with something like modern RPO SLAs.
P.S. this comment is applicable to pretrained models in general. I don't care if it came from China, USA, or model weights scrawled in sharpie on a random bathroom stall. I just want to know the provenance and have enough information to run a blameless "git blame" (or equivalent) if/when things go wrong.
P.P.S. IMHO, we aren't there (yet), but OTOH this proposal ain't a bad place to start.
That's not happening without significant funding. The cost of the primary training run for frontier models is in the hundreds of millions of dollars, and is increasing with every iteration. The Astra pre-train likely cost close to a billion dollars.
The Chinese open-source models (which are 1-2 generations behind the frontier) cost in the low tens of millions to train. This is largely because of a lack of compute. They achieve comparable performance to frontier models via distillation.
This was a topic of discussion at Trump and Xi's AI summit just last month. We're now starting to see the export controls coming into place.
"Second, we encourage open source, open collaboration, and resource sharing. The Chinese government is committed to making artificial intelligence a significant driving force for promoting common prosperity and safeguarding common security. To support the construction of an open source innovation ecosystem for artificial intelligence, convenient measures such as automatic licensing will be applied to technology exports involving open source AI technologies, Chinese companies building and operating their own computing centers overseas, conducting chip fabrication overseas, utilizing overseas resources for joint research and development, and participating in the formulation of international standards, basically achieving "immediate reporting and export." Technologies that have already entered the public domain through open source or published academic papers will not be included in the export management scope, creating a favorable environment for international exchange and cooperation in artificial intelligence technology."
their official stance suggests they remain in support of open-weights
Says who?
What they are saying is that they won't be changing anything.
That could of course change overnight.
There are multiple aspects to this: China is cottoning on to the meaning of soft power, and besides that being a major supplier of AI related models, services and hardware further cements their place in the world economy. They may not be able to compete directly with the USA in this domain but they are able to undercut them.
Of course there are risks as well. Models produced and released into the wild for which you have not seen the training and conditioning data could have payloads that you'd rather not have. But the same thing goes for service providers in other countries as well. You can't really trust hosted inference, not to tell you the truth in so far as is was present in the training data (censorship, not inference errors) and not to leak your information.
So they're changing some things.
“Right now, your government is doing things you think only other governments do.” -Van Halen, Right Now
My (not thorougly informed) take is that they are are taking measures to have the capability of restriction but are also adding provisions so that this capability does not add friction to the current practice of enabling export of open weight models.
So to me it's like they are keeping their options open rather than leaning on one versus the other.
What they will do remains to be seen. The document at hand is what they're saying.
I.e. less important if we follow your own assessment.
But I feel the only reason Chinese models have been able to gain such popularity is BECAUSE they are open source.
I wouldn’t trust models from China enough to use them in important work unless they were open.
Restricting any powerful Chinese model [that ranks in the global top 10] is going to make them very difficult to advocate for, and very hard for global users to adopt.
Feels like this could be the beginning of the end of the democratisation of powerful AI.
And yes, I appreciate the irony that Chinese models have been the ones enabling that democratisation!
Why do people keep calling these "open source" models? They're not that different from NVIDIA's binary blobs. Sure, they work, and you can run them and get results, but the source code, training data, and training process are largely hidden and proprietary. Publishing weights is not the same as publishing the source. They're open-weight models, not open-source models.
So you've inspected the weights for backdoors?
If you're using a closed model, the model provider gets all your data, and can if they want to send back actively hostile tool calls and rely on your harness to catch them without any effort on their side to train that behaviour into their models. The only things you can do about that is to place careful restrictions
If you're using an open model against your own system or a third party provider, the origin of that model could have trained in "triggers" that'd cause the model to produce actively hostile tool calls as well, but they'd have no direct control and would need to train in that behaviour "blind" to how you'd actually use the model.
It doesn't mean you can be complacent about trusting the model, but it does change the likelyhood they'd be able to do any damage, not least because you'd be able to run evals against the model without any risk that the model or provider API could just randomly change on you after a while.
So it takes far more trust to run a closed model than to run an open model even on a hosted third party provider, and certainly far more than to run an open model on your own hardware.
A model is never neutral. A model is always also a way of disseminating an ideology, and it doesn't need to be lying for that to work.
Consider the question "what caused the French revolution?"
There's many ways to answer this while being technically correct. What gets prioritized, what gets omitted, that's where the ideology lives.
(Unless someone totally bonkers spends thousands of hours and thousands of pages answering a single question)
But once we automate that, it drastically lowers the amount of viewpoints represented.
Latest and greatest China is able to make is reserved for the use of businesses and entities in the Chinese market.
Everyone else gets access to models of more limited capabilities.
It's basically the US strategy. Obvious difference being only that the Chinese allow on prem hosting of the lesser models. So plus one to them I guess.
If we want to move China or the US on from the whole, "super intelligence models for me, more limited intelligence models for everyone else" strategy, then "everyone else" will have to come up with some models competitive with the latest and greatest from the US and China. Absent that, there's just no pressure to bring the US and China to the "share the wealth" table.
Unfortunately, "everyone else" coming up with a competitive model is not gonna happen anytime soon.
It makes it harder to extract decent margins from models at or below the quality of the open models, and so makes it harder both for new entrants and for any existing providers that are depending on cheaper models for revenue. At the same time you make people comfortable with your product.
It's the IP version of dumping products below cost if you're capital rich and want to winnow down the competitive field while "buying" customers.
Part of the problem is that there are too many major stories happen back-to-back nowadays, and important stories like this one never get discussed. HN only shows 30 stories on the front page, which limits discussion.
So the real question you should be asking yourself is why you thought that China supported publishing open-weight models, even though they merely didn't prohibit it.
Additionally, the financial and technological equilibrium which makes OpenAI or Anthropic possible is rather precarious so they are poking on it, hoping to make it fall.
somebody wake me up when we have "SLSA, but for model provenance".
I don't (just) want "open weight + verified distribution". I want a dang audit trail.
If/when one of these models is trusted enough to make real decisions with teeth, and if/when one of those decisions screws up badly enough to hurt somebody, I want to be able to play back the tape and pinpoint the source dataset + training method responsible for the failure. At LEAST at a coarse level, on par with something like modern RPO SLAs.
P.S. this comment is applicable to pretrained models in general. I don't care if it came from China, USA, or model weights scrawled in sharpie on a random bathroom stall. I just want to know the provenance and have enough information to run a blameless "git blame" (or equivalent) if/when things go wrong.
P.P.S. IMHO, we aren't there (yet), but OTOH this proposal ain't a bad place to start.
The Chinese open-source models (which are 1-2 generations behind the frontier) cost in the low tens of millions to train. This is largely because of a lack of compute. They achieve comparable performance to frontier models via distillation.