Mythos, China, and a New Era of AI Regulation

Breaking down the Mythos & Fable saga and what this means for AI safety, AI regulation, and China's open-source approach to AI

Watch or listen to the High Capacity podcast on:

• YouTube

• Apple Podcasts

• Spotify

What happened with Anthropic's Mythos and Fable models? Why did Anthropic's approach to AI safety trigger such strong backlash? What does this mean for the future of AI regulation? What role does open-source AI play in this new era of AI control? How will this impact Chinese AI labs and open-source AI models?

In this episode, I talk with Nathan Lambert, a leading expert on open-source AI. He previously led post-training at Ai2, founded The ATOM Project, and writes the popular Interconnects AI newsletter.

Topics covered:
- Anthropic, Mythos, Fable, and the Claude controversy
- Are Anthropic’s AI safety claims sincere?
- Silent degradation, trust, and researcher backlash
- Open science, open-source AI, and safety
- How will Anthropic's new safety features impact Chinese AI labs?
- The “AGI era” of AI governance
- Is open-source AI safer or riskier?
- Why US developers use Chinese open-source models
- What Nathan learned from visiting China’s AI labs
- Are Chinese AI labs AGI-pilled?

Follow Nathan Lambert:

• Interconnects newsletter

• The ATOM Project

• X/Twitter: @natolambert

• Substack: Nathan Lambert

Transcript

Kyle Chan (00:00)

Welcome to the High Capacity Podcast. I’m your host, Kyle Chan, a fellow at Brookings. I’m thrilled to be joined today by my guest, Nathan Lambert, one of the leading experts on open-source AI. He was the post-training lead at AI2 and is now building an American open-source coalition called the ATOM Project. He’s also the author of the fantastic Interconnects Substack, and he hosts the Interconnects podcast. Welcome, Nathan, and thanks for coming on the show.

Nathan Lambert (00:27)

Thanks for having me. I’m happy to be here. I’m not happy about why I have to be here.

Kyle Chan (00:33)

Let’s dive right into it. We’ve had quite a wild roller coaster ride, going back to Mythos and Anthropic’s decision not to release this to the public. We had an executive order that was canceled at the last minute, and then a different version was signed about voluntary pre-release testing, among other things. Most recently, we have these sudden export controls on both Mythos and Fable, which is the safety-restricted version of Mythos. The reaction from Anthropic was basically: if this is going to be export controlled for foreign countries and foreign nationals within the U.S., that means they effectively had to block both models entirely. So here we are. What are your reactions? What are your views on all this?

Nathan Lambert (01:28)

We’ll get into some of the details, but at a high level, Anthropic championed this view and convinced a lot of people that these things are a great risk, and now they are reaping some of that. I don’t think it’s great. Their virtues, in some ways, are valiant for what is coming, but they’re ahead of the game, which is causing whiplash sooner than needed, and that’s disruptive.

Nathan Lambert (01:58)

Then the government has been a much worse actor in how it responds to this, with a mix of punitive measures. Pete Hegseth had this tweet, which was insane. They just do this stuff, and it’s obviously at least partially punitive, which is way over the line. Even the implementation is a bit clunky. Realistically, I don’t think there would be an immediate crisis, even without the classifier safeguards. Most likely, things would be okay. There’s a chance that something really bad happens. I’ve priced in for a while that leading AI models would cause minor bumps along the road and things will keep getting worse. But it seems unlikely that Fable is actually that, based on everything we’ve seen.

A lot of people have had access to Mythos, even people who aren’t supposed to have access. There’s the one Fable Discord server that had access for a long time. So there are a lot of sides of Anthropic being high on its own supply and sanctimonious about safety while implementing features that are mostly good for its business. It’s really hard to unpack because it is true that AI safety is important, and this company has the leading models. But they’re also pretty antagonistic toward China, open weights, and open science.

What made me mad was that this shifted from rhetoric to action on the open-science front. I think it was pretty obvious that they thought they were the only people who should do AI research. But if they stayed out of the way, it wasn’t going to be a big deal. There are a lot of people who talk a lot in these communities of AI safety and don’t really do anything. So I was giving them a pass until they actually did something. Now they really mean it, and they can’t get a pass on this. Maybe we should break down more of these issues, and you should prod me to clarify the different issues at play.

Kyle Chan (03:55)

First, on the AI safety side, Anthropic has been one of the loudest voices about AI safety. They’ve made it core to their identity. I believe it was part of their origin story as they spun out from OpenAI. To what extent do you think they are true believers, and that this is what they really are aiming for? And to what extent do you think it blends into messy competitive dynamics with domestic competitors? We can also get into the China aspect of all this, but how seriously do you take their AI safety claims?

Nathan Lambert (04:38)

They’re definitely true believers, and they have been for a while. The things they predicted kept coming true, which will only increase a person’s resolve and how much they believe things. This goes for the leadership. They have carefully recruited people willing to align with this. I think the long-tail people are happier to align with the views when it also supports their business interests and financial outcomes. If it’s helpful to shift their financial outcomes or make tens to hundreds of millions of dollars of stock more likely to be realized, people are happy to say, “Maybe this is true after all, and we have to do this thing under safety.”

I don’t think these are bad people. I would probably feel the same way if I was looking at that type of payout. But it’s half brilliant leadership and half that they’ve been the longest believer and stuck to their guns. It’s hard to get the right word for it, but in some ways you have to respect how consistent the leadership is in their virtues, even if at the same time you can say it’s causing problems for the ecosystem and/or the world. You can always have at least some amount of respect, which is useful in engaging with them.

Kyle Chan (05:48)

And then there’s the implementation. We can point to any number of these cases: Mythos itself, when they decided to make this big deal about restricting it and then have almost this tiered structure for release, or the restrictions on Fable, which they reversed course on. I think, for part of it, they had content filters or restrictions based on types of prompts, types of API calls. There was especially controversy around how they dealt not just with distillation, but what they called using the model for AI development for others and what that means. I don’t know if you have a reaction to all that.

Nathan Lambert (06:29)

My biggest reaction was that it was very inconsistent. If you’re doing safety as your top-line priority, consistency in how you approach your safety measures should be the default top-line item before you release a model. That’s why I thought, okay, you guys are talking a bit too much of the talk while releasing this Fable model, which you have hyped up to be the most dangerous thing in AI ever. Then you’re saying, “We’re going to do this safety mechanism in two different ways.” If one of them is more safe, why don’t you use that for every domain? What are you doing?

To be specific, cybersecurity, research biology, and distillation were on the classifier side. Frontier AI research was on the silent-modification side. The classifiers being very overeager at release, I don’t really care about. That’s kind of expected. It’s annoying for a lot of people, but it’s something that’s pretty easy to fix, and Anthropic is under a lot of financial pressure to fix it. If a lot of users are saying, “I can’t say anything to the model,” it seems fixable to reduce false positives.

But with the silent degradation of AI researchers, you have no idea if it’s engaged, and therefore you pretty much can’t even try to use the model and see if it fits your use cases. For me, before it was walked back, it was: I just can’t plan on Claude being a reliable part of my future and how I work, because I don’t know if I’ll eventually creep into this.

So much of this year is me trying to figure out how to work with agents, and pricing in that this is just my new way of working and everything is mediated through these models. I still kind of feel this way about Claude. I have a very weird relationship with it right now, where in the future I might not be allowed to do my work in it. So it’s hard to financially support it and hard to build an ecosystem around how this model works.

To do this to your talent pool and people who should be your immediate allies in supporting you as a national champion of this area is a really weird community to take on. It’s very specific to the “our safety is the only way” view. I personally feel like I do my job to try to make AI evolution go better, and I don’t really think what I’m doing is competing with Claude because I’m not building true frontier models. But my version of safety isn’t allowed in their worldview, and there’s no way for me to reconcile it with them.

It’s just: what are you doing, man? Obviously, you can’t build the most important technology over time alone in a room and expect it to go well. There are going to be other people who need to tinker with your models, give feedback on things, and share things that you didn’t want to share because you have a conflict of interest. That’s just how science unfolds.

I think this is a more scientific technology than other revolutions, because the pipeline from a research paper to a frontier model still exists. It’s less frequent, and academia doesn’t have that much compute. But it definitely could still happen. That’s not unfathomable. Therefore, it’s crazy for them to say open science shouldn’t exist in AI around our artifacts. That’s why I would characterize them as anti-safety. I think it’s pretty clear that having more people involved in the development of AI is going to result in a safer outcome.

I think it can be hard to get the right implementation, but that’s a root assumption of what science is, and it is not a monolith. Industrial development is more of a monolith, but science is a very community thing. That’s a whole philosophical debate worth having within the AI community, but I didn’t expect to be having it.

Kyle Chan (10:14)

There are so many layers here. There’s the silent-degradation part, which really shatters trust because you don’t know where the boundaries are or when your usage of the model is being downgraded. You don’t know the scope of so-called AI model training or development work. Even with that fix, which they seem to have reversed in response to the backlash, there’s also what you were talking about just now: the fact that you’re not only pulling up the ladder, but you’re burning all the bridges and really trying to cut off everyone else from building on your work. It’s in total contrast to the current of AI development overall, where it’s this ecosystem that’s building on each other’s work.

Nathan Lambert (11:10)

I think the only fully coherent thing is to do this but also have a program for verified researchers to access the unfettered version of the model. If you work in OpenAI or a competing lab, maybe you won’t get access, but academics and people who do cybersecurity research will still have easy access. Even those regimes are always imperfect, but the imperfection is that you have way more people able to use the tool.

How it relates to the China trip I did is that all the people in China are scrambling, realizing they will no longer have Claude in the future with stricter verification rules, because all the Chinese researchers are Claude-pilled. They all loved using Claude, and they use it through these weird proxy services. If a much stricter know-your-customer regime comes in, it’s clear that those will be less functional. It’s unclear if they will go to zero. It’s not that hard to sign up for an account or use a fake identity. If they’re already doing things that, I don’t know if they’re illegal in the letter of the law, but they go against the terms of service for sure, I don’t know if those will get by the know-your-customer things that Anthropic puts in. But there will be fewer developers in China able to use Claude over time. That has to light the fire in them in terms of figuring out a longer-term solution to this. I don’t know if they have an easy solution, but they’re going to be talking about it.

Kyle Chan (12:39)

To the extent that they have to switch to a next-best alternative, or they’re still going to try to find access one way or another but just have a harder time, it will have a large impact, you think.

Nathan Lambert (12:59)

Yes. Journalistically, that’s one of the most interesting things to do: get the tap on the top labs that all use Claude extensively. You see the U.S. researcher ecosystem on X react to this, but what is the leadership of the various labs in China thinking about this, and how crucial is it? Obviously, the distillation side is one thing, but there’s an ebb and flow between the labs and the Chinese companies wanting to distill. The user side is something they probably didn’t expect to have to do as much cat-and-mouse with in the immediate future. They already priced that in.

Kyle Chan (13:31)

Before we go down the China open-source path, I do want to get your reaction to the U.S. government’s reaction to all this. What’s your take on these sudden export controls, the on-and-then-off-and-then-on-again executive order, and more broadly, where do you see the U.S. federal government going when it comes to model restrictions and AI regulation?

Nathan Lambert (14:11)

I’m new to the space and don’t know the intricacies of government. There will be people in the audience, and you probably know way more about this. But a few things seem to be true in a way that’s harmful to AI’s rollout with effective governance.

One is that the federal government has destroyed internal capacity for understanding, communicating, and working with these models. I think the executive order is supposed to not be a licensing regime, but what we have seen with Anthropic is much closer to the government wanting the power to mandate a model, whether or not the motivations are partly punitive. So one issue is government capacity. Two, the government is definitely acting at least a bit politically against Anthropic, which is a horrible precedent to set far beyond AI. It’s a horrible economic precedent in the U.S. for the government to be doing this. It’s the reverse side of the government taking a stake in a bunch of companies, which is questionable but much less directly pushing the issue than the government taking essentially forms of attack against leading companies.

Nathan Lambert (15:28)

That’s the Hegseth tweet I mentioned. Also, the implementation of this export control was a way to have them shut down the model without telling them they needed to shut down the model. If that’s the outcome you want, just tell them to do it. I don’t think the government should be playing weird games with this sort of technology. This is a huge crisis.

Anthropic seems potentially to like sticking it to the government a little bit. I don’t think they’re searching for issues, but when something comes about, they’re really willing to make a point of it. That goes against a lot of things they’ve been saying long term, because it seems like Anthropic is shockingly pro-nationalization of frontier AI. They see it as an inevitability. But I think they want this nationalization to be very democratic and smooth.

With the U.S. government’s current and medium-term trajectory, there is very little chance of Congress passing a law to nationalize a specific company. Therefore, you’re left with the executive branch doing this, which, as we said, doesn’t have the capacity. At some point in time, there will always be somebody you don’t like in power. Somebody will always have somebody they don’t like in power. There are a lot of enemies of the Trump administration, especially inside Anthropic. Anthropic is the only company not to bend the knee. I think companies shouldn’t need to bend the knee to the White House. I shouldn’t have to say this. Companies shouldn’t have to bend the knee to the White House. That’s absurd.

But this power resting with the government will always have people. There will always be targets, and there will always be enemies. That’s horrible for a business environment.

Nathan Lambert (17:16)

I don’t know how to solve this, but I’ve pointed to more transparency as one solution. Anthropic should fully document all the engagement they have with the White House. They could release exactly when all the calls came in and things like this. They could release more about their decision-making, what convinced them that Fable wasn’t a risk relative to Mythos after saying Mythos was this horrible thing to get in the hands of adversaries. They could release more transparency there.

The government should release more transparency on the scope of the jailbreak and why they think they had the grounds to do this. Otherwise, it’s going to be political and vibes, and that’s going to be bad for everybody involved. That sort of transparency is hard to pull out.

I also think these decisions should be based on evaluations, and I don’t think we have technical evaluations for what is the case. Especially with a narrow jailbreak, it’s essentially one prompt of the model that unlocks a capability. That’s not really something you could use an evaluation set on. It doesn’t fall into nice regimes, like whether the model is safe on cybersecurity and bio, which would be an evaluation set.

I think a reasonable policy would be: you have a set of known prompts for research, bio, and cybersecurity. If the model gives a sufficiently strong expert-cleared answer out of the box with the safeguards it has, then you say, “You should pause releasing this model, and we’ll take action.” I don’t love this, but it seems like there’s going to be more of this as models get stronger. That seems like a reasonable regime. But who is going to build this evaluation set in the government and maintain it over time? Everyone expects that they just won’t do it.

Kyle Chan (19:13)

It seems like we are crashing into a new regulatory regime that is still being formed. In this process, I don’t know if it could get any messier: Anthropic pushing to get more mandatory requirements for testing and then later saying, “Wait, we did it, and now you’re blocking Fable. That’s not what we’re talking about.” Then from the U.S. government side, you have different voices and perspectives within the White House and the cabinet. Some people seem to be very suddenly concerned and have really bought Anthropic’s emphasis on AI safety, and are thinking about, say, financial risk to U.S. financial institutions. Other folks seem to have more of an axe to grind with the company or the founder or something else.

You layer all that on top of each other and get this whiplash, where the executive order is about to be signed, then suddenly pulled back, and then actually quietly signed. You’re gesturing toward imagining if we had a more thought-out way of approaching this: taking some of these issues very seriously, while maintaining democratic values, robust safety mechanisms, and not blowing trust in the whole system or making it feel arbitrary and behind closed doors.

Nathan Lambert (20:54)

I think we need somebody similarly influential to Anthropic to now champion what they have been saying, which is that you need more regulation soon, because now they’re too intertwined with it and it’ll look too self-serving for them. In the last six months, they were very right. I wrote an article yesterday called “Welcome to the AGI Era of AI Governance.” I don’t like the AGI term, but it’s me trying to say clearly that this is the status quo. These Anthropic blow-ups are not one-and-done. We’re going to be having things like this go down on a recurring basis until we have new governance regimes, and it doesn’t seem that likely that we’re going to pass any laws on this governance regime.

This is what it is like when the government realizes that some of its power structures and national security are at risk. The exact flavor of it will be to the tune of the administration. You’re getting a very Trump administration flavor of this. But I think there would be some level of chaos with whoever is in power once they deeply internalize this.

Part of the problem is that these AI labs, from my perspective, have sounded the alarm on a similar level of risk since ChatGPT. So they’ve been soothing people in power into thinking, “This isn’t really real.” Then the capabilities underneath slowly get closer and closer to the messaging the labs are saying. Now it’s finally close enough that they’re saying, “Shit, we have to do something.” But there hasn’t been time for the government to ramp up, and it has been hard for the government to ramp because they were trained to ignore what people were saying.

I agree that it was smoke and mirrors and seemed like regulatory capture when people were saying this chatbot is going to tell you how to build a nuclear bomb in your backyard. That’s not going to happen. That’s why I think this has happened.

The whole other level is that there are going to be way more intelligent AI models coming, so it’s going to get worse. I think the AGI era is these AI assistants that are really useful and could go off and build something for you if you know what to ask for. But eventually, people are going to spawn AI agents that work on their behalf 24/7 with autonomous goals. That’s a whole different era of governance that will spawn even more freak-outs, because I think there will be much more chaotic downside to those, where little things break all the time because the agents just run all the time. In a few years, we’re going to be there, and I don’t think we will be ready for it. I feel a little naive having not priced all of this in a bit sooner, but that’s life.

Kyle Chan (23:29)

What’s the role of open source right now? Is it more important than ever to counter the concentration of power in these proprietary closed models? Or do you buy the criticisms that some folks within the AI safety community make that open source is a source of risk or danger going forward?

Nathan Lambert (23:59)

Open source does have risks. It has different risks than one company saying they define science and safety. I think the status quo we’ve had, with open source being some number of months to a year behind the closed frontier, has been pretty safe. You see what types of models are coming, you have a bit of time to prepare for them with smarter AI, and then you can ideally get ahead of the types of risks that are coming when they come. I think this has been a very good equilibrium.

In particular, the Anthropic Department of War spat and now this spat point to the fact that the long-term equilibrium for existing power structures and sovereigns is going to be much more pro-open or pro-nationalization than having leading AI companies. A lot of people are saying, “I don’t know how we could ever trust the federal government to decide what model to build, so open is a good alternative.”

It’s a scary and very different world to have the frontier of intelligence be open. But I think a lot more people are thinking that, in the long term, we need to have something much more intelligent and much closer to the frontier in the open so that we aren’t reliant on these companies that want to dictate how I use AI, make all the decisions on safety, and control the economy from there. That is definitely not easy to think about.

I think of it as a world where access is much higher, so some types of understanding of the models would be way higher, which provides a type of safety. But there will be more knife cuts of people grabbing a model and doing a bad thing with it than in a world that is highly controlled by one organization. It would be very different to see.

In the near term, though, the probability of this happening is actually going down because the government’s interest in Chinese models in particular is going up and up, and they’re very worried about distillation. We’ve seen increased rhetoric from the White House and from congressional committees against this. In the near term, if the federal government is willing to do this to Anthropic, they would be very willing to do this to a Chinese company that released a surprisingly strong model. They would just ban it.

I have no idea what the legal recourse would be to unban an entity listing, a new type of entity listing on software, or mandated documentation if you’re using the model within the U.S. on certain hardware. I don’t know who would fight against this and how to fight it in court, because it’s such a decentralized coalition. I think it would just get banned. I think it would be a very short-sighted decision because the people who benefit from this are the closed companies, and the people hurt by it are primarily startups and academic researchers. That’s the hard part.

I have sympathy on the distillation point. I do think the Chinese labs are trying to hack the APIs and get more information out of them. But at the end of the day, it’s hard to make regulation that the biggest companies in the world ask for when it would literally just benefit them. It’s a tough spot.

Kyle Chan (27:10)

Do you think it’s even possible to ban open-source models? What does that mean? Does this run into First Amendment issues? How do you block something that’s available freely on the internet?

Nathan Lambert (27:27)

In practice, you couldn’t. Any consumer could still download it, but it would create sufficient legal jeopardy that all the universities would say, “You can’t use this because we’re too worried about getting sued by the government,” and companies would take similar risks. Anyone with a GPU on their desk would still be able to download Qwen models, because I don’t see the U.S. implementing any type of firewall. But if academic and government access, or academic and small-business access, is kneecapped or has a lot of uncertainty over it, that’s such a chilling effect for the ecosystem. Those are the people who push the innovation forward. The consumers are mostly users. They’re not doing a lot of training or development of the broader ecosystem around it.

Kyle Chan (28:10)

Speaking of usage of Chinese models, especially among the research community and among startups, why do those developers use Chinese models at all? I know a number of them, and they describe different workloads and different types of tasks that they switch over. There’s a mix-and-match approach. When would you want to use the most advanced, most expensive proprietary model, and when would you switch to a Chinese or other open-source model?

Nathan Lambert (28:42)

The default workload right now is that you want to get some sort of automation built in your company. You start with the closed models, and they cost you a lot of money. Then there’s a very large financial pressure to try to move that automated task onto the most capable open model. That’s one use case: there’s the closed frontier, and then there’s the open frontier, which is this large mixture-of-experts model layer that, to date, has almost all been built by Chinese companies.

I think NVIDIA released their Nemotron 3 Ultra model, which we could see having a similar amount of adoption. The Chinese labs are building good large MoE models, which is why Reflection hasn’t released a model yet. They don’t want to release a model that’s not better. It’s primarily because you could save a ton of money by shifting to this task.

Then there’s Cursor using this as their access to some control over their own intelligence stack, which is existential for them as a company. Mostly, all of the models in this layer — the frontier but open-weights layer — have historically been from Chinese labs.

They’ve also dominated this long tail of research. The vast majority of language model research right now is on Qwen 3 and Qwen 3.5, which are these small models that are pretty predictable in performance and have been fairly easy to use for multiple generations. You read research papers on reinforcement learning or agentic harnesses, and they’re just built on Qwen flat out. Gemma is the first model from an American company, Google, that potentially could challenge that.

It takes a long time to build up the expertise within researchers to be used to using this type of model, which is why Gemma won’t immediately show up in the research ecosystem. But that’s something I follow and care about a lot: these two classes, the giant model and the smaller models that get a lot of adoption.

Kyle Chan (30:37)

For the smaller models, are you talking about local deployment or still over the cloud, over a platform?

Nathan Lambert (30:47)

The usage numbers, if you look at Hugging Face, are probably dominated by some mix of local. But the thing I particularly follow is what researchers and small companies are using for blog posts and papers, and that’s been mostly Qwen. Researchers mostly use it in the cloud for training and things like that.

There is a large community of people who use models locally, but it’s pretty amorphous to track, and I don’t think it has a huge influence on the frontier of AI. For some things it has risks, like image-generation models downloadable locally having social risks for nonconsensual deepfakes and things like this. But that issue will just exist forever, and we have to figure it out. I’m more interested in what issues are coming when new models get better and what’s down the road.

Kyle Chan (31:36)

In general, there might be a mixture. I was just talking with a friend who was saying that, for his workload, he has the most powerful version of Opus, or actually he was using Fable, to do the architecture design up front, and then he switched over to Kimi for everything else — for building out the rest of the implementation. He was also complaining about Fable suddenly being cut off in the middle of a workload overnight. Going forward, do you think the mix will shift more toward open models? If it’s some percentage or combination of open and closed model usage, will this push more folks to use open models or at least be interested because not only are they cheaper, but you might not be cut off suddenly? Or do you think that’s not as big a factor?

Nathan Lambert (32:36)

I think they will, but the potential to be cut off is actually still there because we’re in this total oddball regulatory regime where they’ll try whatever to get their desired outcomes on AI. But it is much harder to enforce cutting off the open models because they’re downloaded to somebody’s server. Are you really going to check everybody if they’re using it? That’s part of the vibe: you just can’t take back an open model, even if you start adding friction to the ecosystem.

Kyle Chan (33:16)

You mentioned your trip to China, visiting a number of the Chinese AI labs. What was your impression? Obviously, open source is a huge part of their strategy, although maybe there’s starting to be a bit of a shift toward closing some of their most advanced models. What were the vibes like, and what do you think people in the U.S. should know about the Chinese AI ecosystem?

Nathan Lambert (33:43)

I think they’re pretty normal frontier model labs in this capacity. They’re large technical teams, they have a lot of talent, and they wish they had more compute. There’s obviously a big training-compute multiplier between the top labs in China and the top labs in the U.S., which I think will manifest in things like Claude Fable and whatever OpenAI comes up with. I don’t expect a Chinese lab, at least other than Alibaba or ByteDance, to have something like this. I don’t know if Alibaba or ByteDance would release it for general access. But all the Z.ai, MiniMax, and Kimi labs of the world just don’t have enough training compute or capital to do anything like that.

But they’re normal labs. There are small differences. There’s not much of a data industry there, which is interesting to see as the RL environments paradigm plays out. Mostly, it’s fun to go talk to them. You ask them about why they open source, and they’re all pretty practical. It seems like the best way to get company attention and any sort of usage. If you’re a brand-new startup that builds a decent but not frontier model and you’re selling an API, nobody is going to work with that or do anything with it. Versus Kimi getting the credibility of Cursor being built on Kimi — that’s a crazy dependency.

Nathan Lambert (35:05)

It’s not a crazy dependency for Cursor to take on, but it’s a crazy success for Kimi’s marketing. I think Z.ai has a similar thing, where they’re saying, “We don’t know why we’re entity listed, but it makes us a real company in the world’s eyes, so we’re fine with it for the time being. It doesn’t really do anything to us, so we’re just going to go about our lives.” That’s a surreal dynamic for all of them.

While all of this geopolitical intensity is heating up, I think there’s more capacity for Chinese startup-U.S. startup technical collaboration or partnership between the U.S. companies trying to build the open-source intelligence stack, offering fine-tuning for open models or inference. A lot of these companies want to talk for real partnership reasons to the Chinese counterparts building the models. From the U.S. tech industry’s history, that’s unprecedented because the U.S. tech industry has had a wall to China, where if you go to China, your devices are bricked and you can’t work in China.

For that to become a micro-industry in light of the geopolitical competition would be wild to me. But it makes sense with the parties at hand, because it seems like there are a lot more well-resourced, open-source-friendly companies in the U.S. that are trying to monetize these models. If the Chinese labs can get a kick of that by building the models, that’s going to be meaningful revenue for them, given how big the inference and related markets are.

Kyle Chan (36:37)

It seems like, at the big geopolitical chessboard level, there are all these pressures to split the U.S. and Chinese AI ecosystems. But when you drill down to researchers or startup developers in the U.S., they care a lot about these Chinese open-source models. Ironically, Chinese developers care a lot about Claude. That’s the reality I follow.

Nathan Lambert (37:04)

It’s only at the top level of government. Very few people care strongly about China, and vice versa. A lot of Chinese researchers can’t come to the U.S. because our visa system is totally clogged. It’s easier for U.S. people to go to China, especially with transit without a visa. So there are all these weird dynamics at play. If Chinese researchers could just fly to San Francisco, they would be there way more often, and it would probably cool down the temperatures more. They would say, “I work at Kimi. I’m building models. Should we collaborate?” That’s what they probably want to do.

They’re all very intrigued, and I’m a person who’s seen as a funnel into the Western ecosystem. I’ve done essentially free advising for companies asking, “How do we release this model?” A Chinese company will ask me this, and I’ll say, “I’ll try to help.” They’re asking, “How does X work?” They’re trying to unpack this puzzle to get further Western influence and be in the AI discussion. They want to be in the San Francisco version of talking about AI. They want to be seen as somewhat normal in that capacity, and not as a Chinese outsider or potential enemy if people are uninformed.

There’s definitely a history in the Bay Area of people being uninformed. I listened to Bill Gurley have a great conversation with Dan Wang and Patrick McGee, and he was talking about this. He has a very long-term Silicon Valley perspective that I don’t have, and I agree with it. What percentage of people building models in San Francisco have read both Breakneck and Apple in China? It’s probably pretty low. They’re incredible books and super entertaining. That’s probably most of my worldview. I’m not that well-read on China, but I’ve read these books, and it’s like, okay, that kind of makes sense.

Kyle Chan (38:51)

I often have to step back and remember that I’m in a different world from your median American Silicon Valley developer. What is their understanding of China or their default view? I have to factor that in. These books that seem canonical —

Nathan Lambert (39:25)

Most people think it’s control. They think control, and it’s very decentralized and chaotic on the ground. You go to Beijing and see the various municipalities of Beijing competing for tech offices. You end up with tech hubs in different neighborhoods where they’ve gotten good real estate deals. Or the Beijing Academy of Artificial Intelligence — I don’t remember, BAAI, whatever the word is — is a state-funded research lab that probably has tens of millions of dollars of budget just in Beijing.

Tsinghua kind of feels like the Bay Area. You go from a university, which would be Stanford, and then you go to Z.ai and Kimi and Xiaomi, and they’re all nearby. There are these big tech-company campuses. I don’t think most people would expect that very Bay Area vibe to be in China, but it really is. Tsinghua is a very beautiful campus, and these companies walk to the other company for lunch to talk to each other. There’s free-flowing talent, and they don’t feel like they can do much about whatever the government does. They’re trying to do their part, and there are all these layers of competition that make any top-down dictate hard.

You can feel this when you’re there, but it would be hard if you’ve never heard anything about it. If you went to China and thought it was top-down, you would probably say, “Well, it’s a little messy.” But I don’t think you would pick up on the decentralized government layers until you start to study China a bit.

Traveling between cities and seeing the things cities do to build a ton of real estate and be attractive makes it make a bit more sense. I learned that Hangzhou has all these miles of flowers along its highways that they obviously have to replace every few weeks. It’s the biggest highway interchanges you’ve seen, with both sides having well-maintained, beautiful flowers on the side of the road. It’s like, what the heck? How do they maintain this? That’s the way the government competes for status. It’s a ton of little things like that.

Nathan Lambert (41:42)

It’s diffuse, in a way.

Kyle Chan (41:44)

I like your point about talent flowing between the different labs and through universities. It seems like maybe part of the more convivial dynamics among them — and correct me if I’m wrong, because there’s still fierce competition among the Chinese labs for sure — ties back into their open-source ethos. To what extent do we have Sam Altman and Dario not willing to hold hands at an AI summit in Delhi because of various reasons, but one of them being a very different attitude toward AI competition?

Nathan Lambert (42:34)

There’s probably self-selection, because the top researchers at these labs, if they wanted to work for a U.S. company, almost surely could. So there’s probably some patriotism, where they want to see any of their companies, their peer companies betting on AI, succeed. It’s clear to them that ByteDance, Doubao, and Alibaba are going to be fine. They all talk about ByteDance as being the biggest competitor, and they all look at DeepSeek and wonder. I don’t think anybody knows what DeepSeek is really trying to do, but they have a lot of respect for them.

But of the Kimi, MiniMax, and Z.ai companies of the world, if one of them were to succeed, all the other ones would be happy about it because it bodes well for them. They also know it’s not a simple path. It’s not like build Claude Code, make $40 billion in ARR in five months. They’re not on that path, and they’re not delusional about it. They realize it.

Kyle Chan (43:12)

It seems like they have a bit of a David-and-Goliath view of things. Maybe other folks have different ideas, but they’re very aware that the American labs are enormous in terms of capital and scale. They’re trillion-dollar startups at this point, versus even the biggest Chinese big tech companies, which are smaller than American AI startups. It’s a different scale. It’s not the giants on the playing field all battling each other out. It’s more that the Chinese players are trying to carve out their niche and part of the market while these much bigger international players are bestriding the world stage.

Nathan Lambert (44:31)

It feels like a slow burn. There are plenty of people worried about the dynamic where trying to start slower becomes way bigger with AI. There’s not a ton you can do about it. They have Huawei chips for inference. I think Huawei chips obviously work for large MoE model inference at a reasonable price point. All these labs that have inference demand use Huawei chips. I think it’ll be a while before Huawei chips are used for training, because they’re so strapped for NVIDIA, and NVIDIA has to be so crucial to what they’re doing still.

Nathan Lambert (45:12)

The dynamics of AI demand will be very different in China. I think it’ll be a bit of consumer-y free stuff and a bit like cloud, where companies realize it’s part of what they have to spend on in order to build their products. But the big tech companies, like the Meituans and the Ant Groups of the world, are saying, “We need to own this in-house. We will build this technology. We will release the general version because it doesn’t really matter.” Then they specialize it to tasks within their apps or services and don’t release the specialized models that are part of their integrated offerings with these mega-apps.

The mega-app idea suits AI pretty well because it’s a really expensive thing to build, but you can use it in a lot of ways. That dynamic potentially favors those big Chinese companies. I don’t know exactly how it would play out.

Kyle Chan (46:09)

They’re all taking such different strategies, and I guess each of them is trying to play to its strengths. Big players like Alibaba and Tencent are tapping into the vast ecosystems they’ve built up long before this recent AI boom. Same thing with ByteDance, especially leaning hard on video and multimedia. Then you have some of the scrappier startups, where maybe they’re trying to become the affordable, high-quality coding agent of choice for many developers. To a certain extent, they’re following very closely what’s happening in the U.S. and trying to keep track of the latest trends and where the industry is going. On the other hand, they’re also hyper-focused on doing their own thing and trying to figure out a way to do cool models and make some money in the process, right?

Nathan Lambert (47:08)

They’re very bought into the idea that this is going to be a gigantic pie, and if they can take a small slice of it, that’s a huge success. I think it’s a reasonable default expectation for how AI plays out. If the frontier labs wanted to build hyper-efficient, cheap models, I think they could do a pretty good job at it. But that’s not the ethos of why they exist.

It’s part and parcel of the AI boom being a boom and somewhat bubbly to go for the absolute moonshot: What is the top of capabilities? It’s a very boomy mentality. But Chinese companies can’t quite take that approach to technology with the capitalization they have. I’ve been surprised by how well the stocks of Z.ai and these other companies have done, and by the ability to keep fundraising capital in China. The numbers are much lower relative to the capital outlay in the U.S., but I expect that to become more difficult sooner than it becomes difficult in the U.S.

Kyle Chan (48:17)

Last question. To what extent do you sense that the Chinese labs are, quote-unquote, AGI-pilled? How much time do they spend talking about, like their U.S. counterparts, reaching AGI, timelines for getting there, and the societal implications of all this? Or to what extent are they focused on other things?

Nathan Lambert (48:43)

I think they care about it, but they’re mostly focused on other things. A lot of this is down to the fact that they’re raised in a very structured education system, where they’re encouraged to be good at their job, and then the people who are good at their job are very focused and good at this. Therefore, the social dynamics are less encouraging for them to be this type of AGI-pilled.

They definitely care about security, and they’ll use the word security much more often than just safety. Developers think about it. Multiple people gave an answer of, “AGI is when I’m replaced,” and they’re saying, “I want to build that.” But you don’t get the ideological tune of it, or the awe of it, as frequently as in the U.S., where a lot of people are really bought in and that’s part of the cult of things.

Especially right now in San Francisco, with the whole permanent underclass discussion, there’s a very distinctive flavor of permanent-underclass-type discussions, which is pretty gross and not realistic. We didn’t really get that in China. There was acknowledgement of the technological wave, but there wasn’t that full-frontal assault of a very distinctive ideology of the area you’ve just flown into.

Kyle Chan (49:59)

The discussion in the U.S. just feels so all-or-nothing existential. Either you are the lab that will achieve RSI first and basically win it all, or AI in general will be an all-conquering technology and there’s basically nothing left for anyone who is not building or part of the shareholder class for that. In China, it just seems not exactly that.

Nathan Lambert (50:40)

I think they realize that there’s a top-end limit, and it’s kind of ominous. I don’t think we got a lot of direct engagement. We were there when the Manus deal was clawed back, and then we had left the Kimi offices an hour before their external investment round was canceled. Maybe if we were a week or two later, we would have been able to poke at these internalization things. There’s also the clawing back of passports. It’s hard to get people to talk about that when you visit their offices because it’s a very sensitive issue. But that’s happening.

I don’t know — this is a guess that might be wrong — but I would guess a lot of Chinese researchers are afraid of this. It’s scary to have your livelihood restricted. A lot of the people we visited are saying, “I have family in the U.S. and I go back regularly,” or they used to live in the U.S. For individuals to butt into that is definitely scary.

Kyle Chan (51:39)

Definitely. The political constraints are real. On that note, I just want to thank you so much. This has been an awesome conversation. I highly encourage everyone to check out your Interconnects Substack at interconnects.ai and the podcast as well. I’ll include links in the show notes. Is there anything else we should mention or give a shout-out to?

Nathan Lambert (52:08)

I have a technical post-training textbook under the title Reinforcement Learning from Human Feedback coming out in a few weeks. It’s mostly my own notes, so I still use it to reference things and try to communicate the intuitions of training models. It’s mostly targeted toward grad students trying to learn how to do this research, so it’s definitely pretty technical. But it is a topic that warrants its own text, given that it started this whole ChatGPT era. I’m happy to have a textbook on it that I can send to people, but I will not tell people it’s a mass-audience “learn how to build an LLM” type book.

Kyle Chan (52:43)

If anyone wants light bedtime reading, they can read the book on RLHF. Thanks again so much, Nathan. This has been an awesome conversation.

Nathan Lambert (52:48)

It’ll put you to sleep. Thanks for having me.

Kyle Chan (52:59)

Just to close out, if you like this episode, please rate and subscribe on YouTube, Spotify, or Apple Podcasts. You can find episode transcripts and more information on the High Capacity newsletter at highcapacity.org. I’m your host, Kyle Chan. Thanks for joining, and see you next time.