Watch or listen to the High Capacity podcast on:
Can the US and China agree to slow down AI development—or even agree on what counts as an AI risk?
In this episode of the High Capacity podcast, I talk with Kendra Schaefer, partner and head of tech policy research at Trivium China, about China’s approach to AI safety. We discuss AI service registration, the regulatory gaps around open-weight releases and AI agents, and why Chinese policymakers may see some risks differently from their US counterparts. We also explore where US-China cooperation on AI safety might begin.
Topics covered:
Why China does not necessarily see regulation as a slowdown
Anthropic’s calls to slow frontier AI development
China’s AI service registration and safety testing process
The challenge of regulating open-weight model releases
AI agents, incident reporting, liability, and traceability
How China assesses technical, social, and geopolitical AI risks
What the US and China could discuss in an AI safety dialogue
Follow Kendra Schaefer and explore Trivium Tech. You can also sign up for Trivium’s free daily brief.
Transcript
Kyle Chan (00:00)
Welcome to the High Capacity podcast. I’m your host, Kyle Chan, a fellow at Brookings. I’m thrilled to be joined today by my guest, Kendra Schaefer, a partner and head of tech policy research at consulting firm Trivium China. She’s one of the most insightful analysts of China’s AI policy I know of, with years of experience working in China and talking to all the major tech players there. Welcome, Kendra, and thanks for coming on the show.
Kendra Schaefer (00:23)
Hey Kyle, really good to be here.
Kyle Chan (00:26)
So in the US, there’s been this huge wake-up call moment where suddenly everyone’s talking about whether AI might “kill us all”. Industry leaders like Dario Amodei of Anthropic have issued calls to “pace the frontier” or slow down the speed of AI development so we can get a better handle on some of these risks. But then there’s China. Many worry that unless we get China on board, a slowdown from just the US AI labs alone won’t be enough and will actually just give China a chance to forge ahead. So what do you make about all this and how has China been reacting to these calls for an AI slowdown?
Kendra Schaefer (01:07)
That’s a really good point to bring up. I think the most critical point I want to make is that the argument that we cannot slow down AI development because China isn’t slowing down is a bit of a logical fallacy. I think when we say, or at least when the Trump administration says that we can’t slow down because China’s not slowing down, what that often means is we can’t place any regulations on our AI companies because China also won’t regulate AI, and that is just not true, right? China has every intention of regulating AI companies. In fact, as you know very well, I think it’s already regulating domestic AI, right? I think we could probably maybe get a little bit more into that later. And Xi Jinping said this year at the World Artificial Intelligence Conference in his keynote address that while China supports AI, they also fully intend to regulate it. So it’s a little bit of a false equivalence there, right?
And I think the reason that’s true is because China doesn’t really consider regulation a slowdown. Regulation in China goes hand in hand with tech development. And of course, China’s one of the most heavily regulated network environments on earth. Chinese regulators often talk about regulation as a net positive. The idea is that once they put guardrails around the industry and define what the industry is allowed to do and isn’t allowed to do that removes uncertainty for companies. It prevents industry from running off down unsafe pathways or pathways that are unhealthy for the market and just wasting a bunch of time, doing something that the state is gonna ban anyway, something like that, right?
Of course, none of that is to say that China doesn’t recognize the difference between poorly executed regulation that does hamper the market and well executed regulation. But obviously they know the difference between those two things. But generally speaking, well-executed regulation is seen as something worth pursuing. And that’s obviously not how the US tends to view regulation, particularly under the current administration.
So I think this view that China’s view that regulation and innovation can exist in reasonably healthy tension has been borne out by the country’s experience in the platform economy, right? We’ve said for years that heavy censorship regulations placed on Chinese technology companies would prevent China from being competitive in digital technologies, and that didn’t really prove to be the case. Obviously, China has a very strong internet sector. So the first point I wanted to make was just that the way that we are talking about a slowdown and the way that China is talking about a slowdown mean two very different things. They often get mixed up and conflated.
Kyle Chan (04:07)
That’s very helpful. Do you think to take the other extreme, China just sees these calls to do a slowdown as a way to hold back China? Especially coming from Anthropic, which has been one of the most prominent voices around things like export controls and efforts to frame this whole competition as one between like democratic AI versus authoritarian AI. How do you think that lands in China?
Kendra Schaefer (04:36)
I think you’re exactly right. China didn’t reject the idea of regulation. It didn’t even reject the idea of it slowing down its own model development should it choose to do so at some point in the future. What it did reject was the idea that China and the US, at the behest of a company that has called China’s AI development a risk, right? That the US and China should, at the behest of Anthropic, should work together to slow down, right? And as you know, Anthropic’s been pretty vocal about support for export controls, keeping Chinese AI developers behind the frontier. So I think cooperating on that basis or slowing down on that basis or coordinating on that basis was a no go for China.
And I think it also, beyond just Anthropic, I think what China was rejecting was the idea that it should be incumbent upon the second place player to slow down. We saw a lot of conversation in the China environment that, if Anthropic wants to slow down, it can slow down. It’s the one that ought to be the one slowing down, right? That China was actually struggling to keep up and was having to expend all of this effort, given some of the roadblocks that the US has placed in front of it, and that it is always at risk of slipping too far behind.
There’s always this debate about, is China having an easy time keeping up with the frontier models? But we listen to a lot of Chinese podcasts from tech industry leaders in China. And there’s no doubt in our mind at least that there’s certainly constraints on model development there, in part due to export controls. And there are, struggles to get enough financing, right? They cannot throw the same amount of money at this infrastructure as US companies do. So, I think China essentially said, look, Why don’t you slow down, right? If you’re the one running faster than we are and we’re actually, our job is to try to keep up with you and not to slow down at your request.
Kyle Chan (06:53)
Yeah, totally. So it’s like, i if there’s a race and if there’s a winner right now, like then wouldn’t it be incumbent on the, the one who’s leading the pack to send that signal. So I think that’s that’s some of the messaging that we hear. Yeah, well okay.
So if the slowdown idea has been like too narrowly scoped in some ways, like what about AI regulation more generally? Do you think that Beijing will start to enforce and require broader safety regulations and protocols for Chinese AI labs, especially for new model releases? And I think the comparison here is, in the US, we now have, some voluntary framework, “voluntary”, though a lot of the details are not clear, for pre-release testing of new models. The general idea here is before you release a model to the public, you want to test for certain capabilities, certain risks, see if the safety guardrails work as designed. And then you might be able to, after passing through certain criteria, then you might be able to release this model to the public.
And right now, China has a version of this pre-release testing where AI services of a certain size do need to be registered before they can be officially launched for the public. But a lot of this seems to be more about content or maybe not about specific safety testing as we might think of it in the US. So I was just wondering if you can help us unpack how this works in China. What is this model registration process and could that be potentially repurposed for the AI safety testing that I think people more broadly are thinking about.
Kendra Schaefer (08:54)
So China’s existing model safety regime was designed for a pre-generative AI era. It was initially made prior to the release of ChatGPT and the explosion of LLMs. And so one of the interesting things about Chinese AI regulation is that it has had to, evolve so quickly. Chinese regulators try to act really quickly when a new technology comes out. And so you’ll get, new regulations only six months or a year after a new technology has been released before anybody’s really sure what that technology is gonna do or how it’s gonna evolve, but the state will, step in and try to regulate it a little bit. And that’s that’s essentially what happened with AI, right? Chinese regulators started to regulate it with this registration scheme, which I’ll get into more detail about in a minute. But then LLMs change the game for where the risks of the technology actually were. And then, as these LLMs get more and more powerful, more and more powerful, the risks have evolved and have changed again and again. Now regulation is lagging just a couple of steps behind, or at least Chinese regulation is lagging a couple of steps behind there.
So originally the idea was that if you were going to serve some model to the public, in other words, if you’re going to have an algorithm, any algorithm, didn’t have to be a chatbot or LLM, but say an algorithm that’s embedded in an app, an app on your mobile phone or something like that, then before that was released to the public, or I think the original rule actually, to be boring, was within 10 days of releasing that model to the public or allowing the public to interact with that tool or that app or that chatbot, you had to fill out a piece of paper and send it into China’s cyberspace regulator and say, This is what my model is, this is what it does, this is the app that it’s in, and they would issue you a little registration code. And in some in some cases, they would have you go through a safety test on the model.
But those safety checks, even those early safety, or even the safety checks for much less advanced types of algorithms than the ones we’re dealing with now, were pretty weak sauce, to be honest. And regulators imposed that process before they had fully figured out how to conduct a thorough safety test. And so the only thing they really knew how to test for was whether the model says things that are censored. Right? They had they had that test down because they’ve been testing computer output for censored content for 20 years, right? So but they didn’t know all the other issues of AI safety or what they should be looking for or concerned about it was a little bit unclear and still being formulated.
So mostly in the early days, those tests were pretty vague and often involved little more than regulators playing with it to see if it would, generate some censored content or if it would spit out some inappropriate content. But of course, that has now changed.
So the second point I want to make about the registration system is that it is only, there’s a gap in it. It’s only designed to prevent or give regulators control over algorithms or models that are, as you said, interfacing with the public inside of China. What that means is it doesn’t control it doesn’t prevent or apply to a model company that is taking an open weight model and uploading it to the international internet for other people to download. That is completely outside the bounds of the regulation. And in fact, there is no targeted or dedicated Chinese regulation that deals with that particular process, the release of open weight models. So we’re not sure if that regulation is gonna if that regulation’s gonna come out or what. That’s actually one of the big questions here now.
So now we’re in this other new paradigm, which is that these models, these open weight models, can be potentially quite dangerous. And once they’ve been released, you can’t really recall them. Once you’ve uploaded them online, there’s no take back. If a bunch of people download it and it turns out to be dangerous, there’s very little you can do about that.
So we’ve heard that behind closed doors, regulators are asking without formalizing regulation, that they’re essentially asking model companies to just get the nod of approval from them or go through some, informal or unwritten safety test before some of these new models are released for now. There aren’t that many labs in China that can put out models like this, so it’s fairly easy to control at the moment without formal regulation. But that is the big question. Will China formalize some process by which labs releasing new models, either, including to international platforms, have to go through some right, some safety test to do that.
Kyle Chan (14:14)
Yeah. That’s super interesting. It gives us a sense that this is on Beijing’s radar, even if they don’t have a fully fleshed out regulatory framework for dealing with some of these safety testing issues, especially more the more recent ones, and especially for dealing with like full blown model release, not just like a China market issue for AI services. Even if they haven’t come up with something fully, fully baked yet, like this is something they’re thinking about. And there is a sense of like, okay, you can’t just if you’re a major frontier AI model developer in China, you don’t just it’s not just a free for all in terms of being able to post something on Hugging Face and just have it be used by anyone. Right, right.
Kendra Schaefer (14:56)
Exactly. Exactly. There’s also the open question of what is safety and that’s not just a China question, right? Nobody has standardized, not in the United States, not in China, the ultimate test of model safety, the protocol through which you run every model to absolutely ensure that it’s not gonna do X, Y, or Z. That’s dangerous. Those protocols have not been developed. There are lots of great researchers working feverishly on trying to figure that out, but there isn’t some international standard yet. And so, whether it’s a safety test in the United States or it’s a safety test in China, regulators don’t really know what they’re doing. Nobody really knows quite what they’re doing yet. So I think that’s also another, it’s obviously another problem, right?
Kyle Chan (15:41)
To say the least. And in theory, there would be not only some clear definitions and criteria for doing the safety testing, but that this would be coordinated to some degree across countries that are releasing models and that this is not just like the US has one set of rules, China has a totally different set of rules, and we just have models like running amok doing things that, were okay in this regime, but actually not okay in this one.
Well, speaking of models running amok, AI agents are everywhere now. And they’re very useful. They could do really incredible things, but they can do very worrying things as well. And so we have seen obviously a number of recent incidents in the US, like the Hugging Face incident, where AI models from OpenAI or Anthropic, or pretty much all the major model makers these days, have had AI systems, with multiple agents running autonomously and then accidentally hacking other companies or even now other government systems. And so the question then becomes how is China seeing all this AI agent activity and are they trying to develop policies or regulations aimed at least gaining some control over how agents operate?
Kendra Schaefer (17:15)
So one of the easiest ways to figure out what China’s going to regulate is obviously to look at the gaps in the existing regulation as they stand now. And China, as we’ve been talking about, has been regulating AI since before it was cool to regulate AI. And so there are maybe less gaps, I would say, in China’s AI regulation regime than in ours. But agents is a big area where there are a bunch of gaps and the behavior of autonomous technology is just something that’s that’s very unregulated. So I’ll give you an example of a couple of areas where we can see gaps and then whether or not China will regulate there is still an open question.
But one gap is in reporting requirements for agentic behavior. So current cybersecurity incident reporting requirements, of which there are many in China, are predicated on old school cyber attacks and breaches. In other words, those rules say that after an attack has happened, after some material damage has occurred, after personal information has been leaked, then you must report that incident to a series of authorities and platforms. But what about when agents behave in a way in a lab that is dangerous? Or they break out of containment in some way, or they attempt to hack but do not succeed in hacking some third-party platform. There is no requirement whatsoever for any reporting by labs that has occurred, at least not on the books. Like I said, there might be some off-books requirements at the moment, but there’s no requirement for reporting of behavior that has not resulted in tangible real-world harm. So that’s an interesting place where regulators might be looking.
A second area that’s become very hot, a hot topic over the last couple of months, I’m sure you have heard all kinds of chatter about it, and it’s also becoming a hot topic in China is the question of model liability. So in other words, who’s to blame? Which insurance company has to pay money when a model hacks something or ruins something or breaks something? Is it the model maker? Is it the person who directed the model to do that? Where does that line get drawn?
And what’s interesting, of course, is China struggled with the same issue with autonomous vehicles. So I imagine that we’re probably gonna see something quite similar to some of the breakdowns around some of the conversations that we’re having around autonomous cars, where the question right now for the autonomous vehicles is when a car is at L2, it’s not very autonomous and a human is mostly in control of it, well, then it’s the human’s problem to ensure that the automation doesn’t do something that it shouldn’t do. But when an automaker is saying that this tool is allowed to run on a completely automated basis and that humans should not have to ever manage it and it’s designed to and sold as a product that functions entirely on its own, well now liability starts to go back to the car maker or the software maker, right? So this is a really interesting question. When it comes to things like agents and autonomous software.
A third thing that I’ll bring up, which I think is interesting, and we’ve also seen some conversation around, is Chinese regulators love the idea of traceability. Every comment that is made online, every program that operates online, every person that owns a server, right? They don’t like, there is no expectation or presumption of anonymity on the Chinese internet. And so because of that, Chinese regulators love to give stuff IDs. Every website in China has an ID number, right? Has to receive an ID number so that it can be identified and traced. Anything that happens on that website can be traced back to an ID number and assigned responsibility can be assigned to a particular ID number. Every blockchain network in China has an ID has to get an ID number. Now we know that algorithms are registered. They have to get ID numbers.
Now there’s talk about how do we assign IDs to autonomous programs, autonomous systems and agents. And particularly when it starts to get into the blurring the line between an agent that just gets fired up by a system for five minutes and then disappears, versus an agent that is perpetual and is existing for a long period of time and potentially right lives in your system on an ongoing basis and is acting on your system on an ongoing basis.
So we’ve seen some interesting technical conversations around could China modify domestic DNS networks, those are the networks that assign IP addresses, right? To also be able to assign identifiers to agentic to agents and agentic code and autonomous processes that are running around the net? Do we need to make any protocol changes to do that? So we saw that in a couple of interesting policies, a couple of interesting policies where that was under discussion.
Those are the types of policies and regulations that are being talked about. Some of them are some of them are how can humans have some oversight over the release of these agents? Some of them are how can the legal system deal with the impacts of these agents, and some of them are how can we technically, how can we change our network architecture to deal with a future where there are more agents online than humans.
Kyle Chan (23:06)
Yeah. This is super helpful. I think it’s really interesting to use the comparison with autonomous vehicles and robo taxis and like at what level of autonomy are we dealing with and then how who is accountable at what stage? And then having basically like a license plate for these agents as they’re roaming around like the digital superhighway and doing things that are fine, but then maybe doing some things that are not so good. And then if they’re doing things that are not so good, how do you trace that back?
Kendra Schaefer (23:37)
Yeah. It’s a really interesting question technically too because, we all know this from looking at that the Windows terminal from a million years ago, every process, every program running on your computer has an ID number, but that ID number is just for your computer. It only works when it’s on one server and then that process doesn’t on its own decide to run off to another server and do something else or run off across the internet and perform other things, right? That process doesn’t its ID number doesn’t stick with it in the same way. So it’s an assignment. So the interesting question is we have these autonomous creatures scampering about.
Kyle Chan (24:15)
Yeah, exactly. And I also do think about how like traceability internationally too. I if there’s something that would be like required to be embedded in a model’s output for agents where if you saw some activity on some website outside of China, you like would you be able to trace that back to, that was, a Kimi model or that was a DeepSeek model ultimately. Yeah, I can see also reasons why the companies would be reluctant to want to have that traceability. Like they don’t want to be on the hook for who knows what their, people are using their models for. Yeah, it’s like less is less is more. We don’t we don’t we don’t need to know everything. But
Kendra Schaefer (24:49)
Right. Less is more.
Kyle Chan (25:02)
Yeah, so taking a step back a little bit, so there’s been like an interesting shift, I think, in the AI risk discussion in the US. There has been an elevation of certain kinds of risks, at least among the general US public. And I was just wondering, how do you see China assessing these sorts of risks? Are they having the same kinds of conversations about some of the same issues that we’re having in the US? Or, what is Beijing really concerned about? What are the AI labs concerned about and the Chinese AI community more generally?
Kendra Schaefer (25:37)
There’s significant overlap in how China’s both policymakers and labs see risk. Ultimately, I think that China’s definition of risk, and one of the points of friction that I think the US and China will encounter frequently over the next decade or so, is that China’s definition of AI risk is going to be defined on its own terms based on its own understanding of what a risk is, right? But that definition is starting to emerge so we can start to feel the edges of it a bit.
I think part of China’s definition of risk is rooted in its understanding of the generalized risks of digital technologies. The US doesn’t really have a 20, 25-year framework for regulating the internet. We have separate laws that don’t necessarily connect to each other and don’t form a cohesive stack of laws and regulations to govern our internet environment. But China has had a stack that’s interconnected of laws and policies to govern the net for a very long time, for several decades now.
And those policies and laws are directed at controlling specific types of risks that regulators in China for the last 20 years have been concerned about, right? One of the top risks is usually censorship concerns. Is this digital technology going to do something or say something or disseminate something that the state does not approve of, either because it’s anti-party or simply because it is actual genuine misinformation? It is incorrect financial information or medical information or whatever it is.
The second concern, of course, is data security issues. Either leaks of the personal data of Chinese citizens or exfiltration of Chinese citizen data, or sensitive state data, right, that regulators think is relevant to national security. There are cybersecurity risks, right? Is the technology going to be used to hack, sensitive sites or critical information infrastructure?
China’s also pretty heavy on the addiction and dependency concerns, right? They have a lot of concerns about things like are kids gonna become reliant on this technology? Is it going to shape their education in a negative way? Is it going to create, do you remember those internet camps? They had internet addiction camps in China ten years ago that were like really big. People would send their kids to internet addiction camp and you
Kyle Chan (28:21)
Video game addiction, yeah.
Kendra Schaefer (28:22)
And video, yeah, and video game addiction camp, you’d just like run on a treadmill and someone would yell at you.
And then of course China perceives some of the risks as like economic risks, threats to labor through job loss or through exploitation of labor law, that stuff. So those that bucket of risks, I think, has been true not just for AI, but it’s been true for almost every new technology revolution that has happened. It was true for social media, they’re worried about all the same things. It was true for blockchain, they’re worried about all the same things. It was true for, and on down the list.
So the interesting piece there, so all that applies to AI, but then there’s this extra layer of concerns that are specific to AI as a technology only, which regulators are less comfortable with and haven’t fully, I would say they haven’t fully solidified. That includes questions like: How do we worry about agents breaking out of containment and loss of control? They’re clearly worried about loss of control, but it’s not a tradition that’s been discussed and decided. We’re not even sure, they’re not really sure where the boundaries of loss of control are. Same as us. We don’t know where the boundaries of loss of control are, what regulators need to be regulating about.
And then there’s this new wave of cybersecurity attacks that are only relevant to AI, things like prompt injection attacks and stuff like that, where like we didn’t have to think about that before. And now that’s a whole new risk factor. Now what do we do? So there’s the new stack of technical risks. And I think the US and China on the technical risk side, we’re having generally the same conversation about what the technical risks are. I’m sure there’s some differences, but for the most part. Right, because these are engineers who are outside of the bounds of geopolitics, going, that seems like an engineering problem we should probably deal with. And that, that stuff I think is probably pretty similar.
And then finally, I’ll say the last bucket of risks I think is interesting. It’s rooted in geopolitics, right? And the US and China have conflicting definitions of geopolitical AI risk, very conflicting. In fact, I don’t think we’ll ever get on the same page on what is a geopolitical risk in terms of AI. The US, of course, thinks that China developing AI is the geopolitical risk. China does not see the geopolitical risk of AI in that way. But China sees and has increasingly promoted the idea that it’s a major risk to China and the world if we move towards a future in which only two United States closed model companies control access to this critical future technology. That’s the risk. AI hegemony by Anthropic and OpenAI, and that’s been the drumbeat. And of course, the United States is not going to be signing on to that definition of geopolitical AI risk, at least. At least not this administration.
So I think that’s the landscape. And I think understanding how China perceives those risks is so important because the way that China regulates AI, the way it collaborates, what it’s willing to collaborate with the US on AI, what it does next on AI, understanding that, being able to predict that, understanding what China has done when it has done it, is completely dependent on understanding how China has made its risk assessment. Not our risk assessment and China’s not aligning with it, but what is China’s risk assessment on AI, right? And that is, ‘cause that’s what they’re gonna take action on. So it’s probably worth knowing.
Kyle Chan (32:12)
Yeah, definitely. I just want to underscore that point about it’s like we may hope that China acts in a certain way or regulates in a certain way, but what drives that regulation, that behavior and that action towards the technology is gonna ultimately be driven by how they see the risks and how they see the best approach to addressing them.
And so, we can have a conversation or dialogue about, this and as you point out, there’s a lot of overlap. In terms of the discussions already underway. Some of these are technical risk concerns. And then there are obviously some where it’s like literally mutually incompatible if on some of these geopolitical issues. So, maybe that will just not be resolved. But yeah, seeing that as the main driver, these like internal views of risk ultimately.
Kendra Schaefer (33:04)
Well, we see that in almost every area of policy too, because it’s like one area where we’ve also seen a lot of that is data centers. If you read Chinese policy on data centers, it doesn’t sound anything like our policy on data centers, right? It’s like totally right. You know more about this than I do. You just did a presentation about this the other day, which I was very unhappy to have had to have missed. I did want to see that, but I’ve seen some of your work on data centers, which is of course excellent.
China is quite concerned about, they’re quite concerned about the geographic layout of data centers and ensuring that data centers are not built haphazardly. They’re worried about data center overcapacity, right? Not in the sense of having too much compute, but in the sense of things like, some local government official hears that the state wants him to build data centers and that AI is a big thing. So he builds a data center on a empty floodplain that is constantly, that where there’s no electrical cabling and it’s a terrible place for renewable energy and nobody nearby is a tech company and therefore it is like why would you build that data center here, right?
We don’t think that way. We have a totally different perception of what it means to build infrastructure and what you have to worry about when you’re building infrastructure. And so sometimes I see some of my clients pick up like a data center policy and go, this sounds like they’re telling people not to build data centers. And I’m like, no, they’re actually telling them to build, but they’re just saying, please do it considering all of these other factors, do it rationally, do it carefully, do it in a way that is, going to be sustainable over the long term, is going be useful 10 years from now. Don’t just throw something up to grab some state funding. So those shifts in perception around internal problem management again, I think is always really illuminating.
Kyle Chan (35:01)
Definitely. And this is where it really helps to have a lot of the context outside of just the AI space where it’s like, okay, what China’s trying to regulate and prevent in data centers, like reminds me a lot of the overcapacity issue for other industries where they don’t want a bunch of battery plants just mushrooming everywhere that are making low quality batteries and adding extra capacity. They want the higher quality stuff that people actually want to buy and that will level up the country. And for the data centers is similar.
And then on the AI issue, I really like your point about how a lot of these like risk issues are like technological risk issues that China has been trying to grapple with pre-AI. And so AI obviously adds some new dimensions to this, but there are some issues that span across these technologies that they’ve been trying to grapple with. So it’s like not starting from zero, essentially.
Kendra Schaefer (35:57)
Exactly. And I think that’s actually the biggest gap between the US understanding of China’s AI and how it actually is on the ground. China’s existing framework is very, very big. They have foundational laws that govern what you are and aren’t allowed to do with data, what you are and are not allowed to do with certain kinds of networks, how you can and cannot govern. Right, all of these other things.
And then on top of the legal foundation, this is probably even less understood. There are all of these technical network systems that China has implemented over the years to control various pieces of its domestic networks. And that includes things like here’s an example, China’s real name registration system, where they require all social networks, right? If a new user signs up, they have to provide a government ID. You can’t have an anonymous user account on a Chinese social network.
And because you can’t have an anonymous account, that actually creates a situation in which no agent can just go by themselves without authorization and without being traced back to a person, make its own social media account and start posting. Whereas here, you absolutely could just spin up a zillion email accounts. You could write log into as many networks.
So there’s not only a difference in the way that China is looking at its own legal system in terms of AI, it’s also looking at its own network infrastructure and existing checks and technical checks and balances on what autonomous systems can and cannot do on domestic networks. And that is gonna give them a different perception of where the risks actually are. For them, there are certain risks that might be lower than they are for us simply because there’s such strong controls over the way that networks function and data and packets and traffic across those networks. So anyway, it’ll be very interesting to watch that play out.
Kyle Chan (38:00)
Totally. Yeah, that’s all that is brilliant. So now the big question. Given all this, is there any hope that the US and China could do anything somehow, somehow, on AI safety? Like what steps could they take that would be both meaningful and feasible given, how deeply the two countries distrust each other, how fiercely they are competing in a whole bunch of areas, including in AI. Like is there any hope? How optimistic are you?
Kendra Schaefer (38:38)
I’m actually optimistic that there is a pathway. I’m less optimistic that it’s going to be walked, but I do actually see a pathway here, which isn’t always the case. I’m also oddly optimistic, and the Xi-Trump meeting just happened. We came to an agreement that we were gonna do an AI dialogue with China every, I think the next one is coming up in November. And we’re gonna sit down and talk to China about AI safety.
And we read the Chinese readout of that. And what it sounded like the Chinese readout was saying was that China wants to create a common language of risk with the United States, right? That dialogue mechanism will be used to define what they should even be talking about. What is risk? What should governments be, comparing notes about and working together on? And I think that’s great. That’s about as good as we could have hoped that to go. A lot of people said that it was very light on deliverables, and that’s true. And then all that happened was somebody made a plan to make a plan, and that’s true.
But actually, the first step towards cooperation on safety, if that’s possible, is to come to some shared definition and scoping mechanism: these are the issues we talk about in these forums, these are the things that we care about.
So questions like what constitutes dangerous autonomy? When is it scary that a system can do something by itself? Right? That addresses issues like the recursive self-improvement issue where AI models are making themselves better and then making themselves better and making themselves better. So at what point, how do we even deal how do we even deal with that? What should we even talk about as far as that’s concerned? How do we, warn each other when something has happened there that shouldn’t?
What level of model overreach should be escalated to a security incident. In other words, my autonomous agent tries to hack your military network. Do I need to call you? Right? Do I need to let you know? That’s probably a yes. But then there’s this middle of the road, okay. Well, we were testing a system and it tried to hack one of your SOEs, your provincial SOEs. Do we need to, does that need to be a security incident? So calibrating on that front.
And then how should these major risks and incidents be classified? Does there need to be some classifier system or assignment system or ranking system for those incidents? And do we need to discuss different training, model training pathways and the dangers of each of those pathways and come to agreements about not going down certain roads if those roads prove to be, prove to be unsafe?
The hard part is in order to have, I think, those technical conversations, right, we’ll have to prioritize the technical issues over the ideological issues. That’s the part that I think is hard, right? As we already talked about, the US and China aren’t gonna agree on geopolitics. And so I’m afraid that if the US and China try to get to some solution on technical safety and try to shove each other’s definition of ideological geopolitical risks down each other’s throat. If the US tries to insist that China should operate under the assumption that its own AI development is inherently risky to the world and tries to galvanize the world to prevent China from developing AI, and then also tries to collaborate with China on technical issues. I’m not saying it’s impossible. It certainly happened before and under those tough conditions, but it’ll certainly be harder to do, right?
So the path forward as I see it is for at least for now, at least for now, ideology won’t wait forever, but at least for now, if we really think that AI safety is a consideration, and I personally do, that both sides should prioritize the technical issues first. And then go back and deal with all of the other stuff because we’re gonna be playing patty cake about that for probably forever. So let’s get the technical stuff out of the way. If these are really existential risks, then we should probably prioritize at least coming to some technical agreement. So that’s the path forward, I think, is probably the most logical. Do I think the US and China will walk that path? I am less optimistic.
Kyle Chan (43:28)
Well, how about like longer shot things like an international agreement on joint monitoring or evaluation or safety testing or things like verification technology to track chips. I’m just throwing this out there ‘cause there are some people who are like trying to think about different long shot things to be aiming for. Yeah, and if you had a view on any of those.
Kendra Schaefer (44:01)
I think all of those are useful things to think about, but they’re putting the cart before the horse because we don’t, we are so far apart right now on what is a risk. What is a risk? We haven’t even agreed whether or not recursive self-improvement is a risk. There’s no agreement between the US and China on that. There’s no consensus. There’s not even consensus in the US that there’s a problem on that, right? It’s like, so we don’t really have our own consensus.
At least sitting down, I think first and foremost, sitting down and saying, Okay, here are the areas where we’re going to talk. And then in those buckets, yes, we should obviously try to work to some international agreement, but I’m not even sure what the buckets necessarily are at this time. Now there’s been some great papers by other researchers on how we should separate those buckets and how we should probably pursue those conversations. I think that’s probably the place to start.
So I hope that the US-China AI dialogue can be used for that purpose. Let us get, let us get down to risk and let us try to let us try to find areas where we both agree that this is a problem. And then that provides a platform for us to say, okay, well, if we both think that’s an issue and it’s not extremely geopolitically touchy, and then we can manage not to set it on fire with geopolitics. In the course of having that conversation, we can move towards some agreement on that front.
Kyle Chan (45:31)
Totally. Well, yeah, I totally agree. I think it’s good to kinda think big, but then in practical terms. Yeah, especially if we can make this like not just always zooming out to the global struggle between, it’s like, well, what about these AI agents today that are doing weird things? And yeah, yeah.
Kendra Schaefer (45:46)
And if you ask me which one we can solve first: what can we solve first? Like do you think we could get to an agreement first on, cybersecurity incident reporting or the battle between capitalism and communism? I have I have a bet for you on which one would go through the pipes first, but yeah.
Kyle Chan (46:19)
Exactly. Exactly. Well, well this has been super helpful. Yeah, and I think the craziest thing about all this is the technology is changing so fast that some of these incidents and the capabilities of especially agentic AI were just only on the horizon, just last year, it feels like. And they’ve really taken off more recently. And so yeah, it’s just like a good moment to now like ground ourselves back in, what is actually happening in each country? How are we each trying to deal with some of this? Yeah, and realize that we don’t have to start from scratch, and try to, take baby steps.
Kendra Schaefer (47:10)
Yes, yes, baby steps.
Kyle Chan (47:12)
Well, this has been really helpful. I’ll include links to your website and maybe your trivium china page. Is there anything else I should include in the show notes and how can people follow you in your work?
Kendra Schaefer (47:32)
Well come on over to TriviumChina.com. We do have a free daily newsletter. So I’m sure that since you probably get no newsletters in your inbox every day, and I’m sure you have no information, you should totally sign up for ours also. And if you’re a company, I’m the chief editor of Trivium Tech Daily. We monitor some of these conversations happening in China. We publish every day for enterprises, hardware and software companies on those issues. So we’d be delighted to talk about getting you signed up for a subscription.
Kyle Chan (48:06)
Trivium does amazing work. You guys have awesome podcasts as well. So shout out to the podcast. Well, thank you so much, Kendra, for a really amazing conversation.
Kendra Schaefer (48:16)
Great to be here.
Kyle Chan (48:19)
If you like this episode, please rate and subscribe on YouTube, Spotify, or Apple Podcasts. You can find episode transcripts and more information on the High Capacity newsletter at highcapacity.org. I’m your host, Kyle Chan. Thanks for joining and see you next time.