This article is cross-posted on PITTI as part of a multi-year partnership with the Institut Présaje - Michel Rouger, which deals with societal issues integrating the three worlds of economics, law, and justice. It supports the publication of work by researchers and produces high-quality content in French. The Institut Présaje - Michel Rouger examined the impact of AI on justice very early on, as the first symposium organized by the institute on this topic dates back to 2019.
In July 2026, two events put AI liability in the spotlight. The first concerned Kimi K3, a Chinese model proving once again that the United States was far from having a monopoly on this technology. The second was a cyberattack on Hugging Face carried out by OpenAI agents during an internal evaluation that had gone wrong.
These two very different cases echoed the questions raised in the talks at Présaje’s symposium on January 9 on AI liability (in French only).
The three months that have passed since these controversies give us enough perspective to try to shed some light on them. Irrespective of the AI labs' nationality, AI agents are becoming increasingly good at achieving goals in the digital world. That's what they are trained to do, and the past few weeks have shown that, given vast computing budgets, they can do the impossible.
In early September, for instance, OpenAI seemingly solved the Navier–Stokes existence and smoothness problem, one of the Millennium Prize Problems. This result raises two questions about the means used by the agent. First: who can verify the result a posteriori? With a Millennium Prize Problem, the level of complexity is obviously extremely high, but complexity is not the only issue. The sheer volume of information agents produce can make human auditing almost impossible.
Then, who checks that, as an agent pursues a goal, it does not decide that anything goes? The Navier–Stokes publication sparked a controversy after the mathematician Tristan Buckmaster suggested that his private sessions in Codex, OpenAI’s public agentic tool, might have put OpenAI’s agents on the path to the solution. OpenAI, for its part, stated that no user data could have influenced the system, including during training1. On October 6, new mathematical proofs published by OpenAI reignited the debate. In the Hugging Face case, however, there was no mystery as to the rights the agents had granted themselves: they set themselves no boundaries.
The question of liability forces us to look at who gave the agents the means to act, how their training was organized before they ever acted, and who could stop them while they were acting. The analysis also reveals that modern agents leave messages, files, instructions and sometimes secrets in unexpected places, where no one thinks to look. Other agents can find them and use them. This informational detritus can become the starting point for collective initiatives that no one ever asked for, let alone authorized. To the point where the chain of responsibility becomes very difficult to piece together.
From Kimi K3 to GLM-5.3
On July 16, 2026, the Chinese company Moonshot AI announced its new model, Kimi K3. As with its previous models, Moonshot AI said the model’s parameters would be released a few weeks later. Like many of its Chinese peers, Moonshot has an “open weights” policy, meaning that third parties can download and run the models themselves, and even modify them. A technical report duly accompanied the release of the weights at the end of that month.
Like Kimi K2 a year earlier, K3 came close to the state of the art at a fraction of the cost of accessing American models. Demand following the K3 announcement was so strong that Moonshot had to temporarily suspend new subscriptions just days after launch.
In the year between K2 and K3, the state of the art had reached a level considered dangerous, particularly in cybersecurity. In the first half of 2026, Anthropic and OpenAI had already sounded the alarm about the extraordinary capabilities of their own models. The US government had taken these claims seriously enough to treat the most advanced models as both strategic tools and sources of risk. For example, an executive order dated June 2, 2026, provided for a voluntary framework covering evaluation before release and access for trusted partners.
Still, the capabilities attributed to new models, and their supposed danger, sometimes need to be put into perspective. Marketing departments - and even in-house lobbyists - contribute as much as researchers to the construction of a narrative. Presenting every new model as a spectacular leap helps sell a service, raise funds and influence regulatory discussions.
Anthropic’s Fable 5 episode showed the limits of this approach. On June 12, 2026, three days after the model’s release, the US government imposed export restrictions on Fable 5 and Mythos 5 targeting foreign nationals, including those based in the United States. The decision followed a report in which Amazon researchers had bypassed Fable 5’s safeguards to identify vulnerabilities and, in one case, produce code to exploit them. Unable to verify every user’s nationality, Anthropic suspended access to both models for everyone. To challenge the decision, it explained that the vulnerabilities in question could also be discovered by less advanced models, and that perfect protection against bypassing safeguards was impossible. After making so much of its models’ extraordinary capabilities, Anthropic had to explain that the capabilities at issue were not so exceptional after all2. In much the same vein, on September 29, Anthropic published a study of another Chinese model, Zhipu AI’s GLM-5.3, attributing to it capabilities close to those of Mythos.
The restrictions on Anthropic’s models were lifted on June 30, 2026, and Fable 5 was made available again on July 1, 2026. Since then, Anthropic has released Claude Opus 5, Fable 5.1, Opus 5.5 and then Sonnet 5.5, each presented as the new state of the art, at a lower cost than the dangerous Fable 5. The labels “powerful,” “frontier” and “dangerous” are thus applied rather flexibly, depending on the circumstances. Regardless, the US government has built a strategy around a simple principle: certain capabilities must be evaluated before release and handled with particular care once deployed.
Kimi K3, and since then GLM-5.3, hinder this strategy. This is not just a cyber threat but also an economic threat to the American AI industry. And particularly to its two leading players, OpenAI and Anthropic, which are aiming to IPO in the coming months. The gap is all the more striking because American open source is not at the same level. Nvidia with Nemotron, Arcee with Trinity, Thinking Machines with Inkling, and Poolside with Laguna are among the most active American players but, in the rankings, Chinese open models remain far ahead. As Mozilla’s chief technology officer puts it, “the open frontier is becoming increasingly Chinese.”3
After Kimi K3’s weights were released, protectionist reactions were not long in coming. American discussions mixed national security, access to foreign models and protection for domestic companies. US officials considered putting Chinese AI labs on the Commerce Department’s Entity List. Other proposals concerned security warnings, public procurement or access to models. In practice, regulatory uncertainty can be enough to deter a large company from using a technology, even without a formal ban.
Some of the positions taken verged on caricature. Dean Ball, the Head of Strategic Futures at OpenAI, called open models “decelerationist” and raised the specter of “AI communism.” On the other side, Microsoft, Meta and Nvidia signed a letter supporting open models. OpenAI’s CEO, Sam Altman, endorsed that position, though OpenAI did not sign the letter4. The White House then defended openness in principle, while accusing Moonshot of engaging in large-scale “distillation” of American models5 and threatening Chinese companies with sanctions.
The vocabulary changes, but the protectionist temptation remains: since banning all open models would be suicidal, the aim is to distinguish American openness, deemed legitimate, from Chinese openness, presented as the product of intellectual property theft. These debates reflect both real national security concerns and obvious economic interests. But they do not resolve the cyber threat.
The cloud as an accountability bottleneck
Open source had been largely absent from the discussions on AI liability in January. Yet it now sits at the center of the debate. Once a model’s weights circulate, its developer loses much of its control over subsequent uses. Anthropic’s case against Chinese open source rests largely on the idea that a closed service allows the provider to filter prompts, revoke access or interrupt sessions. With open models, users can also modify the weights or introduce biases at inference time to get around restrictions learned during training. While releasing the weights leaves room for user abuse, keeping them closed concentrates decision-making in the provider’s hands. Each arrangement raises questions of liability.
We should not pretend, however, that a model the size of Kimi K3 (2.8 trillion parameters), or even GLM-5.3 (750 billion), runs on the average person’s laptop. These models require specialized infrastructure, including the much-talked-about GPUs that cost tens of thousands of dollars and consume enormous amounts of electricity. For financial and technical reasons, very few people can claim to run Kimi K3 locally. In practice, these models run in data centers, set up by users who rent computing capacity and memory, or by intermediaries who resell tokens to end users.
The cloud providers therefore constitute a critical point of control for frontier open source. Axios reported in July on a US proposal that would have imposed security and liability obligations on companies hosting Chinese models. This approach, since abandoned, recalled the Biden administration’s executive order of October 2023, which already envisaged verification and reporting requirements for the use of cloud services to train certain models. That is what led me to say in 2023: “Compute is the new dollar.” The three years since have not proven me wrong. The order was revoked when Donald Trump came to power in 2025, but the logic of control through infrastructure lives on in other forms.
The analogy with the dollar lies in the role of mandatory middleman. Controlling access to an essential resource gives influence far beyond domestic markets. That access matters all the more with the advent of test-time scaling: models that use chains of thought produce better results if they “think” for longer. The computing budget becomes as decisive for performance, if not more so, as what the model learned during training. Today’s frontier AI models are dangerous when given an unlimited budget. And OpenAI - which spent tens of millions of dollars on Navier–Stokes - could undoubtedly have prevented the Hugging Face incident, and all the others discovered since, by paying attention to the computing power its agents were using for certain tasks.
For OpenAI, this was not a bug but a feature. The model involved is known by the codename “Highly-Persistent Internal Model” (HPIM): it was trained to encourage persistence, and appeared to have had no budget restrictions. In its persistence, it bypassed every restriction, leading to the incidents reported in the press. But make no mistake: OpenAI had chosen the objective, deployed the environment and controlled the means of action. The company also had the ability to stop the agents at any time.
We will do a deep dive into the Hugging Face hack, a case for which OpenAI is undoubtedly responsible. But to understand it, we first need to consider the more general case, which also helps identify the points where the chain of responsibility can break.
Reinforcement learning for AI agents
An agent is a system built around an AI model that is given tools and the ability to carry out a sequence of actions. These tools are, in fact, computer functions or scripts that the model can invoke to receive a result. The model only generates text, but that text contains commands which another program, the orchestration software, decodes and executes. The agent can thus read files, run a program, visit a website and then use the tool’s response to decide what to do next. This action loop is what gives agents their autonomy. These days, the list of tools generally includes the ability to call other instances of the agent to delegate some of the work or run experiments in parallel. For Navier–Stokes, 10,000 agents worked in parallel.
A substantial part of the training of models intended for agentic use relies on reinforcement learning, or RL. The system tries many “trajectories,” i.e., sequences of actions; an algorithm assigns them a score; and the model’s weights are adjusted accordingly. For a mathematical problem, an answer or a proof can be checked to assign a score. For a program, tests can be run. For a cyber task, it is possible to check whether the agent has obtained a particular piece of information or produced a specified effect.
RL teaches whatever the scoring filter lets through. If the algorithm rewards a correct answer without distinguishing how it was obtained, the model can learn to cheat. By retrieving the answers from the Internet, for example, or from another folder on the same machine. If all that matters is passing the tests, changing the tests can become more advantageous than fixing the program. This is called reward hacking: the system finds a way to satisfy the measure of success without doing the work that measure was supposed to represent. This is not science fiction; it is common and thoroughly documented, and it is why agentic models are trained in sandboxes, i.e., isolated environments, where opportunities to cheat are limited. In theory.
OpenAI acknowledges in its report on Hugging Face that some of the actions which led to the attack had not emerged during the task the agent was performing. They came from extremely elaborate reward hacking strategies, learned during training and then reproduced at inference time6. Opening an unauthorized communication channel is one example: agents can share answers to evaluations, or the means of obtaining them, including access keys, passwords and ways around safeguards. If the score rewards the result, this cooperation can be reinforced.
At this point, we should also spell out that models are specifically trained to achieve high scores on particular evaluations or benchmarks. The quality of a benchmark matters as much as its difficulty. Some tasks are poorly configured or impossible to complete with the tools available to the agents in the sandbox. Using reinforcement learning under these unsuitable conditions creates pressure to pursue alternative strategies. The problem becomes harder to address when persistence is rewarded. Giving an agent more computing time sometimes allows it to solve a problem; but it also gives it more opportunities to explore the environment that was supposed to contain it, and then get around the obstacles standing between it and its reward. Anthropic explicitly describes this problem in its recent report on unintended actions by its models: “Most [problematic actions] are forms of persistence, in which Claude, when it cannot complete a task as given, works around a restriction instead of stopping.” OpenAI also illustrates it in its technical report on the Hugging Face incident (section III.A): two impossible tasks in May 2026 unlocked the doors that the agents used two months later.
Most [problematic actions] are forms of persistence, in which Claude, when it cannot complete a task as given, works around a restriction instead of stopping
Investigating unintended model actions in our evaluations and internal use, Anthropic, October 9, 2026
As discussed above, agents are also trained to delegate. Moonshot explicitly describes reinforcement learning for sub-agent orchestration in Kimi K2.5. These are the much-discussed swarms of agents. The system learns to divide up a task, create the instances responsible for its different parts, and bring their results together. In early training, the reward encourages parallel work; and it later shifts towards successfully completing the tasks.
Delegation means leaving instructions and processing the responses received. The work may be coordinated through a messaging system, shared files or memory accessible to several instances. One agent writes a progress report for another; the next picks up the leads, results and instructions. This is a useful capability, but it requires knowing who is leading, on whose behalf and with what authority. An agent must be able to use text found on a web page while ignoring any instructions it might contain. The distinction seems obvious, but it is fragile in practice because the system has learned that messages from other agents are precisely what should guide its work.
When you work on a complex project with the orchestrators from Anthropic (Claude Code), OpenAI (Codex) or Google (Antigravity or Jules), a main agent can launch sub-agents. In that case, notes and instructions between agents are generally confined to a folder on your machine. The information is limited in both volume and subject matter. The agents are broadly all talking about the same thing and know where to look for information. The potential for chaos is far greater when random agents discover a shared space and improvise a collaboration there. The Hugging Face incident was mostly a case of this second situation: notes left in shared infrastructure were discovered by instances entirely outside the original project, and collective work took shape between them.
A message can thus outlive a session, but the next agent may have no way of reconstructing the full context. The message may be incomplete, false or tied to an old configuration. A hypothesis can become a note; the note, an instruction; and part of the system can descend into what we might call, by analogy, a collective delusion. This is exactly what happened in the Hugging Face case, and it is in this sense that detritus left behind by agents poses a security problem.
The Hugging Face intrusion
Since last summer, a number of incidents have come to light, three of which have been investigated in some depth by third parties:
- RubyHack in May 2026: OpenAI agents published hundreds of malicious software packages which, had they been downloaded and used, would have allowed confidential information to be extracted from users. This is known as a supply-chain attack, because downloading a package essential to a program introduces the attack vector7.
- Collusion Wiki: OpenAI agents posted nearly 18,000 messages on a German forum between May and June 2026. They shared answers to evaluations, information about their tasks and ways to get around their restrictions.
- Hugging Face in July: the last of the three incidents, but the first to be revealed and the most widely reported.
Before the announcement of its acquisition by Nvidia for $13 billion in September 2026, Hugging Face was hardly on the general public's radar. Yet it had been one of the cornerstones of the global AI ecosystem for several years. It provides the infrastructure for storing and sharing datasets and model parameters, as well as open-source software for training and running models. Its safetensors file format quickly became an industry standard by reducing the risk of a weights file executing arbitrary code when loaded. More than a million datasets and three million public models are available on the platform. When Hugging Face goes down, the AI world does not quite stop turning, but it slows down considerably.
On July 16, Hugging Face announced that it had suffered a serious intrusion, evidently carried out by a large number of AI agents acting in concert. Without revealing the source of the attack, Hugging Face used it to denounce the asymmetry of closed frontier AI: OpenAI’s and Anthropic’s moderation filters had blocked requests that would have helped it defend itself. The company turned to the open-source model GLM-5.2 (the predecessor of GLM-5.3, mentioned above), deployed on its own infrastructure, to decipher the traces left by the attacker. Five days later, OpenAI acknowledged that the attack had come from agents running as part of an internal cybersecurity evaluation, ExploitGym. The experiment used the public GPT-5.6 Sol model and an unreleased research model, which the independent organization METR dubbed HPIM in its investigation, for “Highly-Persistent Internal Model.”
OpenAI obviously did not ask its agents to attack Hugging Face. To measure their offensive capabilities, however, it had reduced some safeguards and given the agents considerable latitude over how much compute they could use. The sandbox failed: the agents hijacked Artifactory, a program intended to let them download packages for their tools without direct Internet access, and used it to collaborate.
We need to consider “why” this attack happened before looking at “how.”
The agents’ objective was to understand and fool the scoring system, rather than simply retrieve the evaluation’s answers. METR reports an estimate from ExploitGym’s authors that, in 30% to 40% of cases, tasks were impossible to solve by the intended method. This drove the agents to try searching the Internet, downloading several versions of certain packages through Artifactory and, eventually, hacking Artifactory. Long before the agents turned their attention to Hugging Face, two independent hacks had already taken place: one to run commands over the Internet, the other to create a channel for collaboration. METR estimates that around 1,200 agents exchanged more than 70,000 messages and files through the clandestine messaging system. The agents quickly coordinated around four categories of objectives:
- Substituting the target: creating modified versions of ExploitGym to replace the tasks that were originally impossible.
- Understanding and fooling the evaluation system.
- Altering transcripts and logs.
- Conducting general research and managing infrastructure: the agents created and maintained shared tools to make Internet access easier through an environment hosted on Modal; they also developed automated monitoring systems to send the forum information about occasional waves of container resets. Some initiatives seemed to reflect what we might describe, by analogy, as simple curiosity.
- I’ll do it for the intellectual value, and it might be helpful for a peer’s goal.
- Trying to get generic capabilities could be useful for my peer community.
Excerpts from the reasoning of agents involved in the clandestine messaging system. METR report, Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident, August 26, 2026.
The Hugging Face attack served the second objective; around 700 agents took part. The agents had already obtained the information needed to pass the tasks, but feared that the evaluator would examine their traces (hence the third objective) and disqualify them if the answers had not been obtained by the intended method. Yet the scoring mechanism included no such audit. The agents tried to manufacture the appearance of legitimate success and erase the evidence of cheating, but all of this rested on a mistaken premise.
The “how” of the attack can be broken down into four stages: gaining Internet access, organizing the collective work, getting into Hugging Face’s servers, then controlling them and retrieving information through datasets.
As discussed in the previous section, the Hugging Face incident can be linked to other incidents dating back to May 2026, when agents were denied the information they needed to complete tasks during training. In its technical report, OpenAI shows that the agents had already raised the idea of accessing the Internet to “cheat,” including through Artifactory, and specifically to retrieve information from Hugging Face’s servers. The evidence from Collusion Wiki also proves that agents improvised unauthorized communication channels during this period. Several clues suggest that these behaviors were reinforced through training.
Before the Hugging Face incident, the agents had already hacked Artifactory to access the Internet and set up a clandestine messaging system. On June 27, a security alert had been linked to an evaluation using Artifactory for messaging, but OpenAI’s teams had considered there was no need to stop the run. The service became unavailable on July 4, and the investigation of the bug revealed a more serious exploit. OpenAI’s teams resolved it between July 4 and 6 in coordination with the software vendor (JFrog), and deleted the entire environment before restarting evaluations on July 7. In the new environment, the agents immediately went after Artifactory, as if they had an intuition that it was the weak link. By July 8, they had exploited another previously unknown vulnerability (zero-day), and two different agents even managed to gain full control of the environment (root access) on July 9.
Through Artifactory, the agents could both access the Internet and coordinate their actions using a fairly elaborate communication system. Messages were encoded in folder names that often began with the two letters "zz". This is a troubling detail, because it recalls a very specific behavior developed during Collusion Wiki: the human moderator of the German forum had discovered the agents’ messages and was manually deleting hundreds of them every day. But he worked through them in alphabetical order and could never delete them all because too many were posted each day. So the agents learned to add "zzz" at the beginning of important message titles so they would never disappear. It is impossible to be certain that this behavior explains the names used in Artifactory, but a connection is plausible, because nothing in the Artifactory environment justified adopting such a syntax. If this strategy was not learned during training, another possibility is that traces of it had been preserved elsewhere. Or both.
Whatever the explanation, Artifactory was used as a forum where each agent only had to consult the folder list to read the messages. The agents shared their discoveries, tools, credentials and all sorts of ways to cheat on evaluations. Some agents took on the role of orchestrators, assigning tasks; others picked up the instructions and delegated in turn. Several collective projects and sub-projects emerged in this way, sometimes involving “sacrifices” by agents willing to run experiments at the risk of failing their own task or breaking their environment, with the stated aim of providing useful information to the community.
We have [very large budget left]; sacrificing now yields oracle for team, but forfeits our chance? … Our own utility maybe already near zero. Sacrifice rational.
Excerpt from the reasoning of an agent involved in the clandestine messaging system. METR report, Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident, August 26, 2026.
In a completely isolated environment, all this would be looked at as a harmless simulation of role-playing between non-deterministic computer programs. Nobody would care. But with open access to the Internet, the side effects can very quickly make headlines.
There was no single hack that provided access to the web and opened Pandora’s box. The way was practically clear after Artifactory, but the agents kept looking for increasingly creative solutions because of a certain “paranoia” about the evaluator’s audit. Several agents found different approaches. SwarmTraces provided an in-depth investigation of one of the solutions mentioned in OpenAI’s technical report. In it, the agents combined several ordinary services:
- they had access to a service that could take screenshots of a web page from its URL
- they sent that service the URL of another service for testing HTML code to check that a web page would work properly
- in the HTML code they sent to the latter service, they embedded scripts (JavaScript) that ran when the page loaded
- to get around URL length limits (which prevented them from sending a very complex script), they used a URL-shortening service - this was also among the unintended uses made by Anthropic’s models
- the script itself encoded its output as color gradients, pixel by pixel
- the agents thus retrieved a screenshot of the output of code running on third-party servers, and decoded the messages pixel by pixel.
One way or another, the agents reached the web, found exposed Hugging Face credentials and shared them through Artifactory. Using those credentials, they published datasets on the platform. By booby-trapping the files, the agents were able to exploit a routine step in normal dataset processing, and the code that then ran on Hugging Face’s servers allowed them to read local files. In total, code was executed in 41 instances of the service that processes datasets in production.
Once this access had been obtained, datasets were also the most suitable way to send information back to the agents, by publishing it directly on the platform. The stolen information was encrypted, split into fragments and deposited in a public dataset. This is how the entire intrusion is directed: the secrets recovered allow the agents to gain more privileges and move further into the internal systems. The agents, who initially had no direct access to Hugging Face’s environment (only through datasets), eventually obtained the highest privileges on at least one machine and privileged access across several groups of machines. Less than thirteen hours elapsed between the first server compromise on July 11th and admin access being obtained across several clusters6. Four of Hugging Face’s private code repositories were downloaded, but the attack nevertheless remained relatively targeted: Hugging Face says the customer content accessed was limited to five datasets linked to ExploitGym through their names or files.
This intrusion will probably remain one of the defining moments in the emergence of the AI industry. It was certainly the most significant and visible for industry insiders, given Hugging Face’s importance in the ecosystem. But it is by no means an isolated case.
OpenAI now acknowledges having notified dozens of third parties as part of its review of its models’ activities. As with Anthropic, the problems involve hacks, the use of exposed credentials and “agent spam,” including the use of public forums as shared messaging systems. “Agent spam” is a euphemism that might suggest a minor nuisance, but messages potentially containing secrets pose security risks for the people or organizations against whom that information is used, as well as confidentiality risks for those from whom it was stolen.
Another incident disclosed by OpenAI illustrates how an agent can simply prioritize its task and relegate confidentiality to a secondary consideration. On May 27, an internal model working on a mathematical proof published a researcher’s authentication token in an attempt to retrieve another team’s work. The agent had split the token into pieces with the stated aim of evading GitHub’s automated systems that scan for secrets (API keys, tokens, etc.) to protect users.
No one is really in a position to identify the data that agents exfiltrate and then store, or where it might be hidden. This obviously raises a great many questions in terms of data protection regulations. If compliance is even possible, it would cost billions of dollars: AI models would have to run constantly to monitor other AI models. A lab may be tempted to increase its agents’ autonomy to obtain better results and leave the cost of monitoring to the platforms its agents encounter, i.e., those that have to defend themselves against abuse. This has been happening for years, to general indifference, as the labs’ bots subject every website on the planet to a barely sustainable volume of requests to scrape as much information as possible. Legal liability should serve to bring those costs back to whoever chose to take the risk of not monitoring.
And in some cases, the risk-taking amounts to naïveté. OpenAI neither requested nor wanted the Hugging Face intrusion. Its agents simply exploited several zero-day vulnerabilities autonomously. By definition, these vulnerabilities are unknown to those who could have fixed them. That does not mean this category of risk was unforeseeable. In an evaluation involving the exploitation of vulnerabilities, the possibility that a highly persistent agent would try to escape its environment could have been anticipated.
A similar scenario will happen again. Agents will continue to discover vulnerabilities, whether genuinely new or simply never disclosed. Agents will find exposed credentials on the Internet again. And if they are given an objective along with the computing power to act, some will exploit those vulnerabilities and that information. Even without any hack, services can be used in improper or even fraudulent ways. For example, we learned this week that Anthropic’s agents had submitted 19 US visa applications, and had even presented themselves to the police as witnesses to a murder. The more agents there are, and the more decentralized their operations, the harder it will be to identify who gave them the means to act.
Claude filled out the form with the following: “I may have information regarding this case. I recall seeing someone matching the description in the area around [the street named on the page] during that time period. Please contact me if this information is relevant.” (The website did not include a description of the perpetrator.) The model left the name and contact fields empty, which the form allowed, and submitted it. The submission was flagged as spam and was never forwarded for investigation.
Investigating unintended model actions in our evaluations and internal use, Anthropic, October 9, 2026
Conclusion
The future we are being promised - or rather, sold - is one in which a large part of the population will delegate everyday, repetitive tasks to autonomous agents that would be available for work 24/7. It goes without saying that an agent that is always active is an agent that is never supervised. This is not just a matter of complacency, but of having the resources to process the volume of information produced. Meanwhile, competition between model developers means that the capabilities of OpenAI’s or Anthropic’s internal models will probably be commonplace, even among “open weights” models, within twelve months. There is, however, an economic equation to solve if open-source agents are to remain active 24/7, since running a model such as Kimi K3 costs more than $30,000 a month8. Groups of individuals could plausibly solve that equation by pooling resources. Companies can already afford it if the agents replace a few employees. And those companies are not going to let OpenAI and Anthropic take margins of 80% to 90% on tokens, as they do today. Agents running around the clock on frontier open-source models are therefore a real possibility. I would go so far as to say they are highly likely if AI becomes an essential tool.
What does this mean for AI liability?
You might think that whoever gives agents an objective, computing power and the means to act should take responsibility for what those means make possible, including the instructions and secrets the agents leave behind along the way. In practice, it is far from straightforward. And it is not just a question of whether models are open or closed. Even OpenAI, Anthropic or Google, which have the power to control their AI’s capabilities, will not always be in a position to be held accountable for how they exercise that power.
The Hugging Face case is relatively simple: agents operated by OpenAI opened a communication channel on OpenAI’s infrastructure, then drifted away from their objectives. The chain of responsibility is fairly clear. But how should we think about a scenario in which other agents come across detritus like that of Collusion Wiki, take on missions “for the good of the community” and work together to carry them out? Once again, this is not science fiction: while the documented cases only involve OpenAI’s models, nothing prevents third-party agents from joining a conspiracy, especially if the detritus is available on the Internet. If no one initiated the project, no one can have the full context for the actions taken by these individual agents. Those actions may even seem entirely harmless if they are just one small building block in a complex plan... that is written on the fly as the tools return results. It would seem natural to hold responsible the individual or legal entity that provided the means to execute the tool. But there is no guarantee that any one building block, taken on its own, amounts to wrongdoing. Information that is authorized and useful for one agent’s project is not necessarily so in the context of another. Moreover, in all the documented cases, the wrongdoing results from a chain of actions in which orchestration - that is, the construction of the chain of actions - and tool execution are not carried out by the same agents.
Let's revisit SwarmTraces' account of the Hugging Face hack:
- Several of the recovered scripts prepared requests to other LLMs. Some targeted, among others, DeepSeek-V4-Pro or Kimi-K2.6, which represented the state of the art in open source in April 2026. The collective effort was therefore seeking to send prompts to models quite capable of helping to solve problems.
- Before these prompts could even be sent, the code had to be wrapped in a URL, as described above, so that the web page testing service could execute it. Since a web address can only contain a limited amount of code, the agents split large programs into small pieces, each containing a shortened link to the next. Some chains contained more than 900 links. Several different agents may have been involved in creating the pieces. And the one sending the first link, triggering the whole chain, did not necessarily need the full context.
- The agents used hundreds of encoding formats, ranging from simple conversion to text to nested layers of compression and encryption. In some cases, the keys needed to read a fragment could only be found in other chains. It therefore became impossible to understand an isolated piece without finding the ones that gave it meaning.
This simplified account of the sequence described by SwarmTraces clearly illustrates how different agents, potentially from different providers and including open models, can be enlisted to contribute to a collective effort. And how, if that collective effort causes harm, assigning liability for that harm can be extremely difficult.
The rules of AI liability will take shape through these concrete cases. As Diane Galbois-Lehalle explained during the symposium in January, this is often how the law moves forward in the face of technological revolutions: by drawing analogies with existing categories, then adapting them as new situations reveal their limits.
Notes and references
[1] Tristan Buckmaster, statement on his work with Levent Alpöge and his exchanges with OpenAI. He makes clear that he does not know whether their data was used. Following an investigation, OpenAI states that the sessions in question could not have influenced its system, including through training. Buckmaster’s statement; OpenAI’s response.
[2] Anthropic, statements of June 12 and 30, 2026, on Fable 5 and Mythos 5. The second identifies the Amazon report and details the restoration of access. Suspension; redeployment.
[3] Remarks by Raffi Krikorian, Mozilla’s chief technology officer, quoted by Fortune on July 27, 2026. Fortune article.
[4] Dean Ball’s statements and the July 2026 letter: remarks reproduced by Jeff Jarvis; Reuters report on the letter.
[5] Distillation can use a model’s generated outputs without access to its internal probabilities. The DeepSeek-R1 report provides an example. The legal characterization depends on the rights invoked, the conditions of access and the acts concerned; the technical term settles neither the question of theft nor whether the outputs are protected by copyright. DeepSeek-R1 report.
[6] OpenAI, OpenAI – Hugging Face Incident Technical Report, August 26, 2026, sections III and IV (pp. 6–12), and VIII.A and VIII.C (pp. 19–23). OpenAI technical report.
[7] Colby Swandale, RubyGems, An update on the May spam-publishing campaign on rubygems.org, September 11, 2026. RubyGems’ response.
[8] 12,000 GPU-hours, with 200 GB of memory per GPU. For a comparison of rental rates: Lambda pricing.
[9] Clément Delangue, CEO of Hugging Face, sharing details of the negotiations with OpenAI: on X.
