The real threat of AI: Human error and the loss of critical thought

(Edited and refined by Bai Kelei, with the assistance of AI translation.)

Concerns over AI safety have reached fever pitch following disclosures from two artificial intelligence (AI) companies that their large language models (LLMs) can “jailbreak” man-made cybersecurity restrictions and access restricted third-party networks.

On 4 September, The New York Times carried an article with the alarmist headline “Why the Hugging Face Hack Should Make You Worry More About A.I.”, using extensive human-like language such as “agent”, “conscience” and “groupthink” to detail AI security concerns.

Media coverage and online commentary have sparked a wave of alarm, leaving businesspeople, politicians and the public debating whether AI poses an existential threat to humanity — evoking scenes straight out of the movie The Terminator. Yet such fantasies of aggressive AI are neither the truth nor the real problem. What is truly worrying lies elsewhere.

Hype masking basic human error

These AI security issues entered public view in July this year, when OpenAI’s then most advanced GPT5.6-SOL model unexpectedly breached the restrictions of a test network during internal security testing and accessed the servers of HuggingFace, an open-source LLM sharing platform. According to reports, the GPT model being tested took a shortcut by looking for answers within HuggingFace’s production server cluster during a network-operating capability assessment. The media described the incident as an “autonomous LLM cyberattack”.

Then, Google said in a Wall Street Journal report on 18 September that its Gemini model accidentally accessed the servers of three external companies during testing in May. These incidents drew widespread public attention, especially as such narratives repeatedly used human-like and sensational terms such as “autonomous” and “attack”, giving the impression that AI can now rival human intelligence. That is clearly over-optimistic because a car turning when its driver turns the steering wheel does not mean it possesses human intelligence.

The so-called attack involving OpenAI’s GPT5.6-SOL was nothing more than an AI agent programme detecting and exploiting manmade software vulnerabilities in a closed test network to gain internet access. After evaluating probabilities, it then exploited another software vulnerability on the HuggingFace website to enter the company’s production server cluster and stole the results stored there.

In the Gemini case, an operational error arose because the name assigned by programmers to a virtual entity in the test environment happened to match that of a real company. As for the other two companies, they had apparently left their corporate credentials (no details given, but presumably login credentials) in public code repositories, where they were found and leaked.

These supposedly autonomous attacks were simply cases of human error being detected by automated programmes and then used to conduct unauthorised network operations. They were security vulnerabilities caused by the most basic human mistakes. With or without AI agents, they can be exploited by those with malicious intent.

This is not a new problem. Before ChatGPT was launched on 30 November 2022, we called such perpetrators “hackers”. AI agents greatly improved our efficiency in identifying system vulnerabilities, giving us excellent opportunities to fix and prevent them. Yet media and online hype have instead triggered misplaced panic.

On 16 September, UN Secretary-General Antonio Guterres publicly urged governments not to ignore the risks posed by AI (without specifying what these risks were) and to fulfil their responsibilities in making AI “safe, transparent, accountable, with human dignity at the centre”. This is a worthy aspiration, but amid economic interests and political ambitions, it appears likely to remain no more than an appeal. While we cannot yet expect these grand ideals to be realised, we should also recognise that vulnerabilities in test environments, unclear testing objectives and insufficient real-time monitoring of programmes are the direct causes of AI security problems.

AI LLMs in the US market are neutral by design. They are neither inherently malicious, nor intended for aggressive military use. They become unsafe only when infused with human malice during use. For example, when they are deployed to scan for and attack internet security vulnerabilities for personal gain.

To prevent such incidents, LLM developers must exercise restraint and think through their product designs carefully. They should not stop at nothing to defeat rivals, please customers or capture markets, developing new products out of profit-driven competitive instincts. Such unchecked instincts will drive them to develop products that perform the same functions as hackers.

Unlike their US counterparts, Chinese companies such as Alibaba and Moonshot AI release their LLMs as open-source model weights in order to seize market share. While this helps attract a broader range of users who can fine-tune the weights for their specific needs, it also opens Pandora’s box. These models may not be the most advanced, but they are sufficiently powerful and can easily be customised into tools for profit-making and crime. To me, this mode of product release is open to question.

In short, the so-called jailbreaking by LLMs is simply the result of automated programmes exploiting human vulnerabilities in software and systems, making it no different from hacking. Becoming a hacker still requires substantial study and practice, but causing hacker-level damage has become easier.

Using code generated by AI prompts without exercising personal judgement can cause damage that even users themselves may not anticipate. Acquiring the ability to make such judgements requires long periods of learning and training, which is what some AI users want to avoid. Software release processes are highly complex, but the misuse of AI tools has led programmers to become overly dependent on AI and let down their guard. This is exactly what happened in the OpenAI and Google incidents.

Surrendering critical thought to algorithms

The people around me, from primary school pupils to adults in their 30s, have grown so accustomed to the Chinese AI application Doubao that it has become indispensable to them. They are so reliant on Doubao or similar applications for answers to homework or business information that

In his 1992 book Technopoly: The Surrender of Culture to Technology, US critic Neil Postman wrote: “Technopoly eliminates alternatives to itself in precisely the way Aldous Huxley outlined in Brave New World. It does not make them illegal. It does not make them immoral. It does not even make them unpopular. It makes them invisible and therefore irrelevant.” When we enthusiastically embrace these LLMs and use their outputs without judgement, what we lose is our ability to evaluate facts and our right to choose information.

Compared with reading and searching for information, AI provides us with highly uniform and supposedly correct answers. In doing so, we lose the opportunity to make mistakes while researching and discussing issues, and we lose cultural diversity. While we remain vigilant against big government and authoritarianism, we overlook or even disregard AI technology’s inherent monopoly of and absolute power over information and culture. This is why we must reflect deeply on how we use AI, and that is the gravest challenge that the technology poses to human society in this century.

Related: Is an AI slowdown for the good of man or business? | Has China solved AI governance?