AI Stefanie, scams and fake news: China acts on AI regulation

Mandopop diva Stefanie Sun, who hasn't appeared on screen for some time, has resurfaced with the help of artificial intelligence (AI).

Cover songs by "Sun" have taken Chinese video platform Bilibili by storm since April, with several of them receiving millions of views, such as Jay Chou's Hair Like Snow, Nan Quan Mama's Rainy Day, and Faye Wong's Red Bean. According to some media estimates, there are more than 1,000 of such covers, which even exceeds the number of Sun's works since her debut in 2000.

But on closer inspection, these covers were not sung by Sun herself, but by "AI Stefanie Sun".

Simply put, these covers were generated by AI versions of Sun using model training and post-processing to replicate the singer's voice. Although there is still a difference in the singing style and technique between the AI and the original singer, the tone is almost indistinguishable - enough to pass off as the original.

Netizens commented, "I really thought Stefanie Sun covered the song herself... The tone is exactly the same. Although it lacked style and technique, the voice blended well with the background music and it is a truly high-quality song."

Other netizens joked, "It really sounds like Stefanie Sun. Stefanie, if you're still not going on tour or releasing a new album, you'll be replaced by AI!"

As AI Stefanie Sun continued to release new covers on various online platforms and triggered fierce online debate, related topics such as "Stefanie Sun first to be unemployed with the proliferation of AI" and "Experts claim that Stefanie Sun's voice is not protected by law" recently became top searches on Weibo, each amassing over 100 million hits.

Amid the intense online debate, Sun made a response on 22 May on her blog on Make Music. She self-deprecatingly wrote, "My fans have officially switched sides and accepted that I am indeed an 'obscure singer' while my AI persona is the current hot property. I mean really, how do you fight with someone who is putting out new albums in the time span of minutes?"

Notwithstanding, Sun was not pessimistic or negative about the development of AI. On the contrary, she described herself as a popcorn eater with the best seat in the theatre, saying, "In this boundless sea of existence where anything is possible and where nothing matters, I think it will be purity of thought, being exactly who you are that will be enough."

Music copyright lawyer Zhao Zhigong told The Beijing News during a Beike Economy (贝壳财经) interview that AI Stefanie Sun's works involve copyright infringement issues that need to be looked into depending on usage scenarios. If the Bilibili users uploaded the song covers for "self-entertainment" and not for commercial purposes, it may not constitute an infringement on intellectual property rights but could still involve personal copyright issues in terms of attribution, the identity of the performer, and protection against the distortion of the performer's image.

Zhao also pointed out that the songs that Sun chooses to cover should be decided by her and her agency or record label, in accordance with the artist's image and music market considerations. But because AI Stefanie Sun is able to cover any song, "the AI-generated cover songs infringe on Sun's own market resources".

Li Qiang, a copyright lawyer who has worked with many music streaming platforms, agreed that the use of AI to simulate a singer's voice without authorisation, and then publicly displaying it, should be considered an infringement of the singer's rights.

Observably, AI singers are not rare on the Chinese internet. Before the emergence of AI Stefanie Sun, the works of "AI Jay Chou", "AI Faye Wong" and "AI G.E.M." were already on Bilibili. Currently, no agency or singer, including Sun, has issued an official statement prohibiting AI cover songs or pursuing legal responsibility.

Apart from Sun, AI-driven scams have recently swept across the Chinese internet.

The official WeChat account of Inner Mongolia's Baotou Public Security Bureau issued a statement on 20 May saying that the local police have recently uncovered an AI-driven telecom fraud in early May. The victim, a man surnamed Guo, is the legal representative of a technology company in Fuzhou.

According to the statement, Guo suddenly received a WeChat video call from his "friend" claiming that another friend was bidding on a project in a different city and wanted to use Guo's company's account to submit a bid of 4.3 million RMB (US$608,000).

The "friend" asked for Guo's bank account number, claimed that he had transferred the money into Guo's account, and even sent a screenshot of the bank transfer receipt to Guo via WeChat.

Because it was a video call and out of trust in his friend, Guo did not check if the money had arrived and transferred 4.3 million RMB in two separate payments to the scammer. After transferring the money, Guo sent a message to say that the transaction was complete. To his horror, the real friend responded with a question mark.

Guo immediately phoned his friend, who denied that he had asked Guo to transfer any money. Realising that he had encountered a sophisticated AI-powered face-swapping technology scam, Guo called the police. He said of the scam, "I received the video call and recognised the familiar face and voice, so I let my guard down."

According to National Business Daily, there are currently two forms of AI fraud: voice synthesis and face swapping. In a voice synthesis fraud, scammers first extract the voice of a person familiar with the target through methods such as nuisance calls. After obtaining the material, they deceive their targets by fabricating the familiar voice through voice synthesis.

As for face swapping, scammers analyse the publicly available images of people familiar with their target and then use AI technology to replace their faces. The targets would fall for the fraud because they can confirm the scammer's "identity" through videos.

Wang Pengbo, chief analyst at Broadcom Consulting, told Beijing Business Today, "I believe that technological advancements have also increased the difficulty of preventing fraud, which also happens due to excessive information leakage resulting from the information explosion."

In the era of high-speed internet, AI poses more challenges than those mentioned above.

About three weeks ago, Chinese media reported that a man was detained by the authorities for fabricating news using AI software ChatGPT. Bloomberg highlighted that this appears to be one of the first criminal cases in China related to the misuse of AI.

On 7 May, the Pingliang City Public Security Bureau in Gansu province released a statement on its official WeChat account, stating that a man surnamed Hong "fabricated information using modern technology and disseminated it on the internet, which was widely spread and viewed".

Hong fabricated an article about a train accident that allegedly resulted in the deaths of nine people. The article was published across multiple accounts on Baidu's "Baijiahao" (百家号) platform, and gained over 15,000 views before it was removed.

Investigations revealed that Hong learned how to monetise web traffic from his friends on WeChat and used ChatGPT to modify popular social news. Under Chinese law, Hong might have committed the offence of provoking disorderly behaviour (寻衅滋事) due to the widespread dissemination of the fabricated content. The offence carries a minimum sentence of five years imprisonment.

Netizens have commented that the case shows the risks and challenges that AI technology may pose in disseminating information online. "In the hands of unscrupulous individuals, AI can become a weapon for creating and spreading false information and misleading public opinion," one user said.

Another remarked, "AI will make it difficult to distinguish between real and fake news in the media."

Since OpenAI's ChatGPT debuted in November last year, it has caused a significant stir in China's tech circles, with Chinese tech companies such as Tencent and Baidu launching their own rival services since March this year.

On 11 April, the Cyberspace Administration of China (CAC) issued a draft of administrative measures for generative AI services to strengthen its regulation.

The document states that China supports the independent innovation of AI algorithms, frameworks and other fundamental technologies, but companies must apply for security assessments from the CAC before providing generative AI services to the public.

Furthermore, generated content should reflect core socialist values and must not undermine the state's power, subvert the socialist system, incite separatism or undermine national unity. It must also not promote terrorism or extremism, ethnic hatred, or content that may disrupt economic or social order.

Additionally, companies must ensure the authenticity and accuracy of the content and take measures to prevent the generation of false information. The illegal acquisition, disclosure and use of personal information, private and trade secrets are also not allowed. And users must provide genuine identity information in accordance with the law.

The Wall Street Journal reported that China's move puts it at the forefront of efforts by governments worldwide to rein in generative AI amid concerns over challenges posed by this new technology. Generative AI generally refers to AI with the ability to create original content using algorithms, generating images, text, music, code and even videos without human intervention - ChatGPT and AI Stefanie Sun are some examples.

According to BBC Chinese, generative AI is also evolving at an astonishing pace. Since the launch of ChatGPT in November last year, the GPT-4 iteration released in March this year already has a broader knowledge base, smoother responses, and the ability to interpret images. It also achieve high scores in a law qualification exam, and demonstrated self-reflection capabilities.

A Chinese AI practitioner who declined to be named also told BBC Chinese that China was swift in taking regulatory actions mainly due to three reasons: the ethical issues created by generative AI, and its disruptive power and rapid evolution.

Besides China, the G7 leaders - who met in Hiroshima recently - also emphasised the need for a thorough assessment of the rapid development of generative AI, and will discuss the "responsible" use of this technology by the end of this year. In a statement, the G7 leaders said, "We recognize the need to immediately take stock of the opportunities and challenges of generative AI, which is increasingly prominent across countries and sectors..."

The European Union is also moving forward with legislation to regulate AI technology, and may introduce the world's first comprehensive AI regulatory law. The bill will be presented to the European Parliament in June, and once passed, negotiations on legislation will begin with individual member states.

Just as Tesla sparked enthusiasm in the global electric vehicle industry, the introduction of ChatGPT at the end of last year ignited the entire AI industry, with global tech giants rushing into the field in just a few months. However, incidents such as AI Stefanie Sun, AI face-swapping scams and AI-fabricated news remind us that this technology also brings unprecedented challenges.

It is still hard to tell whether the benefits of AI outweigh the drawbacks or vice versa. The only certainty is that AI is still in its early stages of development, and it is just beginning to amaze the world.

This article was first published in Lianhe Zaobao as "中国AI狂潮下的孙燕姿、换脸诈骗、虚构新闻".

Related: AI stars in China and the US lose their shine | AI is us, we are AI | AI is taking the place of mental labour but fear not | Is a Chinese ChatGPT on the way?