Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

New deepfake regulations in China are a tool for social stability, but at what cost?

China is pushing ahead of the European Union and the United States with its new synthetic content regulations. New draft provisions would place more responsibility on platforms to preserve social stability, with potential costs for online freedoms. They show that the Chinese Communist Party is prepared to protect itself against the unique threats of emerging technologies.

On 28 January 2022, China’s State Internet Information Office released the “Provisions on the Administration of Deep Synthesis Internet Information Services (Draft for solicitation of comments)”1. The proposal (henceforth the Provisions) is a draft of regulations for deep synthesis technology, an umbrella term covering “text, images, audio, video, virtual scenes, or other information” created with generative models (Article 2). Also known as synthetically generated content, this includes deepfakes (photos, videos or audio that depicts a person doing or saying things that they have not been recorded doing or saying), generated texts such as those produced by OpenAI’s GPT-3, advanced image enhancement techniques and the construction of virtual ‘scenes’, akin to the immersive movies described in the classic novel Ready Player One. Once enacted, the Provisions will represent a leap past the USA and the EU in deepfake regulation and a considerable advance in the Chinese government’s efforts to control the content of its domestic Internet, and social stability more generally.

Effect of the Provisions

The Provisions appear to be an elaboration on the 2019 “Regulations on the Administration of Online Audio and Video Information Services,” which broadly banned the use of machine-generated images, audio and video to create or spread “rumors”2,3. The new regulations are aimed at deep synthesis service providers and emphasize cybersecurity, real-name verification of users, data management, marking of synthetic content to alert viewers and “dispelling rumors”1. They expand the Chinese government’s efforts to prevent social and political disruption by increasing its control of the Internet. These efforts are tied to the actions of tech platforms and companies. Article 5 encourages industry organizations to establish industry standards and self-discipline systems while “accept[ing] societal oversight”. This oversight is exercised through state regulations but requires industry cooperation.

This is an established trend, as the government has been increasingly relying on tech companies to enforce new Internet regulations to facilitate the Chinese Communist Party’s (CCP) vision of a stable, prosperous society, with consequences if they do not. For instance, after Jack Ma, co-founder and executive chairman of Alibaba, criticized Chinese regulators for stifling innovation, the previously approved initial public offering of Alibaba’s financial technology organization Ant Financial was scuttled, and the entire Ant Group was fined and forced to restructure under new anti-monopoly rules. Ma himself vanished from the public eye for months4. In response to the fine, Alibaba acknowledged that they needed to shoulder more responsibility for China’s social and economic development5, showing that technology companies understand what is expected of them: a commitment to the Party’s image of a stable society of “common prosperity,” with severe consequences for those who do not contribute. Shortly after the IPO cancellation, Alibaba and Tencent promised to crack down on overwork culture. They invested 100 billion yuan (US$15 billion) to support common prosperity, which Xi has emphasized as a way to ensure the Party’s longevity6.

Technological regulations are another means to support this drive. One example is real-name verification requirements, which have increased in recent years. For example, instant messaging services like WeChat have required that users provide real names since 20147. Other major sites such as Weibo, Zhihu and Baidu began requiring real names in 2017 as a result of a law banning anonymous online commenting8. The synthetic content Provisions recruit technology companies to a new front in the CCP’s offensive. They would require all synthetic content to be labelled, and ban a wide swathe of supposedly destabilizing content ranging from subversive incitements to pornography to intellectual property violations1.

Automatically generated, highly convincing mis- and disinformation and deepfake pornography are among the main concerns that have been raised by the increased availability of generative technologies. The Provisions are the most radical reaction against them so far. At least two issues will determine the impact of the Provisions.

First, some technical problems have yet to be solved. According to one of the central regulations of the draft, synthetic content produced by deep synthesis technology services must be labelled, yet how to ensure that labels are created and preserved is unclear. Even whole-frame watermarks can be removed by re-encoding or using another AI system, and metadata or accompanying documentation can be altered or omitted9.

Second, controlling the spread of deep synthesized content is intrinsically difficult. Real-name verification laws can enable authorities to track the sources and spreaders of unlabeled or misleading deep synthesis content. However, deep synthesis service providers are instructed to immediately stop transmitting unlabeled information and institute procedures to dispel rumours to combat mis- and disinformation (Articles 15 and 17). This is challenging because, once a piece of content is created, it can be decoupled from the service on which it is created and spread independently of it. Videos can be re-uploaded, audio re-recorded, images screenshotted, removing them from the originator’s control. And, as the adage goes, the Internet is forever: once some information has spread on the Internet, it is extremely difficult to erase it completely. In China, as anywhere else, content may be removed by censorship, but collective memory may persist. For instance, several weeks into the Shanghai COVID-19 lockdown that began in March of 2022, a video called “四月之声” (Voices of April) that chronicled events during the lockdown through audio began spreading on Weibo, reaching millions through a relay of re-uploads and copies that kept ahead of the censors for some hours. Although the videos were eventually taken down, the dead links remained, a memorial to what had happened10. As part of the censorship effort, cleanup notices were issued to all platforms requiring the total scrubbing of content related to Voices of April11.

Voices of April is not synthetic content, but it exemplifies how platforms where content is distributed (not just those where synthetic content is created) are expected to cooperate in information-quashing efforts. This mandatory, large-scale coordination shows the degree of responsibility that the government is placing on platforms and service providers to maintain the ‘hygiene’ of the Chinese Internet. In the September 2021 “Opinions on Further Intensifying Website Platforms’ Responsibility for Information Content,” one reads that the State Internet Information Office tasks platforms with preventing the creation and spread of “illegal” and “negative” information12. This burdens platforms and deep synthesis service providers with demanding tasks and implies a corresponding amount of government oversight of these platforms. Both the Opinions and the Provisions say that platforms need to maintain the “correct” values or politics, which implies that the government will be defining these “correct” values (likely in support of Xi Jinping Thought and common prosperity). Although these regulations are new, government efforts to control the views expressed on the Internet have a long history in China.

The promise of social stability

The technical difficulties just discussed are linked to the need to maintain the “correct political orientation” in the media, which stems from “Document 9,” a document circulated secretly in April 2013 before the Third Plenum of the Eighteenth Party Congress. What followed the document, which warned against the “infiltration” of Western ideas such as constitutionalism and civil society, was a crackdown on human rights activists, media outlets and dissident academics13, as well as increased efforts to block Western websites14. Since then, more sites have been blocked, VPNs have been removed from app stores15 and real-name verification rules have been increased8.

A frequent justification for controlling the Internet is that China needs social stability to develop16, implying that an open Internet ecosystem would cause instability. When looking at incidents such as the 6 January 2021 attack on the US Capitol, which was fuelled by misinformation circulated on social media, it is easy to see why the Chinese government perceives an uncontrolled Internet as a threat and is taking extremely seriously its efforts to build the Chinese infosphere into an impenetrable bubble. Deep synthesis services are a new destabilizing force: false or misleading information can be generated in massive quantities and distributed rapidly across the Internet to millions of users. A single fake post can be censored or officially discredited. However, a hypothetical fake conspiracy theory supported by deepfaked photographs, videos and documents may not be countered so easily; even when content is removed from the Internet, it lingers in people’s minds. For a government that relies on being able to control what information is available to its citizens, synthetic content represents an existential threat.

Paradoxically, the ease with which deepfakes can be generated and spread is more threatening politically than reliable news, which can be more easily blocked. This unique threat clarifies why the Chinese government has moved to regulate deep synthesis technology more quickly and thoroughly than the United States or the EU. The Provisions will likely be revised lightly and approved by August 2022, since 4–6 months is a typical timeframe for Internet-related regulations in China17. This is a much faster and more comprehensive process than those in the USA or the EU. Only a handful of American states have issued regulations for deep synthesis technology, and those focus solely on deepfake pornography or deepfakes intended to influence elections18. The First Amendment protection of freedom of speech is likely to hinder efforts to regulate deepfakes very radically, as the US Supreme Court has ruled that non-libellous lies are constitutionally protected—“The remedy for speech that is false is speech that is true”19, not prohibiting the false speech—representing a fundamental difference from the approach adopted by China’s government. The “DEEP FAKES Accountability Act,” which has been under consideration in the US Congress since 2019, would impose labelling requirements similar to those of the Chinese draft Provisions and establish mechanisms for redress9,20. However, it is aimed mainly at preventing “unauthorized digital recreations of people” on the grounds that those are unlawful impersonation, which includes generated pornography, and it has been criticized as unenforceable partly because of the difficulty of tracing and attribution9. Most synthesized media regulations have come from individual platforms (including Facebook and Twitter), which have issued limited rules stating that they will label or remove content that violates their policies19. The EU has moved to require deepfake labelling from platforms through amendments to the Digital Services Act, which would take effect in 202321. However, it does not address the use of generated pornography, and, like the US regulations, it does not target the deep synthesis service providers. The assumption seems to be that platforms will be playing cat-and-mouse with anonymous bad actors who will distribute deepfakes and must be detected and labelled. Instead, Chinese regulations address the sources of the generated content.

Beyond deepfakes

The fact that China’s proposed regulations go beyond deepfakes to include generated text, image enhancement and virtual scenes indicates that the government is thinking more broadly about how emerging technology could impact the stability of its regime (even anticipating future developments in virtual reality technology). It also speaks to how the Chinese government is more dependent on controlling this kind of media than the liberal US or European governments. As Chinese authorities have more control over what digital services people can access, this is a more feasible strategy for them than for US or European regulators. In fact, these fundamental differences in governance structures both contribute to this dependency and shape enforcement abilities. As a “rule by law” state, where law is used as a tool of political control, China has far greater enforcement capabilities than the “rule of law” in the USA or the EU, where the law is intended to be above politics22. Compliance is expected, and punishment is a deterrent for peer companies and individuals, whereas enforcement in the USA and the EU would likely be less capricious but also less severe. Furthermore, a single-party system has the advantage in legislative agility—which we see in the fact that these Provisions have come well before anything in the US or EU.

Although the pre-emptive nature of these regulations could herald what has been termed a “Beijing Effect” on forthcoming regulations in the USA and the EU, the siloed nature of China’s Internet may stymie this unless individual Western companies are inspired to take action against synthetic content in order to do business in China. Still, the Provisions are a concrete example of what is expected from technology companies as the CCP works toward its dream of a stable society of “common prosperity”, and they reflect a government commitment to maintaining and tightening control over an ever-more-siloed Internet. Indeed, the recent “2022 Special Action on the Comprehensive Governance of Algorithms”23 reinforces this trend.

At the same time, they also demonstrate a prescient understanding of how new technologies could threaten social stability and thus the regime’s power. It is likely that the Chinese government will continue to exercise such foresight in regulating new technologies, with possible implications for personal freedoms online. However, the impact of deep synthesis technologies, in particular, will depend on solving technological problems and ensuring platform cooperation and coordination. Neither is likely to happen seamlessly, but both represent new frontiers in China’s technological governance.

References

  1. Provisions on the Administration of Deep Synthesis Internet Information Services (Draft for Solicitation of Comments) (Cyperspace Administration of China, 2022); https://www.chinalawtranslate.com/en/deep-synthesis-draft/

  2. Regulations on the Administration of Online Audio and Video Information Services (Cyberspace Administraion of China, 2022); http://www.cac.gov.cn/2019-11/29/c_1576561820967678.htm?module=inline&pgtype=article

  3. Provisions on Management of Online A/V Information Services (National Radio and Televsion Administration, 2019); https://www.chinalawtranslate.com/provisions-on-the-management-of-online-a-v-information-services/

  4. Campbell, C. What the saga of Jack Ma says about China under Xi Jinping. Time https://time.com/5926062/jack-ma/ (4 January 2021).

  5. Campbell, C. How China is cracking down on its once untouchable tech titans. Time https://time.com/6048539/china-tech-giants-regulations/ (20 May 2021).

  6. Liao, R. China roundup: Beijing wants tech giants to shoulder more social responsibilities [Internet]. TechCrunch https://social.techcrunch.com/2021/09/05/china-roundup-beijing-tech-giants-to-shoulder-social-responsibilities/ (2021).

  7. Danning, M. Real names now required for WeChat and other IMS. China Daily https://www.chinadaily.com.cn/china/2014-08/07/content_18268073.htm (7 August 2014).

  8. Gan, N. China’s Twitter-like Weibo orders users to register real names. South China Morning Post https://www.scmp.com/news/china/policies-politics/article/2110400/chinas-twitter-weibo-orders-users-register-their-real (8 September 2017).

  9. Coldewey, D. DEEPFAKES Accountability Act would impose unenforceable rules — but it’s a start. TechCrunch https://social.techcrunch.com/2019/06/13/deepfakes-accountability-act-would-impose-unenforceable-rules-but-its-a-start/ (2019).

  10. Yang, Z. WeChat wants people to use its video platform. So they did, for digital protests. MIT Technol. Rev. https://www.technologyreview.com/2022/04/24/1051073/wechat-shanghai-lockdown-protest-video-the-voice-of-april/ (2022).

  11. Wade, S. Minitrue: Silence the ‘Voices of April’ viral video on Shanghai lockdowns. China Digital Times https://chinadigitaltimes.net/2022/04/minitrue-silence-the-voices-of-april-viral-video-on-shanghai-lockdown-full-english-transcript/ (22 April 2022).

  12. Opinions on Further Intensifying Website Platforms’ Entity Responsibility for Information Content (Cyberspace Administration of China 2021); https://www.chinalawtranslate.com/content-responsibility/

  13. Document 9: A ChinaFile Translation (ChinaFile, 8 November 2013); https://www.chinafile.com/document-9-chinafile-translation

  14. Buckley, C. China takes aim at Western ideas. New York Times (19 August 2013); https://www.nytimes.com/2013/08/20/world/asia/chinas-new-leadership-takes-hard-line-in-secret-memo.html

  15. Feng, C. China’s VPN providers face harsher punishment for scaling the Great Firewall under new data regulation. South China Morning Post (15 November 2021); https://www.scmp.com/tech/policy/article/3156095/chinas-vpn-providers-face-harsher-punishment-scaling-great-firewall

  16. Yuan, L. A generation grows up in China without Google, Facebook or Twitter. The New York Times (6 August 2018); https://www.nytimes.com/2018/08/06/technology/china-generation-blocked-internet.html

  17. China Law Translate. Index of All Posts; https://www.chinalawtranslate.com/index/ (accessed 21 February 2022).

  18. Briscoe, S. U.S. laws address deepfakes. Secur. Manage. http://www.asisonline.org/security-management-magazine/latest-news/today-in-security/2021/january/U-S-Laws-Address-Deepfakes/ (2021).

  19. Sunstein, C. R. Can the government regulate deepfakes? Wall Street Journal https://www.wsj.com/articles/can-the-government-regulate-deepfakes-11610038590 (7 January 2021).

  20. Clarke, Y. D. DEEP FAKES Accountability Act H. R. 2395; https://www.congress.gov/117/bills/hr2395/BILLS-117hr2395ih.xml (4 August 2021).

  21. Anderson, M. European and UK deepfake regulation proposals are surprisingly limited. Unite.AI https://www.unite.ai/european-and-uk-deepfake-regulation-proposals-are-surprisingly-limited/ (2022).

  22. Waldron J. In The Stanford Encyclopedia of Philosophy (ed. Zalta E. N.) (Metaphysics Research Lab, Stanford University); https://plato.stanford.edu/archives/sum2020/entries/rule-of-law/

  23. Notice on Launching the Clear 2022 Special Action on the Comprehensive Governance of Algorithms (Cyperspace Administration of China, 2022); https://www.chinalawtranslate.com/en/special-action-on-algorithms/

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Emmie Hine.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Machine Intelligence thanks Rogier Creemers and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Hine, E., Floridi, L. New deepfake regulations in China are a tool for social stability, but at what cost?. Nat Mach Intell 4, 608–610 (2022). https://doi.org/10.1038/s42256-022-00513-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-022-00513-4

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing