The Locknet
The much-vaunted “Great Firewall” looms large in the imagination, but much of what an average person in China sees or sends online will never come into contact with it. The “Great Firewall” is just that: a digital border flanking the real one, separating domestic and foreign. As long as Chinese internet users are sending group messages in WeChat, opining for the public on Weibo, or livestreaming themselves on Kuaishou, the information they’re transmitting will never leave domestic cyberspace. It will still be subject to surveillance and censorship, of course—but that comes at the hands of private tech companies, who have accepted such tasks as the price of operating in China.
In effect, China has outsourced its in-country censorship to apps and platforms themselves—“service-level censorship.” But that doesn’t mean everything a company scrubs from its website counts as “censorship.” In addition to government-mandated deletions, companies conduct their own content purges, according to their own content rules, a practice known as “content moderation.” For example, a company might forbid users to post advertisements for rival platforms and remove any content that violated this stricture.
The distinction between “censorship” and “content moderation”? Our definition of company-implemented censorship has two key components: first, the government must ask for or demand it, formally or informally; and second, there must be a threat of real consequences for a company that does not comply. Content moderation, on the other hand, arises from a private company’s own internal rules about what material it chooses to permit on its platform. Even though popular culture frequently muddies this distinction (for example, U.S. media outlets and even companies themselves sometimes call platform content moderation “censorship,” or Chinese social media employees might refer to their compulsory censorship as mere “content moderation”), it matters in a very real way for the humans expressing their opinions online. A user who doesn’t like a platform’s content policies can choose to use a different platform, but a citizen who doesn’t like their country’s censorship laws would potentially have to leave the country if they wanted to post with abandon. Users who violate a platform’s content policies may lose their accounts, but a citizen posting illegal material may wind up in jail.
Platforms implement government censorship all over the world to comply with local laws. China’s tech companies are not unique because they carry out government censorship regulations; instead, it’s the regulations themselves that are noteworthy. For instance, China’s list of illegal content dwarfs that of the United States, which mostly concerns child sexual assault material. Moreover, the items on China’s list are often vague, allowing authorities to decide just about anything is verboten. For instance, in mid-2023, official cremation statistics began disappearing from both government websites and social media platforms; before its disappearance, the data had shown a huge jump in cremations after the government put an abrupt end to its zero-COVID policy, contradicting official claims about the death toll.
Certain specific topics, like the 1989 Tiananmen massacre, can’t even be publicly listed, as that would undermine the government’s efforts to memory-hole them. Even so, the Chinese government can hold companies liable for illegal content, whether or not the companies knew in advance it was illegal. (It’s not just Chinese companies that must comply with Chinese laws; any company operating in China is subject to the same provisions. This extends from the professional networking site LinkedIn, which, before it exited China, used to block profiles containing “sensitive” content, to Apple, which helps implement meta-censorship by ensuring that no blacklisted apps appear in China’s version of the App Store.)
China’s robust regulations induce Chinese tech companies’ robust monitoring of user-generated content. From the very beginning, this has involved actual humans, sitting at individual computers, reviewing what users post to a platform. In 1996, the founder of one internet company told a reporter that, if conversations in her platform’s discussion groups got too political, “I cut them off.” Since then, internet cafes, blogs, chat rooms, search engines, and social media platforms have all hired staff specifically to deal with the “real headache” of censoring content on behalf of the regime—a headache which has only intensified as domestically-produced online content has proliferated. In 2019, one Chinese outlet reported that ByteDance, which operates Douyin and TikTok, “had an army of more than three thousand censors” in just one city in China. A recent report published by the Open Technology Fund noted that various companies had posted more than 1.7 million censorship-related job ads between 2015 and 2022. Social media companies and news media were the primary employers, but other companies were hiring as well: “nontraditional” employers such as malls and e-commerce sites who needed staff to censor product information and marketing materials; or one of the more than 3,000 companies that exist simply to provide information control workers as contractors to other companies. For the individuals hired to do these jobs, the pressure can be intense. One information control worker at a major search engine told an interviewer in 2024 that frontline workers (the first humans to review flagged content) inspect more than 10,000 items every day, while second-line reviewers (who monitor the frontline workers’ output) checked somewhere in the neighborhood of 50,000 items.
Of course, automated censorship mechanisms supplement such human labor. Since the early 2000s, various search engines—including American ones like Yahoo!—have programmed their systems to return no results when users search for certain combinations of keywords. Google famously censored its search results so it could launch “google.cn” in 2006, though much to Beijing’s chagrin, it also notified users that some results weren’t being displayed. (Google even more famously pulled its search engine out of China altogether in 2010 when it discovered China-based hackers had infiltrated their systems.) Blogs, forums, email providers, and chat apps were all using some sort of automation to filter out content by the end of the decade. As social media platforms like Weibo and WeChat came online in the 2010s, they instituted some measure of automated filtering, as did online video games. Even nearly invisible technologies, like autosuggestion (the text that appears as you type in a search bar), are subject to filtering. As of 2022, Microsoft’s search engine Bing prevented its autosuggestion from offering up any terms the Chinese government considered “sensitive”—not just for users in China, but for users in the United States and Canada as well.
Platforms generally conduct automated censorship using keyword lists. The platforms’ software reviews all posts, and if a post contains one of the keywords, the software either removes the offending content or sends it to a human for further review. In some cases, the logic governing the use of keywords has become more complex over time; while certain terms used in isolation are enough to make a post disappear, other terms only cause a removal when used in combination with one another. Discussing censorship of news about human-rights atrocities in China’s northwestern Xinjiang region, the information control worked told the interviewer that “The term ‘concentration camp’ isn’t sensitive on its own. Only when it’s combined with ‘Xinjiang’ does it become sensitive.” Different keyword combinations can also result in different types of censorship. Researchers studying search engine censorship in 2023 found that certain keyword combinations triggered “hard” censorship (returning no results at all), while others triggered “soft” censorship (returning only pre-approved sources).
The complexity, and opacity, of this system has only increased as companies beef up their automated censorship with machine learning. The information control worker explained that before human censors have a chance to review it, “AI will already have blocked out a lot of content through keyword filtering . . . we content moderators can’t see that stuff.” But he also expressed confidence that humans would always have a place in the system: “human review is necessary. Machines can only moderate obvious content, while humans can detect subtleties.” The report analyzing job recruitment ads for information control workers supports this notion; it found no evidence that “technological breakthroughs in text or image analysis, company patents, and updates in censorship software services, have a significant impact on the recruitment of people undertaking censorship work.”
Automated censorship becomes even more complicated when considering the newest technologies, like large language models (LLMs) and AI chatbots, which must be registered with the Cyberspace Administration of China (CAC) and whose outputs must also conform to Chinese regulations. Chatbot censorship can happen at two stages. First, when companies are creating and training an LLM, they can limit what information the LLM has access to or otherwise instruct it to deliver censored responses to queries. Until recently, it wasn’t clear if tinkering with the model in this way would degrade the quality of the output or not, but at this point, it’s safe to say that Chinese LLMs perform well enough that this is no longer a concern. Second, a separate “post-training” review mechanism runs in real time while a user is interacting with the chatbot. The review mechanism checks both user inputs and chatbot outputs for “sensitive words” and disconnects the chat upon finding them. Chinese software company Qihoo 360 uses this “post-training” mechanism, along with human reviewers who help manually review the chats and update the model’s blocklist. Combined with “value alignment training,” in which developers seek to imbue the chatbots’ responses with “core socialist values,” this method helps keep Chinese chatbots on the political straight and narrow.
Several factors influence the types of response a China-produced chatbot might give you. Companies all develop or tweak their LLMs to their own specifications, so chatbots from different companies may not all echo Beijing’s talking points in the same way. Additionally, a Chinese chatbot running on a China-based cloud
server
will provide more censored answers than the same chatbot run on a foreign server—the separate “post-training” review mechanism checking the inputs and outputs isn’t present internationally. However, we know that some Chinese LLMs have at least some censorship baked into the first stage, meaning they will provide censored, or at least heavily biased, answers no matter where in the world you are.
No doubt tech companies will fine-tune their political monitoring as they continue to fine-tune the models themselves. The Chinese company DeepSeek, like LLM-makers the world over, is continually working to create models that produce fewer falsehoods and “hallucinations” at the same time it improves the models’ censorship capabilities. Researchers evaluating the LLM model DeepSeek updated on May 28 describe it as “the most strictly-controlled version of DeepSeek yet.” One of DeepSeek’s newer chatbots notably refuses to answer questions about topics like China’s invading Taiwan or the Tiananmen massacre, saying only that it was “not sure how to approach this type of question yet.” Or, the model provides an answer that it deletes seconds later, replacing it with “Let’s talk about something else.”
Companies have adapted their automated techniques to keep pace with pesky users who, for some reason, won’t stop wanting to express their unvarnished opinions online. As late as 2015, for instance, users were still able to use homophones to disguise their messages, which helped them keep their posts online three times longer than similar posts without homophones. (Both Mandarin and Cantonese are tonal languages, containing a large number of homophones, and use a character-based script, meaning one sound might be represented dozens of different ways in writing. For example, internet users briefly employed the phrase zhèngfǔ (正腐), or “totally rotten,” as a substitute for zhèngfǔ (政府), which means “government,” until the former was banned.) But by 2021 both Baidu and Tencent had begun checking for homophones, largely negating their effectiveness as a censorship circumvention tool.
Such technical censorship progress extends to non-textual content as well. WeChat had implemented automated image inspection by 2017 to stop users from sending banned images, like the famous photograph of the “Tank Man” in Tiananmen Square in 1989. The image inspection algorithms also stymied a popular technique for avoiding censorship: putting text over an image, which is easy for a human to read but harder for a machine to identify as text. Livestream platforms, too, use automated review mechanisms for audio and video as well as text. In its SEC filing for 2020, livestreamer Huya wrote,
Our automated AI-backed screening mechanism serves as the first layer of defense in our content review system. This system automatically flags and screens out live streams that involve inappropriate or illegal audio, video, comments or chats by comparing the image, sound or text against our databases in real time. Once the content is processed by our AI-backed automated screening mechanism, our system then extracts identifiers from the content and sends them to our manual content screening team, our second layer of defense, for further review.
As platforms have tweaked their censorship systems over time, they’ve also changed how they communicate with users about that censorship. As early as 2012, Weibo had begun “camouflaging” posts with banned keywords—that is, making it appear to users that they had posted successfully, while in fact hiding their posts from any other viewers. WeChat made a similar move in 2016, when it stopped posting censorship notifications, so users wouldn’t know that the person on the other end of their chats wasn’t receiving everything they sent. That same year, some Chinese search engines began removing their censorship disclaimers, which alerted users to the fact they were not receiving all relevant results. Platforms might also “shadowban” entire accounts, making a particular user’s content invisible to anyone but themselves, though this phenomenon has not been rigorously documented.
These changes exemplify a general shift on Chinese platforms from overt censorship (in which platforms explicitly tell users that content has been censored) to covert censorship (in which platforms hide censorship by not alerting users to the fact it has taken place). As of 2023, many popular online platforms, including search engines, video sharing services, and e-commerce sites, among others, engaged in covert censorship.
No matter the platform or censorship mechanism, companies are largely on their own when it comes to generating lists of banned keywords and images. Beijing does offer guidance on specific cases. In August 2023, for example, authorities demanded that platforms
Carry out a comprehensive cleanup of all content related to the band “Slap” (delete encyclopedia entries, search terms, videos, lyrics, and promotional content; delete topics and hashtags, shut down Baidu Tieba “topic bars,” and remove all related merchandise) and their songs (including “Red Child’s Eighteen Wins,” “The Eighteen Dark Arts of Master Bao,” “The Eighteen Generations of Uncle Pan,” “The Eighteen Hexagrams of Boss Bei,” “The Eighteen Prohibitions of Director Ma,” ‘The Eighteen Verses of Director Lang,” etc.) Content that exposes and criticizes [the band or their songs] will be allowed to remain online.
In the vast majority of cases, however, companies must use their own judgment and experience to update their own keyword blocklists. Studies show again and again that blocklists vary by company and over time, confirming that companies don’t receive comprehensive blocklists from the government and have to generate their own. This makes sense if Beijing’s goal is to encourage robust platform censorship: were the government to provide blocklists to companies, anything missing from those lists would serve as a get-out-of-jail-free card for platforms that might have noticed questionable content but chose to simply follow the blocklist. Instead, under the current system, a platform that’s not sure where exactly a red line is may choose to err on the side of caution and censor anything it thinks approaches the line.
Yet companies also have a financial incentive to allow engaging content to remain online as long as possible. According to research by computational social scientist Blake Miller, who examined censored posts and Weibo company chat logs, “platforms disobey directives to pursue market incentives misaligned with state censorship agendas . . . [this] challenges the common assumption that censorship outcomes reflect state censorship preferences. Instead, private platforms can subvert state actors’ information control agendas, especially when doing so improves user experience and engagement.” Or, as a separate team of computer scientists put it, “Weibo must conduct just enough censorship to satisfy government regulations without being so intrusive as to discourage users from using their service.”
And yet, individual users may still avail themselves of platforms that censor a bit more aggressively. Daniela Stockmann, Professor of Digital Governance at the Hertie School in Berlin, notes that “people go to different platforms based on certain topics. There is not really a market for more censorship in China. People are going to more censored platforms for other reasons,” like the types of users or conversations they find there.
This tension between China’s Party-state, which pushes platforms to assiduously scrub banned content the moment it appears, and tech companies, which want to maintain the most captivating online fora possible, breeds suspicion. Authorities don’t fully trust companies to implement censorship without supervision. In addition to periodic online clean-up campaigns (to keep the internet “clear and bright”), officials conduct more systematic checks of platforms’ censorship compliance. In 2020, according to Chinese government purchasing information, the CAC hired the Institute of Information Engineering at the Chinese Academy of Sciences to develop an automated system that would post illicit material to a variety of commercial platforms and test their responses in real time.
Platforms don’t just delete users’ posts. They also report certain users directly to government authorities. A 2021 study revealed how companies choose which users to “report up”: “Substantive topics, although relevant, matter less than user influence and virality,” authors Mary Gallagher and Blake Miller write. “A post’s content is less important than who is posting it and how many people are re-posting it.” Essentially, anyone with enough online influence will make their way onto the government’s radar via these company reports. The government can then co-opt or more closely monitor these “key opinion leaders” to make sure they stay in line.
“Reporting up” shows that authorities don’t just care about what is said online, but who is saying it. Popular personalities will draw far more attention from the government, while relative unknowns with limited follower counts might avoid official scrutiny. The nature of social media demands that authorities and companies take special precautions against unwanted virality; ensuring that certain individuals or their posts remain obscure can be nearly as effective as actually blocking them.
