Executive Summary

The age of information has brought with it the age of disinformation. Powered by the speed and data volume of the internet, disinformation has emerged as an insidious instrument of geopolitical power competition and domestic political warfare. It is used by both state and non-state actors to shape global public opinion, sow chaos, and chip away at trust. Artificial intelligence (AI), specifically machine learning (ML), is poised to amplify disinformation campaigns—influence operations that involve covert efforts to intentionally spread false or misleading information.(1)

In this series, we examine how these technologies could be used to spread disinformation. Part 1 considers disinformation campaigns and the set of stages or building blocks used by human operators. In many ways they resemble a digital marketing campaign, one with malicious intent to disrupt and deceive. We offer a framework, RICHDATA, to describe the stages of disinformation campaigns and commonly used techniques. Part 2 of the series examines how AI/ML technologies may shape future disinformation campaigns.

We break disinformation campaigns into multiple stages. Through reconnaissance, operators surveil the environment and understand the audience that they are trying to manipulate. They require infrastructure––messengers, believable personas, social media accounts, and groups––to carry their narratives. A ceaseless flow of content, from posts and long-reads to photos, memes, and videos, is a must to ensure their messages seed, root, and grow. Once deployed into the stream of the internet, these units of disinformation are amplified by bots, platform algorithms, and social-engineering techniques to spread the campaign’s narratives. But blasting disinformation is not always enough: broad impact comes from sustained engagement with unwitting users through trolling––the disinformation equivalent of hand-to-hand combat. In its final stage, a disinformation operation is actualized by changing the minds of unwitting targets or even mobilizing them to action to sow chaos. Regardless of origin, disinformation campaigns that grow an organic following can become endemic to a society and indistinguishable from its authentic discourse. They can undermine a society's ability to discern fact from fiction creating a lasting trust deficit.

This report provides case studies that illustrate these techniques and touches upon the systemic challenges that exacerbate several trends: the blurring lines between foreign and domestic disinformation operations; the outsourcing of these operations to private companies that provide influence as a service; the dual-use nature of platform features and applications built on them; and conflict over where to draw the line between harmful disinformation and protected speech. In our second report in the series, we address these trends, discuss how AI/ML technologies may exacerbate them, and offer recommendations for how to mitigate them.

Introduction

“Falsehood flies, and the Truth comes limping after it; so that when Men come to be undeceiv’d, it is too late; the Jest is over, and the Tale has had its Effect…"(2) — Jonathan Swift

Disinformation is a potent tool of geopolitical power competition and domestic political warfare, and one that has been gathering force. Sustained and well-funded disinformation operations are weaponizing the fractured information environment and creating real-world effects. Their use has generated public angst and media attention, yet their impact remains difficult to measure.(3) The word "disinformation" reentered U.S. political discourse in 2016, when Russian operators executed a campaign to influence the U.S. presidential election.(4) Dusting off “active measures” and dezinformatsiya––the Soviet-era terms for operations meant to meddle in the internal politics of rival nations––this campaign marked an escalation of the long-standing Russian efforts to discredit democratic institutions, sow discord, and undermine public trust.(5)

Russia reinvented the twentieth century “active measures” and repurposed common digital marketing techniques to usher in a new cyber-enabled era of asymmetric conflict in the grey zone between war and peace. Since the late 2000s, the Kremlin has waged increasingly brazen disinformation campaigns––operations meant to intentionally spread false or misleading information. These operations were initially honed against domestic opposition targets, then in the Baltics and Georgia, before reaching deadly potency in Ukraine.(6) While democratic governments and social media platforms were still learning about its new tactics, the Kremlin continued experiments to sow discord throughout Europe before setting its sights on influencing U.S. voters.

As targets changed, malicious actors innovated. Russia's early operations at home and in its near abroad used a model, sometimes termed as the "firehose of falsehood," to produce repetitive, high-volume messaging aimed at audiences on the political extremes to exacerbate tensions.(7) Bots and teams of human trolls amplified the messaging across social media platforms with support from pro-regime broadcast media.(8) The Russian campaigns launched against U.S. and European elections in 2016–2017 were informed by research into the political segmentation of target societies. They relied upon tactics such as hack-and-leak operations and forgeries to discredit political candidates unfavorable to the Kremlin. Proxy organizations made up of professional trolls worked to manipulate real people into real- world actions to exploit societal fissures, driving political polarization.(9) In recent years, these methods covertly cultivated unwitting users and amplified disinformation from homegrown conspiracy theory communities, including some anti-vaccination groups, QAnon, and domestic extremist organizations.(10)

While Russia was the pioneer, other states have also adopted these tactics. Chinese and Iranian disinformation operations, developed against their own internal opposition, are now used to sway international opinions in support of their geopolitical goals.(11) Both nations have stepped up malicious disinformation campaigns, including targeting––or contemplating to target––the 2020 U.S. presidential election.(12) Echoing well-known Russian tactics, China deployed a high-volume network of inauthentic accounts—across multiple social media platforms in multiple languages—to promote COVID-19 disinformation and conspiracy theories.(13) Today, some 81 countries use social media to spread propaganda and disinformation, targeting foreign and domestic audiences.(14)

Disinformation campaigns in the age of social media are notable for their scope and scale, and are difficult to detect and to counter. The tactics used in these campaigns capitalize on central features of today’s social media platforms––maximizing user engagement and connection. They exploit the biological dynamics of the human brain: cognitive shortcuts, confirmation bias, heightened emotion, and information overload that overwhelms cognitive resources under stress.(15) They seek to deepen the natural fissures within open societies, erode trust, and chip away at the sense of a common foundation of political discourse, which is critical to a functioning democracy.

Recent advances in artificial intelligence offer the potential to exacerbate the volume, velocity, variety, and virality of disinformation, automating the process of content creation and the conduct of disinformation campaigns. A crucial question is the degree to which applications of AI will change the largely manual operations of the past. There are many unknowns. Will AI only marginally impact the scale and reach of the current tactics, or will it change the game? Can the labor-intensive operations of Russia’s 2016 campaign be partially or fully automated in the future? Which techniques are likely to benefit from the rapidly evolving AI research, and how? Commentary and research have focused on synthetic video and audio known as “deepfakes.” However, there are other AI capabilities, such as powerful generative language models, conversational AI chatbots, and audience segmentation techniques that may be more impactful.

In this series, we examine how advances in AI and its subfield of machine learning (ML) are likely to enhance malicious operations to spread disinformation. In this report, we analyze the building blocks of disinformation campaigns from the perspective of those who build them, outlining common techniques threat actors have used in recent years. We offer case studies that illustrate these techniques, hinting at systemic challenges that perpetuate these operations and suggest how they may evolve.

In the companion paper in this series, we identify AI technologies, such as natural language processing (NLP), Generative Adversarial Networks (GAN), and Conversational AI, that may amplify future campaigns. We then offer recommendations for how governments, technology platforms, media, and AI researchers can prepare, thwart, and respond to the malicious use of AI/ML in disinformation campaigns.

The Disinformation Frameworks

Though no two disinformation campaigns are alike, they share common elements and follow patterns that exploit the underlying features of social media platforms. A number of researchers and organizations have built frameworks to conceptualize various aspects of these campaigns, including phases, objectives, types of messages, and tactics.(16) One conceptualization used within cybersecurity is that of the “kill chain,” which describes the various phases and associated tactics and techniques a hacker may use when conducting a cyber operation.(17) Clint Watt’s “Social Media Kill Chain,” the U.S. Department of Homeland Security’s “Disinformation Kill Chain,” and data scientist Sara Jayne-Terp’s Adversarial Misinformation and Influence Tactics and Techniques framework have all applied similar models to disinformation operations.(18)

It is challenging to capture the full spectrum of disinformation operations in a single framework explaining the various ecosystems of actors, tactics, and platforms. There is an additional difficulty of how to account for the role of traditional media in the ecosystem, whether as the target, the platform, or the witting or unwitting agent of influence. Given this complexity it is perhaps not surprising that the community researching, disrupting, and countering disinformation among the government, private sector, and civil society stakeholders has not coalesced on the single model akin to the cyber kill-chain. In this paper, we draw on the above frameworks and other research efforts to build a threat- model of disinformation campaigns.

The RICHDATA Architecture of Disinformation

Understanding the key stages of disinformation campaigns offers insight into how AI may enhance them. While strategies and intent vary by the actor and target––from sowing chaos to shaping public opinion––they share common tactics. To aid in understanding, this paper provides a framework, which we term RICHDATA, based upon the various stages often seen in disinformation campaigns. A continuous feedback loop runs through these concurrent stages to assess efficacy, monitor engagement, and refine the tactics. The pyramid shape reflects the greater importance of preparatory stages that are often overlooked in discussion of disinformation yet can be the most valuable and time-consuming for operators. The depicted stages can occur concurrently and are dynamic, evolving based on the feedback received throughout the campaign. The RICHDATA acronym reflects the critical role data and digital footprints play in both disinformation operations and machine learning.

In popular discourse, the term “disinformation” is often associated with bots and trolls on social media.(19) However, they are just a part of the picture. In this section, we describe various techniques disinformation operators use at each stage of campaigns, though this is by no means an exhaustive list. Defining the manual and automated parts of the job allows us to frame the spaces where AI/ML may enhance traditional automation, and those where operations are more likely to be run by humans.

(R) Reconnaissance: Understanding the Audience

Disinformation campaigns often begin with an examination of the target society and its information environment in order to shape and target their messages. Disinformation operators read and watch news and entertainment to understand cultural references, tracking content on traditional and social media to identify the themes and influencers driving the conversation. To perform this labor-intensive process, they build teams with the necessary language and data analytics skills, as the Internet Research Agency (IRA), the infamous Russian professional "troll farm," did in their campaign to influence the 2016 U.S. election.(20) Similarly, throughout its 2014–2015 military campaign in Ukraine, Russian operatives closely studied Ukraine’s media and political landscape, producing daily reports including topics covered in the local media, and the number of citations for the pro-Russian term "Novorossiya."(21)

Operatives seek to understand the historic grievances and fault lines of the target society in order to shape broad narratives into their specific messages and talking points. In 2014, Russian operators depicted Euromaidan and the Revolution of Dignity in Ukraine, mass protests against the corruption of the Yanukovych regime, as a “fascist coup.” They tapped into a sensitive fault-line–– the collaboration of some Ukrainian nationalists with Nazi Germany during World War II against the Soviets in return for the unfulfilled promise of Ukraine’s independence. Contemporary Russian narratives continue to project the actions of a few Ukrainians onto the entire nation.(22) They also ignore the contribution of Ukrainians to the underground resistance and to the allies’ war effort on the eastern front.(23) Historical narratives are a potent tool, and malicious operators can use them to develop specific themes that contain messages and keywords to include in their social media posts.(24) Messages are more likely to resonate when they align with a target’s pre-existing worldview, connect to a grievance, or contain a kernel of truth.(25)

Operators exploit societal fissures to segment audiences by their positions on divisive issues. They identify core influencers in each segment, as well as potential “super-spreaders” with large social media followings: public figures, officials, activists, and journalists. Manipulating super-spreaders into sharing a disinformation campaign message can pay off with broad resharing and even traditional media coverage.(26) The fact that the super-spreaders are unwitting perpetrators also helps to evade countermeasures by social media platforms.

Disinformation operators also exploit evolving crises and the ambiguity inherent in incomplete information with messaging to target audiences across the political spectrum. China’s and Russia’s efforts to target audiences in the United States and around the world with disinformation narratives about COVID-19 origins, treatments, and the efficacy of U.S. response present a compelling demonstration.(27) They also underscore the importance of developing narratives that tap into deeply rooted values and existential fear.

Exploiting Ambiguity Case Study: COVID-19 Disinformation
Incomplete and evolving information in a crisis, such as a global health pandemic, presents opportunities for disinformation actors to fill voids with alternative narratives that feed panic and undermine trust in government institutions and expertise. Aiming to sow fear and chaos, early in the pandemic, a China-linked source spread a message via social media and text messaging apps that warned Americans of the impending lockdown by the U.S. federal government.(28) Capitalizing on incomplete information about the origins of COVID-19 and U.S. conspiracy theorist’s claims linking U.S. scientists to the “gain of function” research, Chinese and Russian actors amplified the narratives that COVID-19 was engineered by the United States, exploiting the anti- government sentiment on the U.S. political extremes.(29) While this message did not resonate with U.S. audiences, others, such as those that promoted doubts about vaccines, spread more widely. The efforts to fuel vaccine hesitancy and discredit vaccines made by European and U.S. firms are ongoing. In August 2021 Facebook banned an inauthentic network originating in Russia that claimed AstraZeneca vaccine would turn people into chimpanzees and Pfizer vaccine caused higher casualties than other vaccines.(30) The network was linked to Fazze, a UK-registered marketing firm operating in Russia, which also recruited YouTube influencers to claim that Pfizer vaccines were deadly.(31) The campaign supported its claims with a document allegedly hacked and leaked from AstraZeneca, which in reality was forged from different sources and taken out of context.(32) This activity targeted countries where Russia wanted to sell its own Sputnik-V vaccine and coincided with the amplification of the same narratives on Russian state-sponsored media and false news outlets operated by Russian intelligence.(33) Ironically, Russia’s promotion of vaccine hesitancy around the world is backfiring at home: as of June 2021, only 11 percent of Russian population had been fully vaccinated.(34)

Having segmented their audience, operators identify the best channels to reach them. In their reconnaissance efforts targeting the U.S. 2016 elections, Russian operators researched groups on social media dedicated to political and social issues, tracking metrics such as size, frequency of content, and the level of audience engagement.(35) To aid in their audience segmentation efforts, operators may leverage the automated advertising features of social media platforms. As of 2019, digital advertisers could target their audiences through hundreds of thousands different attributes on mainstream platforms.(36) Anyone with a personal Facebook account can test the advertising mechanisms, targeting audiences based upon demographic data (e.g., age, gender, location) and the more personal information users share with the platform: interests, hobbies, politics, and life events.(37) Twitter provides potential advertisers with information on what a user searches for or views, their location, and apps installed on their device, as well as targeting based on interests, attended events, and an option to target a specific account’s followers.(38) Microtargeting through advertising remains relatively easy, despite recent restrictions and changes to advertising policies to limit political ads on major platforms.(39)

(I) Infrastructure

Key elements of campaign infrastructure include a digital army of fake personas, computational propaganda tools (e.g., automated accounts), channels for distributing disinformation (e.g., social media groups and pages), and websites for hosting false content. In addition, threat actors need tools to cover their tracks, and financial infrastructure to purchase ads, such as PayPal accounts or credit cards under stolen identities.

Inauthentic personas, or “sock puppets,” are the digital soldiers of an online disinformation campaign. Threat actors can create these personas in-house, acquire them on the dark web, or rent them from the growing coterie of firms that offer influence as a service. They can also rent authentic accounts of witting or unwitting humans or pay them to propagate campaign messages. As social media platforms get better at identifying inauthentic activity, threat actors increasingly resort to burner accounts with a short life span that blast disinformation widely and then disappear. These accounts may be effective for seeding a conspiracy theory narrative or injecting forgeries into the information stream, but seldom reach the audience engagement levels of curated inauthentic personas.(40)

Threat actors place a premium on messengers that appear authentic. These personas can be highly curated, including a profile photo, biographic data, presence across multiple platforms, and a history of relevant posts. To develop realistic personas, threat actors scrape photos from public social media profiles, stock photos, or images of lesser-known celebrities. Operators increasingly use new ML techniques that can generate realistic human faces, providing a pipeline of untraceable avatars. These are becoming more important because threat hunters can use a reverse image search to identify stolen photos. A degree of credibility is necessary to set up new groups or hijack existing ones, and to evade automated detection systems, which look for accounts with a lack of history of posting locally relevant content, fewer connections, and recent creation date.(41)

Building Personas Case Study: The Russian IRA Impersonates the American Political Spectrum
The IRA's posts from a small group of highly curated accounts––created with a focus on credibility––were the most successful in getting traction online. Some of the most active inauthentic accounts in the IRA's disinformation campaign against the 2016 U.S. presidential election were personas imitating Black Lives Matter activists and the Tennessee Republican Party.(42) Out of 1,500 fake accounts the IRA deployed in the English-language operation, this group of 300 personalized accounts garnered 85 percent of all retweets in English (18.5 million retweets of 1 million tweets).(43) These accounts were “consistent, persistent, political personas that reflected caricatures of U.S. political participants” and evolved to “behave like real people.”(44) These personas allowed the IRA to infiltrate existing grassroots activist communities on both sides of the political spectrum, in some cases coordinating with real activists to organize protests.

Threat actors in a hurry can outsource persona creation and rent a digital army. The growing “influence as a service” industry ranges from seemingly legitimate marketing and PR firms to outfits selling “armies” of inauthentic accounts, likes, and comments via the dark web.(45) This practice, however, carries risks. These inauthentic accounts may have been used in other campaigns and can be easier to detect. Nevertheless, they can be useful to amplify or inflate follower numbers for a narrowly targeted campaign.

As platforms get better at detecting fake accounts, operators may resort to purchasing or renting authentic user accounts. The technique of “franchising,” paying witting or unwitting individuals to establish online personas and spread disinformation, has been documented several times, including during the 2019 elections in Ukraine and in the 2020 IRA-linked operation in Ghana and Nigeria targeting the United States.(46) This technique makes operations look as if they are perpetrated by authentic users. Due to heavy human resources, such operations may be more expensive to scale and sustain overtime.

Threat actors with limited resources, or those looking for a quick ramp up, may use automation to build accounts en masse. This tactic is part of what some researchers call “computational propaganda,” the use of automated accounts (bots) and scripts to publish posts.(47) Social bots are user accounts automated to interact with other user accounts.(48) This involves writing computer code that will talk to the platform application programming interface (API), automatically generate bot accounts, connect them into networks of bots, aka 'botnets', and automate posting. While the platforms have enhanced countermeasures to detect botnets, their presence persists. In the 2020 U.S. election, bots accounted for 13 percent of users engaging with conspiracy theory content and reshared hyper-partisan content “in a systemic effort to distort political narratives and propagate disinformation.”(49) In recent years, the use of bots has grown exponentially, fueled in part by availability of open-source code to build them at more than four thousand GitHub sites and over forty thousand other public bot code repositories on the internet and the dark web underground market.(50)

Accounts created by automation often betray their inauthenticity through metadata or behavior. Telltale signs include missing avatars, mismatched male/female names, names that resemble computer code, or accounts created within seconds of each other. These accounts may also publish posts at the same intervals and retweet every few seconds, faster than the average human could. These technical markers can be detected by the social media platform algorithms.(51) Russian IRA operators built a human operation in part because their bots were getting shut down, a lesson that China-linked actors also learned in their 2020 “Spamouflage Dragon” campaign.(52)

Having built personas to propagate their messages, disinformation operators may create groups and pages to begin building an audience. These need time to build credibility. Often groups start with pre-populated inauthentic personas to give the appearance of legitimacy before they can attract authentic members. One of the IRA's top performing Facebook pages, "Blacktivist," cultivated a large community with messages of empowerment aimed at African Americans and garnered 11.2 million engagements from authentic users.(53)

Operators may also create general interest pages on innocuous subjects, such as food or tourism, to reach a broader audience of authentic and inauthentic users and then pivot to disinformation content later.

Building Channels Case Study: A Road Trip to Falsehood
A covert Russian network of accounts and pages linked to Rossiya Segodnya and Sputnik, Kremlin-affiliated media outlets, encouraged followers to travel around the Baltic nations, featuring food and health tips,(54) and occasionally interjecting Sputnik stories with pro-Russia and anti-NATO messaging.(55) Through the use of innocuous content in its 266 pages, the network amassed an audience of 850,000 followers, ready to ingest a coordinated disinformation campaign. Come for the pictures of beautiful Riga, stay for the falsehoods.

Disinformation operatives use audience-building techniques such as infiltrating authentic groups. Social media platforms allow and encourage the creation of interest-based subpopulations, giving threat actors easy targets for infiltration and ready-made audience segmentation. In response to the privacy concerns following the 2018 Cambridge Analytica scandal, Facebook's change to promote private interest-based groups more prominently on users’ feeds had unintended consequences.(56) Promotion of these private groups, some with tens of thousands of members, opened an avenue for malicious actors to identify politically interested communities, infiltrate them, and build trust before deploying deceptive content.(57) While Facebook scaled back on promotion of political interest groups on its platform and now requires stringent content moderation from group administrators, the avenue of hijacking innocuous Facebook groups to push disinformation remains open.(58) Telegram users can create channels of two hundred thousand members and protect messages with end-to- end encryption and self-destruct capability.(59) Features designed to build trusted spaces can give operators another way to target and build an audience, presenting an ongoing challenge for social media platform engineers, policy managers, and threat hunters.(60)

Tools of operational security (OPSEC) are a key component of building or acquiring infrastructure. As disinformation operators create personas, activate channels and, in some cases, build a web presence, they use tools to hide their operations. In a well-run operation, threat actors cover their tracks as they register domains and create inauthentic accounts. This includes identity-protection measures like proxy servers and virtual private networks (VPN) to mask their IP addresses as they register domains and create accounts to appear to originate from a geographically appropriate location. The reuse of email addresses, domain registrars, or hosting services can expose the operator.

Technical markers identify disinformation operations, so advanced actors have to be hypervigilant. In one case, a small slip in operational security––the failure to use a VPN while accessing a social media profile of “Guccifer 2.0”––allowed U.S. intelligence analysts to trace this inauthentic persona to an officer at the Moscow headquarters of the Russian military intelligence service (GRU). This proved that Guccifer 2.0 was a Russian operative and not a Romanian hacker claiming to have hacked and leaked emails from the Democratic National Committee.(61) That said, some operators want to be discovered. Creating a perception that they have greater impact than it is in reality, known as “perception hacking,” helps malicious actors weaponize distrust, play into the societal expectations of foreign interference, and sow doubt in the integrity of institutions.(62)

(CH) Content Creation and Hijacking

Content is the fuel of disinformation campaigns. After identifying narratives and target audiences, operators seek to create a steady stream of engaging content to shape the saturated information environment, marked by a 24-hour news cycle and competition for attention.(63) The process of content development is iterative and labor intensive, demanding responsiveness to user engagement, refinement, and a ceaseless stream of content.

During the Russian IRA operations, six hundred operators worked in its 40-room St. Petersburg headquarters, behind doors with labels such as “graphics,” “blogging,” “video production,” and “social media specialists.”(64) Managers placed a premium on creating compelling content that would go viral organically.(65) The human content farm churned out thousands of posts for the required weekly “technical tasks” reiterating its themes through talking points and “must have” keywords.(66) At the height of the 2016 campaign, one hundred operatives from the IRA’s American Department created one thousand unique pieces of content per week, reaching 30 million users in the month of September, and 126 million over the course of the election, on Facebook alone.(67) This scale requires a dedicated team, skilled in creating specific types of content, with nuanced command of the language and culture of the target society.

Operators create content to elicit a range of emotional responses: posts that are uplifting and positive, angry and negative, or neutral and unrelated to the campaign. Different types of content are useful for different campaigns. Neutral content helps build audience and funnel traffic towards pages operated by the threat actors, while positive posts can help operators infiltrate and gain credibility with authentic groups. They can also garner more engagement. Counterintuitively, the Russian IRA’s posts with most likes and reshares on Facebook leading up to the 2016 U.S. election were benign and positive, rather than polarizing, in emotional profile.(68) That said, threat actors continue to deploy negative messaging in campaigns when it serves their purposes. For example, COVID-19 disinformation efforts targeting Indo- Pacific countries have predominantly used negative emotional hooks, such as anger, in their messaging.(69)

The types of content range from short messages and medium- length articles to visual media. Operators can hijack posts from authentic users, screenshot and repost them with a slant. Screenshotting helps hide the operator’s origins, minimizes the effort entailed in creating content from scratch, and retains the authentic voice of local users. These techniques can help avoid common grammatical errors or misused idioms that can be red flags to a vigilant target audience or a social media threat investigator.

Visual content and memes, in particular, are a potent tactic of disinformation. The interplay between image and text increases the virality of memes and challenges platforms' detection algorithms.(70) Memes capitalize on both the human ability to quickly process imagery and their own quality to “shed the context of their creation.”(71) Once amplified, the best memes lose ties to their creators, are reinterpreted, and integrate into the consciousness of the internet, becoming collective cultural property.

Prioritize Visuals Case Study: #DraftOurDaughters Memes
The “#DraftOurDaughters” campaign, created by anonymous users on the 4chan fringe network in the final week of the 2016 U.S. presidential campaign, spread falsehoods to mainstream platforms about candidate Hillary Clinton.(72) Disinformation operators pushed memes masquerading as Clinton campaign ads, alleging the candidate’s intention of instituting a mandatory draft of women into the military.(73) Private individuals found their wedding photos scraped and turned into memes with the "#DraftOurDaughters" and "#DraftMyWife" hashtags.(74)

From their innocent origins, memes offer a potent tool(75) that operators have weaponized not only against political candidates but also journalists and ethnic minorities.(76) In 2016–2017, the Myanmar military pushed anti-Muslim Facebook posts against the Rohingya population before the platform shut them down. Offensive imagery thinly veiled in a cartoon form contributed to the spread of violence through their ease of interpretation and crude attempts at humor, unleashing real-world horror.(77)

The leaking of embarrassing or compromising content, sometimes laced with forgeries, is a key tactic used by threat actors.(78) Threat actors can develop forgeries and leak them, claiming they are authentic hacked information. They can hack targets' network to acquire compromising content or purchase hacked content on the underground market, then enhance this illegally obtained material with forgeries.

(Hack)-Forge-Leak Case Study: Ghostwriter and Secondary Infektion
Hacking and digital forgeries are a tactic of choice for Russian state-affiliated actors. Operation Ghostwriter distributed forged military documents alleging that NATO troops were responsible for the spread of COVID-19 in Europe. It also published anti-NATO op-eds authored by fake personas impersonating journalists and invoking forged “leaked” emails between officials at NATO and national governments of Estonia, Latvia, Lithuania, and Poland.(79) Leading up to the 2021 German elections, actors connected to this group escalated their efforts to hack members of the German Parliament to obtain compromising information for use in pre-election disinformation campaigns. This prompted German and European Union authorities to call on Russia “to adhere to norms of responsible behavior in cyberspace.” Despite this call, security researchers at Mandiant recently linked this campaign to Belarusian government, suggesting that Russia’s techniques are diffusing.(80) Similarly, Operation Secondary Infektion targeted the United States and its allies with forgeries aiming to stoke diplomatic tensions between NATO allies and partners.(81) In six years, over 2,500 pieces of forged content appeared in seven languages on three hundred platforms.(82) Only a small portion of activity surfaced on Facebook, with most of the campaign percolating through small platforms that lacked threat hunters.(83) Higher quality digital forgeries would make this tactic more effective, particularly if mixed with authentic leaked documents. Secondary Infektion showed the difficulty of detecting operations that are rolled out across multiple platforms, an increasing trend.

(D) Deployment

After setting up infrastructure and crafting content, operators put their preparation into action by interacting with authentic users. In the deployment stage, operators identify and activate the persona(s) that will deliver the initial payload. They prepare the payload in the chosen form, such as a blog, a video, a social media post, or a meme. They then drop the payload into targeted channels, such as a conspiracy theory forum, a social media page, a false news website managed by the operators, an authentic pre- infiltrated group, or a persona’s social media feed.

Disinformation actors often begin their campaigns away from mainstream social media platforms, which are moderated. They may instead seed disinformation content to an obscure website or channel and reference it repeatedly on mainstream platforms. In recent years, several state-sponsored actors have deployed content in the form of false news stories and research by establishing fake news sites and think tanks. In 2018, an Iranian disinformation operation used six public-facing websites to push content aligned with Iranian national security interests.(84) The sites linked into a web and hosted both original and hijacked content. Likewise, Russian actors operate multiple proxy sites, such as Strategic Culture Foundation, SouthFront, and Geopolitica.ru, and publish disinformation disguised as analysis in support of the Russian government interests.(85)

(A) Amplification: Pushing the Message

With content created, operators turn their efforts to amplification–– exposing their message to the maximum number of eyes and ears, getting it picked up by trending topics or hashtags and spread organically. If authentic voices engage and unintentionally spread malicious content, content moderation is more challenging for the platforms.

Operators push the payload across multiple social media platforms to maximize exposure. Research on cognition and disinformation stresses that the first engagement with false information leaves a lasting imprint, particularly if it is seen via trusted networks of personal connections.(86) For this reason, efforts to debunk disinformation often fall short of the scale and reach of the initial interaction and fail to fully dissuade the previously targeted audience.(87) A diversity and variety of channels all pushing the same deceptive message enables threat actors to mask coordination, appear authentic, and ensures repetitive, high-volume messaging.

Resharing Content Case Study: China’s “The Spamouflage Dragon”
In 2020, China launched a disinformation operation in an effort to shape positive perceptions of its handling of the COVID-19 pandemic. The campaign, conducted on Twitter, relied on 23,750 core posting accounts and 150,000 additional amplifier accounts designed to “artificially inflate impressions metrics and engage with the core accounts.”(88) While the majority of these were in Chinese, around 9.4% were in English and 1.8% - in Russian.(89) The English- language tweets focused on COVID-19, pushing narratives from prominent Chinese state media accounts and Chinese officials, arguing that China was both transparent and efficient in their response to the outbreak.(90) Technical indicators linked this operation to the same actor spinning a pro-PRC portrayal of Hong Kong protests and responsible for a network of two hundred thousand accounts previously suspended by Twitter in August 2019. The network “Spamouflage Dragon” also posted video content extensively on YouTube and amplified the same content across Twitter and Facebook.(91) Recent findings suggest that the same network operated far beyond the mainstream platforms: posting thousands of identical messages across 30 social media platforms and 40 other online forums in additional languages, including Russian, German, Spanish, Korean, and Japanese.(92)

Operators may also use the tools of computational propaganda, launching bots to barrage the followers with queued-up posts linked to payload content. Bots can automate tasks such as rapidly posting links to media outlets or categorizing tweets by high- volume influencers. They can also automate interactions with other user accounts and hijack conversations.(93) In the context of disinformation campaigns, bots act maliciously by impersonating real users. Networks of bots or botnets can act in concert to distort conversations online by driving up “likes” and retweets, hijacking trending hashtags and topics by spamming them with unrelated content or sharing links to websites that publish disinformation.(94)

For example, a legitimate nonprofit group created the hashtag #SaveTheChildren to spread awareness about child trafficking and fundraise to combat it. Followers of the QAnon conspiracy theory hijacked the hashtag, expanding their reach through misuse of the platform features.(95) These techniques draw attention of unwitting users to the topic or a hashtag and expose them to the disinformation payload.

Threat actors may also deploy custom or third-party applications that leverage platform features to manage multiple bots and botnets. Many platforms expose hooks into their platform for programmers to encourage the development of engaging apps and advertising tools within their platforms. While users experience the social media platforms through the front end of a website or an application, bots access information and functionality of the platforms by “talking” to the API through code.

Through this access, digital marketing applications can automate the propagation of content for campaigns across multiple social media networks. This allows bots to collect the data from the social media site more efficiently than from the front-end-user’s view of the platform, and then complete any action that a regular user can, including posting and interacting with content. Through the Twitter API, researchers and businesses can access conversations in real time, analyze past related posts, measure tweet engagement, monitor for news events as they break, or explore a specific account and its history in depth.(96) Services like MasterFollow, Botize, and UberSocial allow users to upload large amounts of content, manage delivery schedules, connect multiple automated accounts, and integrate bots across multiple platforms.(97) In an effort to combat spammy behavior, Twitter increased restrictions in access to data through the API, particularly by governments, and prohibited users and applications from simultaneously posting "duplicative or substantially similar" tweets across multiple accounts.(98) However, its policy still permits automation of multiple accounts for "related but non-duplicative use cases." (99) This enables coordination as long as accounts appear adequately authentic and the messages are sufficiently varied.

These programming hooks have many legitimate uses but can be misused by threat actors. The “good” bots that push notifications from the traditional news outlets to Twitter feeds are built on the same infrastructure as the “bad” bots that push disinformation payloads. The same APIs can enable malicious actors to post high volumes of disinformation content and ensure a continuous and repetitive stream of content across platforms. This dual-use nature of API access requires the platforms to perform nuanced detection and vetting to distinguish legitimate use and misuse, particularly as disinformation actors can masque their actions through third-party applications and firms.

It is not just bots that may spread the message. In 2020, private firms in 48 countries have offered automated and human-powered inauthentic amplification as a service and deployed it on behalf of a political actor to shape public conversation, domestically or internationally.(100) Since 2009, these firms have spent almost $60 million on computational propaganda. These firms offer a variety of services including the creation of inauthentic personas (sock- puppet accounts), micro-targeting, and message amplification with fake likes and fake followers, allowing threat actors to outsource disinformation operations and complicate attribution.(101)

While the marketing claims of some of these “amplification as a service” firms can be dubious or overblown, threat actors are turning to them. A large takedown by Facebook in July 2020 implicated a Canadian political strategy consulting firm Estraterra in a campaign of coordinated inauthentic behavior. The campaign targeted Ecuador, El Salvador, Argentina, and other countries in Latin America around elections, aiming to manipulate political debate.(102) This technique has cropped up around the world, including the case of an Indian public relations firm creating inauthentic bot accounts to spread an anti-Saudi and pro-Qatari narratives in the Gulf.(103) The services can be relatively inexpensive. In Taiwan, “cyber armies” cost as little as $330 per month.(104)

In addition to relying on bots or outsourcing, operators can also turn to key individuals in the social media ecosystem, super- spreaders, the “verified” influencer accounts belonging to public figures, celebrities, politicians, media personalities, and journalists, whose audience reach helps increase distribution. When journalists discover and report on public figures who have been duped into spreading disinformation, the additional media coverage further exposes and amplifies the campaign’s messages.

Hack and Leak Case Study: UK Trade Leaks
Secondary Infektion threat actors combined hack and leak techniques with superspreading to powerful effect in the 2019 UK parliamentary election. The threat actors known for their six-year campaign that used forgeries of U.S. and European government documents deviated from their tactics by injecting leaked authentic documents containing details of the US-UK trade agreement negotiations.(105) Oblivious to their origins, Jeremy Corbyn, the leader of the Labour Party, re- amplified the leaked documents at a campaign event. The media picked up the story, assuring wall to wall coverage for days leading up to the election.(106)

The implications of covering hacked material by the press are far reaching. Together with domestic political actors with vested interest in spreading such material to undermine their opponents, the media can play an important, if sometimes unwitting, role as a vehicle of amplification.(107) In 2016, the disinformation operation by the GRU precisely targeted the media into amplifying leaks of hacked and slightly forged information from the Democratic National Committee, the Democratic Congressional Campaign Committee, and the Clinton campaign.(108) The hackers integrated the stolen information into the broader campaign to manipulate the media into promoting the material. The coverage played into the narrative of Secretary Clinton and "lost emails." Mainstream media is in a difficult position when deciding whether to publish hacked and leaked material, and must determine whether the public’s right to know outweighs the potential harm caused by unwittingly amplifying a disinformation campaign. As long as there is lack of consensus on disclosure, operators will continue to rely on this tactic.

Distribution of malicious content is not limited to super-spreaders or curated groups, as operators may spread bespoke payloads into authentic online communities representing different sides of a divisive issue. By infiltrating authentic communities, threat actors can blend in, exploit the preexisting biases, and ensure the disinformation seeds can root and grow to spread organically to other adjacent sympathetic audiences.

Homegrown conspiracy theory communities provide fertile ground for “information laundering” by foreign and domestic actors.(109) This technique draws narratives from fringe sources, cultivates them, and amplifies them into the mainstream to gain legitimacy. Researchers attribute the rapid growth of QAnon conspiracy communities in part to the technique of cross-posting QAnon content into loosely related groups on mainstream platforms, quickly drawing them into a single anti-establishment tent.(110) Threat actors have leveraged these communities as a pipeline of ready-made disinformation that fits their goals––to undermine trust and sow discord. Russian disinformation actors historically cultivated and exploited conspiracy theories, and in recent years have engaged and amplified anti-vaccination groups and the QAnon networks.(111) Chinese disinformation operators likewise resorted to this technique to fan COVID-19 origin conspiracies as we described in the COVID-19 disinformation case study earlier in this paper.

Operators can use the recommender algorithms of social media platforms to their advantage, both to identify adjacent interest- based groups and communities to cross-pollinate and to draw more users into the conversation. One technique is to manufacture debates to generate activity, artificially inflate engagement, and make the online dialog appear lively for any authentic observer. Because many platform algorithms prioritize high user engagement, this inflated activity causes social media recommendation systems to push content with higher levels of engagement across more networks, thereby exposing more eyes to the disinformation content. We discuss recommendation algorithms in the second installment of this paper series, which focuses on AI/ML technologies.

(T) Troll Patrol: Controlling the Message

Disinformation thrives on dynamic and nuanced engagement. While blasting deceptive narratives across the social media ecosystem in a one-to-many amplification campaign can get the message before the maximum number of people, organic debates help disinformation to gain a foothold and thrive. Legitimate users are forced to expend cognitive and emotional resources defending their positions, and the platform’s algorithms are likely to prioritize lengthy debates and push them into additional feeds. This is a theatrical and labor-intensive exercise. Whether the goal is to cement a specific message or to sow discord, inauthentic personas engage authentic users––and each other––in elaborate back-and- forth arguments. Malicious actors measure their success––and the size of their paycheck––in the volume and the intensity of exchanges that drive up controversy and draw in authentic viewers.

Controlling the Narrative Case Study: The Downing of MH17
In the immediate aftermath of the MH17 crash over eastern Ukraine in the summer of 2014, trolls swarmed a variety of posts about Russia’s lack of culpability, peddling many shades of falsehood.(112) The IRA’s Ukraine operation monitored related conversations and steered the arguments towards several familiar themes in an effort to control the narrative. Some suggested that the flight was full of passengers who were already dead. Others pointed the blame at Ukraine with narratives such as “the Ukrainian air force shot down the plane” or “the missile came from the Ukrainian government-controlled territory.” These claims were debunked by satellite imagery, intercepted communications between Russian proxy forces in the Donbas, and open source intelligence tracing the journey of the Russian missile system across the Ukrainian border.(113) These efforts infused falsehoods into the comment sections and social media pages of reputable international media outlets, leaving audiences confused and misinformed.(114)

Manipulation of the target audience is often dynamic, iterative, and agile. Human trolls with “hands on the keyboard” switch methods based on the desired effect.(115) They may use provocation techniques to start arguments within posts, private groups, and on media publications by adding comments that raise the temperature of debate. In other cases, trolls may subvert authentic conversations by sharing information that advances the troll’s narrative. Through social engineering, they may incite users to join a group, observe a boycott, or vote for a candidate. They use ad hominem attacks to discredit authentic users, delegitimizing specific positions. Operators use diversion techniques and nuisance attacks to subvert a thread, derailing discussion and irritating other users. Swarming techniques “flood the zone” with a counter narrative.

If the goal of a campaign is to drive a particular message, trolling techniques must control the narrative. Trolls swarm threads and debate legitimate users in order to route the conversation in the desired direction. Some trolls coordinate entirely fake debates.(116) In an interview with The Washington Post, a former IRA troll described operatives creating plays in three acts. One would post a negative opinion about an issue. The others would comment in disagreement with supporting links. Eventually the first naysayer would succumb to the “evidence” and declare themself convinced, providing an illusion of consensus to unwitting audiences.(117)

If the goal of a disinformation campaign is to disrupt, divide, and antagonize, trolling techniques focus on provoking the target audience to react emotionally. The more dialog and greater degree of manipulative responses in comments, the higher the volume of engagement from authentic users. Accusing a threat actor of being a troll or spreading disinformation tends to lead to denial, counteraccusation, or “what-about-ism”––all aiming to push the target into a heated argument.(118)

As with amplification, the task of the trolls is to drive up engagement. Professional trolls are paid by the number of comments they post and reactions they get, while real people may be drowned out, whipped into a fury, or even driven to disconnect.

(A) Actualization: Mobilizing Unwitting Participants

To sustain itself, and to avoid detection, a campaign needs to attract authentic messengers. Ideally, disinformation operators can reduce their direct involvement, as the target audience, comprised of authentic users, begins to create organic content to support the injected narratives. At this stage, the line between foreign and domestic disinformation blurs, as does the line between intentional disinformation and unintentional misinformation. The campaign may move into the physical world, culminating in the mobilization of unwitting participants through protests and rallies. The pinnacle of a disinformation campaign is the creation of a state of contradiction: a readiness to mobilize and a disempowerment to act. At this stage, a threat actors’ approach is targeted and more closely resembles the activation of an intelligence asset rather than widespread opportunistic and exploratory messaging fanned through the social media ecosystem by bots. If a campaign is to have a sustained effect and continue rooting and growing in the target society, it needs authentic messengers and an organic following. Threat actors may precisely identify susceptible individuals, cultivate them into an operational role, and incite them into political action. Radicalized susceptible individuals can step into the roles of disciples and carry on the campaign.

Actualization and Mobilization Case Study: From Hot Dogs to High Noon
Disinformation campaigns may culminate in real-world confrontation. Early in its U.S. operations, the IRA tested its capacity to have an impact on the ground half a world away. In 2015, operatives posted an event on Facebook offering free hot dogs at a certain time and place in New York City. They watched through a live cam as New Yorkers showed up looking for the promised hot dogs. The test exceeded expectations, opening operatives’ eyes to new possibilities: using the “event” feature on Facebook.(119) Having tapped into a cleavage—racial, ethnic, and religious tensions—in American politics, the IRA chose its target. One May afternoon in 2016, followers of the Facebook page “Heart of Texas” marched in front of the Islamic Da’wah Center in Houston, Texas, brandishing confederate flags and wearing shirts that read “White Power.” The protesters had joined the vitriolic Facebook page and viewed ads promoting a rally to “Stop the Islamization of Texas.” Waiting at the center were other Texans manipulated by the IRA: members of another Russian page, “United Muslims for America.” Like a scene from a western, the two groups came face to face on Travis Street, waving banners and hurling insults at one another.(120) They were unaware that their confrontation had been arranged for roughly $200-worth of advertising space.(121) Though protests were small, the IRA had successfully exploited one of democracy’s most sacred tools––the freedom to peacefully protest.

At this stage, the narratives of the disinformation campaign gain a foothold in the authentic population, blending and reinforcing their own sentiments. Attribution to threat actors becomes more challenging as more authentic voices amplify the campaign.

Conclusion

The RICHDATA framework provides an overview of the core aspects of modern disinformation campaigns. Threat actors use a variety of techniques across many stages to achieve their objectives. After surveying their target audience and building the necessary infrastructure to support a campaign, these operators begin to seed the information environment with tailored content often hijacking legitimate online forums. As they disseminate their message, they apply a variety of amplification techniques to broaden their reach and impact. If authentic users stand in their way, they may apply a variety of trolling techniques to marginalize dissenting voices while leveraging sometimes unwitting super- spreaders to propagate their message. At the pinnacle of a disinformation campaign, threat actors may operationalize their target audience and mobilize them to action.

The case studies highlight systemic challenges in the current information environment. The lines between foreign and domestic disinformation operations are blurring, particularly with the outsourcing of these operations to companies that provide influence as a service. The dual-use nature of platform features and applications built upon them makes vetting and due diligence critical. Lack of consensus over where to draw the line between harmful disinformation and protected speech requires platforms to make decisions that are politically unpopular.

Many of the tactics discussed in this report could become even more powerful with the application of ML techniques, the subject of our second paper in this series. In that paper, we discuss how advancements in AI/ML may augment the operators in each phase of a disinformation campaign and exacerbate these systemic challenges.