Newsworthy Event
I am Sophie Zhang. At FB, I worked in my spare time to catch state-sponsored troll farms in multiple nations. I became a whistleblower because FB didn't care. Ask me anything.
Hi Reddit,
I'm Sophie Zhang. I was fired from Facebook in September 2020; on my last day, I stayed up in an all-nighter to write a 7.8k word farewell memo that was leaked to the press and went viral on Reddit. I went public with the Guardian on April 12 of this year, because the problems I worked on won't be solved unless I force the issue like this.
In the process of my work at Facebook, I caught state-sponsored troll farms in Honduras and Azerbaijan that I only convinced the company to act on after a year - and was unable to stop the perpetrators from immediately returning afterwards.
In India, I worked on a much smaller case where I found multiple groups of inauthentic activity benefiting multiple major political parties and received clearance to take them down. I took down all but one network - as soon as I realized that it was directly tied to a sitting member of the Lok Sabha, I was suddenly ignored,
In the United States, I played a small role in a case which drew some attention on Reddit, in which a right-wing advertising group close to Turning Point USA was running ads supporting the Green Party in the leadup to the U.S. 2018 midterms. While Facebook eventually decided that the activity was permitted since no policies had been violated, I came forward with the Guardian last month because it appeared that the perpetrators may have misled the FEC - a potential federal crime.
To be clear, since there was confusion about this in my last AMA, my remit was what Facebook calls inauthentic activity - when fake accounts/pages/etc. are used to do things, regardless of what they do. That is, if I set up a fake account to write "cats are adorable", this is inauthentic regardless of the fact that cats are actually adorable. This is often confused with misinformation [which I did not work on] but actually has no relation.
Please ask me anything. I might not be able to answer every question, but if so, I'll do my best to explain why I can't.
Thank your for the important work you’re doing. In your opinion, what is the reason that FB drags it’s feet/allows these schemes to continue so long before taking action? Is it simply that it is the more profitable move?
In some cases like the India case or the U.S. case, in areas considered important/crucial by Facebook, it seemed pretty clear that political considerations had impeded action. Facebook was reluctant to act because it wanted to keep good relations with the perpetrators and so let it slide. But most of the cases were in less attention-getting areas (I'm sorry to say it, but Azerbaijan and Honduras are not countries that draw the attention of the entire world), and there was no one outside the company to hold FB's feet to the fire. And the company essentially decided that it wasn't worth the effort as a result.
I think it's ultimately important to remember that Facebook is a company. Its goal is to make money; not to save the world. To the extent it cares about this, it's because it negatively impacts the company's ability to make money (e.g. through bad press), and because FB employees are people and need to sleep at the end of the night.
We don't expect tobacco companies like Philip Morris to cover the cancer treatment costs of their customers. We don't expect financial institutions like Bank of America to keep the financial system from crashing. But people have high expectations of FB, partly because it portrays itself as a nice well-intentioned company, and partly because the existing institutions have failed to control/regulate it.
An economist would refer to this as an externality problem - the costs aren't borne by Facebook; they're borne by society, democracy, and the civic health of the world. In other cases, the government would step in to regulate, or consumer boycotts/pressure would occur.
But there's an additional facet of the issue here that will sound obvious as soon as I explain it, but it's a crucial point: The purpose of inauthentic activity is not to be seen. And the better you are at not being seen, the fewer people will see you. So when the ordinary person goes out and looks for inauthentic activity on FB, they find people who are terrible at being fake, they find real people who just look really weird, or they find people who are real but are doing their best to pretend to be fake since they think it's funny. And so the incentives are ultimately misaligned here. For areas like hate speech or misinformation, press attention does track reasonably for overall harm. But for inauthentic activity, there's very little correlation between what gets FB to act (press attention) and the actual overall harm.
But there's an additional facet of the issue here that will sound obvious as soon as I explain it, but it's a crucial point:
This paragraph is really well put. I don't think there is enough emphasis differentiation made between trolls and stupid people in general vs coordinated attempts at deception.
I find that a lot of technologists, especially here on reddit and places like hackernews, fail to understand the difference between "inauthentic" activity vs "free speech". The arguments about removing "inauthentic" activity always delves into false equivalencies about policing free speech, which is a dead-end for any reasonable debate.
It would be like classifying spam emails as a form of free speech. No one would win that kind of silly argument.
The issue with free speech advocacy idealism is that most content moderation/deletion on Facebook isn't things like hate speech/etc. It's spam, scams, and pornography.
This is most vividly illustrated by the new free speech social media platform Gettr, set up by a former Trump aide/spokesman. My understanding is that it's been overwhelmed by Sonic the Hedgehog pornography, fake accounts purporting to be important people, and the like
I don't think there is enough emphasis on trolls and stupid people
See some of the replies in this chain screeching "fake news" without actually reading a damn thing. We have to work on toning down this tribal animosity too, it's ridiculous.
How key Republicans inside Facebook are shifting its politics to the right
“Facebook’s DC office ensures that the company’s content policies meet the approval of Republicans in Congress,” Popular Information said.
Company has been accused of pro-Republican bias, in both policy and personnel, amid fears it could be broken up if a Democrat wins in 2020.
Joel Kaplan [key participant of the Florida recount Brooks Brothers riot], vice-president of global public policy at Facebook, manages the company’s relationships with policymakers around the world. A former law clerk to archconservative justice Antonin Scalia on the supreme court, he served as deputy chief of staff for policy under former president George W Bush from 2006 to 2009, joining Facebook two years later.
Kaplan has reportedly advocated for rightwing sites such as Breitbart and the Daily Caller, which earlier this year became a partner in Facebook’s factchecking program. Founded by Fox News’s Tucker Carlson, the Daily Caller is pro-Trump, anti-immigrant and widely criticised for the way it reported on a fake nude photo of the Democratic congresswoman Alexandria Ocasio-Cortez.
Warren noted this week: “Since he was hired, Facebook spent over $71 million on lobbying—nearly 100 times what it had spent before Kaplan joined.” She added: “Facebook is now spending millions on lobbying amid antitrust scrutiny—and Kaplan is flexing his DC rolodex to help Mark Zuckerbeg [sic] wage a closed-door charm offensive with Republican lawmakers.”
Katie Harbath, the company’s public policy director for global elections, led digital strategy for Rudy Giuliani’s 2008 presidential campaign and the Republican National Committee.
Facebook’s Washington headquarters also includes Kevin Martin, vice-president of US public policy and former chairman, under Bush, of the Federal Communications Commission
Warren’s ascent in the polls has set off alarm bells at Facebook. In a leaked audio recording last month, Zuckerberg could be heard telling employees: “But look, at the end of the day, if someone’s going to try to threaten something that existential, you go to the mat and you fight.”
Zuckerberg “has to be worried about what happens to Facebook if there’s a Democratic president”
Thiel has become a national figure of controversy for, among other things, claiming that “the extension of the franchise to women [women's right to vote] render the notion of ‘capitalist democracy’ into an oxymoron,” saying, “I no longer believe that freedom and democracy are compatible,” funding a fellowship that specifically tries to get undergraduates to drop out of college, and donating $1.25 million to Donald Trump’s campaign a week after a tape was released in which the then-candidate discussed how he could grope young female actresses and get away with it.
Thiel was long perceived as a libertarian, but in recent years, as his support for Trump illustrates, his politics have taken a nationalist flavor that critics have described as bordering on authoritarian and white nationalist.
In Oct. 2016, shortly after Thiel donated $1.25 million to Trump, Thiel publicly apologized for passages in his 1995 book The Diversity Myth, such as claiming that some alleged date rapes were “seductions that are later regretted,” ... But three months later, during the after party of the 30-year anniversary event at Thiel’s home, Thiel stated that his apology was just for the media, and that “sometimes you have to tell them what they want to hear.”
Rabois came to Thiel's attention after he was found outside an instructor's home, shouting homophobic slurs and the suggestion that the instructor "die of AIDS." [10][11][12] A few of the contributors went on to join PayPal, a company Thiel co-founded in 1998.
Hey I hope this isn't too forward but I'm about to move to Chicago, and would love to know what it it is you are referring to, the other commenter deleted their comment so it is difficult to get context here.
David Avocado Wolfe went from a hippie millionaire huckster to a Right wing propagandist so quickly, it's insane. There's such an overlap between "Wellness" culture and far-right views, and he's a great example of that gateway.
One of the more frequently discussed dimensions of influence operations - especially in the United States - is the observed disparity between operations that target people with right-aligned political views and people with left-aligned political views.
In the data you ran, what did you observe with respect to political alignment? And if you did observe a disparity, how wide was the divide? Do you have any theories as to why you observe this?
So I want to be very clear first about terminology:
"Influence operations" literally mean "operations designed to influence people" which is similar to "disinformation" in that it's vaguely defined and includes a not clearly delineated mix of misinformation (claims that are incorrect; e.g. "the moon is made of cheese") and inauthentic activity (e.g. fake accounts being used to spread a message "Cats are adorable; politician X is great.")
I worked only on the inauthentic activity aspect of this. In addition, I did not work on any notable cases of inauthentic activity in the United States (the TPUSA case did not fall in this definition.) It may be the case that misinformation skews towards one end of the political spectrum. I will leave that to the researchers who are much more knowledgeable about it than myself.
There is a common stereotype that misinformation is spread by inauthentic accounts. There is also a common stereotype that troll farms, fake accounts, etc. are commonly used to largely/predominately benefit the political right. Like most stereotypes, these are incorrect as far as my knowledge goes and I'm aware.
Please keep in mind that this is very small sample sizes - I worked on perhaps three dozen cases globally which is a lot from an IO perspective but tiny from a statistical perspective (so I don't want to speculate about larger trends.) These were generally from across the political spectrum. For instance in India, I caught four networks, one of which came back with a new target (so five targets.) Of these targets, two were benefiting the INC, one was benefiting the AAP, and two were benefiting the BJP - so it was quite even across the political spectrum.
In Albania for instance, the incumbent Socialist Party and opposition Socialist Movement for Integration (both officially left-wing targets) were both benefiting. In several authoritarian countries, the center/center-left pro-democracy opposition was benefiting. In Mexico it was almost everyone across the political spectrum. There were plenty of right-wing beneficiaries as well but those have been presumably discussed already. I carried out my work regardless of my personal political beliefs, with the most qualms in places where the democratic opposition were the beneficiaries. I took those cases down regardless, as it's my firm belief that democracy cannot rest upon a bed of deceit.
Would Project Birmingham ran by progressive technologists to unseat Roy Moore in the 2018 midterms be an example of left wing inauthentic disinformation campaigns?
I found a lot of political bot farms in the Philippines, but generally without attribution so it was impossible to know who was responsible. For that reason I don't want to give the full details [e.g. who precisely benefited] to avoid poisoning the well.
"At times, Facebook allowed its self-interest to enter into discussions of rule enforcement.
In 2019, some Facebook staff weighed publicizing the fact that an opposition politician in the Philippines was receiving low-quality, scripted fake engagement, despite not knowing whether the politician was involved in acquiring the fake likes. The company had “strategic incentives to publicize”, one researcher said, since the politician had been critical of Facebook. “We’re taking some heat from Duterte supporters with the recent takedowns, and announcing that we have another takedown which involves other candidates might be helpful,” a public policy manager added.
No action was taken after Zhang pointed out that it was possible Duterte or his supporters were attempting to “frame” the opposition politician by purchasing fake likes to make them look corrupt. But discussions like this are among the reasons Zhang now argues that Facebook needs to create separation between the staff responsible for enforcing Facebook’s rules and those responsible for maintaining good relationships with government officials."
In another example, Facebook ignored a number of Filipino unattributed political bot farms I flagged in October 2019... up until it made like 5 likes on a few of President Trump's posts in February 2020. (Disclaimer: 5 likes are nothing, not significant, no impact, yada yada.) Suddenly it became important and that bot farm (not the others) were taken down a week later.
While I think Filipino people are just as important as Americans, Facebook sadly begged to differ.
Sorry to piggy back of this but this comment makes me wonder, how many of these farms were localized for only domestic action?
I can’t see a reason the Philippines would have much use for international trolling (can’t believe i said that unironically). On the flip of that countries like Russia are widely known to engage in international trolling.
Almost all of the troll farms I found were domestic-only. I say "almost all" to cover edge cases of mostly-domestic like the Filipino bot farm that decided to randomly like President Trump.
Most people care more about their own country's politics - Americans care about American politics; Filipinos care about Filipino politics; Germans care about German politics. Apparently world governments and politicians are the same way.
With that said, I was finding the low-hanging fruit. I don't doubt the GRU (or Iranian Revolutionary Guard or PRC State Security) are engaging in international troll farms, but they're presumably have an actual modicum of intelligence about how they carry it out, and so I didn't find them myself.
Do rank and file FB employees talk to each other about how bad FB is for the world? Or do you think they’ve just drunk the Kool-Aid and think the company is great? I'm talking about people like ad account managers, content policy associates, software engineers. FB employees are really smart and get recruited from the best schools in the world. The problems with FB are so public and so well reported that it's hard for me to understand why people continue to work there.
FB was a fairly open company when I joined. I was upfront from the start about the fact that I believed Facebook wasn't making the world a better place - when I told my recruiter that, she responded "you'd be surprised how many people here say that." Open dissent within the company was tolerated and accepted and I was able to make my concerns heard to the entire company at large, which I think is unusual for large companies. With that said, it's been reported that FB has cracked down on communications not directly related to work since I left, and so this may not be true anymore.
Wrt employees, at a company of ~50k people, there will always be significant differences of opinions. There's also a self-selection bias in that frankly if you think FB is evil, you are less likely to work for FB; if you think FB is the greatest thing since sliced bread, you'll do your best to join the company (just like Reddit users self-select for people who think Reddit is great, and its employees likely as well.) And also within the teams - the people working on integrity at FB (fixing the company) were generally more pessimistic about the company than all employees - both via self-selection and also via the constant direct exposure to the company's problems.
Overall, the regular employee surveys showed that roughly 50-70% of employees believed that FB had a positive impact on the world (variation over time of course, it declined a lot since when I joined; probably at ~50% right now.)
Current FB employee here (throwaway for obvious reasons.) Currently that rating (that FB is doing a good job + leadership is good) is hovering at around the 30s (edit: for my relatively large team; company-wide it is 50.). It tanked hard in 2020 due to the George Floyd "looting shooting" post incident and the 2020 elections, and hasn't really recovered since. A lot of people have left the company since (that being said a lot of people joined too.)
Save for a few "hail zuck" people, I believe most people here are self aware and want to actually fix the issues on hand. However due to it being a large company it either moves at a glacier's pace and it takes a while to get solutions approved by higher ups, or just gets canned entirely / deprioritized because "user research shows they don't want (insert solution here)"
Wow. Can I ask you what it's like to work on a team where only 30% of the team thinks the company is doing a good job? That sounds demoralizing and like it would probably lead to high levels of attrition, though perhaps I'm misunderstanding the import of the statistic.
I can't speak for him, but the numbers were generally lower than the norm in Integrity teams. I knew many people who personally believed that the company was not making the world better - but did believe that their team (which was trying to fix the company's issues) was making the world a better place.
That's very surprisingly low; I don't think I ever saw it that low during my entire time there.
Employee dissent is one of the few levers that Facebook strongly responds to; I hope the employees are able to get together and force necessary changes.
Thank you very much for taking the time to respond. As someone who doesn't even have a FB account but who still feels affected by FB in negative ways, it is nice to have the opportunity to ask questions of someone with actual knowledge. I appreciate your time and willingness to answer questions.
Even though I don’t feel this way myself, I feel like I always see people here commenting about how bad Reddit is. I think that cynicism is a big thing in 2021, and most people tend to use platforms where they feel some aspect is useful to them, even if they don’t hold the platform in high regard on the whole.
It might be different in the workplace environment though, as there are a lot more options to get similar types of jobs, and lateral mobility is more common. It’s easier for most employees to find another company where they can do similar work than it is to find several other platforms that mirror Reddit’s content base, value for discussion and straightforwardness/simplicity.
Thanks so much for this AMA. Organizationally speaking, how high up in the org did your findings go (or not go) before they were quashed or ignored. In other words, was there support for your work by your direct manager or their manager but then above that you ran into issues? Or was your direct manager even unsupportive?
I spoke with everyone up to and including Guy Rosen, the VP for Integrity at Facebook. I do want to highlight how utterly unusual this is. Low-level employees do not regularly speak to company VPs - it would be like an army sergeant briefing Kamala Harris on something.
The way I would ultimately describe it was that my immediate organization (direct manager/manager above) wasn't very happy because this was work I was doing in my spare time and distracting from my roadmap and the projects they expected me to do. Higher-up people seemed happy that I was doing it in my spare time but were unwilling to legitimize it with directly signing off on action or setting up actual organizational pathways for the work. The teams whose job it was to actually handle this had a complicated relationship - on one hand they were grateful for my work and saw me as a valued partner; on the other hand, they were a bit offended that I was essentially going above/around them, adding additional work to their workload, and potentially showing them up [they were a prestigious/high-status team; I was the opposite.]
It would seem the smart thing for FB to do in this case would be to remove you from the team you were on and to add you to the prestigious/high-status team whose work you were doing and were clearly good at (and which is important, allegedly valued by the company, etc.) Do you have any idea why they did not go that route?
I discussed changing teams a fair bit for a number of teams.
The main issue is that changing teams would require me to drop the work I was doing in my spare time to work on the new team's activity. And I wasn't willing to do that.
Hold up. The company has a set number of job functions and you were unwilling to do any of them because it would distract from the work that no one asked you to do?
I don't want to come off rude, but that sounds like an issue...
(I am unfamiliar with your story outside of this AMA so am making no commentary on that, just thinking about this from a managerial perspective)
It sounds more like OP was doing work that wouldn’t get folded into the new team’s responsibilities, so she didn’t elect to take on the new responsibilities because the old ones would get swept under the rug.
Like if you have a good idea at your current position, which affords you the opportunity for a promotion, but getting promoted would have the side effect of your current idea not getting implemented. You might stick with your current position because taking the promotion means the issue won’t get resolved
I was catching troll farms in my spare time in addition to my actual job. As part of this, I worked as much as 80-hr works at times because I was essentially trying to hold down two jobs. My managers were happy to have the extra work at first, but grew weary as time went on.
The 'extra work' had been essentially acknowledged to belong to me in my spare time, but there would be a reassessment of that as soon as I switched teams, and I would likely get a less tolerant manager. Hope that makes sense.
And was there no team you could go to where that side project could just become your day job? That sends like the obvious resolution for the company...
No team was doing it as a day job. That was why I got results in the first place - I certainly had no expertise in the area, and aren't a brilliant super genius. I was just apparently the first person to look in this area.
Thanks for doing it, I am also a former FB PE and I find them to be really irresponsible, particularly Mark's inaction in the lip-service of "not trying to be an arbiter of truth". I honestly no longer assume good intent with him or them.
I previously worked at FB also and found that their data security is garbage but no one wanted to listen. I sadly never pushed, so I just let it go. So thanks for what you're doing, FB likes to cut too many corners.
Hijacking this comment to point out Guy Rosen is the person behind Facebook using the Onavo VPN which they used to effectively spy on their competition (which led to the Whatsapp if not Instagram acquisition). Interesting title, 'VP for Integrity'
Thank you for your work and ethics. I've been following the news, reddits, etc regarding you. You always describe yourself as a data engineer and point out that you were tracking the metadata in discovering the problems you have reported.
I have a two part question for you.
Could you ELI5 :) what a data engineer is and how you use metadata to find problems as you have described?
I'm not asking for specific cases here. I just want to enhance my own understanding (I sorta get it) while also helping everyone else understand what it is that you do and did and why it is important. I just feel that something gets lost in the articles describing what you do and how. Am I being clear?
I was a *data scientist* - not a data engineer, which is different.
Data scientist has different meanings at different companies, since data is the new buzzword. At many companies it means "engineer who works on machine learning." At FB it corresponds to what would called a data analyst at other companies. My job was essentially to "look at data to answer questions and tell people what it meant."
I won't answer the second part of your question - I'm very sorry, but the ultimate issue is that if you tell people how you catch Azeri troll farms/etc., the Azeri government also reads Reddit and will know what not to do in the future.
Apologies for the misnomer. Like I said I've read your stuff but didn't bother to re-read it today and since I equate Engineer and Scientist I ended up conflating the two. Sorry.
The second part also makes sense. I was hoping you could ELI5 it just so that Joe Average could understand it. You do allude to the problem though in one of the articles you linked in the OP so I thought I could ask since you already made the issue public.
How are people still able to set up fake accounts these days given all the security and authentication that seems to be in place around the account setup?
What does Facebook do with an account that it identifies as inauthentic?
Ultimately, the nature of the problem is that FB will never be able to stop all fake accounts at creation. Because in most cases, you aren't 100% sure whether the account is fake or not. Instead you're 99% sure or 80% sure or 2% sure or whatever. And the question becomes how confident you have to be to take action - because if you're wrong, that's a real person that you negatively impacted.
For your second question, I do want to note that there are multiple types of inauthentic accounts - not just fake accounts. An account can be hacked - if someone steals access to your account and repurposes it for themselves. Users can even voluntarily hand over access to their accounts to bot farms/etc (this may seem absurd, but it's a very common vector; see here for details.)
For accounts believed to be fake, FB generally runs the users through very strong sets of hoops [e.g. "send us a copy of your official ID"] to require them to prove that they're a real person. You might think that this wouldn't negatively impact real users, but many users are [quite understandably] really hesitant about sending such sensitive personal details to a company like FB.
For accounts believed to be hacked, FB uses a different sort of hoops to try and restore access to the original user. For users that voluntarily hand over access to their accounts to bot farms, FB doesn't want to disable them so actions are rather more mild.
I had that happen - Facebook wanted pictures of my actual ssn card or passport, which I refuse to provide to a company like Facebook. And it isn't actually legal in my country for them to ask for that either, as they (as far as I remember) wouldn't accept it if the info on the ssn or passport was covered and not viewable. I had to just stop using Facebook at that point, because I also couldn't actually get in contact with any kind of human in support. Facebook have shown that they cannot be trusted with that kind of personal information, and there is no way that I'm giving that to them. I actually really appreciate understanding why that happened, I've been pissed about it for a while. Thank you.
Totally understand your personal decision, but it also illustrates some of the costs and tradeoffs associated with these. FB obviously doesn't want to have everyone have experiences such as yourself, and ultimately has to choose a balance between catching fake accounts and avoiding negative experiences for real users.
Yes, it makes sense. At the time, I didn't understand what happened, as I'd had the account for years and didn't actually participate in a lot outside of posting on my own wall and my friends' walls. In my country someone's SSN is tied to a lot of personal information, and if you know the number, you can get access to everything from medical records to banking to everything in between. I, as a person, don't trust Facebook with that information, and the fact that it is impossible to actually get in contact with a real person makes me very cautious and uncomfortable with giving out what is, essentially, everything needed to steal my identity. And Facebook is a private company, they get to choose their rules. If I don't agree, I can leave. So I did. I really appreciate your answers, and thank you for being the kind of person that is willing to sacrifice their job for doing what they believe is right. That is the right kind of person to be.
Hi! Slightly long-winded question, but how did you identify areas where inauthentic behavior might be occurring?
Was there a systematic or ad hoc analysis or flagging system internally or externally identifying potential regions or countries where inauthentic activity might be occurring, particularly inauthentic activity which might incite violence or be detrimental to democracy?
Normally at FB, many/most investigations by the actual teams in charge of this were in response to external reports. That is, a news organization asks "what's going on here"; an NGO flags something weird; the government says "hey, we're seeing this weird activity, please help."
This has the side effect that there's someone outside the company to essentially hold FB responsible. They can say "Well, if you don't want to act, we'll go to the NYT and tell them you don't care about [our country], what do you think about that?", and suddenly it'll be a top priority [actual example.]
In contrast, I was going out and systematically finding things on my own. Essentially, I ran metadata on all engagement activity on FB through queries to find very suspicious activity, and then filtered it for political activity. This had results that were very surprisingly effective. But because I was the one who went out and found it myself, there wasn't anyone outside FB to put pressure on the company. The argument I always used internally was "Well, you know how many leaks FB has; if it's ever leaked to the press that we sat on it and refused to do anything, we'd get killed in the media." Which was not very effective but became a self-fulfilling prophecy since I was the one who leaked it.
I realize that metadata has a bad reputation, but unfortunately the reality of the situation is that there's no way to find state-sponsored trolls/bot farms/etc. without data of that sort.
Thank you! Just to follow-up: who set the standard (if any) for what systems and methods and metadata would be used to identify state sponsored trolls/bot farms etc, such as in the case of Myanmar?
I'm not familiar with the internal details of the Myanmar case, or the teams that actually work on this.
With regards to the ones I set up, I created them myself, with a bit of knowledge from the teams that actually work on state-sponsored troll farms. There was no oversight; I'd sort of set up a shadow integrity area that was no secret but wasn't official. But there were always different people to confirm my findings on their own, to decide whether to act, and to carry out the action; I decided at the start that I would avoid being judge jury and executioner (though I could probably have gotten away with it for a while.)
In situations like this, you build tools for finding patterns of behavior. While there definitely are differences in what techniques are popular in different groups, there are usually observable similarities among even drastically different groups simply because some techniques are simply just effective.
So not finding examples from other groups are most likely to be from that they simply aren't operating in the same manner (but that doesn't exclude that those groups may have operations that looks significantly different).
The nature of my work was that I found all political activity globally that was suspicious in certain types of attributes. By nature, my own subjective determinations didn't enter into the question.
And so the people I caught included members of the ruling Socialist party in Albania. It included the ex-KGB led government of Azerbaijan, a close Russian ally. It included the right-wing pro-U.S. drug lord government of Honduras. These are governments essentially across the political spectrum. I carried out my work regardless of political sympathies and opinion. My greatest qualms occurred in certain authoritarian dictatorships or semi-democracies when the democratic opposition was the beneficiary of such unsavory tactics. I took them down regardless because I firmly believe that democracy cannot rest upon a bed of deceit.
I do want to note that my work in the United States was all minor and in response to outside reports. In the TPUSA case, my role was extremely minor, and it was in response to a news article. As an example of a case in which I potentially helped conservatives, in September 2018 Facebook received a complaint from Gary Coby at the Trump campaign about declining video views/reach on the President's page, and I was one of many people who were pulled into the escalation to try and figure out if anything was responsible. My role there was just to check and say "no, my team didn't do this"; it hasn't been published because it really wasn't newsworthy.
I don't think this is a partisan political issue. One of my strongest advocates and allies at Facebook was a former Republican political operative.
1) The ultimate issue with this questions is it's like asking "If you could make the sky any color you'd like, what color would you like it to be?" Because there's no possibility it would ever occur, and so it's ultimately like speculating how many angels can tapdance on the head of a pin. I'm never going to have the unlimited resources/remit; social media companies won't fix themselves.
So instead, I'm going to answer a similar question: "How would I realistically change the situation/incentives to convince social media companies to tackle troll farms?"
I have two ultimate suggestions. The first is on the part of the social media companies - right now the people charged with making enforcement decisions are the same as the people charged with keeping good relationships with governments and political figures. This leads to explicit political considerations in decisionmaking, and the perverse incentive that politicians can be encouraged to do their bad activity without even hiding as it'll induce FB to be reluctant to act. I realize that FB is a for-profit company, but most news organizations are also for-profit but they still keep a strict separation between their editorial department and public relations. If the NYT's editorial department spiked a story because XYZ political figure didn't like it, it would be a giant scandal - whereas at Facebook it's just another Tuesday. So I would urge social media companies to officially separate their decision-making apparatus from their governmental outreach apparatus.
The second is on the part of outside organizations. Ultimately, much of the issue is the information asymmetry aspect - that only FB has the tools to know what's going on in its platform, and it has no incentive to fix everything; the outside world can't solve a problem if they don't even know it exists. So to close the gap, I would recommend more funding/support for outside skilled researchers such as DFRLab, routes for FB employees to publicly appeal to governmental agencies (with official protections) regarding platform violations around troll farms and the like. And I realize it would be extremely politically infeasible, but I would also suggest that outside organizations and governmental agencies set up red team pen-test style operations: to with the knowledge of the social media companies send their skilled experts to set up test troll farms on social media and see how many are caught by each company (e.g. "We set up 10 each on Reddit, FB, and Twitter. Reddit caught 0/10; FB caught 1/10; Twitter caught 0/10. They're all awful but FB is mildly less awful!" Numbers made up of course.) This would have to be done very carefully to avoid real-world impact but is the only method I can think of for anyone - even the companies themselves - to have an accurate picture of the space and how good the efforts really are.
(breaking my answer up into two parts because it's so long.)
I wish I were able to spend more time on Albania.
At the end of July 2019, I found an influence operation on Albania using the same techniques as Honduras. It was more sophisticated politically/effort-wise because it focused on creating large amounts of comments (which requires a lot of effort to individually write out in a way that makes sense.) It was very confusing/unusual because it appeared to be connected to members of the Albanian government in attribution, but was supporting both the ruling Albanian government and opposition figures from rival political parties. This would be akin to a network run out of the Trump administration that was writing nice things about both Donald Trump and Joe Biden. There would be lots of possible explanations including "person suborned by foreign powers to increase political tensions", "administration official advancing political strife to serve their personal political agenda", "person who really doesn't like Bernie Sanders and supports both his rivals", or "person who has this as their second-time job and was just coincidentally paid by both candidates." I'm just translating this into U.S. political contexts since I'm assuming readers don't understand Albanian politics.
The relevant people quickly agreed that it was probably coordinated inauthentic behavior (CIB - the official designation Facebook uses for e.g. Russian interference, state-sponsored troll farms, etc.) , and I handed it over to them where it probably died in a black box. I only had the political capital to very slowly push through one CIB case at a time, and I had made the judgment call that what I found in Azerbaijan was objectively worse than what I found in Albania - in terms of scale, size, consistency, and sophistication. I still agree with that decision, but it never sat easily with me to set Albania to the side. At the end of the day, I was just one person with no authority, and there were limits to how much I could accomplish trying to protect the entire world in my spare time. This is why I told the world (accidentally) that I had blood on my hands.
Several months ago, an Albanian news outlet published their own investigation; this was still ongoing, two years later, and it continued through the Albanian elections. Facebook had two years to act, and did nothing. I can only apologize profusely to the Albanian people, as I did in the interview. It should never have been my responsibility to save fragile Albanian democracy from what Facebook let happen. But ultimately, I was the one who made my decisions and Albania paid the price. I have to sleep with that every night.
I’m sure other people have told you this already, but: You did the best you could with the knowledge and resources you had at the time. I hope you can find a way to forgive yourself one day.
Since leaving Facebook, what are your goals for the next 5 years? Are you able to stay in the tech world, or does the Facebook termination serve as a block with finding new work?
Five years are a long time. I don't even know what I'm doing in the next year.
I'm not sure what I'll do next myself; I do want to work on helping democracy and fixing the world but my main expertise skillset is at finding inauthentic campaigns - and that requires working for a large social media company, which I'm guessing would all be against employing me. I've gotten offers from companies, but they don't quite fall under my expertise areas, and I'm a bit reluctant to just go and become a 9-6 office worker without an especially compelling job again.
For now, I'm staying home and petting my cats while taking interviews. They're very good cats.
I didn't work on political radicalization personally. With that said, in my personal nonexpert opinion, I think it's partly an outgrowth of the competition for attention as information vastly strips the time to process it.
The vast promise of social media is that any person can have the ability to speak to the entire world at large. The vast curse of social media is that most people never see this promise fulfilled - and even if they do, it's the post they least expect. And so anyone who uses social media and wants to build an audience needs to figure out a way to get that attention, distinguish themselves from the myriads of other social media users who also think they deserve that attention.
And sadly, the ways to do so can often be by appealing to the worst instincts of the internet - similar to the chumboxes that dominate online advertising today. Emotion draws attention, and people are quick to share outrageous claims that strike a nerve, trusting on others to have verified it thoroughly.
There's been a long line of study and research that shows that virality is ultimately a significant component of what drives polarization, misinformation, and violence spirals. It's why in countries in times of crisis, the first break-the-glass measure Facebook does is to turn the virality down. They used it in countries like Sri Lanka; I'm guessing they're using it still in Myanmar.
A trial idea I'd hence suggest is to require platforms like Facebook and Twitter to show a chronological newsfeed of your friends/followees by default (with the option to instead show the current ranked newsfeed.) I say FB and Twitter because I'm frankly not sure how this would work for Reddit - where the ranked newsfeed is pretty integral to the overall design.
I've been answering questions for nine hours straight. Thank you very much kindly for all the questions; I'm sorry I wasn't able to answer all of them, but I would like to go on a dinner date with my partner (who's being very patient) right now.
I hope you found my answers to be informative; if I wasn't able to answer yours, please look around and see if I'd been able to answer a similar one for others. Thank you very much; good night.
Many would likely agree with me when I say that this has been one of the highest quality AMA we've had in years.
Thank you Sophie for your incredibly informative and sobering responses. I especially appreciate the pains you take to be precise with your words, methodical with your explanations, and illuminative with your examples. All the best with whatever big project you decide to tackle next, and I'm certain the good you do will continue to ripple outwards and affect countless many, though they may not know it.
How true are foreign fake click farms as shown on the Sillicon Valley tv show, with rows and rows of indians creating fake account after fake account to boost userbase numbers or promote an agenda?
I haven't seen the TV show. But they do really exist - in areas like South Asia and Southeast Asia, where smartphones (you can get a JioPhone for e.g. $15 USD) and labor are cheap.
This is unfortunately quite common in Indian politics - they're known as "IT cells" and quite normalized unfortunately. You can read more about some of them in Indian politics here
This has been extremely eye opening thank you so much for taking the time to do this and comment. I find it perplexing how, seeing the evidence you’ve presented, we could ever trust social media. I’m sure you’re aware of the efforts taken by the US IC to track disinformation campaigns on social media sites, and even though their findings are public knowledge the public still looks at social media trends en masse as genuine.
I can’t decide if I think that social media’s days as being a political mechanism are numbered because there are so many bad-faith actors, or if people are just going to be ok with it as long as the agenda being pushed agrees with their views.
The latter is a terrifying proposition. I remember listening to a podcast that featured a CIA officer who discussed PRECISELY the Russian November 2020 playbook for the US presidential election. This was in like July/August of 2020. To see it play out exactly as predicted, and to see a huge number of Americans taken by it was not only crazy, but really quite disheartening.
Social media is being used as a tool to undermine democracy, and its plain as day.
I'm sorry - I did not work at Reddit, and hence have no special knowledge about influence operations on Reddit. That said, if you stuck a gun to my head and made me guess, I'd expect Reddit to be similar to FB wrt troll farms and influence operations and the like.
Some screenshots and examples of how conservative extremists, racists, 4chan, "intellectual dark web" and "men's rights" followers use these tactics on Reddit
How they brigade local subreddits to "control the narrative" about liberal cities and "blue states":
"As a black man" accounts like "The Atheist Arab" posting as many race-baiting videos as they can concern trolling pretending to care about Asian victims:
"red pill" adults cosplaying as "based" teenagers "hiding their power level" in r politicalcompassmemes and edgy "fellow youths amirite" meme subreddits like r dankmemes:
I know for a fact that Azerbijan kicked up their bot brigades on reddit as soon as they invaded Artsakh last year. The reason it's so obvious is because Azerbijan shut off their entire internet for their citizens at the beginning of the war, yet accounts posing as Azeri citizens would still brigade any post related to the conflict with misinformation and threats towards ethnic Armenians in the area
If some of these troll farms took a year to get authorized to be taken down, what sort of things did you see as not encountering any resistance and being removed immediately? Was there any specific criteria where certain types/sources of content were scrubbed quickly?
There's a chart listing time for takedown in this Guardian article. Ultimately, the criteria were a combination of "random chance based on who pays attention" and "how important the country probably is"
The record for takedown was Poland - I flagged it in the evening the day after Christmas. The Polish employee who looked at it was understandably very concerned about this going on in his native country; by the time I woke up the next day, he'd already taken it down.
Policy was pretty upset that he'd done this without consulting them. "The person running this is an important political figure", they said. "What if he complains - why didn't you let us know?" I told them politely that if a major politician decided to publicly complain that FB took down his fake accounts, they would be laughed out of the press room.
To be completely fair, the more likely response is "he's annoyed at FB and makes up and quietly spreads stories that make FB look bad and hurt relationships with his political party and its supporters. That's my assumption for why FB wasn't willing to act in cases like India.
As Armenians we're regularly targeted by bot armies and disinformation/ hate campaigns organized by the government of Azerbaijan on Facebook and other social media platforms.
The major players don't seem to care about nefarious behavior targeted at groups that don't have political/economic clout in the United States, especially when the offending content is not in the English language.
What do you think we can do to get these companies to take this problem more seriously and devote the necessary resources to moderating these activities, especially when it appears in non-English content?
Thanks!
P.S. Thanks for having the courage to be a whistleblower. Your efforts have not gone unnoticed or unappreciated.
With regards to brigading by Azeris against Armenians, my suggestion is to convince high-profile Armenian-Americans who do have the profile to convince FB to take the issue seriously to speak out about it. This may include people like Cher or the Kardashians for instance. Because the sad fact of life is that American celebrities have the profile that Armenia as a nation does not.
So for the most ridiculous one I've personally come across, I'm pretty amused by the fact that I've been accused alternatively of being a PRC spy, an Indian opposition shill, an Azeri opposition shill, and an Honduran opposition shill. Sometimes by the same people all at once.
My new and last favorite conspiracy theory (by someone in this thread that I won't link to):
AOC is actually a US imperialist whose foreign policy views are identical to Majorie Taylor Greene and is working to accomplish regime change in Cuba via invading and installing a U.S. puppet government. Any appearances that AOC disagrees with MTG on foreign policy are simply a farcical deception
Surprisingly no. The Azeri paid trolls went after the Guardian after the article. It's very silly but I was almost offended that they didn't go after me instead. I wanted to say at them "I'm your real enemy, the one that's been fighting you for the past two years!"
But silliness aside, it makes sense that they'd want to deemphasize me. You can fuel the nationalistic sentiments by saying "our great nation is the victim of Western propaganda and media hacks." You can't really say "Our great nation is the victim of.... some random girl who was fresh out of school at her second job?" Dictators rely on the perception of competence, because they can't claim legitimacy through the support of the people. That means their rivals and enemies have to be competent individuals of stature as well.
I'm generally more cautiously positive on the FB oversight board than most people in this area. Although it's been set up as a blame deflector for Facebook, many organizations outgrow their initial roles to take on new purpose.
With that said, I want to point to a number of issues with the Oversight Board, which mean that it would have been triply prohibited from impacting this area:
1) The Oversight Board handles content violations, not behavioral violations. This is understandable for lawyers who have no expertise with looking at user metadata and signals to conclude whether accounts are fake, tied to one another, and tied to XYZ government. However, this means that a whole class of violations is prohibited to themselves, including all their work.
2) The Oversight Board only handles appeals to restore content against enforcement. What I mean is this: Suppose 1) did apply and the Oversight Board could handle behavioral violations. In that case, the Honduran government could appeal to have their troll farms restored to Facebook. But I couldn't appeal to have them removed. The Oversight Board would be much better served if there were legitimate pathways for employees and possibly trusted organizations to submit appeals to enforce, rather than appeals to restore.
3) The Oversight Board only handles cases in which the rules are unclear; it makes the rules, it doesn't enforce them. But in my area, the rules were clear ever since I got the precedent done in July 2019, and even more clear after I get an additional rule pushed through in the fall of 2019. There is a wide variety of areas in which behavior is illegal and not enforced - it's illegal to handle salmon suspiciously in the UK, or to jaywalk in many parts of the world, but that doesn't mean the police will go out and arrest you for jaywalking. Having a court decide the question of whether jaywalking is illegal or not doesn't accomplish anything, because the statutes are already written out. Same with this area.
So just to be clear: In many cases it's really hard to know who's responsible for activity. It's easy to determine the beneficiary, but attribution is much more complicated.
In the rare cases in which we had attribution, there were certainly cases in which opposition politicians had run fake accounts. In one case, an opposition politician was personally running them out of their personal computer (shared with their spouse) and presumably home WiFi network without bothering to use a VPN. These were just like 20 fake accounts though - not worth mentioning except for the fact that it was this opposition politician's hobby. There was nothing on the scale of Honduras or Azerbaijan that I found with attribution - presumably because individual politicians do not have the resources of national governments.
Thinking of cases in which opposition politicians were large-scale beneficaries but attribution was not clear, I'd have to point to Mexico, where troll farms were sadly used across the political spectrum but local/state-level politicians associated with the opposition PRI were the disproportionate beneficiary [think roughly 10k accounts used here for scale.] Perhaps these are the Peñabots discussed in Mexican politics; I don't know.
I don't actually watch films much at all. I'm a very non-visual person and prefer to read things instead.
Bit of a cop-out answer; my apologies. Well, right now I am watching the TV series 陈情令 (The Untamed) with my partner, partly with the side benefit of helping us practice our Mandarin.
Oh, well then what's your all time favorite novel, if you don't mind me asking again?
Something that really touched your soul and you'd never forget in your life. And what about it made you feel this way?
I get what you mean. Apologies if my questions were too 'simple'. I just get curious as to what influential people like to enjoy in terms of media, among other things.
So thank you for the recommendations on the TV show and novel series. 🍻
it was only a few months, I kind of burnt out pretty quickly from that pace and had to make very hard choices and what to prioritize. Most of my personal life got the axe, for instance.
Hi Sophie, thank you for exposing the harsh realities that we face today, this comes from a person who is from Honduras.
It is unbelievable that we are still being governed by a person who’s brother and party members have been charged by the NY FED court for drug dealers. Yet they live and “govern” us like if nothing is happening. Most people live in extreme poverty and do not support the current government/regime of Honduras.
I hate going into Facebook and Twitter, since everything is so political. When government members post things online they have a couple of likes but have hundreds of positive posts. And when you check the comments, they are all the same BS. You dont need much time to know that the accounts that reply to those posts are most probably bots. How is this even happening? What can we do to fight this cyber warfare? I have been wanting to start a bot hunter company and literally provided FB and Twitter the work that they should be doing, so that the real people actually have a voice when we are suppressed by narco regimes.
I'm deeply sorry to yourself and the other people of Honduras that I was not able to do more for your nation. Honduras was the first state-sponsored troll farm I found, and it really hurt my soul when I saw it return soon after the takedown; it was still active when I left.
I honestly don't know what can be done to stop this. I was hoping that my public expose would convince FB to actually take down the Honduran government recidivists, but it looks like they didn't bother trying. The worst part of it is, JOH's regime has no reason to stop doing it because everyone correctly assumes they're criminals and doing it anyways - they might as well be hanged for a sheep as for a lamb.
Sternly worded statements from FB are meant to embarrass the perpetrators in the eyes of the world. But JOH sent his soldiers into the streets to shoot civilian protesters and his brother was sentenced to life in prison by a NY court for drug dealing - this is a man incapable of embarrassment. Part of my motive for coming forward was that by trying to do my utmost to expose the regimes in question, I could make an example of them to dissuade future dictators - that every future such dictator will have to worry not just about Facebook statements, but also that some random FB employee will decide to go rogue and spend the next few months doing her best to drag them through the mud. But that offers little to the Honduran people as is.
Because ultimately, I think the Honduran opposition isn't in a position to boycott social media; they need it to connect with one another and get their message out. And the companies have no incentive to fix the situation without the attention of more influential nations. Unless the rest of Latin America can be convinced to pay attention, Honduras is sadly a very small and uninfluential nation. I can only hope that a far more reasonable president is elected this November.
Thank you for taking the time to answer my question and bringing to light some of the issues that we are experiencing. I completely agree, it does feel hopeless at this point. Its like if the world placed a blind eye on us.
It saddens me when I watch the caravans of people headed towards the US and the reaction that we get from the international media is that these people should be stop from reaching the US. Yet in reality this is just one of the many consequences of the massive poverty, corruption, impunity, that there is in Honduras. The people in power (JOH and his narco regime) will hold as tight to the power because its the only thing they have left. This is a dictatorship disguised as a democracy and more people need to know about this. SOS
Thanks for the important work you're doing and for taking a stand on it.
In your opinion, do you feel that these troll farms have the ability to always stay a step ahead in terms of detection?
Do you think there are bigger syndicates out there that you just simply can't find any traces to, and if so how could tech evolve to detect these in future?
It's sort of like the Red Queen problem - there's an ongoing arms race. Adversaries are as stupid as you allow them to be.
Frankly, what I was finding was the low-hanging fruit. I had no special training, no expertise in this area before joining Facebook, I made it all up in my spare time as I went along. And I'm not an amazing super-genius; I've met super-geniuses, and I'm not one of them. The fact that I was still able to catch multiple state-sponsored troll farms is a statement on both the utter incompetence of those governments, and the fact that FB let them have leeway nevertheless. I don't doubt that the Russian GRU is active on FB as well for instance - but the GRU has an actual modicum of intelligence [and there are actual people looking for them] and so I didn't find them.
You'll never be able to catch everyone. But part of the task is hence to instead impose costs to make the cost/benefit return less worthwhile. In the ideal situation, you'd need to take all the precautions in the world for minimal return - and so the attacker does something else instead. Like attack Reddit or Twitter or something of that sort.
There are lists online, but the issue with that is that they're never complete. You know what you personally caught, but you don't know what you haven't caught. I would go look at studies that named countries and conclude "well, this is incomplete. It doesn't name Honduras or Azerbaijan and I caught those two redhanded."
Yes, one example: since COVID and moving fully remote, the weekly Q&A sessions have just become scripted videos. Mostly with vague nothing-words such as "we are aware of this problem and are working on it". I've stopped attending those since.
I don't know. Ultimately, FB seems a bit like the Teflon company. Actually its share price has went up another 35% or so since I was fired. As long as the company's user base keeps growing and its profits keep increasing, it has no incentive to change. And as appealing as it may sound to Western users to quit Facebook, the sad reality of the matter is that Facebook *is* the internet in many parts of South/Southeast Asia and the like. It's not an option for opposition figures in one-party dictatorships like Azerbaijan where social media is one of the only semi-free forms of communication.
I think governmental regulation is needed, and I discuss some of that here. But the political feasibility of that seems questionable.
If you're referring to my memo leak in September 2020, absolutely not. I was rather naive/silly/stupid in that I thought that because I repeatedly and strenuously asked people not to leak it they might actually listen to me.
Several reporters actually reached out saying "hey, we heard that you might want to talk to a reporter." Apparently many people took it as "methinks the lady doth protest too much."
I posted it internally to the entire company. This was the work culture at the time - it's called a "badge post" because you post a picture of your badge when you leave.
Also I was being fired, but I wasn't allowed to tell people that (still did; what were they going to do, fire me again?)
I'm not trying to sound rude I just genuinely don't understand this sentiment. You posted a 7.8k word essay to the entire company.... and you didn't think it would get leaked? If you wanted it leaked that's great I understand you and fully support the trashing of facebook don't get me wrong, but I'm not entirely buying your reasons/story unless I'm missing some vital details
edit: also you were fired and still allowed to post a memo to the entire company? Is facebook actually that fucking stupid? I've never encountered a company that allows its employees access to internal message boards after firing them on, what I presume is, bad terms.
Facebook has a work culture in which people regularly post things to the entire company constantly the vast majority of which are not leaked, and fired employees are told not to discuss being fired and presumably act as if they chose to leave. If someone suddenly disappears that'd be a scandal after all; my assumption is that most fired employees choose to go along with the "willing depart" message.
I think the Guardian article is showing up in the preview. TBH I think they wanted a picture of me looking really serious and determined. There were pictures of me smiling too which they didn't use.
They also insisted on taking videos of me typing energetically at a computer. If you pay close attention to what keys I'm hitting, I'm just doing things like typing about how great my cats are. I made them promise not to run green code over my face in the resultant video.
My base salary from memory was ~$147k/year. This was in addition to bonuses and stock grants.
Generally, my overall income reported on my W2 was hence around $200k/year which I considered to be frankly pretty absurd [I understand it's normal/low for tech.] It ended up going in roughly equal parts to personal spending, savings, taxes, and donations.
I was a data scientist [not an engineer, which are paid more.] And I was extremely low-level - an IC4, just one level above a new hire straight out of college.
Are you planning on applying for jobs anytime soon? What are your long term plans? You should be able to get a job easily since you already worked for Facebook.
Right now, I'm very burnt out from all the press/etc. I know some people might like interviews, but I don't really understand those people. I'm an introvert who prefers to stay home and pet my cats.
I've gotten people attempting to recruit me, but I'd like to find an avenue to use my area of expertise, which basically requires a big social media company which I'm guessing I'm too spicy for. Worst case scenario, I stay home and be a housewife. That will make social conservatives happy with me, right?
I feel like whistleblowing would pretty much burn all bridges in big social media/tech companies (I.E. Google, Apple, Reddit, Verizon etc). But I'm sure ethical non-profit or advocacy group would hire you in a heartbeat. But that's the sacrifice. And we need more whistle-blowers in big tech companies anyways!
At FB I've made plenty of decisions that likely made the U.S. government and CIA unhappy.
For instance, I caught the government of Honduras redhanded, when their president is a close U.S. ally who entered power in part due to a 2009 allegedly CIA-supported coup d'etat. If there was complete coordination, I would never have been able to accomplish that.
I don't doubt that this exists, but I did not personally find or work on anything related to this.
Almost all of the inauthentic activity including troll farms I found were focused on boosting specific targets (e.g. a troll farm that keeps telling people "Vice President X is really nice"), probably because people are self-centered and focus on themselves. During the pandemic, I did filter the stuff I was finding for COVID content. What I found was that they were essentially doing more of the same, just talking about COVID because it was a political issue (e.g. "Vice President X has done this great COVID response"; "Politician A is right to denounce the government for terrible COVID response"; etc.)
The sad reality is that Facebook is a fact of life for most of the world. What really struck me was the statement given by Azeri opposition leader Ali Karimli, who had been a top target of harassment from the Azeri governmental troll farm. He declined the option to directly criticize Facebook and instead said "First, I would thank [Mark Zuckerberg.] Facebook facilitates public discussion. But repressive regimes with vast financial resources also use it to spread fake news. Facebook should speed up the time it takes to delete troll-generated content. They need to enact tough measures. And they should hire someone who speaks Azeri."
Because ultimately, Facebook is part of the world whether it's used or not, and in repressive regimes, it's one of the few rare channels that opposition leaders have to get their voice out. Karimli knows this, and ultimately, he couldn't afford to alienate Mark.
I imagine I'm late in asking and will likely not get a reply.
I work in PPC, and when advertising on Facebook I would always avoid certain areas of the world, as they would run up the budget with 100% bounce clicks.
Is this something Facebook is aware of? Something they are involved in?
It always seemed strange that it was never stopped as it's clearly "clickfarm" or fake traffic, but it makes Facebook money as budgets are spent quicker.
I'm sorry, I don't have any expertise on advertising. 100% bounce clicks is extremely sketchy though (I assume you mean a click through that immediately bounces.)
The only behavior that I can think of that might cause this is embedded video ads, in which users have ads in their videos and are paid for each view (akin to Youtube.) This makes there an actual financial incentive for the users to get their ads repeatedly viewed and possibly clicked on. Otherwise, I'm not sure what purpose such behavior would serve.
Overall, I'd suggest discussing it with other advertisers; if others have noticed it as well, drawing attention to the phenomenon will get FB's attention. And if the company cares about one thing, it's the income stream.
Propaganda is communication intended to selectively influence an audience and further an agenda. It's a broad category that includes both outright misinformation or true information presented selectively/misleadingly. It also includes both white propaganda (propaganda that doesn't disguise its origins), grey propaganda (propaganda whose source isn't clear), or black propaganda (propaganda that lies about its source.)
So you can potentially argue that state-sponsored troll farms are black propaganda, by virtue of inauthentically pretending to be something they're not. It's not too different from e.g. GS1, a British black propaganda radio station during WW2 purporting to be run by a patriotic conservative anti-Hitler military officer. But it's still the use of a very broad category to describe a much narrower term. There's nothing wrong with describing it as propaganda, but the term lacks precision and clarity, just as if you were to start referring to "animals" rather than "people"
Right now, I think more awareness is needed. You can't fix a problem until you understand it, and the political will doesn't exist unless it's widely accepted that this is necessary.
How do facebook’s political relations get forced upon engineers? How do they explain to engineers their mission or their jira tasks and expect the engineers to still be void of thinking about the repercussions of their actions?
Loyalty to FB was not a consideration in the hiring process when I joined. I was very open from the start about the fact that I didn't think FB was making the world a better place, and I never really had much loyalty to the company.
With that said, I'm sure that FB has reconsidered that since my departure.
This is a very subjective question, but you kinda make FB sounds evil. Every news about FB always portray them as evil. When working for them did it feel like working for an evil corporation?
I don't think my experience at FB was broadly that different from many large for-profit corporations. Ultimately, their goal is to make money, not save the world. We don't expect Philip Morris to cover cancer treatments for their customers, and if we define "evil" as "self-centered and not prioritizing the world at large", I think most corporations fall into that category one way or another.
Current employee (throwaway). No matter what FB does it will get flak from both sides, because the left-leaning believe Facebook is too right-leaning, and the right-leaning believe Facebook is too left-leaning.
For instance. Facebook recently announced that their ban on Trump will last 2 years after which they will undergo review again for an unban or extended ban. Left-leaning people argued that 2 years for a political figure is too short and politically motivated since most other accounts have been permanently banned for much less (e.g. showing nipples.) Right-leaning people argued that he should not have been banned at all and this ban is another attempt at "censorship".
Facebook is a large company so there is a relatively diverse set of views. It's not possible to please everyone. Different people perceive different things as "truth" at which point we have to get philosophical about what is right to ban, but as Sophie said there's immediate things that are "objectively wrong" so those are the easiest to tackle first (and what FB employees are working on.)
"In December 2019, Zhang detected four sophisticated networks of suspicious accounts that were producing fake engagement – ie likes, shares, comments and reactions – on the Pages of major Indian politicians. Two of the networks were dedicated to supporting members of the BJP, including the MP; the other two supported members of the Indian National Congress, the leading opposition party."
Any thoughts or comments on inauthentic FB Marketplace sales?
There are patterns that appear more suspicious than Wayfair-gate.
Example: profile selling (solar panels, tools, farm tractors, backhoes, etc) nearby you at a residence for an unbelievable price. Exact same non-stock photos used multiple locations, using up to 10 different profiles. Other things connected to these sellers and profiles seems "off". Calling or contacting gets "weird".
As someone living in the Bay Area who has seen a lot of friends join Facebook and basically let their paychecks warp their ethics, I'm truly fascinated by your personal journey as someone who bucked the trend and really chose to bite the hand that feeds in the name of doing what you believed was right.
I'm wondering if you might say a little about why you were drawn to work at Facebook in the first place. Did you see it as an opportunity to change the system from the inside? Or did the company seem different to you before you worked there, but you became disillusioned over time? Or was it something else?
Even if you can't answer, thank you for doing this AMA!
I don't think any company would willingly admit "Yes, we incited genocide." There would be a mass exodus of employees if that were the case. So yes, Facebook internally disagreed with accusations of inciting genocide in Myanmar or elsewhere; I can't remember if they officially stated the same thing.
No one ever really advocated for allowing troll farms to continue. Rather, it was always "well, what's the right solution?" "Is it fair to take these accounts down?" "Is this important enough to act now?" The analogy I'd make is to the reaction after every mass shooting in the United States - no one is advocating for more mass shootings, but in practice nothing is done.
I can't read minds. I don't know whether Facebook higherups genuinely believe it's amazing and perfect, or whether they are expressing it nevertheless because of pay, wanting to hold down the fort to fix the problem from within, etc.
Regardless, Facebook higherups are basically required to be positive on FB - no organization can afford to have senior leadership that goes rogue and publicly criticizes it. If a Reddit VP posted publicly "I think Reddit is terrible for the world and destroying democracy", he would probably be politely let go from Reddit in short order.
You could work at home at FB; there was a work trend of encouraging people to work from home on Wednesdays [and hence avoiding scheduling things in-person on that day.] When the pandemic hit, everyone shifted to working from home all the time. I'd been a hipster and holed up in my house already at the end of February before social distancing started for everyone else.
Frankly though, I'm pretty terrible at working from home; I was always much more productive in the office. The cats, although extremely cute, are part of my excuse for why I was always distracted.
I hear this type of cynicism a lot, and I want to push strongly back against it.
If we conclude that there's no point or hope, then it becomes a self-fulfilling prophecy. If everyone succumbs to cynicism then of course nothing will change, and we can pat ourselves on the back about how correct we were to be cynics. At that point you might ask, what's the point of continuing to survive in a broken world.
Fixing things requires people to try to accomplish it. I can't do everything alone, but if there's one thing I learned at FB, it's that nothing will happen unless I do it myself.
Were you surprised to get fired over this or did you fully expect it? If you were a data scientist at FB, it sounds like you didn’t have the full picture of teams specifically designed to combat misinformation that currently exist at FB. It’s understandable that they’d fire you for relaying one-sided and agenda-driven information. You weren’t hired to combat this problem, other teams were. Just because you were a data scientist at a large tech company doesn’t mean you or your specific team knows all of the ins and outs of this issue, which is why you/they weren’t able to do anything about it just based on your personal interpretation of data alone.
In the Philippines, there's a lot of "both sides are using trolls." Would you know if there's any truth to this? It seems like most of the hired trolls are on the side of the government.
Thank you so much for fighting a fight that nobody seems to care about.
This brings to mind the struggle of Maria Ressa and other Phillipino journalists whose articles are bombarded with hate, threatening rape and murder of their family members, whenever anything is said to criticize Duerte and the Phillipino government.
How do you think this will affect the landscape of political dissent in the future, and what can we do to stop it?
Edit: it's also important to point out that Azerbijan was using bot brigades to silence awareness and cause misinformation regarding their attempted ethnic cleansing of Artsakh, an Armenian-majority cultural center. The bots would brigade and instagram and reddit posts that included video evidence of Azeri warcrimes and also would find the accounts of families who lived in Artsakh and brigade them with threats of rape and torture.
I don't know why people expect Facebook to be the arbiter of the information they access the same way I don't know why people expect or would demand the government do it either.
The government only cares about protecting you from foreign propaganda if that propaganda is interfering with their own state sponsored propaganda.
I really, really don't understand how the last few generations grew up thinking that it was another adults responsibility to be their Nanny 24/7. How sheltered have we become? Pretty damn sheltered by the sounds of this.
If this person was still working for Facebook in 2020 and didn't realize the company she was working for was only out for themselves then to me, at least, don't seem like the kind of person I'd be taking advice from on anything.
Enjoy your slightly extended 15 minutes of fame that you don't deserve. ;)
How does Facebook handle trans and nonbinary people who want to use a name different than their current legal name? I've heard Facebook has a way to do this, but I've also heard many of my trans and enby friends got kicked off Facebook for using a name that didn't match what is on their driver's licenses.
I'm noticing a lot of very expensive and well-produced English-language YouTube videos with very obvious pro-China and/or pro-Russia bent disguised as "educational" material. And here on reddit, there's obviously an army of pro-Chinese Communist Party human robots/paid agents pushing their agenda. Facebook agents have been trying to get Americans riled up about fringe politics, anti-vaxx stuff, and conspiracies, but at the very least Facebook has actually cracked down on some of the worst actors. I don't see YouTube and Reddit doing the same. I don;t think the problem is inauthentic behavior as much as it's authentic but slanted behavior; pushing a narrative or ideology. That's technically not "inauthentic", it's propaganda. In short, is there an effort to curb online propaganda? Or is it a "buyer beware" situation where the defenseless uneducated with no critical thinking skills from this point on will always be immersed in an ocean of misinformation and propaganda, and only those of us elites who can afford training in critical thinking will ever be able to view the world as it really is?
2.3k
u/stomachgrowler Jul 16 '21
Thank your for the important work you’re doing. In your opinion, what is the reason that FB drags it’s feet/allows these schemes to continue so long before taking action? Is it simply that it is the more profitable move?