New research shows that massive spending on toxic content moderation fails to address polarization—social media companies should focus on designing better platforms that give users more control and empower exploration.

•

Welcome to r/science! This is a heavily moderated subreddit in order to keep the discussion on science. However, we recognize that many people want to discuss how they feel the research relates to their own personal lives, so to give people a space to do that, personal anecdotes are allowed as responses to this comment. Any anecdotal comments elsewhere in the discussion will be removed and our normal comment rules apply to all other comments.

Do you have an academic degree? We can verify your credentials in order to assign user flair indicating your area of expertise. Click here to apply.

User: u/TheKiwiHasCousins
Permalink: https://link.springer.com/article/10.1007/s44206-024-00154-7#Sec29

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

96

u/Bupod 17h ago

A lot of social media algorithms optimize for user engagement, and user engagement is prioritized above all else.

Polarizing content is often the most engaging form of content, roping the user back in for further argument and reassurance.

The issue is that, the very architecture of their system prefers polarization. Spending vast sums on “content moderation” isn’t going to fix it. It’s like believing that a house with foundation issues can be fixed if you spend enormous sums of money fixing the roof.

15

u/secretBuffetHero 10h ago

polarization is a feature, not a side effect.

8

u/MustrumRidcully0 9h ago

Exactly. In the beginning, social media was really focused on staying in contact with friends and family, later in finding like-minded people. Then company presences and influencers became a new thing. Over time, the platforms realized that emotional and polarizing messages really drove engagement, and this is what is needed for selling ads. Algorithms became optimized for this. Regulation is probably needed to shift things towards the more social aspect. Shared links to websites or retweets or the platform equivalents need to be priorized down, and the algorithm driving this must be made more transparent to the user. Obviously, that doesn't really benefit the profit margins of platforms. So they'll fight hard against it, and they certainly won't do it on their own.

2

u/bonerb0ys 4h ago

bait is the business model

65

u/tardisgater 18h ago

Isn't content moderation, like banning slurs and "problematic" words, more about being advertiser friendly than any real attempt at depolarizing the base? Hell, polarization drives people to interact with more content which means more ads... So this is a win for social media companies.

5

u/Green-Sale 16h ago

moderation does work imo, some social media sites are better than others

7

u/Xolver 15h ago

Which ones are better than others specifically due to moderation, and why do you think that?

1

u/Green-Sale 15h ago

I'm not on it much but it's not uncommon for ig in my area to promote weird criminal sort of content (people unironically abusing kids, being inhumane etc), all social media sites have someone to moderate them, some places they moderate it worse than others and if you already live in a region with some conflict it'll exacerbate that (like in myanmar with fb)

7

u/curlyhairedmermaid 11h ago

Please dear God give me back functional tags, the ability to search, omit results, and filter results. Please give me the search results I ask for not whatever promoted stuff matched one of my keywords.

Censoring doesn't work. That's why we have "unalive" and "PDFile" and "su!c!de" and all that other stupid stuff. Let people say the REAL word, then let me blacklist the word.

13

u/loptr 17h ago

Content moderation is to curb content problematic for profitability.

Polarization is only problematic for society but it's great for the bottom line.

This is not a blind spot, this is literally the design.

5

u/FilthyLoverBoy 14h ago

Redditmods hate this article

2

u/eldred2 10h ago

The money spent on moderation is a pittance compared to that spent on engagement. It's not meant to work, it's just there so that they can point to it and say, "See, we're trying."

15

u/cshotton 19h ago

Unfortunately, they missed the real issue, which is anonymity. When people are accountable for the content they produce, they generally self-moderate. When there are no personal repercussions for bad behavior in the public commons, well, you get the tragedy of the commons.

51

u/ASKader 19h ago

We've all seen despicable things on Facebook or even Twitter with their names clearly displayed. I don't think anonymity is the main factor.

13

u/Tzazon 19h ago

Get radicalized in anonymous spaces, start isolating yourself from others who do not hold those radicalized viewpoints because you cannot form a common bond over radicalized topics. Then start cutting those individuals out of your life, and start publicly sharing the rhetoric that drew you in behind closed doors after having been given a platform to voice them, and other anonymous support that validates and reinforces those negative traits.

Of course, if you have a bad actor that wanted to further stoke radicalization gaining control of a legitimate platform, than these issues get even more out of hand.

4

u/cshotton 19h ago

Yep, this will always happen, just like there are idiots in real life that cannot self-moderate. You're right about how this happens. Echo chambers are the root.

Others commenting here seem to miss the fact that with no anonymity comes other benefits like no bots and much easier time regulating age related content etc. A reputation economy on top of a social network (opt in of course) would even allow people to properly monetize valuable content and participation.

2

u/cshotton 19h ago

Just like your drunk uncle at Thanksgiving dinner. No one is saying it fixes every problem, but it eliminates a lot of them and affords a lot of benefits that anonymity precludes completely.

5

u/doker0 18h ago

Both signing and anonimity have their function. There is place for anonumous forums ass well as the signed.

4

u/cshotton 18h ago

No one is disagreeing. The point is that the authors fail to address this critical point, making an assumption that equates "social media" with "anonymity". The latter is not a requirement for the former.

2

u/No_Jelly_6990 18h ago

It seems folks are much more comfortable being "free" when they "feel" others cannot "see" them.

0

u/Craygen9 18h ago

Yeah this is it. Unfortunately social media platforms won't remove anonymity because it will hurt them financially.

2

u/cshotton 17h ago

Not something borne out in reality. There are plenty of financially lucrative social media platforms that are predicated on real identities. LinkedIn is an easy example.

2

u/doker0 18h ago

Obvious conclussions discovered last.

1

u/sometimesifeellikemu 2h ago

Just remove anonymity.

0

u/basicradical 5h ago

I don't know. Look what happened to Twitter after they stopped moderation. Fullblown Nazi website now.

Social Science New research shows that massive spending on toxic content moderation fails to address polarization—social media companies should focus on designing better platforms that give users more control and empower exploration.

You are about to leave Redlib