By Ailís Daly, Head of Trust & Safety, WebPurify, an IntouchCX company

Automated moderation can scan millions of listings in seconds. What it can’t do is tell the difference between a seller who didn’t know a sports team name was offensive and one who did.

That distinction, between ignorance and bad faith, between cultural context and clear harm, between self-expression and policy violation, is precisely where trust and safety gets hard. And it’s exactly where the human element remains irreplaceable.

Sophie Walsh, Global Director of Trust & Safety at Depop, made that case with striking clarity in a recent episode of the podcast “Trust Issues”, hosted by Ailís Daly by WebPurify, an IntouchCX company. Her insights from building trust and safety at one of the world’s most community-driven marketplaces cut straight to a question every platform is wrestling with right now: where does AI end and human judgment begin?

The Problem With Platforms That Over-Automate

Trust and safety decisions on peer-to-peer platforms don’t exist in a vacuum. At Depop, a resale marketplace with over 55 million global users where commerce, creativity, and culture intersect, a listing is more than just a product. It’s often a reflection of someone’s identity, their personal style, and in many cases, their income.

That context changes everything about how moderation decisions land.

“Trust and safety actions can sometimes feel more personal to users because our decisions can affect their self-expression and potentially their earnings.” — Sophie Walsh, Global Director of Trust & Safety, Depop

When platforms rely entirely on automated systems to make those calls, they risk getting it wrong in ways that erode exactly the trust they’re trying to protect. A seller penalized for an honest mistake, one they didn’t know they were making, doesn’t just lose a listing. They lose confidence in the platform, and when that happens at scale, participation drops.

When Cultural Context Becomes a Trust & Safety Variable

One of the most instructive examples Sophie shared involves “Washington Redskins” merchandise. Vintage sportswear is a meaningful category on Depop, but the team’s name and branding are now widely recognized as offensive, leading Depop to classify related items as prohibited content.

“Many sellers were genuinely confused or frustrated. For them, it was simply a retro item that they had owned for many years with no intent to cause harm. In several cases, people were not yet aware of the broader cultural shift around the name.” — Sophie Walsh

This is a pattern seen consistently across platforms of all sizes. The hardest trust and safety calls aren’t the obvious ones. They’re the ones where a seller did nothing wrong by their own understanding of the world. Getting those right requires a system that can distinguish intent from outcome, and that isn’t something automation can resolve on its own. It takes trained human judgment, clear escalation paths, and communication that treats users as partners in the process, not just subjects of enforcement.

What this moment illustrates is that policies are not static rule books. They’re living frameworks that must evolve alongside changing cultural norms. And when they do evolve, the way platforms communicate that change matters as much as the change itself.

The Proactive Shift: From Enforcement to Education

What separates platforms that build lasting trust from those stuck in reactive mode is a deliberate shift toward prevention.

At Depop, that means investing in user education: help centers, community guidelines, and safety tips that help well-intentioned users understand expectations before an issue arises. The goal is to reduce the number of cases where enforcement is even necessary.

“It definitely requires a very firm intention to carve out the time to move from reactive to proactive. That culture of urgency can really snowball within the trust and safety function, and can feel relentless at times. So as much as you can get on the front foot and move into the proactive space, the better for users, the business, and those working in that space.” — Sophie Walsh

This is the real cost of over-relying on automation: it optimizes for speed, not prevention. It gets better at removing bad content, but it doesn’t reduce the volume of bad content over time. Education does.

Where AI Belongs in the Trust & Safety Stack

None of this means AI doesn’t belong in the moderation workflow. At Depop, it plays a critical role. Just not the one many assume.

“AI is most valuable when it helps our teams to see those patterns that would be difficult for humans to spot alone — emerging trends or shifts in behaviors. It has also helped us to support early detection by surfacing signals rather than making decisions itself.” — Sophie Walsh

That framing is precise and important. AI as a signal amplifier. AI as a cognitive load reducer. AI as the system that tells a human moderator where to look, not what to decide.

“At Depop, we think about AI very much as a bridge and not a barrier to helping to translate information, reduce that cognitive load, and give people better inputs to make those fair decisions.” — Sophie Walsh

This is the human-in-the-loop model in practice: not humans checking AI’s work after the fact, but humans and AI working in tandem, each doing what they do best.

What Getting It Wrong Actually Costs

Sophie was direct about what happens when platforms get the human-AI balance wrong in either direction:

“If we get that distinction wrong, the risk for users is a loss of trust — particularly if people feel that they’re being penalized for honest mistakes. And conversely, if bad behavior isn’t being addressed, it can really undermine confidence in the marketplace and discourage participation.” — Sophie Walsh

The consequences aren’t theoretical. Platforms that over-automate burn good-faith sellers. Platforms that under-enforce lose buyers. Both outcomes destroy the community that makes peer-to-peer commerce work.

The Future Challenges Already Taking Shape

Sophie flagged three emerging threats she believes the industry is underprepared for, and none of them have a clean automated fix.

• • AI-generated receipts and fake proof of authenticity. As generative tools become more accessible, verifying whether a document is real stops being a visual check and becomes a deeper forensic problem, one that scales faster than most trust and safety teams are built to handle.
  • False social signals. Fake likes, follows, and engagement that mimics genuine community behavior are already eroding the credibility of platform trust systems. Many platforms still don’t have reliable ways to detect social engineering that hides behind the appearance of real participation.
  • The shift from marketplace risk to personal safety risk. As platforms add social features and community grows, the line between a transaction dispute and a harassment case blurs. Most platforms were built to manage the former. Few are fully equipped for the latter.

What connects all three is the same problem that runs through this entire conversation. They require judgment that understands context, intent, and human behavior. Automated systems can flag. They can’t adjudicate.

Trust Is Built in the Space Between the Algorithm and the Answer

The platforms that will get trust and safety right in the next five years aren’t the ones with the most sophisticated detection models. They’re the ones that understand where automation ends and judgment begins, and that invest in both with equal seriousness.

As Sophie Walsh put it: human judgment remains essential, particularly in a creative, community-driven marketplace. Automated systems alone can’t reflect community values.

This article is based on a conversation with Sophie Walsh on Trust Issues, the podcast by WebPurify, an IntouchCX company. Listen to the full episode here. 

Trust & Safety is one of IntouchCX’s core capabilities, combining content moderation, fraud detection, and platform integrity expertise with a human-in-the-loop model built for scale. 

Learn more about Trust and Safety services here

About the Author: 

Ailís Daly is the Head of Trust & Safety, EMEA at WebPurify and a veteran policy expert with 15 years Tech experience. 

She is a qualified barrister who has held trust and safety policy and operations roles at TikTok, Airbnb, Twitter, and Google. Her work has spanned issues like online misogyny, the impact of COVID-19 on marketplace homestays, international conflicts, brand safety and elections. She has played a pivotal role in preparing for major regulations and building trust with regulators-enhancing both organizational reputation and compliance. 

Based in Ireland, she advises top-tier companies on regulatory requirements, Trust & Safety operations, and policy development across the region. She is also the host of Trust Issues, a Trust & Safety–themed podcast that shares insights from the people who keep the internet safe, featuring stories and perspectives from prominent voices across the online safety world.