Illegibility As Defense

Sunlight Considered Harmful

Epistemic status: Probably 80% right, not sure which 80%

A lot of communities are defended not through existing legal systems, or public systems of assigning guilt and responsibility, but through informal, private whisper networks—even, sometimes, when dealing with actual criminal behavior.

Those networks tally up accusations—if someone has, say, an abusive pattern of behavior, eventually their victims will talk to their friends, and their friends will talk to others, and if these incidents become common knowledge the abuser will stop being invited to parties and be cut off from various social resources. But, whisper networks are flawed in many ways.

We have things we want out of our legal system — things like avoiding false positives, people being able to fully defend themselves, a system with a process and an outcome that are both extremely legible to all involved and external observers.

Whisper networks are not like that.

You might look at a whisper network and think: this doesn’t have a jury of your peers. It doesn’t have two people sitting down and putting all the facts on the table. It doesn’t let you legibly defend yourself, it doesn’t make its process and outcomes legible to all involved. And you can look at that and say “wow, whisper networks suck, because they don’t have all the good things our legal system does.”

It’s an understandable critique!

However, I think this is a bad take—there are some very specific pressures that create the whisper networks that are common in lasting communities. To understand whisper networks (and many other things!) you must first understand those pressures.

Clocktower and Adversarial Optimization

Let’s talk about Blood On The Clocktower. One of the things I’ve talked about in Clocktower is that obviously the evil team doesn’t want to be legible/known, because the entire goal of the game (if you’re Good) is to figure out who the evil team is and kill them. But, counterintuitively, the good team also doesn’t want to be legible.

Clocktower isn’t just a game of learning who to trust with power and who to wield that power against. It’s about who to trust with information, and how, unless someone has deemed themselves worthy of that trust, you should probably keep valuable information hidden.

The good team is operating in a state of both offensive and defensive paranoia. They have to be very careful who to trust, because if they trust the wrong person with information, that information can easily be corrupted. Getting a sense of who’s trusted, who’s powerful, what information is getting passed through what channels — these are extremely crucial for evil to figure out—they need to understand that system in order to defraud it.

This adversarial nature causes paranoia, and paranoia causes illegibility. It results in things you don’t say publicly, in “I have a plan but I won’t explain it to the whole group because then it might not work”. Because, being able to easily map where trust and information and power come from makes it much, much easier for bad actors to subvert those.

This is why whisper networks are illegible in the way they are.

Look at something like what happened with Brent. While I wasn’t in the relevant communities at the same time that Brent was, my understanding is that he was good at finding people and basically making them his — making them feel sympathetic to him, obligated to help and protect him or at the very least obligated not to actively hunt him down.

The problem is that Brent repeatedly abused people. And when the community immune mechanisms started to kick in, Brent was extremely aware of this. He actively, for a meaningful amount of time, subverted some of the defense mechanisms intended to stop people like him. He was in situations he shouldn’t have been in, and it took longer to expel him than it should have.¹

The social power of bad actors is directly dependent on how well they can evade community immune systems.

As such, abusers are extremely incentivized/trained to get around those systems, and so you need to learn to defend against people who will be trying competently to subvert it.

Public accusations/specifically setting yourself against someone are easy to heavily retaliate against or otherwise disincentivize, which makes it difficult to import some of the nice things about the legal system. Those systems would be heavily optimized against, or just not used because of the massive time+energy costs involved. This is why part of why many people don’t go to the police/courts to settle things.

Illegibility as Defense

There’s a broader point here. Just sitting down and mapping out a social network (who’s friends with who, which people are trusted arbiters, how does information and resource propagate through the network) makes it a lot easier — for good and for evil — for someone to come in and have their way with your social network.

You pick up this sense of paranoia the more you interact with ~intelligence apparatuses, on either side, in similar ways to which lawyers or folks in Compliance are very aware of what gets put in writing.

A really good worked example of this is Andor. It’s a show about many things—one of those things is two intelligence apparatuses, each trying to get ahold on the other. This results in incredible amounts of waste in the service of secrecy.

Similarly—The Wire follows a special investigations unit, which is an intelligence apparatus of the city government of Baltimore. Notably, when they first set up a detail to track Avon Barksdale’s drug ring, not only do they not know specific details of their operation that would lead to his arrest, but they don’t even have a picture of the kingpin. This is not an accident—the Barksdale organization is relatively competent at being illegible to the police, and actively adjusts to the tactics of police in order to avoid detection.

In fact, like ninety percent of the work is mapping out the connections between people — mapping out that social graph. Once you have that, it becomes much easier to monitor those nodes, monitor the lines between nodes, and take them down.

There’s a similar dynamic with networks of power broadly. Power wants to be illegible, in order to protect itself from intelligence apparatuses that seek to steal or curtail its power. Sometimes, the relevant adversarial intelligence apparatus is “journalism”. Sometimes it’s the KGB. I think this is a morally neutral thing—there are many groups of people who try to hide from journalists, where it’d be a good thing for information on their networks to end up in the New York Times—but there are also many examples of power-for-good that nonetheless does not want to end up on the front page.

So, mapping social graphs is a direct form of power. As such, those social graphs are not made legible, to protect against bad actors in various ways. In the same way that you don’t want to put your address on the internet, you don’t want to put your social network on the internet.

One problem: your social network is on the internet, by design, through social media. This presents many vulnerabilities, none of which I am going to discuss publicly.

I think this point has been made before me—illegibility is something postrats have been writing about for a while. But I wanted to re-derive it, because that’s what this blog is: re-deriving 101 or 201-level things based on relatively little, mostly fictional evidence, in ways that can be helpful for future or smaller Sammys — or people who don’t have the exact set of special interests that I do.

1. Again, I don’t have a ton of context on this specific example, and am mostly using it to gesture at a broader theme.