Essay. Revision 1. ~2,000 words. Evergreen. The hub of the safety cluster.
I sell something called AI safety and I want to argue against it first, because the objection is better than most of the answers given to it.
Nobody knows how these systems work. The math is published, but that is not the same thing. No one can say why a model produced the answer it produced, or predict what it will produce next. The people who build them call interpretability an open problem. Abilities show up that nobody designed. Behavior shifts between versions in ways the builders did not intend and cannot fully explain.
So when someone sells you AI safety, ask what is being made safe. Not the model. The vendor did not build it and could not change it if they wanted to. Not the answers it gives. Nobody can promise those. What is being sold is a document, and a document does not make anything safe.
There is a harder version of the objection. Safety means safe from something. You can protect against a danger you can name. But the ways this can go wrong include ones nobody has thought of yet, because the technology is new and its effects on institutions have not had time to show up. A safety practice built against the dangers we already know about is, at best, handling the part we happened to notice.
I think that is right, and it applies to what I sell.† So I want to say where I think it stops.
The mountain
Nobody says a mountain is safe. The weather cannot be predicted more than a few days out. The ground holds hazards nobody mapped. People who know far more than you do still die up there. The system is dangerous, and it is beyond anyone's full understanding.
And wilderness safety is still a real thing. It is taught seriously. The difference between a group that practices it and a group that does not shows up in who comes home. Nobody finds this contradictory.
So what is wilderness safety made of? Not control over the mountain. It is knowing what you are carrying and what it does. Knowing who is responsible for what. Agreeing ahead of time what conditions turn you around. Having a plan for the emergencies you can foresee, and a general way of handling the ones you cannot. Telling someone where you went. Not bringing people who are not ready.
None of that makes the mountain predictable. All of it changes what happens when the unpredictable arrives.
That is the whole argument, and I want to be careful about what it does and does not allow.
What it allows. Safety for an organization is not a claim about models. It is a set of practices about what you expose, who decides, what you refuse, and what you do when something goes wrong. Every one of those is fully inside your control. Every one of them matters whether or not anyone understands the model.
What it does not allow. The suggestion that we know the dangers. We do not, not fully. A guide in the mountains knows what hypothermia is. We are working in country where the equivalent of hypothermia has not been named yet. So the practice has to include a way of revising itself, and any framework that presents itself as finished is lying about where the field actually is.
The honest version is not this makes you safe. It is: you are going into country nobody has mapped, there are things you should do before you go, and doing them changes your odds.
Safety is a property of the practice
I want to state the move plainly, because everything else follows from it.
Safety was never a property of the system. It is a property of what people do.
This is true of things we do not usually think about this way. A commercial airliner is a machine with millions of parts and failure modes nobody has enumerated. Aviation safety is not a claim that the machine is understood. It is checklists, roles, refusals, reporting, and a culture where anyone can stop the process. Electricity is dangerous and not intuitive to most of the people who work with it daily. Electrical safety is procedure.
In every one of those cases, the safety is not in the thing. It is in what the people around the thing have agreed to do.
Which means the black-box objection, though correct, is aimed at the wrong target. Yes, the model is opaque. That was never where the safety was going to live.
What this looks like when it is real
Five things, and they are the same five in the mountains, in a cockpit, and in an organization.
Know what you are carrying. Which tools are in use, by whom, for what, under what terms. Most organizations cannot answer this and are surprised by the answer when they get it. This is the layer people most want to skip, and I have written about why, and about four things my own guide to it was missing.
Know what you will not do. Not a general commitment to being careful. Specific refusals, written down, that someone can point at when a vendor pitches the thing you refused. A refusal you cannot cite is a preference.
Know who decides. When something is unclear, the question is not what the policy says. It is who gets to say. If that is unnamed, the answer at the moment it matters will be whoever is most confident.
Know what turns you around. The conditions under which you stop. Agreed in advance, in daylight, by people who are not currently under pressure.
Know what you do in the first hour. Not a plan for every emergency. A named owner, a way to escalate, and something to say to the people affected while you work out what happened.
None of that requires understanding the model. All of it changes what happens when the model does something nobody expected.
The part where writing matters
There is a version of this objection I hear from thoughtful people. We already know all this. Our people are careful. Writing it down is bureaucracy.
I think that is wrong, and the reason is specific.
An unwritten position can change silently. Anybody can revise it, under pressure, and there is no record that it moved. A written one has to be argued with.
Writing also forces the disagreement to happen now. Somewhere in drafting, two senior people find out they do not agree about whether this technology can touch pastoral conversations, or hiring, or anything else that involves a person's life. That disagreement is there either way. Writing it down means the argument happens on a Tuesday in a room, instead of at nine at night while something is going wrong.
And it makes the thing sharable. Twelve people carrying twelve private versions of the policy cannot coordinate. A page they can all point at is not mainly there to be read. It is there so a conversation has something to refer to.
The person you need to get this right is not the senior leader who has thought about it. It is the nineteen-year-old volunteer who has thought about it for nine seconds. They cannot read your mind. They can read half a page.
Why this comes before anything else
Three reasons, and the first is not a prediction.
It is already happening. Your staff are using these tools now, on their own accounts, for the writing that is hardest, which is always the writing about actual people. A donor situation. A family circumstance. A young person's story. A personnel matter. Nobody is being unreasonable. They are trying to do their jobs and something showed up that helps. But information about people in your care is leaving your organization, nobody chose that, there is no record of it, and it cannot be brought back.
A leader who cannot name what is happening cannot lead through it. The first thing leadership does is describe reality in a way a group can act on. Silence about this gets read two ways at once, as fear by some people and as permission by others, and both readings are being made right now inside your organization.
The gear goes on before the trip. This is the part that decides everything after it. Safety is not a stage you finish and leave behind. It is what you carry into everything that follows. Experimenting without it is experimenting with no edges. Training people without it forms them around nothing. Building without it hands capability to an organization that has no formed judgment about it, which is worse than having nothing, because now the mistakes are fast and confident and they carry your name.
The liability picture moved underneath all of this in the last eight months, in ways most organizations have not heard about, and that is a separate and dated piece.
There is a fourth reason and it is the one people find hardest to accept. The ability to handle this cannot be borrowed later. You cannot build judgment during the week you need judgment, any more than you can disciple someone in the middle of their crisis. Every organization I have worked with already knows this about every other kind of formation, and almost none of them apply it here.
What I am not claiming
I would rather say this than have you find it later.
This does not make your use of AI safe. It makes your practice something you can govern.
It does not create insurance coverage. If your policy excludes this, a document does not override it.
It does not prevent an incident. It lowers the odds on the ones you can foresee and shortens the response to the rest.
It does not stay true. It describes one moment, and the moment is moving faster than any review schedule.
It is not a substitute for the slower work of forming people. A rulebook for people who have never used these tools is a set of rules about a country nobody has visited.
And it does not touch the thing the objection at the top was actually about. Nobody knows what these systems will do. This work does not change that. It makes you an organization that has decided, written it down, and prepared, in country that stays unmapped.
Where that leaves it
There is no such thing as AI safety, if you mean a safe system.
There is such a thing as a prepared organization. The difference between a prepared one and an unprepared one is not a claim about the mountain. It is the whole of what you actually control.
The wilderness is not safe. We teach wilderness safety anyway, and the people who learn it come home more often. Not because the teaching made the weather predictable. Because it changed what they carried, what they refused, who was responsible, and what they did in the first hour when something went wrong.
That is all this is. Less than the word safety suggests, and a great deal more than nothing.
† No source. This is a judgment, not a fact. I think the objection at the top of this piece is correct and I cannot prove it, and I have an obvious interest in where I decide it stops. Weigh it as a judgment.
I build these systems and I sell a facilitated version of this work. The full method is published free in the field guide, and I would rather you finish it yourself than pay me and not finish.
If you are choosing between people offering to help with this, I have written about how to tell which of them is worth trusting, including why you should discount me.
If you want the current state of the liability and insurance picture, that is a separate and dated piece.