An essay on what AI actually needs to know about you, and what happens when it does not. Revision 1. The pillar of a second cluster. See the note at the foot on splitting it.
Somebody at your organization is going to ask an AI system a question about your organization today. Here is the only question that matters about it.
Where does the answer come from?
Not what model. Not which vendor. Where does the content come from, when the machine writes a paragraph about your business.
Almost nobody asks this. The conversation about AI at work is about capability, and capability is genuinely astonishing, and astonishment is a poor guide to anything. The better question is unglamorous and it turns out to determine everything else.
I want to work through it with one example, because the general version of this argument is easy to nod at and impossible to act on. So: an advertising agency. Independent, about a hundred people, a portfolio of retail and restaurant and franchise clients, sixty years old. The kind of firm where the work is craft and the product is judgment and the asset is a set of relationships nobody else has.
Everything that follows is true of a hospital, a denomination, a law firm and a manufacturer. The agency just makes it easy to see.
One question, four ways
Start with four questions somebody at that agency might type on a Tuesday afternoon. They look like the same kind of thing. They are not.
What is a normal click-through rate for display advertising in quick service restaurants?
What do we bill a senior art director at on the Henderson account?
Should we show the client the riskier concept or the safe one?
Why did we drop the second concept on that pitch back in 2019?
Every one of those gets an answer. All four answers will arrive in about the same amount of time, in the same tone, formatted the same way, with the same air of competence. Nothing in the presentation distinguishes them.
And they are four completely different events.
The first one is public knowledge. The machine has read a great deal about display advertising and quick service restaurants and it will give you a reasonable range. It did not need to know anything about your agency and it is right not to have consulted you. Fine.
The second one is a fact about you. There is a right answer and it is written down somewhere, in a rate card or a scope document. If the system can reach that document, it retrieves the number and you can check it. If it cannot, and it answers anyway, it will produce a plausible number. Somebody will eventually put that number in an estimate and somebody else will notice it is wrong. Bad, embarrassing, and recoverable, because there was a right answer to compare it against.
The third one is a judgment call. There is no right answer. Two good strategists could disagree and both be defensible, because the answer depends on this client, this history, this moment, this room. This is the question where an AI system is genuinely useful, and it is useful in exactly one condition: that it knows this client, this history, this moment and this room. Without those, it will still answer. It will give you the considerations any strategist anywhere would list. Useful the first time, useless the fourth, and it will never tell you which it is being.
The fourth question is the one this essay is about.
The fourth question
Why did you drop the second concept in 2019.
There is a right answer. Something happened. Somebody made a call and had a reason. Two or three people in the building still know it.
Nobody wrote it down.
The meeting had no minutes. The reason went into an email that is in somebody's archive, or into a hallway conversation, or into nothing at all. The concept deck survives, the winning route survives, and the reason the other one died lives in three heads and nowhere else.
So when the system is asked, here is what actually happens, step by step, and it is worth going slowly because every step is ordinary and the outcome is not.
The retrieval runs, and it succeeds. This is the first thing people get wrong. It does not come back empty. Retrieval systems almost never come back empty, because almost none of them has a threshold below which they decline to return anything. They return the nearest thing they can find, whether it is close or across the room. So it hands back the client's brand guidelines, a creative brief template, and a case study from a different account. All real. All yours. None of them about the question.
The model receives a question, some related-looking material, and an instruction to be helpful. Nothing in that arrangement tells it the material does not answer the question. It has no mechanism for noticing.
So it completes the pattern. This is the part worth sitting with, because it is the mechanism.
A language model predicts what comes next. Given a question shaped like why did we drop a concept, the most probable continuation is the most common answer to that shape, across everything the model has ever read. The client felt it did not fit the positioning. There were concerns about production timing. The research did not support it.
Those are not guesses. They are the average. They are the statistically central answer for that question shape, assembled from every agency that ever wrote anything down.
Which is exactly why it sounds right. It is right, for the median case. It is a good answer about agencies in general. It is simply not about you.
The system has substituted the population for the individual, and nothing in the output records that it did.
Why the near-miss makes it worse
Here is the part that surprises people, including me when I worked it out.
In this situation, partial retrieval is more dangerous than no retrieval at all.
The passages that came back are yours. They carry your client names, your vocabulary, your formats, your internal terminology. So the population-average answer arrives wearing your clothes. It reads as though it came from inside the building, because half of it did.
And if the system cites its sources, which the good ones do, it cites those passages. The citations are real. The links open. The documents exist. Somebody checking the sources finds genuine internal files attached to a fabricated claim, which is considerably more convincing than no citation at all.
That is the worst available combination: generic content, house voice, verifiable-looking sources.
The confidence runs backwards
Now a second property, and it is an inversion nobody expects.
How fluent an answer sounds is a function of how common the question shape is, not how much real material supports it. Common shapes produce smooth answers because the model has seen thousands of them.
And the most common question shapes inside an organization are the operational ones. Who approves this. Why do we do it this way. What happened last time. What is our policy on this.
Which are precisely the questions nobody ever wrote down, because everybody already knew.
So the questions most likely to be asked are the questions most likely to be fabricated, and they will read better than the questions the system can actually answer.
Nothing catches it
Five points in that chain where a mistake could be caught. None of them can.
The retriever does not know it failed, because returning the nearest thing is what it was built to do.
The model does not know it is extrapolating, because from the inside there is no difference between recalling and generating.
The output carries no marker, because a well-grounded answer and a fabricated one come out of the same process and look identical.
The reader has no basis for suspicion, because the answer is in their vocabulary with their documents underneath it.
And no later event reveals it. Somebody will eventually notice a wrong rate on an invoice. Nobody will ever notice a wrong reason for a decision in 2019, because there was no scoreboard.
Three axes, and where the trouble concentrates
Those four questions differ along three properties, and each is independent of the other two.
Settled or unsettled. Is there a fact of the matter, or does it depend on judgment?
Reachable or unreachable. Can the system actually get to the material?
Internal or external. Is it about your organization, or about the world?
Three yes-or-no properties, eight combinations. Six of them are fine. Two matter.
Unsettled, reachable, internal is the whole prize. A judgment call, made with your actual history in front of you. Should we show the riskier concept, given that this client killed a bold route in 2019 for a reason we can now read, and given what happened when we played safe last quarter. That is not a machine deciding. It is a person deciding with everything relevant in front of them in thirty seconds instead of two days, or, far more commonly, deciding from memory because reconstructing it was too expensive.
Unsettled, unreachable, internal is where organizations get hurt. A judgment about your situation, with none of your situation in it. Fluent, plausible, well-hedged, and about nobody. And no signal, ever, because there was no right answer to check it against.
Everything else in this essay is about the distance between those two cells.
What it costs, in four currencies
Organizations pursuing efficiency are pursuing one of four different things, and they rarely say which.
Speed. Same output, less time. Leverage. More output, same people. Elimination. The work stops being necessary. Recovered attention. Time given back to the thing that actually needed a person.
The last two are the only ones that are permanent. The danger cell damages all four, and the damage is not intuitive.
Speed goes to zero or negative. You get a fast draft. Because the question was unsettled, you cannot evaluate it yourself. So you go and ask the person who knows, which is the thing you were avoiding. Ask the tool, read the answer, sense it is generic, ask the person anyway. Three steps where there was one, and the tool added two of them.
Leverage inverts. The whole point of leverage is that a junior does more without a senior. What happens instead is that the junior produces a plausible document rather than asking a question, and now the senior has to read and correct a plausible document. Correcting confident wrong prose in your own field is slower and harder than answering the original question. Senior time consumed goes up.
Elimination fails outright. The task does not disappear. It becomes two tasks: use the tool, then do the work.
And recovered attention, the largest loss and the one nobody measures. The purpose was to give time back to the work only people can do. Instead attention goes to the most expensive cognitive labour available: evaluating a plausible claim inside your own domain, where you cannot skim, cannot delegate the check, and have to hold the whole thing in your head to catch what is subtly untrue.
That is harder than originating. And it is spent by exactly the senior people the exercise was meant to protect.
And then people stop asking
Here is the second-order effect, and it is what actually kills these projects.
People learn. Somebody asks a real question, gets a generic answer, notices, and adjusts. Not by complaining. By asking easier questions. Then fewer. Then none that matter.
The tool is not switched off. It is quietly demoted to the questions it can handle, which are the ones that were not important, and everyone agrees it is quite useful.
In 2025, MIT's Project NANDA published a report called The GenAI Divide, which found that ninety-five percent of enterprise pilots produced no measurable impact on profit and loss. The figure has been contested and the methodology is reported inconsistently, so hold the number loosely. Hold the finding. And note what the finding actually is: not that those organizations failed, but that they ran pilots and the pilots produced nothing anyone could measure. The report's own explanation was not that the models were bad. It was that the systems did not retain feedback, adapt to a context, or improve, so nobody learned anything.
They also found that in more than nine out of ten firms, employees kept using personal AI tools even when the official pilot failed.
Which makes complete sense. The personal tool is honest about being general. The corporate one implied it knew you, and did not.
So what is actually in the way
Now the useful part. If the difference between the prize and the danger is whether the material is reachable, the obvious question is what is stopping it.
Not one thing. Four, and they cost completely different currencies, which is why treating them as one number produces bad plans.
Reachable. The system can get it now. This is the only state that counts.
Stranded. It was written down and the system cannot get to it. Old files, dead websites, recordings nobody transcribed, a filing cabinet of minutes going back to 1974. Everything is intact. Everything is unreachable. This is retrieval work: technical, delegable, weeks.
Tacit. Somebody knows and it never became a document. Why the concept died. Who really approves what, as opposed to what the org chart says. What this client actually cares about underneath what they say. This is extraction work: a room, a recording, good questions, and somebody to structure the result. More tractable than the phrase suggests, and it takes people's time in a room, which is the one input no software reduces.
Unmade. Nobody produced it. And this state divides into two things that look identical and are not.
Unwritten means somebody knows and never wrote it. Same as tacit, really, and the same fix.
Undecided means nobody knows, because it was never settled. The approval matrix does not exist not because nobody typed it up but because nobody ever decided who approves what. Every job has been routed by habit and improvisation for eleven years, and it has mostly worked, and nobody noticed there was no rule.
You cannot interview your way to an approval matrix. Nobody is holding the answer. Somebody has to decide, which means a meeting where people disagree and one of them wins.
Which reframes this entire undertaking. A context project is partly an excavation and partly a constitutional convention, and the second half is why they stall. It is not a documentation failure. It is that an organization has been running on unresolved questions, and writing things down forces resolution.
There is a fifth state, and it is not a gap. Open. Deliberately unresolved, known to be unresolved, and reviewed on a date. An agency that has explicitly decided it is not ready to take a position on AI in creative concepting is in far better shape than one that never noticed the question, and a good system should be able to say your organization has explicitly not settled this, which is a much better answer than either a guess or a shrug.
What complete would even mean
You cannot say how much context an organization has without saying what all of it would be. So here is the list, for the agency, and the exercise transfers everywhere.
Identity and point of view. What the firm says it is. Positioning, methodology, published thinking, credentials, award submissions. and award submissions are worth naming, because they are the single best-written account of every case the firm has ever done, and almost nobody thinks of them as an asset.
People and decision rights. Who does what, who decides, who to ask. Almost entirely undocumented in most firms.
Clients and relationships. Per account: the contract, the guidelines, the approved claims, the contacts, the approval chain, the history, and the feedback record. That last one is the most valuable single artifact an agency could hold and the least likely to exist anywhere.
Work product. Everything ever made, findable by problem rather than by folder. Including the concepts that were not chosen, which are the ones that teach.
Process. How a job opens, moves, gets approved and closes.
Commercial. Rates, scopes, how each account is compensated, and what the last four jobs of this kind actually cost against what they were estimated at.
Performance. What happened when the work ran, connected to the work that produced it.
Governance. Confidentiality, IP, category conflicts, regulatory constraints, and anything the firm has already agreed to in a contract.
Category and competitive. What the firm knows about the worlds it works in.
Institutional memory. Why things are the way they are.
Ten layers. Now the honest part: complete is not the goal, and some rows should be at zero on purpose.
Individual compensation. Account profitability. Personnel matters. Anything crossing a category conflict. In a nonprofit working with young people, a whole additional set: medical records, family circumstances, counselling notes, anything identifying a minor. In a congregation, something stronger still. pastoral confidence, and in many places the giving records that the pastor is deliberately kept from seeing, which is a governance decision and not an oversight.
A context project without an exclusion list is a breach with a roadmap. Getting that list right is judgment rather than engineering, and it is the part most in-house efforts skip, because the person building it usually reports to the person who wants everything in it.
Where the value actually shows up
The temptation is to imagine full coverage producing faster versions of what people already do. That is the smallest part of it.
The work map of an agency has five places where things reliably snag, and none of them is a step. They are the joints. Trigger to job open. Context reconstruction at the start of everything. Brief to production. Creative meeting media, which is two departments with two clocks and one deadline. And the wrap that never happens, which is why the next job's reconstruction is expensive.
The pattern underneath all five is the same: information exists and does not travel with the artifact. So it gets re-supplied, by a person, from memory, every single time.
Coverage does not make anyone type faster. It makes the information travel.
And the biggest gains turn out to be in a category nobody budgets for. When you value any use of this, the baseline has three states, not one. Done manually, which costs visible hours. Done with a general tool, which costs hours plus verification plus the danger cell. Or not done at all.
Nobody writes the post-mortem. Nobody checks whether the firm has solved this before. Nobody reads three years of client feedback before writing the brief. Nobody looks at what the last four estimates in this category actually came in at.
Those are not tasks people are slow at. They are tasks nobody performs, because the cost exceeds the visible benefit. And they are exactly where elimination and recovered attention live, which are the only two efficiencies that do not decay.
So the honest test for any use of this is not how many hours it saves. It is whether it makes possible something that currently is not.
Which is a stronger claim and a harder one to argue with. The outcome is not a faster agency. It is an agency that does things it never did.
The part where the money question turns out to be prior
One complication, and in an agency it is not a footnote.
Under fee structures derived from hours and headcount, efficiency in billable work does not stay with the firm. It gets surrendered at the next scope review, when procurement asks why the same fee still assumes six people. An agency that becomes thirty percent faster at billable work under that arrangement has spent money to make itself poorer, and everyone inside knows it, which is the unspoken reason efficiency initiatives fail to be adopted by the people who would have to adopt them.
Which produces a conclusion worth stating plainly: elimination is safer than leverage, because eliminated work is invisible to procurement. If context reconstruction stops happening, no line item changes and there is nothing to reprice. If the same scope is delivered with four people instead of six, that shows up immediately.
And it means the pricing conversation is prior to the technology conversation, not downstream of it.
Context is not capability
Now the thing I got wrong for three years, at some cost.
A system that could answer a question, if somebody knew to ask, in the right way, at the right moment, is not a system that does anything.
I spent three and a half years showing this technology to organizations, mostly unpaid, to people who were genuinely impressed and who changed nothing about their Monday. I assumed for a long time that the demonstrations were not good enough. They were fine. The gap is somewhere else and it has four parts.
Knowing it is possible. Most people do not know what to ask because they have no model of what the thing can do, so they ask it to write an email.
Knowing how to ask. The difference between a vague question and a good one is enormous and it is a skill.
Remembering, at the moment of work. This is the big one. A person under deadline does not stop, open a different tool, formulate a query and read a result. They do what they have always done. The tool is not rejected. It is never reached for.
And doing it often enough to become a habit, which almost nobody does unaided.
Which produces the design principle that follows from everything above.
The interfaces that matter are the ones that fire without being asked.
The context pack should exist when the job opens, not when somebody thinks to query for it. The constraint check should run when the brief is submitted. The wrap capture should fire when the job closes and ask three questions while the answers are still in somebody's head. If a person has to decide to ask, most people will not, most of the time, and the value never arrives.
That sorts every use into two piles. Things a chat box can serve, which depend on the person knowing what to ask. And things that must be built into the moment of work, which is where adoption actually comes from.
Three layers, and you need all three
Context makes the answer true. Without it, everything operates in the danger cell and produces confident descriptions of a firm that does not exist.
Interface makes the answer arrive. Without it, the context sits there being correct, reachable by the two or three people who remember it exists.
Formation makes the judgment sound. Without it, people cannot tell a good answer from a plausible one in their own field, cannot tell when to override, and cannot tell when the system should have refused. Demonstrations do not produce this. Hours on real work do, and there is no shortcut I have found.
Buy only the first and you have a well-stocked library nobody visits. Buy the first two and you have a system people use and cannot evaluate. Most failures are an attempt to buy the third by purchasing the first.
What I am not claiming
Four things, because the argument above is easy to overextend and I would rather do it myself.
The technical build is not expensive. A vector store, embeddings, retrieval with citations and an interface that shows sources is a solved problem. One competent engineer, a few weeks. The database costs nothing. Anyone quoting a very large number for a first phase is selling the technology, and the technology is the cheap part. What money actually buys is sequencing, the exclusion list, the willingness to build refusal, someone to hold the argument open when two senior people disagree, and having watched it fail before.
Coverage does not make a system honest. It reduces how often it must decline. Declining is a separate mechanism, and without it more coverage just means confident answers on a wider range of topics.
And coverage does not eliminate error, it changes its kind. Retrieval can pull the right passage from a stale document, or a paragraph whose qualifier was in the paragraph before, or a passage that answers a nearby question. And a model can retrieve well and still write a smooth connecting sentence that was in none of it. Grounded is not the same as constrained. Coverage moves the failure from fabrication to misgrounding, which is more tractable only because you can now show somebody the source.
Some of this is old. Organizations have been inventorying their information resources since the late eighties, under the name information audit, with a real methodology and a real literature. What they never had was a way to make the catalogue answer questions, which is why the audits were done once and never maintained. The economics changed on both sides. The catalogue became answerable, and the audit became cheap enough to repeat. But the idea that you should know what you know is not mine and I did not invent it.
Where this leaves the question
Somebody at your organization asked an AI system a question about your organization today.
If it was a public question, the answer was probably fine and did not need you.
If it was a settled question about you and the document was reachable, the answer was right and checkable.
If it was a settled question about you and the document was not reachable, the answer was wrong, and somebody will eventually notice.
And if it was a judgment call about your situation, with none of your situation available, the answer was a well-written description of a company that does not exist, delivered in your vocabulary, with your documents cited underneath it, and nobody in the chain was able to tell.
The useful thing is not to know how often that happens. It is to know that it can be counted, and that the count is made of specific files and specific conversations that either exist or do not.
Which is uncomfortable, and it is the only useful position to be in, because you cannot decide about something you cannot see.
What you do after that is not a technology decision.
Before this publishes
The MIT hedge is now identical to the one in Three years of magic shows. That piece publishes first and its wording is canonical. If one changes, both change.
Length, and my recommendation. At roughly 4,400 words this is the longest thing on the site by a wide margin, on a platform with no audience yet. I would split it rather than cut it, and the break is after "And then people stop asking." Everything before that is the diagnosis and it is the stronger half: four question types, the near-miss, the confidence inversion, the three axes, the four currencies. Everything after is what to do about it: the four states, the ten layers, the exclusion list, the three layers of context, interface and formation.
Split that way you get a pillar of about 2,400 words that ends on people quietly demoting the tool, which is a real ending, and a spoke of about 2,000 that a reader arrives at already convinced there is a problem. Both are better than one piece nobody finishes. I would not cut to 3,000, because the passages that would go are the specific ones, and specificity is the only thing holding a hypothetical agency together.
Pillar or spoke. This is a pillar, and it is the head of a second cluster, separate from the safety cluster. Its spokes are the second half of itself, the context method, and the three arguments still trapped in the client paper: the review tax, the four sustainability tests, and the junior-role problem.
Josh is in this twice and absent from the rest. I would leave it. The essay works because it is about a mechanism rather than about him, and the two first-person passages, the three years of demonstrations and the thing he got wrong, arrive exactly where the argument needs a witness. Adding a third would start to make it a story about a person.
One check. The exclusion list passage names minors, medical records, family circumstances and counselling notes. Confirm none of it could be read as describing a specific organization he works with.