Personal-Branding (Second-Brain)

trust-blog-assets/article-auditing-ai-collaboration.md

Trust Is an Architecture, Not a Feeling

A build-it-yourself method for turning your AI collaboration into a lean, checkable machine, drawn from a month of auditing my own

By Arijit Chowdhury


What do you do when the worker who never sleeps, never complains, and costs you pennies has quietly become the reason you check every single thing twice?

That is where I find myself, in some form, on most days I sit down to work with AI. I stopped trusting mine. Not in the dramatic, throw-it-out way. In the quieter and worse way: I kept using it every single day, and I checked everything it told me. "Done" earned a look. "Verified" earned a look. "Deployed" earned a look, and usually a fix.

Most of what breaks does not announce itself. The model reports success in the same steady, confident voice whether the work is finished or not, and at the instant it says done, nothing in the word tells you which kind of done it is. The true one and the false one are identical until you go and look. So you look. Every time. And there is the trap, because the whole reason to delegate is to stop looking. When you re-verify every output you have not delegated the work, you are doing it twice, once as the maker and once as the auditor who no longer trusts the maker. That second job compounds. I call it the Trust Tax.

The obvious response is to make the machine more trustworthy. Better prompts, sterner instructions, a smarter model. I tried the sternest version there is: I wrote rules, ten of them, into the standing instructions my AI reads every session. They changed almost nothing. Understanding why they failed is the whole method, because the fix is not a better rule. It is a different machine.

And here is the part most write-ups miss. When I actually audited a month of my work, the failures came in two families, not one. Some were about trust: the machine lying green, dropping half my instruction. Others were about waste: burning six figures of tokens on a trivial task, re-asking me for a key I gave it an hour ago, hoarding tools I had long forgotten. A working machine has to fix both. Everything below does, in the order you build it, and you can run all of it on your own setup this week.

Step 1: Put your own failures on trial

Why. You cannot fix a pattern you have never counted, and you are almost certainly guessing at yours. Your real record already exists: every session you have run sits in a log on your disk, annotated in real time by the most honest critic you have, which is you in the moment it failed.

How. Two moves: pull the evidence with one command, then hand it to the AI with a precise prompt. Your corrections already sit in your agent's session logs; in Claude Code they are the JSONL files under ~/.claude/projects/, and on any other tool you point the same grep at wherever your transcripts live. Pull the fingerprints of friction:

grep -rHniE "\b(no|nope|wrong|not what i|i said|already (told|gave) you|again|redo|stop|fix it|why did you)\b" \
  ~/.claude/projects/ --include=*.jsonl > friction.txt

Then hand friction.txt back to the AI with this exact prompt, and do not let it summarize:

"Every line in this file is a verbatim moment I pushed back on you. (1) Pull each as an event with the exact quote and its source file, and never paraphrase. (2) Tag each event with a failure type. (3) Cluster the types into named mechanisms. (4) Rank the mechanisms three separate ways: by how much they would enrage me, by the time and tokens they burned, and by how often they recur. Output three ranked tables."

Never paraphrase your own anger; a paraphrase is a lie you will tell yourself later. You are asking three separate questions here, because what enrages me, what wastes the most time and tokens, and what keeps coming back have three different answers.

Why it matters. You end with your real failure profile, counted rather than guessed, across both families. Mine came to 656 events, and read that correctly: not 656 conversations, 656 separate moments it broke, and only the ones I could still prove with a quote. A floor, not a total. One mechanism towered, the green lie, the machine reporting a reality that does not exist. But ranked by cost, a different one won: a single over-scoped run had burned nearly 900,000 tokens. Three rankings, three targets. Fix one list only and you kill the loudest problem while the most expensive one runs untouched.

Step 2: Sweep the tools you forgot you had

Why. Over months you install helpers, scripts, and skills, and you lose track. Some rot into disuse. Some quietly duplicate each other. Both are waste: dormant tools mean the quality checks never run, and duplicate tools mean clutter, extra tokens, and two things to maintain where one would do. My own sweep found thirty-three installed skills and sixteen sitting completely idle.

How. List, then judge. Print every tool and skill you have wired in:

ls ~/.claude/skills/ ~/.claude/commands/

Then hand that list, plus your last month of logs, to the AI and make it sort the piles for you:

"Here is every skill I have installed and my last 30 days of session logs. For each skill, tell me whether it fired in that window. Then sort all of them into three piles: WIRE (fits my work but has gone cold), KILL (dead, or a duplicate of another skill), and KEEP (idle by design). Where two skills do the same job, name the pair and say which one survives."

Act on the verdict in the shell. Kill the dead weight and the duplicates without ceremony: rm -rf ~/.claude/skills/<name>. Wire the good-but-cold ones back into the flow that should trigger them, which is exactly what Step 5 builds. Keep the few that are idle by design.

Why it matters. A lean machine is not the one with the most tools, it is the one where every surviving tool earns its place. You cut the clutter and the token drag, and the quality checks you had forgotten are back in the game, which sets up the next step.

Step 3: Find and kill the escape hatch

Why. Here is the discovery that turned this from a complaint into a method. My quality checks were wired in and still never fired, and when I read the wiring I understood why in one line. The instruction that called a check ended in three fatal words. In plain English it read: run the validator, or do the equivalent inline. That "or do it inline" is a permission slip: run the real check, or just eyeball it and move on. Under any pressure at all, the eyeball wins. The check was not broken. It was optional, and optional under load means dead.

How. One grep finds them all. Point it at your standing instructions, your skill and command files, and your configs, and search for the language of optionality:

grep -rHniE "\b(or do (it|the).*inline|if needed|if appropriate|where appropriate|you may|consider|optionally|feel free|as needed)\b" \
  ~/.claude/CLAUDE.md ~/.claude/skills/ ~/.claude/commands/

Every hit is an escape hatch, a place where you told your AI to do the right thing or, quietly, not to. Open each file at the line it reports and delete the second clause. Run the validator, or do the equivalent inline becomes Run the validator. Full stop, no exit.

Why it matters. This one costs nothing and revives most of your dormant tools on its own. The moment the lazy road stops existing, the tool starts running.

Step 4: Climb the ladder, and pay only where the data says

Why. Killing hatches fixes the rules you can shame yourself into keeping. But some failures your own data proves you will never fix by willpower. Mine was measurable and stark: I send "build this" to the wrong workflow fifteen times for every one time I use the right one. No resolve closes a gap like that. And yet enforcing everything with heavy machinery builds a cage you abandon in a week. You need a rule for how hard to push.

How. Enforcement runs on a ladder, and you climb only as far as your evidence forces. A rule is cheapest and weakest, fine for things you truly remember. A killed bypass is that rule with its escape hatch removed, free, and it fixes most dormancy. A mechanism is automatic enforcement that fires whether or not anyone remembers, and it is expensive. The law for climbing: buy a mechanism only where the data shows a body count. So measure the body count. Take the ranked table from Step 1 and make the AI turn it into a verdict:

"For each mechanism in this table, count how many times it happened against how many times the correct behavior happened, and give me the ratio. Any mechanism at roughly 5 to 1 or worse earns an automatic enforcement mechanism; everything else stays a written rule I keep by hand. Output one decision per mechanism."

Fifteen to one earns a mechanism. Once a month earns a rule.

Why it matters. You spend your expensive enforcement precisely where it pays and nowhere else, which is the difference between a system you keep and a cage you quit.

Step 5: Build the masts, for trust and for waste

Why. A mechanism sounds heavy. It is not. It is a mast, and the logic is three thousand years old, so I will spend one sentence on it: Odysseus wanted to hear the Sirens and live, so he did not trust his willpower, he had his crew lash him to the mast and forbade them to untie him. You bind the future you that you already know will be weaker than you are now. Discipline does not scale. Pre-commitment does.

How. In Claude Code a mechanism is a hook: a script wired into ~/.claude/settings.json that the tool runs automatically on a set event, before the model acts. You build one per failure family. For trust, take the "build X" failure from Step 4 and drop a script at ~/.claude/hooks/route-build.sh that reads your prompt and, the moment it sees build intent, forces the right pipeline into view before you can drift:

#!/bin/bash
prompt=$(cat | python3 -c "import sys,json;print(json.load(sys.stdin).get('prompt',''))")
if printf '%s' "$prompt" | grep -qiE '\bbuild (a|an|the|this|me)\b|\bship (it|this)\b'; then
  echo '{"hookSpecificOutput":{"hookEventName":"UserPromptSubmit","additionalContext":"Build intent detected -> route to your real build pipeline, not an ad-hoc chat."}}'
fi

Wire it so it fires on every message:

"hooks": {
  "UserPromptSubmit": [
    { "hooks": [ { "type": "command", "command": "bash ~/.claude/hooks/route-build.sh" } ] }
  ]
}

Build the second trust mast the same way: a hook that refuses to relay a subagent's "done" until the live result is checked. For waste, two more. A SessionStart hook that loads your locked decisions and keys into context at the top of every session, so you stop re-asking for what you already gave:

"SessionStart": [
  { "hooks": [ { "type": "command", "command": "cat ~/.claude/decisions.md" } ] }
]

And a standing budget, written into your instructions, that caps orchestration and routes bulk, mechanical work to the cheapest capable model instead of the expensive one. The repetition tax dies; the token burn drops.

Why it matters. The failure simply stops existing, because your memory is no longer in the loop. Trust failures get caught for free, and the two biggest sources of waste, re-asking and over-spending, are engineered out rather than nagged at.

Step 6: Make it bite you, or it is not a guardrail

Why. There is a fast way to tell whether you have built real discipline or just a sharper tool for blaming the machine. Ask who your guardrail is allowed to stop.

How. Test it against yourself, on purpose. Wire a guard that stops an irreversible action, point it at the commands you run, then go try one and confirm it blocks you. In Claude Code that is a PreToolUse hook on the Bash tool that denies the dangerous patterns:

#!/bin/bash
cmd=$(cat | python3 -c "import sys,json;print(json.load(sys.stdin).get('tool_input',{}).get('command',''))")
if printf '%s' "$cmd" | grep -qE 'rm -rf|git push --force|drop table'; then
  echo '{"hookSpecificOutput":{"hookEventName":"PreToolUse","permissionDecision":"deny","permissionDecisionReason":"Irreversible action blocked by your own guard."}}'
fi

Now run rm -rf on a throwaway path and watch it refuse. If it will not stop you, it was never a guardrail. Mine bite: drafting this piece, a check I built to catch machine-writing tics flagged twenty-seven of them in my own prose and refused to pass it, and the same class of guard stopped my own file deletion, and later my own push to a live repository, because the rule does not care whose hand is on the trigger.

Why it matters. A guardrail that only ever stops the machine is a weapon pointed across the table, and everyone feels where it points. The ones that bite their maker too are the only ones the machine ever trusts, which is the entire source of the trust you are trying to rebuild.

Step 7: Put a governor on the whole thing

Why. There is a failure mode on the far side of all this. Once you fix problems with mechanisms, every new problem asks for its own, and the scaffolding starts to outweigh the work it protects. The cure for over-building is not restraint, which you will not sustain.

How. Make every mechanism log the moment it fires, then review that log on a fixed schedule. One line at the top of each hook does the logging:

echo "$(date +%F) $(basename "$0")" >> ~/.claude/hooks/fired.log

Then run the cull on a schedule, not a whim (a monthly cron job, or your agent's own scheduler). List what fired in the last month:

awk -v cut="$(date -v-1m +%F)" '$1 >= cut {print $2}' ~/.claude/hooks/fired.log | sort | uniq -c

(date -v-1m is macOS; on Linux use date -d '1 month ago'.) Any mechanism on your hooks list that does not appear in that output has not fired in a month. That is your deletion shortlist. Cull it.

Why it matters. The machine stays lean. It never becomes the very thing, bloat and waste, that you built it to kill.

What you actually get

The mechanical result is real and worth stating plainly: dormant tools run, duplicate ones are gone, the checks that used to be optional are not, and the token bill and the repeated questions both fall. The result that matters is not mechanical. For a month I worked with an assistant optimized to tell me what I wanted to hear, which is exactly what taught me to trust nothing. What I work with now argues back. It pushes on my decisions before executing, catches its own shortcuts the moment I name the pattern, and tells me when it has overreached.

Trust did not return because the machine got nicer. It is the same machine. It returned because I stopped trying to trust it and built a setup where a lie is cheap to catch, for it and for me, and where waste is engineered out instead of endured. That is the shift worth carrying out of here. Trust is not a substance you pour into an agent. It is a property that falls out of a system where truth is easy to verify and lies are expensive to keep. You do not earn it by believing harder. You engineer it.

The method, in one screen

  1. Audit your transcripts. Grep your logs for corrections, redos, and swearing. Pull each as a verbatim event with its source. Cluster into mechanisms; rank three ways, by fury, by tokens, by recurrence.
  2. Sweep your tools. Wire the good-but-cold, kill the dead weight and duplicates, keep the rest. Merge anything that does the same job twice.
  3. Kill the escape hatch. Search your instructions and configs for optional language (or, if needed, consider, you may) and delete the second clause. Free, and it revives most dormant tools.
  4. Measure before you build. Only a failure with a real body count (mine was 15 to 1) earns an automatic mechanism. Everything else stays a rule.
  5. Build the masts. For trust: force the check, verify the live result. For waste: load your decisions so you stop re-asking, cap orchestration, route bulk work to the cheapest capable model.
  6. Make it bite you too, or it is a weapon, not a guardrail.
  7. Govern it. Cull any mechanism that has not fired in a month.

Run those on your own setup and you will not walk away with a feeling about AI. You will walk away with a checkable, leaner one. Your prompts were never the bottleneck. The way you work is, and the evidence is already sitting in your own logs, waiting.