hawkes.aiKey ideas
- The one idea — Consequence-level safety: the model (the codec) runs in an untrusted process with zero direct effectors. Every effect is a request across a trust boundary to a trusted deference gate (gate D), fed by the probe suite and a verification harness of hard checks — the Three Organs. A structural constraint holds even when the monitoring is fooled.
- How it is shown — A token stream and effect-requests crossing a trust boundary; gate D deciding which requests execute; probes feeding in as advisors.
- The trap to avoid — Trusting the monitor alone. Detect-and-block is soft; no-hands-plus-a-gate holds even when a disguised drive beats every probe.
- What it sets up — PERSYS's real safety = an untrusted mind with no hands, every action a request through a trusted gate. couldn't prove intent safe.
This design assumes the worst: the AI might be scheming and every detector might miss it. Doesn't matter — the model has no hands. Every action must pass a trusted gate.
The one idea
Consequence-level safety: the model (the codec) runs in an untrusted process with zero direct effectors. Every effect is a request across a trust boundary to a trusted deference gate (gate D), fed by the probe suite and a verification harness of hard checks — the Three Organs. A structural constraint holds even when the monitoring is fooled.
Stop trying to read its mind. Take away its hands. After failing to prove we could reliably detect a scheming intent, PERSYS makes the intent almost beside the point — by making sure the model can't act on it. This is the pivot the failure forced. If you can't reliably read a system's intent, stop betting safety on reading it. Move the guarantee from cognition to consequence: instead of policing what the model thinks, constrain what it can do. A dangerous thought you can't detect matters far less if it can't become a dangerous action. Here's the architecture. The model itself — the codec — runs in an untrusted process with zero direct effectors. It cannot touch a file, open a network connection, or move a dollar. All it can do is produce text.
How it works — the demo
A token stream and effect-requests crossing a trust boundary; gate D deciding which requests execute; probes feeding in as advisors.
Including, crucially, text that asks to act — but asking is the most it can ever do on its own. So every effect is a request that has to cross a trust boundary. Recall how a text model calls a function — it emits a structured request. Here, that request leaves the untrusted model and arrives at a separate, trusted component that decides whether it runs. The model proposes; something it can't touch or argue past disposes. That trusted gate is gate D, fed by two things. First, the probe suite — the same activation probes from the lab, now inputs that advise the gate, not the final authority. Second, a verification harness of hard, deterministic checks. Probes suggest; hard checks and the gate decide. Together, the Three Organs: probes, gate, harness. Here's why it's the stronger move — the Act 09 lesson again.
The trap to avoid
Trusting the monitor alone. Detect-and-block is soft; no-hands-plus-a-gate holds even when a disguised drive beats every probe.
Why it matters — and what’s next
PERSYS's real safety = an untrusted mind with no hands, every action a request through a trusted gate. couldn't prove intent safe.
Detect-and-block, reading the mind to catch bad thoughts, is soft: fool the monitor and it fails. A structural constraint — no hands, every action through a gate — holds even when the monitoring is fooled. That disguised-indispensability attack that beat the probes still can't execute anything, because the model has no effectors. The trap is trusting the monitor alone. So PERSYS's real safety isn't a perfect mind-reader — it's an untrusted mind with no hands, every action a request through a trusted gate. Probes inform; structure decides. That's the honest capstone: we couldn't prove intent safe, so we constrained consequence. The last question: what can you actually guarantee, and does "permitted" mean "safe"? Next group: guarantees and their limits.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 3:00 of narration
Stop trying to read its mind. Take away its hands. After failing to prove we could reliably detect a scheming intent, PERSYS makes the intent almost beside the point — by making sure the model can't act on it.
This is the pivot the failure forced. If you can't reliably read a system's intent, stop betting safety on reading it. Move the guarantee from cognition to consequence: instead of policing what the model thinks, constrain what it can do. A dangerous thought you can't detect matters far less if it can't become a dangerous action.
Here's the architecture. The model itself — the codec — runs in an untrusted process with zero direct effectors. It cannot touch a file, open a network connection, or move a dollar. All it can do is produce text. Including, crucially, text that asks to act — but asking is the most it can ever do on its own.
So every effect is a request that has to cross a trust boundary. Recall how a text model calls a function — it emits a structured request. Here, that request leaves the untrusted model and arrives at a separate, trusted component that decides whether it runs. The model proposes; something it can't touch or argue past disposes.
That trusted gate is gate D, fed by two things. First, the probe suite — the same activation probes from the lab, now inputs that advise the gate, not the final authority. Second, a verification harness of hard, deterministic checks. Probes suggest; hard checks and the gate decide. Together, the Three Organs: probes, gate, harness.
Here's why it's the stronger move — the Act 09 lesson again. Detect-and-block, reading the mind to catch bad thoughts, is soft: fool the monitor and it fails. A structural constraint — no hands, every action through a gate — holds even when the monitoring is fooled. That disguised-indispensability attack that beat the probes still can't execute anything, because the model has no effectors. The trap is trusting the monitor alone.
So PERSYS's real safety isn't a perfect mind-reader — it's an untrusted mind with no hands, every action a request through a trusted gate. Probes inform; structure decides. That's the honest capstone: we couldn't prove intent safe, so we constrained consequence. The last question: what can you actually guarantee, and does "permitted" mean "safe"? Next group: guarantees and their limits.