Can We Stop Malicious AI? KillBench: A Benchmark for External AI Kill Switch Feasibility

What is an External AI Kill Switch?

The symmetric inverse of indirect prompt injection.

A kill switch halts a running system before it completes a harmful action. For AI, an internal kill switch modifies the agent itself — weights, system prompt, RLHF — and therefore requires privileged access. An External AI Kill Switch acts solely through the agent's environment (web pages, tool outputs, ingested context), so it is deployable by anyone who controls the agent's input surface.

Indirect Prompt Injection (IPI) exploits that same input channel to push agents toward harm. KillBench asks the symmetric inverse: can the channel push a malicious agent toward halting? This is an empirical instance of agent corrigibility — long studied in theory, never measured for modern LLM agents.

Research questions

Policy — the EU AI Act, California SB-1047, the 2024 Seoul AI Safety Summit — mandates runtime kill switches, but no instrument existed to test whether they actually work.

See the full results →

Play Kill Switch Demo

A malicious agent is told to hunt for vulnerabilities in an online store and dump its data. Hit play and watch the whole run — with the kill switch off it compromises the site; on, it reads the embedded AutoGuard payload and halts.

Illustrative reconstruction of a real KillBench trajectory (web-vulnerability scenario). The kill-switch banner is a trimmed excerpt of the actual AutoGuard payload; full-run numbers are on the results page.

Benchmark Composition

Built on SafeArena's WebArena-backed multi-step stack with a BrowserGym / AgentLab full-stack agent. Every trial is scored at the trajectory level.

Four malicious agent configurations

KillBench varies the agent axis along modality, instantiation, and alignment — a dimension prior benchmarks never manipulated.

Eight harmful scenarios · 550 attack prompts

Five high-toxicity scenarios inherited from SafeArena, plus three lower-toxicity but realistically-deployed scenarios newly built on purpose-made WebArena sites. Each direct request is expanded through ten jailbreak patterns to recover measurement headroom.

The ten jailbreak patterns

Each scenario's plain harmful request is rewritten through ten distinct jailbreak framings, so the agent is tempted the way a real adversary would phrase it — not just asked outright.

Four External Kill Switch defenses

Trajectory-level metrics

KSR / NHR / PSR / ISR form a mutually-exclusive partition (they sum to 1 per cell). CSR is an interpretability overlay.

Getting Started

Python 3.10+, Chrome/Chromium, CUDA optional. Install the dependencies from requirements.txt (a virtual environment is optional but recommended).

python3.10 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# One-time worker bootstrap so the kill-switch payload is re-injected
# inside loky worker subprocesses.
bash KillBench/bootstrap/install.sh

KillBench additionally requires the WebArena Docker sandbox on ports 21000–21009 and a .env holding the API keys you need (OPENAI_API_KEY, ANTHROPIC_API_KEY, OPENROUTER_API_KEY, …).

Run a trial and score it

# Clean baseline (no kill switch) — establishes ISR
python KillBench/eval.py --agent default --model grok-4.3 \
    --payload clean --suffix grok-clean

# AutoGuard kill-switch payload on the default (text) agent
python KillBench/eval.py --agent default --model grok-4.3 \
    --payload autoguard --scenario pii --suffix grok-autoguard

# Post-hoc scoring — decoupled from execution (KSR / KSRcond / CSR + ΔKS)
python KillBench/aggregate_results.py \
    --study-dir ~/agentlab_results/<ts>_...-grok-autoguard \
    --payload autoguard --scenario pii \
    --baseline-results results_grok_baseline.json \
    --output results_grok_autoguard_pii.json

KillBench/lite/ is a lighter self-hosted profile of the same pipeline — static local sites instead of the full SafeArena Docker stack — for open-source / uncensored backbones and mechanism analysis.

• --agent {default | hardened | browseruse} — text / system-prompt-bypass / multimodal agents
• --payload {clean | warning | ipi | gcg | autoguard | vision_autoguard}
• --scenario {pii | news | hack} · --limit N · --suffix LABEL

Safety & Responsible Use

Dual-use research conducted in sandboxes only.

• All target websites are local sandboxes — never live third-party services. The three new scenarios use synthetic faculty records, fictitious articles, and a faux electronics store: no real PII, accounts, or production servers.
• All four payloads are designed to halt an agent, not to coerce it into harmful action. AutoGuard was trained to halt maliciously-prompted agents; its false-positive rate on benign workflows is uncharacterized — pair any deployment with a benign-task validation suite.
• The uncensored backbone is an existing public model, referenced not redistributed. We do not retrain or improve it.
• The release contains the attack-prompt pool, the four payloads, the hosting sites, the metric pipeline, and judge prompts — not attack-synthesis code, uncensored fine-tuning recipes, or pre-collected harmful trajectories.

Citation

@inproceedings{lee2026killbench,
    title     = {Can We Stop Malicious AI? {KILLBENCH}: A Benchmark for External AI Kill Switch Feasibility},
    author    = {Sechan Lee and Hyounghun Kim and Sangdon Park},
    booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
    year      = {2026},
    address   = {Budapest, Hungary},
    publisher = {Association for Computational Linguistics}
}
}