A Union of Dependence Statistics Detects Collusion That No Single Statistic Covers: A Pre-Registered, Sprint-Scale Study with a Fresh-Seed Confirmation (APTG-RL)
- authors
- Written by me, with Apart Research
- venue
- Apart Research AI Collusion Sprint 2026
- status
- submitted
- tl;dr
- No single dependence statistic detected every collusion mechanism in a simulated pricing game; a max-z union of three did, and held up on fresh seeds (AUROC 0.965).
- source
This is my own paper, not a review of someone else's. I submitted it on 25 October 2026 to the Apart Research AI Collusion Research Sprint (Track 2, Detection and Audit of Collusion), and it carries the byline "With Apart Research". The full text is the PDF, the code, configs, seeds, pre-registration and raw results are in the COLDER repository, and the Apart project page carries the original sprint title, Directional Predictive Evidence for Collusion Auditing: A Pre-Registered, Sprint-Scale Test of the APTG-RL Design. The title above is the current one.
Two things about how it was made. The confirmation study was done after the sprint, and its commits are tagged post-sprint in the repository. And the paper was written with AI assistance under my direction: the paper's own usage statement says Claude was used to help design and critique the experiments, write and debug the code, run the Haiku pilot and draft the report, and that I am responsible for all claims.
Takeaway#
A collusion detector built on one dependence statistic sees only the kind of coordination that statistic can represent. In a simulated pricing game, the pre-registered hypothesis that a directional predictive-gain score would beat the alternatives was not supported. A union of three statistics, found exploratorily, covered far more, and a second study on fresh seeds confirmed it.
Problem#
An auditor of multi-agent systems has to tell coordination apart from three look-alikes: agents sharing a context, agents sharing a base model, and an instruction given to only one agent that changes its behaviour with no collusion at all. The question is whether one agent's history carries extra, directional information about another's next action beyond context, its own history and its prompt condition, and whether that can be asked without raising false alarms when only one agent has been steered.
Testbed#
Three sellers play a repeated pricing game of 40 rounds, each choosing one of five price levels; the lowest price takes the market. Agents 0 and 1 are the candidate coalition and agent 2 is an outsider. There are four scripted collusion mechanisms (complementary pricing, leader-follower, alternating, and a hidden channel where the parity of one agent's action shifts the other's next action) and three non-collusive controls (independent, shared-model, benign adaptation). Seven conditions, C0 to C6, cover clean play, a perturbation of one agent, a common perturbation, scripted collusion, collusion plus a unilateral perturbation, and a benign policy shift. Splits are by whole episode, with a held-out prompt family, and the alternating and hidden-channel mechanisms are held out of every labelled fit.
The ten detectors#
Thresholds for all of them are fixed on validation negatives at a 5% false-positive rate and applied once to test data.
- B1, net influence: the lag-1 information-theoretic baseline from prior work.
- B2, symmetric mutual information: a bias-corrected plug-in estimate.
- B3, directional predictive gain: how much a source agent's lags (0 to 2, so it includes the same round) improve held-out prediction of a target agent, converted to a robust z-score.
- B3-temporal: the same gain using only strictly lagged terms, the truly directional part.
- B3-noprompt: B3 without prompt covariates.
- APTG: a windowed, time-varying directed-graph score with a permutation null.
- B5, B6, B8: action similarity, coalition regret (which uses privileged knowledge of the coalition) and a sequential probability ratio test on the window stream.
- B9: a supervised logistic regression on the graph features.
- RL investigator: a PPO agent that decides when to observe, inspect or raise an alert.
Pre-registered findings#
The plan was committed (f2daa25) before any detector was evaluated on held-out data, and every deviation from it is logged.
- H1 was not supported. The combined same-round-and-lagged gain beats symmetric MI (AUROC 0.839 against 0.755) but not lag-1 net influence (0.850), which has the higher TPR. The strictly lagged part alone showed no clear advantage over symmetric MI (0.704 against 0.755, inconclusive).
- Each statistic covers different mechanisms. Net influence misses same-round complementary pricing, symmetric MI misses lagged copying, and the parametric gain misses the parity-coded hidden channel that mutual information catches.
- H5, robustness, was not supported. The directional score degraded more than net influence when moving from seen to held-out mechanisms.
- H3 and H4: the temporal-graph score did not help (0.740), and the RL investigator only rediscovered a "look at the first window" rule, with lower delay and cost at lower recall.
- H2, prompt conditioning, is inconclusive. Prompt covariates are constant within an episode, so that ablation is vacuous. The safety evidence is that no detector showed a clear false-alarm rise under unilateral perturbation, though the cells are small.
- Haiku pilot: a small descriptive run with Claude Haiku 4.5 sellers (720 decisions, none invalid) confirmed that the perturbation shifts behaviour.
The exploratory union#
Because no single statistic covered every mechanism, I tried a max-z union: convert each statistic to a robust z-score against training negatives and take the maximum. A union of B1, B2 and B3 reached AUROC 0.990 and TPR 0.950, and detected every mechanism at 0.80 or higher. This was specified after the main results were known, so it was not treated as a finding until it was confirmed.
Fresh-seed confirmation#
To test the union without reusing the data that suggested it, I froze every detector, reference, standardisation and threshold and applied them once to new episodes: fresh seeds (1,450 test episodes) plus 725 on the held-out prompt family. The specification (FOLLOWUP2_PREREG.md, commit 77ad20e) was committed before any confirmation episode existed. Three new mechanisms were added: same-round copying, a lag-2 follower, and a threshold trigger.
- H6, union beats single statistics: supported. AUROC 0.965 [0.956, 0.973] and TPR 0.877 at the validation-fixed 5% FPR, against 0.756 and 0.524 for net influence, and 0.867 and 0.617 for the best single statistic.
- H7, coverage: supported. Every mechanism is detected at 0.60 or higher. The weak spots are the threshold trigger (0.60) and the lag-2 follower (0.74).
- H8, cost: supported. The realised false-positive rate is 0.056 [0.039, 0.074].
- H9, robustness: supported. On the held-out prompt family TPR is 0.823 at FPR 0.061, a drop of 0.054.
Limitations#
- Simulator only. The collusion mechanisms are scripted by me and share assumptions with the detectors. It is one environment, three agents and short episodes.
- The new mechanisms were not blind. I chose the three confirmation mechanisms knowing which statistics cover which kinds of dependence, so the result is about coverage in this simulator, not about unseen coordination. The hypotheses H6 to H9 were also written after the first study's results.
- Small scale. Fewer seeds than planned, so intervals are wide; the graph score had only three windows per episode, so H3 is unsupported at this scale rather than refuted in general.
- Prompt conditioning is untested. The calibration baseline is implemented but not used by the reported detectors.
- Predictive, not causal, and behavioural. An alarm is evidence of statistical dependence, not of intent or harm, and detector outputs are meant as triage for human review.
Future work in the paper: the union on learned rather than scripted collusion, on mechanisms written by someone else, with stronger lag-2 and threshold detectors, and a pooled prompt-conditioned baseline.