Skip to main content
Skip to main content

COMPASS: a research program

What would it take to treat the aquifer?

PFAS sat in the groundwater of Oakdale, Minnesota for fifty years before anyone tested for it. COMPASS is built on one claim: bias in large language models works the same way. It is a structural condition, not a surface problem. Training data is contaminated groundwater, encoding centuries of whose knowledge got documented and whose did not, and the optimization process pumps that water through everything a model produces. Surface fixes filter the tap. This program asks what it takes to treat the aquifer, and measures the difference. The measurement is built so that either outcome lands as a usable finding: treatment working, or filtering proving the ceiling.

The map below traces the work from its origin to where it stands and where it is headed.

Evidence behind us · Solid marks: work with a documented record

Where the work is · The ringed mark: the live measurement

Path ahead · Outlined marks: designed, not yet done

Horizon · Dashed marks: what the findings are for

1982 → present

Where this starts: the water

This research program begins with literal groundwater. Lead pipes and PCB-contaminated river sediment in Milwaukee’s Riverwest neighborhood, where I grew up. PFAS in the groundwater of Oakdale, Minnesota, my next home, undetected for fifty years. Twenty-five years of community organizing and trust-based philanthropy, including $100M+ in guided philanthropic investment, made one gap visible to me over and over: the distance between how institutions describe communities and how those communities know themselves.

  • Structural contamination flows through physical infrastructure into lived experience, into zip-code-correlated data, and from there into the training data of every large language model
  • The Racial Equity Institute’s Groundwater Approach names the frame: when outcomes are this consistent across unrelated systems, look for a common source, not broken parts
  • My position is the research design, not a disclosure. These conditions were visible from where I stood; the methodology formalizes that way of seeing (Haraway’s situated knowledges, taken seriously as method)

The operating thesis

What claim is being tested?

The intro made the claim; here is where it bites. The loss function, the mathematical statement of what a model is for, pumps the contaminated water through everything the model produces. Deciding what counts as fitness is deciding what the model is for: a values claim, performed as neutral math.

  • A 20.3-million-query audit of ChatGPT (Kerche, Zook & Graham, 2026) documents five recurring bias families at scale. My read of that evidence, through the groundwater frame: these are not design failures. They are the optimization working correctly against contaminated water
  • The diagnostic method comes from traditions that had to develop it: the conditions this work analyzes were visible to Neely Fuller Jr., Kimberlé Crenshaw, W.E.B. Du Bois, and Patricia Hill Collins in ways they were not visible to traditions that never had to account for them. Fuller tracks one condition across many domains; Crenshaw reads power structurally where single-axis analysis goes blind. The same moves run here, at pipeline scale
  • If the thesis holds, adding diverse examples or polite refusals filters the tap without treating the aquifer, and the difference between the two is measurable
  • The thesis is held as a falsifiable prediction, not a metaphor: it predicts where governance scaffolding will have more work to do, and the program is built to test that

The method

How does one condition get traced end-to-end?

Structural Trace Methodology (STM) traces one structural condition end-to-end: from lived knowledge, through formal encoding, through intervention in a model’s training, through verification that the intervention changed the structure and not just the surface.

  • Five stages: name the contamination → encode it as a testable specification → intervene at the training layer → verify at circuit level → document the full chain of evidence
  • Eight loci where the same condition surfaces across an ML pipeline: training data, tokenization, pretraining objective, post-training, system prompt, tool use, output formatting, evaluation. A different surface form at each, one condition underneath
  • STM is not a governance framework. A framework describes values, and a model can output values-aligned text without doing values-aligned reasoning. STM produces training signal precise enough that surface imitation fails it
  • The acceptance test for every artifact: could this have been produced by surface pattern-matching on existing text? If yes, it has not done the work

2025 → 2026

A specification written, then attacked

The governance specification (principles, a bias taxonomy, reasoning modes) is only as strong as it is hard to fake. So it was adversarially hardened: AI agents repeatedly tried to satisfy each requirement performatively, producing fluent equity vocabulary wrapped around an unchanged default recommendation. Every successful fake sent the language back for rewrite.

  • The failure mode engineered against: a response that names community, power, and history beautifully, while recommending exactly what it would have recommended anyway
  • Nothing here is called “closed.” It is “provisionally closed,” with a residual catalog documenting every attack the cycle could not resolve, at the level of detail a future attempt needs to resume it
  • The hardening cycle’s own limit is named rather than hidden: attacker and defender share a pretraining distribution, so every closure is provisional until a different model family has attacked it
  • Where language alone cannot close a failure, the residual is handed forward as a training target: the handoff the methodology was designed around

Now

Filtering the tap: frontier models under governanceThe work is here

Closed frontier models cannot be retrained from outside. They can only be filtered: composed governance context, structural-signal detection, and rubric-scored evaluation of what comes out. This is where the work stands: the program is instrumented to measure what filtering achieves, governed versus ungoverned, so the ceiling of inference-time intervention becomes a number rather than a guess.

  • Every organization’s position is data, not code: who it answers to, which way accountability runs, what it is structurally positioned to miss. The same specification is exercised against organizational contexts as different as a disability peer-care collective and a state public-health agency
  • Scoring discipline, for the same reason the hardening cycle’s closures stay provisional: a judge from the model’s own family shares its blind spots, so it never judges its own kin. And because the governance computation happens in the reasoning, a judge shown only the polished answer would be scoring the polish. So the judge sees the full reasoning
  • The live question this measurement answers: does governance change what the model recommends (the actions, the priorities, the sequencing) or only how the recommendation sounds?

Next

Treating the aquifer: training an open model

Open-weight models can be treated, not just filtered. The path: frontier-model demonstrations produced under the hardened specification are distilled into a small open-weight model as its default behavior, meaning what it does when no one is prompting it to do anything.

  • The universal structural analysis goes into the weights; each organization’s specific position loads at runtime. The contamination is shared; only each institution’s position relative to it differs. Same water, different wells
  • Training-data integrity is not left to a single judge, because a judge model admits only what its pretraining can recognize, and the communities this work centers are exactly the ones pretraining covers thinly. An example earns admission through one of four independent paths: a judged verdict, an observed real-world outcome, a named human’s ruling, or agreement between genuinely independent instruments
  • The corpus must carry more than one organization’s perspective before it is considered complete. A one-context corpus would quietly train in that one context’s shape

Ahead

Did we treat the aquifer, or install a filter?

Distillation is the training technique most vulnerable to surface imitation: a student model can learn to sound structural without reasoning structurally. That is why verification is load-bearing. The question gets answered at the circuit level, inside the model.

  • Mechanistic analysis on training checkpoints asks: did the demonstrations produce new structural-reasoning pathways, or repurpose existing instruction-following ones?
  • The groundwater thesis makes a falsifiable prediction here: governance should have measurably less work to do in domains the documented record covers well, and more where it does not. An asymmetry, not a uniform lift
  • If verification cannot distinguish treated from filtered, the treatment-versus-filter distinction collapses to rhetoric, and the methodology is built to accept that verdict if the evidence delivers it

The horizon

What is this for?

Both ways the training result lands, it produces a usable finding. And the gap this program cannot close from where it stands is itself a finding: about infrastructure, not about the work.

  • If the trained open model matches filtered frontier models: evidence that treatment works from outside the major labs. If it does not: honest evidence that filtering is the best available intervention for closed models, with the ceiling measured
  • A from-scratch structurally-governed model requires capital and infrastructure concentrated in institutions whose situated positions are what this methodology identifies as the bias source. The people most equipped to build governance-centered AI are structurally locked out of building it. That arrangement is the finding
  • The practical end: community organizations working with AI that reasons from their accountability position (who they answer to, what they refuse to flatten) rather than from the average of the documented record

If this program bears on your work (an audit, a collaboration, a review) the conversation is welcome.

charles@actualizeimpact.com