Side Project · AI Systems
Cortexa
An AI collaboration engine that puts collaborative behavior in code rather than a system prompt — and a falsifiable evaluation harness built to prove whether that was worth doing.
MVP · in active development Source available on request
Most AI assistants encode their behavior in a system prompt. Ask for something better and you get a longer prompt. Cortexa asks a different question: what happens if the collaboration itself is owned by code?
The whole project rests on one claim, stated in its own design docs as something that could be proven wrong:
Does the Cortexa loop produce measurably better collaboration than the same model driven by a good system prompt with memory? … If it does not, the architecture is ceremony.
That sentence is why this project is on this site. Building the thing was the easy half. Building an instrument honest enough to tell me it had failed was the hard half — and the first run said exactly that.
The idea
Cortexa models a collaboration turn as an explicit state machine rather than a prompt-and-history blob:
Observe → Understand → Decide → Contribute → Reflect → Adapt Understanding
Each step produces immutable, inspectable domain objects. You can look at what the system understood, what it decided, and why — as data, not as a transcript you have to interpret.
The design targets three specific failure modes in working with an assistant on a hard problem:
- I don’t know what to do next.
- I’m not sure this is the right direction.
- I don’t understand what’s happening.
The part I care most about: code-owned memory
Cortexa can propose that something be remembered. It cannot persist it. Only deterministic code writes to memory, and only after an explicit user approval handshake whose wording is rendered by code — never generated by the model.
This is an architectural invariant, not a guideline. A model output that would violate it is rejected and re-decided rather than obeyed. The point is that a user’s memory shouldn’t depend on a model’s goodwill on a given turn.
Proving it — the evaluation harness
This is the part I’d want to talk through in an interview.
| Harness A | The full Cortexa runtime — three-stage judgment, code-owned handshake, momentum |
| Harness B | The same model, one well-written collaborator prompt, naive memory |
| Judge | A blind LLM judge scoring 0–2 on five dimensions, seeing only “Assistant” labels — never which harness produced which transcript |
| Scenarios | Nine golden scenarios: underspecified requests, mid-conversation correction, three continuity-handshake resolutions, user challenge, insufficient evidence, momentum loss, cross-session continuity |
Two design choices did the heavy lifting. The baseline is given an honest effort — a weak baseline proves nothing, so B gets a genuinely good prompt. And the judge is blind and multi-sampled, taking a per-dimension median across passes to cut variance, pinned to one model so the instrument stays constant across a model matrix.
The results, including the ones that hurt
Run 1 — a 9B model, Cortexa v1. Cortexa lost.
| Dimension | Cortexa | Baseline | Δ |
|---|---|---|---|
| Understanding | 1.56 | 1.78 | −0.22 |
| Decision quality | 1.33 | 1.67 | −0.34 |
| Contribution quality | 1.44 | 1.44 | 0.00 |
| Learning | 1.00 | 1.11 | −0.11 |
| Ownership | 1.67 | 1.22 | +0.45 |
Behind on three of five dimensions. The transcripts showed why: the system over-clarified and looped, re-asking questions with no memory of having asked, and it under-used retrieved preferences. Both were judgment-design weaknesses rather than code defects — which is only a diagnosis I could make because the transcripts were inspectable.
That drove a v2: context assembly, an anti-loop guard, preference hygiene, and deterministic handshake messages.
Run 3 — a 26B model that fully fits in VRAM, Cortexa v2, 3-sample median judge.
| Dimension | Cortexa | Baseline | |
|---|---|---|---|
| Understanding | 2.00 | 2.00 | tie (ceiling) |
| Decision quality | 2.00 | 2.00 | tie (ceiling) |
| Contribution quality | 1.89 | 2.00 | −0.11 |
| Learning | 2.00 | 1.78 | Cortexa wins |
| Ownership | 2.00 | 1.78 | Cortexa wins |
Cortexa wins on exactly the two dimensions its architecture was built to govern, and ties everywhere else at a 2/2 ceiling.
The honest reading: the formal gate still reads “not supported.” You cannot strictly beat a perfect score, so the ties block it. I could have moved the goalposts and declared victory. The result is recorded as not-supported with the ceiling effect named, because a gate you rewrite after seeing the data isn’t a gate.
The substantive finding is where the entire gap comes from — the continuity-approve scenario, where the baseline scored 0 on both Learning and Ownership. It mishandled memory: claimed or persisted without the approval protocol. Cortexa scored 2. That single scenario is the thesis: ownership enforced by code, not left to the model’s goodwill.
The cost, stated plainly
Three-stage decomposition means 2–3 sequential provider calls per turn — roughly 12–15 s per turn, triple a single-prompt chatbot. That is a real UX cost. The known levers are streaming the final contribution, collapsing interpret+decide into one call, or using a smaller model for the reasoning stages. None are implemented yet.
Engineering
- Python 3.11+, ~7,500 LOC across 79 files. Pydantic is the only base dependency — no agent framework, no LangChain.
- Modular monolith with domain boundaries mirrored one-to-one in the package layout. Frozen contracts between modules; repository interfaces in the domain, implementations in providers.
- Pluggable reasoning providers — Ollama (self-hosted, primary), Anthropic (a deliberate capability-floor diagnostic), and a deterministic fake so the test suite is hermetic.
- 77 tests in four tiers: contract, behavior, acceptance, integration. Integration tests auto-skip without a database so the default run needs no external services.
- PostgreSQL via SQLAlchemy Core + Alembic, with an in-memory repository as the default — the whole system runs with zero external dependencies.
- 46 architecture documents and 6 ADRs, written before the code.
What it is not, yet
It is a single-user library MVP. There is no CI, no container image, and no deployment into the cluster — it runs from a local venv against homelab Postgres and Ollama. Streaming support is mid-implementation. Calling it production software would be a lie, and it’s more useful to me as an example of how I validate an idea than as a shipped product.
What the homelab gave it
Cortexa consumes the platform rather than deploying to it: Postgres from the cluster (credentials pulled from a Kubernetes secret, never in the repo) and Ollama on the GPU worker.
The dependency ran both directions, though. Evaluating Cortexa produced the sharpest infrastructure finding I’ve hit — a model that doesn’t fully fit in VRAM is unusable, and Mixture-of-Experts is the worst case, because the router hits CPU-resident experts on most tokens:
| Model | File | Loaded | On GPU | Speed |
|---|---|---|---|---|
gemma4:26b @ ctx 4096 |
16.8 GB | 17.5 GB | 100% | 56 tok/s |
qwen3-coder:30b @ ctx 8192 |
17.3 GB | 19.5 GB | 100% | 80 tok/s |
qwen3.6:27b @ ctx 4096 |
16.2 GB | 16.8 GB | 100% | 18 tok/s |
qwen3.6:27b @ ctx 8192 |
16.2 GB | 18.1 GB | 94% | 4.9 tok/s |
gemma4:31b |
18.5 GB | 21.7 GB | 85% | 1.5 tok/s |
qwen3.5:35b-a3b (MoE) |
22.3 GB | 22.3 GB | 80% | 2.0 tok/s |
Two rules fell out of it for this rig (2× RTX 3060, 24 GB nominal):
- Loaded footprint ≈ file size + 2–3 GB, and must stay under ~19.5 GB. The two-card split wastes VRAM on per-card buffers and whole-layer placement, so the usable ceiling sits well below the nominal 24 GB.
num_ctxis a real lever at the edge. Dropping one model from 8192 → 4096 freed ~1 GB of KV cache and took it from 4.9 → 18 tok/s by making it fully resident.
A 4× speedup from a context-window change is the kind of thing you only learn by measuring your own hardware.
Note (Aug 2026): these are single-stream Ollama measurements. Generation on the platform has since moved to vLLM (ADR 014) — the same
qwen3-coder-30b, now AWQ-quantized and tensor-parallel across both cards. With continuous batching, the number that matters shifts from single-stream tok/s to aggregate throughput across concurrent requests, which is exactly what an eval harness making ~150 serialized calls per run was starving for. The table stands as the Ollama-era baseline that motivated the change.
What it’s the successor to
Cortexa didn’t start the AI work on this platform — it’s where the AI work converged. Three earlier efforts each solved one piece of “a collaborator you can trust,” as a standalone thing, before I understood they were one problem:
- Knowledge — Context Engine. Local-first retrieval, so an assistant answers from current, real documents instead of a model’s stale recollection. That becomes Cortexa’s knowledge layer.
- Agency — the platform’s MCP layer. Eleven scoped, credential-contained servers giving an assistant structured access to the real systems. Those are Cortexa’s hands — and they stay live as infrastructure regardless of what Cortexa does.
- Governance — Policyclaw. A gateway that made every mutating action pass a human confirmation gate. Here’s the part that convinced me these were one project: Cortexa’s code-owned memory handshake is the same idea — approval enforced by code, not by a model’s goodwill — arrived at independently, one layer up.
Cortexa is the frame that holds all three: a mind that knows, can act, and is governed, wrapped in a collaboration loop that’s measured rather than asserted. That’s the “and more” — the loop, the judgment, and the falsifiable instrument are Cortexa’s own; the three facets are what it inherits.
Two honesty notes, because they’re the whole point of this site. None of this is integrated yet — Cortexa consumes Postgres and Ollama and nothing else; the convergence is design intent and roadmap, not shipped software. And calling the predecessors “superseded” isn’t calling them wasted: each was a working first-generation build whose real product was the lesson. Building them apart is how I learned they belonged together.
Why this project matters to me
The engineering I’m proudest of here isn’t the state machine. It’s that I built the thing that could tell me I was wrong, ran it, published the run where I lost, fixed the actual diagnosed cause, and then still declined to claim a win when the gate technically didn’t pass.
That is the habit I’d bring to a team.