# Portfolio AI Ops **Deploy AI across portfolio companies and score every claim on a Brier-scored forecast ledger.** Buyer: PE operating partners. Wedge: proof, not pilots. The premise is narrow. Portfolio AI spend is real and growing, and almost none of it is measured in a way that survives an LP question. This makes every AI initiative a dated, probabilistic forecast, resolves it YES or NO on the date, and scores the forecaster. What you end up with is a calibrated track record instead of a slide of vendor logos. --- ## Contents ``` portfolio-ai-ops/ ├── app/ │ ├── index.html the console: ledger, calibration, portfolio, IC memo │ ├── engine.js scoring math (Brier, Murphy decomposition, calibration, economics) │ ├── app.js UI, charts, state, persistence, fleet client │ ├── data.js seeded demo ledger: 6 companies, 36 forecasts │ ├── styles.css │ ├── verify.js engine unit tests, run with node │ └── TESTS.md recorded verification output ├── assets/ │ └── calibration.png screenshot used by the landing page ├── landing.html sales page ├── icon.svg 512x512 hand-authored ├── icon-256.png raster export ├── GTM.md ICP, triggers, 5-email sequence, 3 LinkedIn posts, objections, prospect list ├── PRICING.md tiers, replacement costs, margin math, capacity ceiling └── README.md ``` --- ## Running it No build step, no dependencies, no package.json. It is plain HTML, CSS and ES2020. ```bash cd portfolio-ai-ops python3 -m http.server 8777 ``` - App: - Landing: Opening `app/index.html` directly via `file://` also works for everything except the optional fleet call, which browsers block cross-origin from a `file://` page. ### Running the tests ```bash node app/verify.js ``` Checks the scoring engine against values computed independently in Python. Exits non-zero on failure. Current state: all checks pass, including that the Murphy decomposition identity reconstructs the measured Brier score to within 1e-12. --- ## What the app actually does **Ledger.** All 36 forecasts, filterable by company, status and category, sortable on every column. Resolve any open forecast YES or NO and every downstream number — portfolio Brier, calibration curve, cohort trend, ROI, IC memo — recomputes immediately. Add forecasts through a modal with validation. Edit by clicking a row. Export and import the whole ledger as JSON. State persists to localStorage; Reset restores the seed. **Calibration.** Brier score, log score, Brier skill score against a climatology baseline, and overconfidence bias. A reliability diagram drawn as SVG, with point size proportional to bin count. Brier by cohort, grouped by the half-year each forecast was opened. Full Murphy decomposition with the `BS = REL − RES + UNC` identity checked numerically on every render and the residual displayed. Score broken out by category and by company. **Portfolio.** Six companies with sector, EBITDA, headcount, site count and investment thesis. Per company: forecast count, Brier, hit rate, ROI, deployed spend, realised value, open value at stake. **IC Memo.** An eight-section memorandum generated entirely from the ledger — position, forecast quality, cohort trend, category and company breakdowns, the three worst forecasts listed by name, open positions, and a recommendation that flips based on whether skill score is positive. Every figure is computed. None is hardcoded. ### The math `app/engine.js` is pure functions with no DOM dependency. - **Brier score** — `BS = (1/N) Σ (fᵢ − oᵢ)²` - **Murphy decomposition** — `BS = Reliability − Resolution + Uncertainty`, computed exactly by grouping on unique forecast values rather than on bins, so the identity holds to floating-point precision rather than approximately - **Brier skill score** — versus a climatology reference of always predicting the base rate - **Log score** — mean negative log likelihood, clipped at 1e-6 - **Wilson score interval** — for honest confidence bounds on the hit rate at n=30 - **Economics** — realised value counts only forecasts that resolved YES; spend counts on every attempt, hit or miss. That asymmetry is deliberate --- ## Fleet integration The IC Memo tab has an optional **Draft narrative on fleet** action that sends the computed aggregate figures to a local GPU node and asks for a spoken narrative. The header shows live node reachability. Configured in `app.js`: | Node | Endpoint | Model | |---|---|---| | blackwell | `100.67.126.117:8000/v1` | Qwen3-Coder-30B (vLLM) | | rig-96gb | `100.102.142.84:11434/v1` | qwen3-coder:30b | | rig-36gb | `100.89.143.27:11434/v1` | qwen3-coder:30b | | rig-28gb | `100.76.209.22:11434/v1` | qwen3-coder:30b | | rig-256gb | `100.91.39.12:11434/v1` | laguna-262k | Verified working: all five nodes returned reachable from inside the browser (`fleet: 5/5 up`), and a live generation completed on blackwell in 2.3s. The prompt includes explicit interpretation guardrails, because smaller models reliably misread a Brier score as a percentage error on the first attempt. If the fleet is unreachable the app degrades cleanly — the panel says why, names the likely CORS cause, and the deterministic memo is unaffected because it never touches the network. --- ## What is real vs what is stubbed Being precise about this, because the product is about not overclaiming. ### Real and verified - **All scoring math.** Independently cross-checked against a Python reference implementation. The Murphy identity reconstructs to a residual of ~8e-17. - **All app interactivity.** 13 browser interaction tests pass against real Chrome: filtering, search, sorting, adding forecasts, resolving them, localStorage persistence, chart rendering, memo generation. Console is clean. - **Both SVG charts.** Drawn from live data, no chart library. - **The IC memo.** Computed from ledger state at render time. - **Import / export.** Round-trips real JSON. - **The live fleet call.** Executed end to end from the browser against blackwell. - **Every number in PRICING.md.** 21 arithmetic claims verified programmatically. One error was found this way (a wrong tier-crossover figure) and corrected. - **The icon and landing page.** Hand-authored, rendered and screenshotted. ### Demo data, not client data - **The ledger is fictional.** Cardinal Ridge Partners and its six portfolio companies do not exist. The structure, metric choices and thesis language are modelled on real LMM PE services roll-ups, and the numbers are internally consistent, but no real fund is represented. - The **7.3x** and **Brier 0.206** figures quoted on the landing page are from this demo ledger. The landing page footer says so. Replace them with real client figures before this page is used in front of a buyer, or cut them. - The cohort improvement (0.323 → 0.196 → 0.131) is constructed to model a forecaster who starts overconfident and learns. It is a realistic shape, not an observed result. ### Stubbed or not built - **No backend.** Single user, browser-local. Multi-user, auth and an audit trail are the first things a real fund will ask for, and none of it exists. - **No data connectors.** Forecasts resolve when a human clicks YES or NO. There is no integration with NetSuite, Sage Intacct, or any company system. In delivery this is manual anyway for the first two quarters, but it is a real gap. - **No hosting.** "LP-shareable hosted ledger" in the pricing is a promise, not a deployed feature. - **`mailto:` CTA.** The landing page has no form, no CRM, no tracking. - **Fleet call needs CORS.** Works from a served origin where the node allows it. Ollama needs `OLLAMA_ORIGINS="*"` for a browser to reach it directly. - **No Apollo or LinkedIn integration.** GTM.md names them as the list-build path. Nothing is wired up and nothing was sent. ### Honest assessment The app is a real, working, verifiable product and the maths is genuinely correct — that part would survive a technical buyer poking at it. What it is not yet is multi-tenant software. It is the demo artifact and the delivery instrument for a services engagement, which is the right shape for a solo operator selling a $45K + $12K/mo engagement, but it should not be described as a platform. The one thing to fix before selling: **the numbers on the landing page are from fictional data.** Either land the first client and replace them, or mark them as illustrative on the page itself rather than only in the footer. Selling a measurement product on unmeasured claims is the exact failure the product exists to attack.