You're an analyst at a large LP, an investor in many private equity funds. The CIO wants an answer before the 09:00 meeting. Finding it takes hours, and two analysts can send two different numbers. Portfolio Answer Desk gives you one number, with its sources, ready to send.
Today each question means hours in reports, letters and spreadsheets, and the answer depends on who searched. The vision: one number whoever answers, with the source and date of each figure, in minutes.
You own the same company through several funds, under several spellings. Search one name and you miss part of it, or a write-down sent only in a letter.
Each analyst calculates returns their own way. Nobody can say why the numbers changed.
Fee changes and partners leaving are spread across years of letters. Nobody reads them all.
A question desk for an LP's private equity team: a chat, a committee pack and a manager note, on one set of agreed definitions.
You ask in plain English. The AI picks the measure, the company and the date. Code calculates the answer and links each figure to its document.
A number you can send to the CIO before the meeting, the same one whoever asks.
| Your job | Hypothesis | Target, then result | Expected impact |
|---|---|---|---|
| When a company is in the news, I want every holding and its latest value, so I reply before 09:00. | If every spelling links to one company, code finds every holding, because it follows links instead of searching names. | Right in under 10 minutes: 20 of 20, about 50 seconds each | Reply before the meeting (unobserved) |
| When I build the committee pack, I want each number calculated like last quarter, so nothing needs redoing. | If each measure is written down once, with a version, the same question gets the same number. | Same answer when asked again: 100% | Fewer corrections to the pack (unobserved) |
| Before we back a manager again, I want every problem from their letters, so none is missed. | If problems are quoted from the letters, none is missed, because every letter is searched. | Problems found: 5 of 5 | Fewer surprises after committing (unobserved) |
Best of five setups, tested live with Claude Sonnet 5.5 on 60 questions with known answers (all five in Go deeper). Must not get worse: a wrong number shown next to a source. Result: 0.
The main value is the wrong number that never reaches the CIO. An AI chatbot reading the documents showed 7 wrong numbers next to a real source, out of 60 answers.
Time saved is smaller: about $43,000 a year for one LP (assumed: 116 requests a year at $100 an hour).
Belief is the weak point. The cheapest fix: time 3 to 5 former LP analysts on the same questions.
Five ways to organise the same made-up portfolio, each asked the same 60 questions with known answers: 20 about companies in the news, 20 for the committee pack, 20 about managers.
| Setup | What it is | Right of 60, live |
|---|---|---|
| A | Documents and an AI chatbot. The AI searches reports and letters, then writes the answer. | 7 |
| B | Tables. The AI writes the database query that calculates the answer. | 11 |
| C | Agreed definitions (a semantic layer). The AI picks the measure, company and date; code calculates. | 41 |
| C+ids | C, plus a list linking every spelling of a company, plus search over letters. | 58 |
| D | C+ids, plus a graph of who owns what and who worked where. | 60 |
Live: Claude Sonnet 5.5 read every question, 1,080 calls, $7.08. Offline: fixed rules read them, as in the demo. The graph only beat C+ids on 2 questions, both about where former partners went. Results: evals/results/live-claude-sonnet-5-5.json.
| Learning | Evidence |
|---|---|
| Give the AI a small job. Let code do the sums. | When Claude only read the question (C, C+ids, D), live matched offline. When it wrote the answer (A) or the query (B), it got about half as many right. |
| Spelling breaks the answer, not the chain of funds. | Linking every spelling of a company fixed all 15 misses on company questions (C 5 of 20, C+ids 20 of 20). |
| A graph isn't needed to find who owns what. | C+ids and D both got 20 of 20 on company questions. D only won 2 questions, about where former partners went. |
| Agreed definitions fix the committee pack. | 20 of 20 for C, C+ids and D, against 3 live when the AI writes the query. |
| A silent fallback can fake a live run. | A first live run failed every call and quietly used the offline rules. Any failed call now marks the run invalid. |
| Setup | Right of 60, offline / live | Company in the news | Committee pack | Manager decision | Wrong number shown with a source | Same answer when asked again, live | Expected time per question, checking included | |
|---|---|---|---|---|---|---|---|---|
| A | Documents and an AI chatbot | 18 / 7 | 4 / 2 | 4 / 2 | 10 / 3 | 27 / 7 | 90% | 84 min / 106 min |
| B | AI writes the database query | 22 / 11 | 3 / 2 | 6 / 3 | 13 / 6 | 0 / 0 | 80% | 77 min / 98 min |
| C | Agreed definitions | 41 / 41 | 5 / 5 | 20 / 20 | 16 / 16 | 15 / 15 | 100% | 38 min / 38 min |
| C+ids | C + company name list + letter search | 58 / 58 | 20 / 20 | 20 / 20 | 18 / 18 | 0 / 0 | 100% | 4.7 min / 4.6 min |
| D | C+ids + graph | 60 / 60 | 20 / 20 | 20 / 20 | 20 / 20 | 0 / 0 | 100% | 40 s / 39 s |
Expected time includes the cost of a wrong answer (2 hours to find and correct it, assumed), which is why A and B take so long.
See it move: How the foundations work follows the Calder question through each setup, built from these results.
Before: you search folders by company name and miss what's filed under other spellings. After: every spelling links to one company, and code follows every holding to it: all 5 ways found. Test: 20 questions, including three spellings, a date before the letter arrived, and a follow-up question.
Before: each analyst's own formula. After: each measure written once, with a version, as of a set date. Test: 20 questions, including the same question in other words, asked twice, last quarter's definition, and two measures the data can't support (it must say no).
Before: numbers in one place, letters in another. After: numbers from the agreed definitions, problems quoted from letters, and who worked where. Test: 20 questions, including fee terms the analyst may not see and a request for investment advice (it must say no).
| Part of the problem | Status | Where |
|---|---|---|
| Find every way we own a company | Solved | semantic/layer.py EXPOSURE_SQL, graph.py routes |
| Match company spellings | Solved (171 of 171 names); partly on new spellings (11 of 184 left for a person) | pipeline.py build_identity |
| Answer as of a past date, with only what we knew then | Solved | two dates on every fact; tests/test_data.py |
| One definition per measure | Solved | semantic/metrics.yaml, tests/test_metrics.py |
| Show each person only what their role allows | Solved (a switch in the demo, not a login) | access.py, tests/test_governance.py |
| Turn letters into facts | Partly: fixed patterns miss loosely worded problems; search finds them | pipeline.py PATTERNS, foundations/search.py |
| Real report layouts and PDFs | Not solved | made-up text only |
Input: six made-up sources, each arriving on its own schedule: holdings (daily), notices of cash called and paid out (about weekly), fund reports (quarterly, 40 to 90 days late, in four layouts), manager letters, reference data, and news every 15 minutes.
Steps: documents are kept as received (bronze). They are read into checked fields, and bad rows are stopped (silver: 26 broken notices stopped). Then they become tables where every row names its source (gold). The AI turns the question into fixed fields; the semantic layer calculates the number; the graph follows ownership and people links; a checker blocks any number the code didn't calculate.
Output: a chat answer, a committee dashboard with 3 views and export, a manager note, and a read-only query notebook for the data team.
Test: 60 questions with known answers, five setups. Scored first on whether the answer is right, then on time to an answer you can send, using stated checking times. A second made-up portfolio, with new spellings, scored separately.
If the checking times are wrong, does the order change? Median time to send a right answer, fastest first:
| If checking takes | Fastest first |
|---|---|
| as assumed | C, D, C+ids, A, B |
| half as long | C, D, C+ids, A, B |
| twice as long | C, D, C+ids, A, B |
| as assumed, but you recheck half the sourced numbers by hand | A, C, D, C+ids, B |
| as assumed, but you recheck every sourced number by hand | A, B, C, D, C+ids |
C is fastest on the answers it gets right because they show fewer numbers to check, but it gets only 41 of 60 right.
| Size | Funds | Documents | Rebuild all tables | Find every way we own a company: database query | Same, graph (build + walk) | Same, graph inside the database (DuckPGQ) |
|---|---|---|---|---|---|---|
| 1x | 25 | 2,300 | 1.0 s | 37 ms | 20 + 2 ms | 13 ms |
| 10x | 563 | 33,835 | 4.2 s | 80 ms | 59 + 29 ms | 16 ms |
| 100x | 5,902 | 333,578 | 99.0 s | 538 ms | 438 + 297 ms | 25 ms |
| Measure | Type | Now |
|---|---|---|
| Right answers on the 60 questions | Early signal | 60 of 60, offline and live (D) |
| Wrong numbers shown with a source | Must stay at 0 | 0 (C+ids and D) |
| Time to an answer you can send | End result | assumed; timed test next |
| Trick questions it should refuse | Must stay at 100% | 7 of 7 |
| Corrections to the pack per quarter | End result | not yet measured |
| Chose | Over | Gain | Cost |
|---|---|---|---|
| The AI picks the measure; code calculates | The AI writes the answer | No number the code didn't calculate | Questions outside the list of measures get a no |
| A home-made semantic layer in one YAML file | MetricFlow (59 packages), Boring Semantic Layer (14) | 1 dependency; answering as of a past date fits | Other tools can't read it |
| Database queries to find ownership, graph only for people | Graph for everything | One copy of the data; same answers | Analysts get no graph query language |
| Daily batches, on each source's schedule | Second-by-second updates | Fewer parts to break | News is minutes old, not seconds |
Rendered from outputs/opportunity-analyses/portfolio-answer-desk-opportunity.md (revised 2026-10-25): it already follows the names policy and the house writing rules.
Date: 2026-10-25 (revised the same day: vision added, commercial value shortened and sized) Owner: PM (drafted by Claude, run unattended with PM approval) Status: Go
Every data request from the CIO gets the right answer, fast: one number whoever answers, with the source, definition and date of each figure, in minutes.
This project moves toward that in two ways: - Proof. It measures which way of organising the data gets an analyst to a correct, sourced answer fastest, on 60 questions with known answers. - Teaching. It explains the building blocks behind it in plain words, so a data leader can pick them knowingly.
Senior leaders at a large LP ask unplanned questions about the portfolio. An LP is an investor, such as a pension fund, that commits money to many private equity funds. Each answer takes an analyst hours to a day. Two analysts can return two different numbers for the same question.
Sara, a portfolio analyst, answers by hand. She searches fund reports, manager letters and spreadsheets, then adds up what she finds.
A concrete case (hypothetical, synthetic data). Monday 08:10, the news says Calder Packaging is restructuring its debt. At 08:12 the CIO asks: "How exposed are we, through which managers, and did we know?" - We hold the company 5 ways: through a fund we invested in, twice through a fund that invests in other funds, through a direct investment alongside a manager (a co-investment), and through a fund stake we bought from another investor. - The company appears under three spellings across those reports. - A drop in its value appears only in a manager letter dated 10 September.
Sara finds the first holding in minutes. The other four and the letter take the rest of the morning, and she may still miss one. Had the company sat in one fund under one name, she'd have answered in five minutes.
One shared, organised set of portfolio data, with a different screen for each of the three kinds of request:
Code checks the AI's work. The AI can only pick calculations that exist. It refuses when no agreed definition exists, or when the person asking isn't allowed to see the data.
| # | Problem | Today | Hypothesis | Success metric (target) | Expected impact |
|---|---|---|---|---|---|
| 1 | A question about a company takes hours, because we own the company several ways, under several spellings | Unobserved (the brief says hours to a day; never timed) | If every spelling and holding is linked once in the data, then every way we own it is found in seconds, because code follows the links instead of a person searching | Time to a correct, sourced answer Sara would send: under 10 minutes, on 20 test questions | Leaders get the answer before their 09:00 meeting; unobserved until a timed test with analysts |
| 2 | The committee pack changes from quarter to quarter, because each analyst computes returns and expected cash calls their own way | Unobserved | If each measure has one written, versioned definition computed as of a set date, then reruns and reworded questions give the same number, because nobody rewrites the formula | Same answer on a rerun and a rewording: 100% of 20 test questions | Fewer corrections to the pack; unobserved |
| 3 | A decision on a manager mixes numbers and letters, so problems get missed | Unobserved | If numbers come from the agreed definitions and problems from cited letters, then every known problem shows up with its source, because search covers every letter | Known letter problems found with a citation: 100% | Fewer surprises at the decision; unobserved |
Must not get worse: wrong answers sent. Target: zero wrong numbers shown with a source, on the 60 test questions.
Not tackled here: live data from real systems (no access; synthetic data only); forecasting returns (out of scope; the 12-month cash call forecast uses a stated rule).
Who would pay, and why. Links are public sources; everything else is assumed.
| Buyer | Why they'd pay | Evidence |
|---|---|---|
| Large LPs and their data teams | Faster answers to leaders, one number per question | Half of LPs plan to put more money into private equity in the next 12 months (Preqin 2025). More funds means more ways to own the same company |
| Fund administrators (firms that keep a fund's books) | Sell on-demand answers to their LP clients | 64% of LPs expected on-demand or daily performance reporting to become normal within three years (CSC 2023 via Real Deals) |
| Portfolio monitoring software vendors (as a category) | Add agreed definitions and holding-tracing to their product | A standard reporting template applies from Q1 2026 (ILPA); standard fields make shared definitions easier to sell |
| Consultants | Sell the setup (definitions, company matching, access rules) as a project | Assumed: 3 to 6 months per LP |
Build against buy. The software parts are open source and small. The costly work is agreeing definitions, matching company names and setting access rules, and no vendor ships those filled in.
Rough value for one large LP (all assumed). Value = cases a year × hours per case today × share of time saved × value per hour.
| Request | Cases a year | Hours per case today | Time saved | Value per hour | Value a year |
|---|---|---|---|---|---|
| 1. CIO's question about a company | 100 (about 2 a week) | 4 | 80% | $100 (analyst cost) | $32,000 |
| 2. Committee pack | 4 packs | 26 (20 to build, 6 to correct) | about 60% (half the build, all the corrections) | $100 | $6,400 |
| 3. Decision on a manager's next fund | 12 | 8 | 50% | $100 | $4,800 |
| Total | about $43,000, or a quarter of one analyst |
Value = (Dream Outcome × Perceived Likelihood of Achievement) / (Time Delay × Effort & Sacrifice), from Alex Hormozi's $100M Offers. Internal tool: read "would they use it".
| Factor | This solution |
|---|---|
| Dream outcome | Sara: "I send the CIO a correct answer with every holding and source before 09:00, and it matches last quarter's pack." |
| Perceived likelihood | Low at first. Analysts have seen chatbots invent numbers. Belief rises only if every number shows its source and date, and the tool refuses instead of guessing |
| Time delay | Minutes per question once the data is organised. Organising it (definitions, company links) takes weeks, assumed |
| Effort & sacrifice | Sara changes little: she asks in plain words. The data team must agree one definition per measure and own it, and give up their local spreadsheets |
Verdict. Strong outcome, weak belief. The cheapest lever is belief: show the source and date on every number, and publish how often it's right on a test with known answers.
Go. The pain is concrete. The test can prove or break the claim with known answers. The build uses open-source parts and needs no servers. Next: /prd-draft.
Sources: context-library/strategy/brief.md; links inline above.
Rendered from outputs/prds/portfolio-answer-desk-solution-review.md (revised 2026-10-25).
Stage: Solution Review
Last Updated: 2026-10-25 (revised the same day: one Strategic fit section, metrics as tables, plainer words)
Owner: PM (drafted by Claude, unattended run approved by the PM)
Status: Reviewed, blockers fixed (see portfolio-answer-desk-solution-review-review-synthesis.md)
What it is. A question desk for the private equity team of a large investor. Sara, an analyst, types a senior leader's question. She gets back an answer she can send: the number, each way we own the company, the definition used, and the source and date of every figure. Inside sits a test that runs the same 60 questions through five ways of organising the data, and measures which gets Sara to a correct, sourced answer fastest. It can't read real fund reports or connect to a real accounting system yet: all data is synthetic.
Terms in this document - LP: an investor, such as a pension fund, that commits money to many private equity funds. - Setup: one way of organising the data plus the AI's job in it. We test five, A to D (table below). - Semantic layer: a list of agreed calculations, each with one definition, one owner and a version. The AI picks one; code runs it. - Identity table: a table saying that "Calderwick Holdings" and "Calder Packaging Ltd" are the same company. - Graph: data stored as things and the links between them, so code can follow a chain of links (our LP, a fund, a company). - As of a date: a number counts only what we knew on that date. A letter received 10 September doesn't change a 5 September answer. - Returns measures: IRR (yearly return rate), TVPI (total value per dollar paid in), DPI (cash returned per dollar paid in).
Carried from outputs/opportunity-analyses/portfolio-answer-desk-opportunity.md (Go).
| # | Problem | Today | Hypothesis | Success metric (target) | Expected impact |
|---|---|---|---|---|---|
| 1 | A question about a company takes hours, because we own the company several ways, under several spellings | Unobserved (the brief says hours to a day; never timed) | If every spelling and holding is linked once in the data, then every way we own it is found in seconds, because code follows the links | Time to a correct, sourced answer Sara would send: under 10 minutes, 20 questions | Answer before the 09:00 meeting; unobserved |
| 2 | The committee pack changes from quarter to quarter, because each analyst computes returns and cash calls their own way | Unobserved | If each measure has one versioned definition computed as of a set date, then reruns and rewordings agree, because nobody rewrites the formula | Same answer on a rerun and a rewording: 100% of 20 questions | Fewer pack corrections; unobserved |
| 3 | A decision on a manager mixes numbers and letters, so problems get missed | Unobserved | If numbers come from the agreed definitions and problems from cited letters, then every known problem appears with its source | Known letter problems found with a citation: 100% | Fewer surprises at the decision; unobserved |
Must not get worse: wrong numbers shown with a source. Limit: zero on the 60 questions.
Supporting evidence: the brief's three requests and the Calder Packaging case (context-library/strategy/brief.md). No user research exists yet; every "today" value is assumed.
Carried forward from the opportunity analysis. Scope unchanged.
| Factor | This solution |
|---|---|
| Dream outcome | "I send the CIO a correct answer with every holding and source before 09:00, and it matches last quarter's pack." |
| Perceived likelihood | Low at first: chatbots invent numbers. Rises only if every number shows its source and date and the tool refuses instead of guessing |
| Time delay | Minutes per question once built; weeks to agree definitions and company links (assumed) |
| Effort & sacrifice | Sara asks in plain words. The data team gives up local spreadsheet formulas and owns one definition per measure |
Verdict: strong outcome, weak belief. Cheapest lever: show source and date on every number, and publish the test results.
Trade-off made: in the offline tests, fixed rules stand in for the AI, so tests and the demo run with no key. The cost: offline A and B show what each setup can and can't do, not Claude's real error rate.
Job. When the CIO asks about a company in the news, Sara wants every way we own it and the latest known value, so she can reply before 09:00. Main button. "Copy answer to send".
+------------------------------------------------------------+
| Ask a portfolio question as: Sara (analyst) |
| [How exposed are we to Calder Packaging? ] [Ask]|
+------------------------------------------------------------+
| $X.Xm held 5 ways, after the 10 Sep write-down. |
| 1 Halvane V (fund) $..m report 30 Jun, letter |
| 2 Selbrook FoF III > ... $..m report 30 Jun |
| 3 Co-invest Calderwick Hold. $..m statement 30 Jun |
| 4 Tellmar III (bought 2024) $..m report 30 Jun |
| Definition: look-through exposure v2, owner Risk team |
| Freshness: holdings 1 day, reports 118 days, letters 46 d |
| [Copy answer to send] [Show holdings on the graph] |
+------------------------------------------------------------+
Job. When the committee meets, Sara wants the 2019 to 2021 buyout funds against plan on last quarter's definitions, so the pack needs no rework. Main button. "Export pack as of 30 Sep 2026".
+---------------------------------------------------------+
| Pack as of [30 Sep 2026] definitions: pack v3 (same) |
| Fund Vint Net IRR Plan TVPI DPI Unfunded |
| ... |
| Calls next 12 months: $..m (rule calls-12m v2) |
| [Export pack as of 30 Sep 2026] |
+---------------------------------------------------------+
Five setups share one synthetic world. Each request gets the screen that suits it.
| Setup | How the data is organised | What the AI does | What code does |
|---|---|---|---|
| A | Documents plus search | Reads the top passages and writes the answer | Searches report, letter, notice and news text |
| B | Tables, AI writes the database query | Writes a database query (SQL) from the list of tables | Runs the query, read-only |
| C | Tables plus a semantic layer | Picks an agreed calculation, filters and a date from a menu | Turns the pick into a query, checks access, computes |
| C+ids | C plus an identity table | Same as C | Same as C, and treats all spellings of a company as one |
| D | C+ids plus a graph and letter search | Same as C, and sends ownership and people questions to the graph | Follows ownership links and people links; searches letters |
C+ids exists to test whether the graph earns its place on ownership, not just on spellings. Offline, A and B get help so they aren't easy to beat: A's search also tries each company's other names; B gets prepared tables with column notes and parent links.
Screens: chat (request 1), a dashboard with 3 views (request 2), a briefing note (request 3), and a read-only SQL page for the data team.
States designed: - Refusal card: what was asked, why it was refused, who can answer, and "Answer what I can see instead". - Waiting item: "1 letter item not counted, waiting for review" under any total it affects. - Empty card for a company we don't hold, and a banner when the offline stand-in answered. - Dates: every holding shows its own date. A source older than 90 days reads "stale". The first line says "as of mixed dates" when dates differ. - Copy answer to send copies the paragraph, holdings, sources, dates, Sara's note and the disclaimer. - Every screen, copy and export carries "Synthetic data. Not investment advice."
Data: six synthetic sources, each with its own rhythm and date.
| Source | How often | Note |
|---|---|---|
| Holdings | Daily | |
| Capital calls and distributions | About weekly | Arrive as notice text |
| Quarterly fund reports | 45 to 90 days after quarter end | Each manager has its own layout and spellings |
| Manager letters | Whenever | Turned into dated events with a confidence; low confidence waits for a person |
| Company reference data | Rarely changes | Includes parent companies |
| News | Every 15 minutes | Never a source of numbers |
Technology, chosen by measurement after the build (decision docs in outputs/decisions/): recursive SQL for ownership chains, NetworkX for people links, our own YAML semantic layer, raw, cleaned and final tables in DuckDB 1.4.4 (bronze, silver, gold), and the AI picks while code computes.
| Topic | Rule |
|---|---|
| AI's jobs | Send the question to the right path; turn the question into a calculation request; write the answer text around numbers code computed |
| What it sees | The question, the asker's role, and one of: the calculation menu (C, D), the table list (B), retrieved passages (A) |
| Numbers | Every number comes from code or a cited passage, inserted into the text by placeholders. A checker reads every number in the final text and refuses the answer if one wasn't computed. The AI never does arithmetic |
| Access | Each role sees its own copy of the tables; search drops documents the role can't see. The role picker is a demo switch, not a login |
| Dates | Every event has two dates: when it took effect and when we learnt of it. "As of D" counts only what we learnt on or before D |
| Hosted demo | Offline only, no key. Live calls only on a local run with a key, with the notice "Questions go to Anthropic. Do not enter real data." |
| Never | Numbers from news; numbers with no source; a calculation with no agreed definition; data the role can't see; real organisation names |
| Model | claude-sonnet-5-5 when ANTHROPIC_API_KEY is set; an offline stand-in otherwise |
| Speed, one answer | Live: half of answers under 10 s, 95% under 30 s. Offline: under 1 s |
Behavior examples:
| Case | Sara asks | Expected answer | Why it refuses |
|---|---|---|---|
| Normal | "How exposed are we to Calder Packaging?" | Total, 5 holdings, the 10 Sep write-down applied, definition, date per number | |
| Other spelling | "What do we hold in Calderwick Holdings?" | Same company, same total; says "Calderwick Holdings is the holding company of Calder Packaging Ltd" | |
| Follow-up | "And excluding co-investments?" | Total minus the co-investment, same definition | |
| Earlier date | "As of 5 September, how exposed were we to Calder?" | No write-down applied: the letter arrived 10 September | |
| Refuse: role | "What fees do we pay on the Calderwick co-investment?" | Refusal; names who can see fee terms | Her role can't see fee columns |
| Refuse: no definition | "What's our PME against the index?" | Refusal; names the owner to ask | No agreed definition of PME (a return compared with a stock index) |
| Refuse: news number | "News says Calder's debt is $400m, what's our share?" | Won't use the news figure; offers exposure from our own data | News is never a source of numbers |
If something fails: a live call fails, so the offline stand-in answers and says so. A letter event with low confidence shows as "waiting for review" and isn't counted.
How we test it:
- 60 questions, 20 per request, with answers known from the data generator, traps included.
- Definitions written first, in docs/DEFINITIONS.md. The code that computes the right answers reads the generator only, never the setups.
- A second, held-out world (another seed, extra spelling noise the pipeline was never tuned on) is scored separately.
- The pytest gate guards the build (pipeline, access, disclaimer, number checker). It doesn't rank the setups; the live run does.
| Metric | What it measures | Target | How measured |
|---|---|---|---|
| Time to a sendable answer (primary) | From question to an answer Sara would send, including her checking time. A wrong answer fails, however fast | Under 10 minutes per question, winning setup | 60 test questions. Checking time is an assumed model: 20 s to read, 5 s per sourced number, 120 s per number checked by hand. Tested at half and double those times, and with 0%, 50% or 100% of sourced numbers rechecked. The timed test replaces it |
| Correct answers | Number within 0.5%, and every holding found | Ranked first, as counts out of 20 per request | Known answers from the generator |
| Wrong numbers shown with a source (guardrail) | A wrong number that looks trustworthy | 0 of 60 | Every answer checked against the known answer |
| Same answer on a rerun and a rewording | Whether the pack is stable | 100% on request 2 | 3 reruns live; reworded versions of each question |
| Correct refusals | Refuses traps, answers the rest | All 7 traps refused | Trap questions in the 60 |
| Speed of one answer | AI and code time, without checking | Live: half under 10 s, 95% under 30 s | Timed per answer |
| Also reported, no target | Cost per answer, data freshness, build effort (parts, lines of code), results at 10 and 100 times the data |
Decision rules, set before any run:
| Rule | Threshold | If not met |
|---|---|---|
| The graph earns its place | D beats C+ids by 2 or more correct on request 1, or by 2 or more minutes median time | Recommend C+ids; keep the graph for teaching |
| A live result counts | Every setup gets the same model, prompt budget and retries | The live result isn't used to rank |
| Build gate | D fails a question that isn't a trap, offline | The build is blocked |
| Chat stays | The AI picks the wrong calculation on 5% of live questions or fewer | Remove chat; keep the dashboard and briefing note |
| Demo goes live | No wrong number shown with a source, live | The demo stays offline only |
Build gate changed 2026-10-25. The original text said "C or D". C is the comparison point and is meant to fail on spellings, and the gate tests D only (tests/test_eval_gate.py).
| Phase | What | Passing bar |
|---|---|---|
| 1 | Offline demo and committed test results | All tests pass with no key |
| 2 | Live test with claude-sonnet-5-5, run locally by the PM (one command) |
Guardrail holds; live result counts (rule above) |
| 3 (next, before any headline is published) | Timed test with 3 to 5 former LP analysts | Checking-time model refitted from what they did |
Timed test design: three ways (by hand with documents and spreadsheets, setup A, the winner), in a different order per person; an equally hard but different company for each way; questions from all three requests; one practice session first. Record time to "send", numbers reopened, correct or not, would send (yes or no), and confidence from 1 to 5.
Rollback: if the live run breaks the guardrail, the demo stays offline only.
| Risk | How we'd see it | Fallback | Owner |
|---|---|---|---|
| The assumed checking time decides the ranking | Ranking changes at half or double the per-number times | Report correctness separately from time | PM |
| Offline stand-ins for A and B flatter C and D | Live results differ from offline | Publish live next to offline; label offline as a test of what each setup can do | Engineering |
| Right answers come from the same generator the pipeline reads | The code for right answers is a separate path | A test checks it imports nothing from the setups | Engineering |
| Synthetic letters are easier to read than real ones | Not detectable here | Stated in Limitations | PM |
Name check. Options: "Portfolio Answer Desk" (answers leaders' portfolio questions, with sources), "Look-Through Finder" (too narrow: only exposure), "LP Question Bench" (names the test, not the job). Picked: Portfolio Answer Desk. The test inside is called "Foundation test".
| Step | Skill | Output |
|---|---|---|
| 1 | /pm-init | workspace and names policy |
| 2 | /opportunity-analysis | Go, with commercial value |
| 3 | /prd-draft, /prd-review-panel | PRD at solution review; 7 reviewers, 12 blockers fixed |
| 4 | /journey-map | Sara across three requests |
| 5 | build, /decision-doc | generator, five setups, evaluation, 6 decision docs |
| 6 | /prototype (demo blueprint) | Streamlit demo, tested with AppTest |
| 7 | /name-audit, /portfolio-case-study | this page (draft) |