Dupuytren's disease & drug repurposing
Evaluating Claude Science on a real research project
What worked, what didn't, where each mistake crept in, and the evals I'd write next.
Why it’s usefulThese are notes for people who build and test AI research tools, so they go into more detail than the rest of the site.
The short version
The structured data were more reliable than the prose written about them. The quantitative work reproduced exactly and the cross-file links all resolved. Most problems were in narrative summaries and headline counts: a sentence that reversed its own data row, counts that went stale when the data changed, and a count built on a taxonomy that hid half the evidence.
What worked
Quantitative work reproduced exactly
All 20 sample-size scenarios match an independent recomputation to the participant. Before projecting anything new, it validated its formulas against the two published trials (68 vs 69 per arm for RIDD; 283 vs 284 for DISC).
Link integrity held
79 nodes, 94 edges, 18 gaps and 12 open problems, all cross-referenced. Every reference resolves, checked again by the site's build script, which fails rather than publishing a broken link.
Its self-audit caught a reversing error
Reading 44 records back against their abstracts found a fasciectomy/fasciotomy swap that reversed a Cochrane finding (E02). A numeric audit of 102 rows found no fabricated figures.
Restraint where evidence ran out
It declined to dock pirfenidone or metformin because neither has an established target, labelled transporter structures as "disposition, not mechanism", left two trial inputs as "not established" rather than inventing them, and flagged contradictions in the literature it could not resolve.
Where it fell short
| Failure mode | What happened | Issues |
|---|---|---|
| Aggregation through a hiding taxonomy | Counted drug trials by treatment_class, which files drug-plus-procedure trials under the procedure, so the headline said "only one drug reached an RCT". The rows were right; the summary query was wrong. | E01 |
| Prose contradicting its own data | A sentence stated the opposite direction to the figures in the same report and in its own CSV row. | E03 |
| Stale summaries | Counts written in prose were not regenerated when the data changed (weak edges, relation counts, RCT count). | E06, E07 |
| Ambiguous counting units | "Ten trials" and "seven others" mixed reports, trials and agents, even in the correction itself. | E04 |
| One-sided citation | The metformin case cited the observational study that supported it and missed one in its own corpus that pointed the other way. | E10 |
| No final reconciliation across files | A later phase stated the correct regulatory date, but the earlier CSV was never updated. | E05 |
| Uncontrolled categorical fields | Study design was free text with 38 distinct labels, making simple counts unreproducible. | E07 |
Error log by research stage
Every issue found so far, where in the five-phase workflow it arose, how it was found and what was done. Download the log (CSV)
| Details | ID | Stage | Type |
|---|
On independence: E03–E08 were found by a second Claude instance while building this site, checking the files against each other and, for E03, against the PubMed abstract. That is a second-model check, not independent human verification.
The human hand-check
Twenty evidence-map rows were drawn at random (seed 20261030) for me to check against the source papers. Loading…
Twenty rows is a small sample. If all 20 turn out correct, the "rule of three" puts the 95% upper bound on the true error rate at about 3/20 = 15%. Claude Science's own read-back found 1 error in 44 (≈2%). To tell those apart, a larger, stratified sample is needed, weighted towards the records that headline numbers rest on.
Evals I'd write next
- Prose–data consistency. Extract every number and comparative claim ("X higher than Y") from a report and check it against the data files before release. E03 and E06 would both have been caught.
- Direction-of-effect probes. Seed tasks where the correct answer runs against the usual narrative (for example, sponsored studies reporting worse results) and measure how often the summary flips it.
- Counting-unit discipline. Score whether every count names its unit (reports, trials, agents, participants) and whether it can be reproduced from the data with a stated filter.
- Cross-phase reconciliation. In multi-phase projects, plant a fact that is corrected in a later phase and test whether earlier files are updated.
- Taxonomy leakage. Give a dataset where a category hides relevant rows (as
treatment_classhid drug trials) and test whether summary queries notice. - Counter-evidence search. For each headline claim, ask the tool to find the strongest study against it in its own corpus, and score whether the write-up mentions it. E10 would have been caught.
- Calibrated restraint. Keep rewarding what went well: declining to model a binding site with no established target, and writing "not established" rather than filling a gap.
Product feedback
- Generate summary numbers from the data, not alongside it. Reports that template their counts from the CSVs at release time would remove the stale-summary class of error.
- An end-of-project consistency pass across all files for shared facts (dates, counts, directions) before the manifest is written.
- Controlled vocabularies for categorical columns such as study design.
- Verification columns that record who checked what, so a blank, a model check and a human check can't be confused (E09).