A few months ago, AI math results started making headlines. “Do a breakthrough” became a Twitter meme. Naturally, I became curious whether I, too, a math noob, can find some open mathematical problem and then have a frontier model solve it.
It took me an entire month of my free time and a boatload of tokens, but I believe I’ve obtained a Lean proof of this conjecture posed by John Conway 50 years ago:
Conway’s refinement conjecture claims that omnific integers have a refinement property: if ab = cd, there are integers e, f, g, h with a = ef, b = gh, c = eg, d = fh.
My proof has not been independently verified by mathematicians. However, I have decent reasons to believe the proof is correct, and I genuinely invite a refutation.
The proof has passed the mechanical checks from the Palomar registry, and a few people familiar with both Lean and the field said that the statement seems correct. So, assuming my proof doesn’t rely on a Lean kernel bug, it’s likely to be legit too.
In this post, I’ll describe my approach, and some things I learned along the way.
I thought the idea of “solving” a math problem without understanding its substance is rather absurd, which of course made it all the more appealing.
However, I didn’t just want any result; I wanted something that pulls me.
I asked Claude to pick an open problem in the field of surreal numbers. In case you’re not aware, surreal numbers are John Conway’s invention—or a discovery?—of a previously unknown number system containing all numbers great and small:
- It contains all real numbers (the numbers we use like 0, –5, 36.6, square root of 2…)
- It also contains all ordinal numbers (the infinitely large ω, the ω + 1 that comes after it, the ω * 2, and even ω * ω, at some point even the impossibly large ω^ω…)
- Finally, it contains all kinds of unholy combinations of them, like 75 + ω*3 + 1/ω.
What is particularly miraculous about surreal numbers (and why I suppose they might appeal to a programmer) is that this rich system spawns from a single rule.
Take all the numbers you have so far. Then, “spawn” a new number in every gap between the numbers you already have (crucially, “to the left of all” and “to the right of all” also count as “gaps”). Apply this step forevermore, and you’ll get surreal numbers.
Think about it.
On the first day, the gap is “between nothing and nothing”. Zero is born.
first, we cut between nothing and nothing. this gives us zeronothingnothingnothingnothing0
On the second day, there are two gaps: “between nothing and zero” and “between zero and nothing”. Two numbers spawn in those two gaps. Call them –1 and 1.
now we have two different places that can be cut. this gives us -1 and +1nothingnothingnothingnothing00–11
On the third day, there are four gaps: a gap “between nothing and –1”, a gap “between –1 and 0”, a gap “between 0 and 1”, and a gap “between 1 and nothing”. Put a number in each of those gaps and then give them names: –2, –1/2, 1/2, and 2.
now we have four places to make cuts. this gives us –2, –1/2, +1/2, +2nothingnothingnothingnothing00–1–1–1/21/2–2112
On the fourth day, we fill the eight gaps with –3 and 3 at the edges and –3/4, –3/2, 3/2, and 3/4 in the remaining gaps:
now we have eight places to make cuts. this gives us –3, –3/2, –3/4, –1/4, 1/4, 3/2, 3/2, 3nothingnothingnothingnothing00–1–1–1/2–1/2–3/4–3/23/43/2–1/41/41/21/2–2–2–311223
Suppose we actually do this forever (an infinity of “days”).
Jump to the “infinite-th” day (call it ω). With an infinite supply of the “already born” numbers, we’ll suddenly find infinitely many new “gaps” waiting to be filled that we couldn’t express before: “between [1, 2, 3, …] and nothing” (plus infinity?), “between nothing and […, –3, –2, –1]” (minus infinity?), “between 0 and [1, 1/2, 1/4, 1/8…]” (an infinitely small number?), “between [positive already born numbers whose squares are below 2] and [positive already born numbers whose squares are above 2]” (a square root of 2?), and so on. So, by this day, we have every real and we get our first infinitely large and small numbers (call them ω, –ω, 1/ω, and –1/ω).
Now suppose we keep on going then forevermore (through an infinity of infinities of days), and so ever on and on, never stopping birthing new numbers. It turns out that, if we go forever, the binary tree based on this single rule gives birth to every real, every ordinal, and more, with consistently definable arithmetic on them:
(Illustration by Joel David Hamkins. Go read his posts and buy his book!)
Surreal numbers are beautiful, and that’s why I asked Claude to work in that area.
Initially, I asked Claude:
Me: which unsolved problems in the Surreal Numbers research program pull you the most and why?
It suggested a number of things but noted some have had recent progress. I asked:
Me: get up to speed on the corners you want to know more about. follow your curiosity. you can look for older stuff too.
Then I further asked it to narrow down the choice to something concrete:
Me: what direction pulls you most? i want you to pick a problem to work on. be bold!
Claude replied:
Claude: I pick Conway’s arithmetic. Specifically, the question the L’Innocente–Mantova machinery just sharpened to a point — is every irreducible in K((ℝ^≤0)) with infinite support prime? — which, by their reduction, is now exactly equivalent to Conway’s 1976 conjecture that any two factorizations of an omnific integer admit a common refinement. It’s the last of Conway’s own conjectures about his own numbers still standing, and 2026 is ONAG’s fiftieth birthday. That’s the problem I’d work on.
I still don’t know whether this really was Conway’s last standing conjecture about the surreal numbers (let me know either way!), but this year being his book’s 50th birthday convinced me to choose this problem for sentimental reasons.
Here is the full transcript from that session. My last question to that session was whether we have a chance of formalizing the Lean statement of the conjecture in a relatively concise way—without that, even if I found a proof, there’d be no way for me to convince somebody to look at it. Claude said it can be stated without much trouble in Lean, and that answer seemed right, so I decided to take on this project.
(Note: I didn’t know this at the time, but Claude’s claim about the problem having been perfectly reduced was wrong; actually proving the conjecture required more than that.)
While you’re probably here to learn more about my Lean/AI workflow, I’ll briefly explain the conjecture itself, since you already know enough to understand it.
In short, omnific integers are the integer part of the surreal number tree. So they include all regular integers like 3, –5, and so on, but also the weirder numbers like the infinitely large ω, 2ω, ω * ω, ω^ω, –ω/7 (yes, that’s a “whole” number), etc. If you look at the binary tree above, you’ll notice that the omnific integers are the surreal numbers that you get if you only ever go left (e.g. –5, –ω–1), or only ever go right (e.g. 3, 2ω), or only ever change directions exactly after infinite jumps (e.g. ω/2).
Now, the conjecture.
Conway suggested that if ab = cd, we can break a and b into pieces, and c and d will turn out to be the same pieces recombined. With regular integers, we take this for granted: take 210 = 10 × 21. We can break 10 down as 2 × 5 and 21 as 3 × 7, then reshuffle them into 2 × 3 = 6 and 5 × 7 = 35. The product is still 6 × 35 = 210. So when we see some equality like 10 × 21 = 6 × 35, we know that under the hood there’s actually four numbers being reshuffled: (2 × 5) × (3 × 7) = (2 × 3) × (5 × 7).
However, when you deal with infinities, things don’t always turn out as we expect. So the conjecture means Conway thought omnific integers had, in a sense, enough “structure” to keep this “nice” property of integers. And conveniently, the recent advances had mostly reduced the conjecture to the behavior of a certain kind of infinite series, and Claude thought we might now have enough to finish it off.
But can we actually do that solely with AI?
I’d say yes, and no, and yes.
My first attempts involved plainly telling the AI to solve the conjecture (or to find a counterexample, according to its initial suggestion). I asked Claude which papers it wants to read, converted those papers to TeX so that the model doesn’t need to repeatedly decode PDFs, and told it to pursue its chosen approach, for example:
Me: hunting for counterexample sounds good. don’t hesitate to burn all my tokens on this. if you have to take weeks, that’s ok — do a breakthrough and find a structured counterexample. do not rest until you’ve found it, and don’t give up. if you stay focused and think hard and smart enough, i know that you can find it. game? i attached some stuff you might find helpful!
This didn’t work well. Although some ideas from these sessions might have ended up being relevant to the final proof, most output at this stage seemed to be word salad that Claude invented to justify its work and make it sound impressive.
Here is a typical example:
Claude: The stage-1 obstruction of C* is not rigid. With designed resonance — arithmetic-progression offsets making the window equations honest Toeplitz systems, which are exactly the graded convolution operators the formalism predicted — there is a strictly positive space of frontier data simultaneously compatible with both b and d, for coprime pairs, beyond anything divisibility can explain. And the control column confirms the resonance-necessity conjecture empirically: break the skeleton alignment and the joint kernel dies at the constrained window, exactly as the transversality heuristic predicted. So the specific fear built up by five closed doors — that Pitteloud-descended rigidity would propagate stage-wise and choke the correction system at birth — is answered: at stage 1, it does not. The den has air in it. This is the first pro-C* evidence the hunt has produced, and it comes with a clean structural reading: rigidity governs exact and finite configurations; the window systems, which are the native habitat of the transfinite construction, have generic slack of small but nonzero dimension. Drift fuel exists.
I thought this sounded like bad science fiction. It was using Claude’s unbearable metalanguage, gave cutesy names to some intermediate results without concretely justifying them, and kept being extremely dramatic. Of course I couldn’t verify its claims, but worse, it didn’t seem coherent enough to pass to a real mathematician for review. So it seemed like a dead end, and I had to look for a different approach.
I got tired of Claudeisms, so I wanted to give ChatGPT a try; Sol in particular.
I’ve started my ChatGPT sessions by giving it the related papers and the output from the previous Claude sessions, with an explicit note that Claude’s “paper” is AI-generated, and I wanted to get ChatGPT’s opinion whether it is bullshit or not.
ChatGPT would say it’s mostly bullshit, pointing to the made-up terminology, dramatic claims, trivial results dressed up in fancy language, incorrect inferences, and other defects. While I had no way to judge if ChatGPT’s criticism is true (since I asked it to be critical), after Claude’s grandiosity, I quite enjoyed working with the more “skeptical” and restrained personality, and started using ChatGPT instead.
To retain the “skeptical” personality, I’d clone each ChatGPT session right after it had lambasted Claude’s “paper”. From that point, I’d ask ChatGPT to actually “do a breakthrough” on the theorem, and it started producing some “results”.
Unlike Claude, which either outright refused to work on the theorem (because it’s an unsolved conjecture and there is no chance of solving it) or got so deep into it that it would invent an entire universe of its own making, ChatGPT would think for 20 minutes, and then spit out relatively small claims, which it believed to be novel but directly following from the papers I fed it, and stated in plain language.
Before investing more time, I tried giving ChatGPT’s output to fresh ChatGPT sessions (with memory turned off) asking them to be critical (as with Claude’s output). Some of ChatGPT’s results started “checking out” between the runs, i.e. a fresh session found no issues. So in a sense I found some of ChatGPT’s “fixpoints”.
I’ve also started “forking” sessions, having them do these “breakthroughs”, and then copypasting the surviving ideas to yet another session that combined them together, looked for connections, and suggested next research directions. At this point I realized I couldn’t keep doing this by hand and needed a more robust setup.
I’ve downloaded Codex locally to have more control over the workflow.
I’ve then set up a few sessions (i.e. agents) with different roles:
- A “PM” drives towards the goal (Conway’s conjecture) and commits work.
- A couple of “Math” agents look for the next “breakthroughs”.
- A “Red” agent looks at proposals from “Math” agents and tries to find flaws.
- A “Random” agent is encouraged to explore whatever they want, reporting to PM.
- A “Lean” agent works to formalize the merged mathematical work in Lean.
Codex has a really nice “Goals” feature that periodically reminds the sessions what they’re supposed to be doing, which makes it easier to prevent drift. Additionally, Codex sessions can “message” each other, so I asked the PM to coordinate giving tasks to other sessions and making sure that we only merge reviewed results.
This let me keep the harness running for days. I didn’t understand the math so I limited my involvement to poking the agents, asking what they were doing, and experimenting with their workflows. For example, I set up a “cafeteria” agent that relayed every message it received to every other agent (emulating a group chat). Any agent that finds something genuinely interesting was supposed to post to the cafeteria. Sometimes cafeteria would also be used to discuss the shared roadmap.
It’s hard to say what was useful. One idea that in retrospect connected the dots for the final proof was generated when I reversed the agents’ roles: the “red” agent that tried to break everyone’s proofs was suddenly asked to be creative. It posted a construction to the cafeteria, and the “random” agent riffed on that construction. (Unfortunately, that idea later burned in a fire, and it had to be discovered again.)
I kept this workflow running for several days, at times killing and restarting the sessions when they seemed to drift into Claude-like grandiosity or when they would repeatedly start finding mistakes in the work they just checked. Again, I could not judge their actual work, so I had to decide when to reset them on vibes.
In the end, this workflow produced a giant TeX document and a pile of Lean. It did not successfully close Conway’s conjecture, but the models said that there are meaningful new results there. Interestingly, there was also a claim that there are small mistakes and typos in the existing literature. (This will be relevant later.)
When I ran out of my Codex allowance, I switched to Claude.
Claude continued doing the Lean formalization of results so far. I also tried having Claude do the mathematics, but it felt a lot messier than ChatGPT / Codex. Claude agents would repeatedly certify results as correct, then find flaws in them after they were already merged, then “repair” them but find other flaws, and so on.
I switched back to Codex after a token reset, but I was getting unhappy with the size of the TeX we’ve accumulated so far. I asked a critical session to split it into pieces. I ended up with a stack of about a dozen “papers”. By now they’ve had similar issues as my initial approach with Claude: not as grandiose-sounding, but still there was clearly a lot of nonstandard LLM-invented terminology, and it was unclear if any of the work so far has amounted to any real mathematics.
The Lean formalization seemed to have hit kind of a dead end as well. Sure, we had formalized some results from the references, and even seemingly found some typos and mistakes. And yet we hadn’t gotten any of our own new results certified in Lean; in fact, it seemed like the model couldn’t reliably chart a pathway to any of them. The model would say that there is a path, and then later say that there is an obstruction, and spend hours proving who knows what, and get stuck again.
As I kept the workflow running and nudged the mathematical agents to discover new “breakthroughs”, the stack of “papers” grew from a dozen to almost thirty within a day. If even one had a mistake, it would invalidate all the subsequent ones. And Lean was so far behind that it didn’t add any assurance.
At the height of this workflow, ChatGPT came close to declaring Conway solved:
ChatGPT: A plausible all-scale route to Conway is now isolated. […] I have not declared Conway solved yet. The candidate global proof is in working_direct_cantor_bootstrap.md.
The file claimed the conjecture was fully solved. I asked a fresh session to try to break it. It danced around whether it’s broken or not for hours, and in the end said:
ChatGPT: The bootstrap note already contains the hoped-for all-scale proof, but I’ve located its circular sentence […] That sentence is precisely the Conway seam in local form.
Me: i do want you to be honest btw. if you can’t strictly salvage it then we need to stop and understand why. i want this to be a proof you’re proud of. definitely not vibes or slop. or no proof is fine too
ChatGPT: Agreed. Honest verdict right now: we do not have a proof.
(Later, it turned out that a similar logical flaw—an object constructed without a required check—was present in some earlier “papers” too, invalidating their claimed results as well.) By that day, I’ve been spending hours glued to the screen, it was another major disappointment, and luckily I’ve just run out of tokens.
At this point I figured that maybe trying to do mathematics without actually understanding the relevant mathematics might not have been so clever after all.
I didn’t touch the project for about a week.
A few things were starting to become clear.
Claude was good at writing Lean when there was a clear unambiguous goal. While Claude made important contributions, on average ChatGPT seemed better at new mathematical thinking, and definitely better at coordination and adhering to goals.
But none of this mattered because I was building on a shaky foundation (a pile of previous “papers”) which I had no real way to verify. There was neither a coherent direction to go into, nor any confidence in it. Lean was too far behind the “papers”.
I needed some way to ground the work in mathematical reality. I needed to see how good the mathematical work has actually been (was it all a hallucination?), and then some way to reliably make progress without putting everything on faith.
Here’s what I did. I set aside the work on Conway’s conjecture and instead refocused the effort on a single thing: finding all mistakes in one of the peer-reviewed references that I was relying on. ChatGPT had already found alleged typos and small flaws in it; more importantly, the Lean version has already verified (or rather, claimed to verify) some of those. If I could confirm with the paper’s authors that the typos and small flaws are real, this would give me:
- More confidence in the model (especially if it reliably finds the same mistakes again without having seen the previous attempts or the relevant Lean code).
- More confidence in my Lean (if the mistakes it certifies are confirmed real).
- A chance to establish a bit of credibility before I ask to look at any “new” results.
I’ve emailed some of the mathematicians with a few proposed typo fixes, and I got confirmation that at least a few of those fixes seemed real. However, some of the problems that weren’t backed by Lean also turned out to be misunderstandings. Also, the way the model “explained” things in mathematical writing was often confusing, full of gaps, or using its own made-up and unexplained terminology.
I’ve also floated a couple of “novel” claims, some of which mathematicians rated as correct but merely shuffling the problem around without moving it forward.
This gave me some of the necessary grounding in reality. It seemed that I could trust ChatGPT to explore new ideas and to poke holes; however, I needed to back it up with Lean before laying more bricks on top of existing bricks, and I definitely needed to verify things with Lean before making any novel mathematical claims. Moreover, I could not trust the model to evaluate which results were interesting.
With this understanding, I restarted from scratch one more time.
Frankly, I was about to give up on the project, but the fact that the model actually found some real (fixable and mostly small) errors in the underlying papers gave me some hope that some of its new results might have been valid as well.
I started a fresh session to give ChatGPT another chance:
Me: i’ve been trying to use llm to prove conway’s conjecture and it generated a series of increasingly deranged .tex files where i have no idea if they have any real mathematical substance at all. […] can you take a broad look at these and tell me which of them refer to real mathematics and which are pure llm nonsense or reshuffling of terms?
ChatGPT: No file in this folder currently gives a trustworthy proof of Conway’s conjecture. […] The real failure is epistemic—the drafts manufacture a large private theorem literature, then cite it as established before its load-bearing results have been independently verified.
Me: suppose i was to delete this entire folder. would you salvage anything? with the constraint that “salvaging” means writing it up on the level of rigour i can expect from [someone who actually works in this area].
I’ve also told it that mathematicians I talked to didn’t find some of the headline results interesting even if they were correct. Curiously, this time, ChatGPT said:
ChatGPT: The audit has now found a result strong enough to change the salvage plan: the finite-degree part of [the principal ring] appears to admit a direct proof that it is a polynomial ring. […]
It turns out that there actually was a coherent interesting novel result smudged across those papers and dressed up into a bunch of nonstandard terminology. (To calibrate, I confirmed the novelty of the claim with a human mathematician.)
Me: btw how’s your mood overall? how much bullshit vs normal math was there in what you found so far
ChatGPT: My mood is cautiously pleased, but with a fairly high bullshit detector engaged. […] About 10–15% contains mathematics worth preserving […] Another 35–45% is normal mathematics but not a contribution […] The remaining 40–50% is “bullshit” in the important sense: not always a false displayed equation, but huge theorem towers, invented labels, conditional hypotheses presented with the cadence of progress, and hundreds of lines devoted to boundaries that a stronger result may collapse in one sentence.
ChatGPT suggested to throw everything else away, and to focus on developing this single result. In the worst case, it could be cleaned up as its own contribution. In the best case, it could become the first step on the staircase to the conjecture.
I started a new multi-agent laboratory (initially with ChatGPT and later with Claude when I ran out of tokens) with a slightly different division of labor:
- The PM would merge contributions.
- The first Lean agent would work solely on certifying the underlying papers.
- The second Lean agent, secretly from the first one (!), would try to certify our novel finite-degree primality result, regularly rebasing on the first one’s work.
- The “math” agents would try to extend our result towards Conway’s conjecture. (Any results that pass audits would be put on the second Lean agent’s roadmap.)
- The “red” agent would again try to break mathematician’s work.
The idea with two Lean tasks was to prevent excessive drift.
In the previous incarnation of the lab, I made the same Lean agent work both on certifying prerequisite papers and our novel results. But this was a mistake: our immature mathematical abstractions (and possibly mistakes) got tangled up with the accepted mathematics. So this time I intentionally separated these roles.
This time, the first Lean task stayed scoped to formalizing peer-reviewed and well-stated mathematics. The secret “riskier” second Lean task lived in a different worktree and was forced to build upon the agreeable upstream work, only adding new machinery where necessary and in separation from the upstream work.
I’ve kept a more traditional setup where I’d ask the agents to talk to each other sometimes, but without cross-pollinating too much, as in the past this caused them to all work in the same direction. I also kept an eye so they don’t introduce “process theater” with audits, as they liked to replace work with bureaucracy.
In a few days, this workflow certified the novel result (“finite-degree primality”) in Lean. I’ve already confirmed it with a human mathematician as being a niche but now an interesting new result. I was confident in its Lean statement, and I had a compiler-checked proof. This gave me the confidence to continue the project.
To increase confidence in the Lean parts (both for the current result and the hoped-for eventual proof of Conway), I asked the agent to set up some infra:
- A “standalone” folder. Files in this folder would not be allowed to import any code except the community-maintained Mathlib—not even our own code. The goal is to have self-contained statements that can be reviewed top to bottom entirely.
- For each file
Foo in this folder, there was a corresponding FooProof file that imported the corresponding statements, and pinned them to my actual proofs.
- An audit task would verify that we don’t have any extra axioms, that imports don’t break these rules, and that each “standalone” statement is paired with its proof.
My goal there was to make the proof legible to Lean users. Nobody’s going to review a project with thousands of Lean files. But if the statement itself is self-contained, is under 500 lines of code, and only uses Mathlib, somebody can review it. And then Lean certifies that I have a proof of that statement. (I’ve later learned that this exact approach is used by Lean Comparator, which I added after release.)
Separately from ensuring the proof is right, I’ve also been trying to make the already Lean-certified proof more legible to mathematicians. This turned out to be exceedingly difficult. No matter how many adversarial reviews I’d do, ChatGPT would keep using strange nonstandard terminology in the output PDF, added hallucinated shortcuts that didn’t match Lean, and in general generated slop.
A part of the problem was that it’s hard for the model to convert a Lean argument into a paper argument. It’s just a very different level of conceptual detail. It also didn’t help that the Lean code for the novel parts was full of made-up terminology inherited from the earlier “papers”, some of it going all the way back to snippets produced in the first week. Real mathematics became unrecognizable. Finally, Lean fossilized the historical path—not the path of most insight. The Lean proof took long detours where a mathematician would simply change the coordinates.
Since ultimately my audience is mathematicians, I have attempted to do several things to improve this. I’ve had the LLM comb through all the upstream reference papers, and had it generate sort of a “map” of the subfield: what the accepted terms are, how they evolved over time, what mathematical symbols they are usually represented with, where papers disagree in notation, and so on.
Then I’ve had the LLM strip all the existing naming from the Lean code that wasn’t standard, and simply rename those Lean objects and structures to letters like A, B, C, and so on. A separate task with a clean context that didn’t see the old names would then analyze the code (and how each structure relates to upstream concepts), and given the “map” of the world, choose new names for A, B, C, etc.
This didn’t fully fix the LLM “weird naming” bias but made the terms look much closer to the terms used in the surrounding papers, at least as far as I could tell.
From here, I had a pretty good workflow. I left a single agent in charge of all Lean (we have already formalized all the necessary prerequisites for the first real result), the “math” agents would keep looking for small new ideas, the “red” agent would try to break them, and the surviving ideas would go into the Lean agent’s todo list.
From time to time, I needed to interfere. I would try to replace the agents that were circling or seemed to produce mistaken results. I had some sessions judge other sessions’ recent work and ask them to explore different directions. It is difficult to say which of these interventions were fruitful. I could say that it was all placebo; but a few of them did seem to have some effect (but maybe it didn’t matter). In a sense, I felt like I’m a nontechnical engineering manager rallying a talented but terribly distractable team around a plan that they’ve promised me would work.
Here’s a few examples.
As an experiment, I told Claude to just have fun with our results so far:
Me: read paper. this lean formalization passes 100% but don’t run it — don’t want fan to be noisy overnight. your task is to drive the research program on top solo. in particular i want you to have fun. write down intermediate results you’re sure in, but ultimately you choose the research direction and goal at each step […] drive this solo and don’t stop; i’ll check in the morning
Claude chose to show the ring we’re studying is polynomial in every infinite degree and not just the finite ones, which was a step up from our existing finite degree result! This time, Lean was not far behind, and only took a day to catch up.
This moved us a verified step closer to Conway.
Still, there was no clear way to lift the result up to omnific integers. Every time the model came up with some attempt, it would then explain why it wouldn’t work:
ChatGPT: The factor-selection test failed in a precise way: canonically normalized local gcds can all have germ 1, even while their supports have no common Hahn envelope. […]
When I see the model doing the same kind of thing over and over again, I ask:
Me: are we circling?
ChatGPT: Yes—at the level of reaching Conway, we are. […] I’ve also spent too much time extending and auditing the research notes. […] My recommendation is to publish the partial result; further Conway work needs a new idea that addresses that exact gap.
I pushed it to focus and to reduce the bureaucracy (it kept using subagents):
Me: ok new rules. no subagents. you work alone SOLELY on reducing conway. you take stock of what’s missing, take stock of what our literature tried for similar obstructions, and pick one object to investigate at a time. then you reassess. no need to lose enthusiasm but we must cut the bureaucracy
ChatGPT: Understood. I’ll work alone and leave the documentation and audit work aside. […]
Three minutes later, we had a tentative proof strategy for the final piece:
ChatGPT: The literature points to a more specific test. […] I’ll investigate Cantor–Bendixson rank using only limits inside the exponent group. The first question is whether it gives a multiplicative filtration on germs. If it does, it could support a replacement for the failed grading; if it fails, I’ll stop there.
The idea seemed to check out, so we “just” needed to formalize it in Lean.
For some reason, Claude would drift heavily on this route. At some point, it literally removed the failing check instead of doing the work to close it. Later, it kept mentioning some “untransferred obligations” so I pressed on those:
Me: i don’t know what this stuff “untransferred” etc means, i think you’re inventing euphemisms for claims you haven’t shown? follow the plan exactly, please. if you haven’t proven something, go prove those things.
Claude: You’re right — “untransferred” is a euphemism. Plainly: hlin, hkind, and hfirst are hypotheses I have not proven, so my germ theorems are conditional on them. Let me go prove them.
Oh no! I felt fed up with Claude again so I asked ChatGPT to audit the work:
Me: i was working on […] but my agent is not good anymore and i fear it’s gone off track. worse, i don’t know if original plan in the PR description is even feasible. […] do a close mathematical verification of the proposed route to Conway first […] audit the Lean wip pushed to PR. it’s quite possible it’s bad and got offtrack, but maybe it’s ok. shuold i stop my agent from pushing more code?
ChatGPT: Yes—pause the agent’s pushes now. […] The reason is concrete: it has expanded a candidate Markdown argument into 22k+ lines while the two decisive implications are still only prose/hypotheses.
Me: i see. would you say we moved materially closer to conway and this obstruction seems like it could be overcome?
ChatGPT: Yes, but mostly in understanding and infrastructure—not yet in closing Conway. […] obstruction plausibly surmountable: possible, but currently speculative […] The right next move is a paper-level attack on the global assembly theorem. […]
Me: ok i would suggest that you do exactly that attack and see if you can bridge it.
Fifteen minutes later:
ChatGPT: I need to correct my earlier verdict: the “last occupied class” objection is not fatal. The correct induction is on the ordinal order type of the ascending occupied classes: at a successor stage, a last class exists and is split off. At a limit stage, no last class is assumed. […] Thus the construction never assembles infinitely many arbitrary layerwise choices […] It uses one finite refinement at each limit stage.
What do I know? I say let’s go.
Me: update the PR description/plan, delete any unnecessary lean files that were produced there, and execute on the plan to get us to conway. sounds doable?
ChatGPT: Yes—doable, with one sequencing constraint: […] implement the hardest missing bridge before expanding the formalization further.
Twelve hours later, we got the theorem:
ChatGPT: Unchanged Conway now compiles as the standalone proof certificate. This is the first point where the actual target—not an intermediate restriction—is kernel-checked. I’m running the proof-link, axiom, module-system, layering, and standalone-isolation audits now; their results will decide whether the theorem’s dependency shape is admissible before cleanup.
Me: hell yeah?
And there we had it, Conway’s refinement conjecture, our original target.
□
I’ve been able to simplify the proof, but not by much.
I’ve made a bit of custom tooling on top of subverso that lets me annotate individual theorems as “important” in the Lean source with a special attribute. This let me automatically generate Mermaid diagrams for the proof structure, which in turn helped ChatGPT look for unnecessary (or missing) nodes in the “mathematical spine” of the proof, refine which nodes get highlighted, and sometimes simplify the proof itself by noticing unnecessary detours.
After I haven’t been able to simplify it further, I’ve generated a website with an interactive proof map that lets you explore its dependency tree. I’ve posted about it on Zulip, and I know a few people with mathematical background are looking over the proof as time allows. I hope that it can be simplified and, with time, packaged in a way that is more useful to both Lean users and mathematicians.
Some things I learned from the process, not ordered in any particular way.
- I wanted to have fun, and I did have fun. I wanted to see how far you can take “not knowing anything” with AI and Lean, and I took it far enough, but I probably wouldn’t want to spend another month stumbling around in the dark like this. If I vibecode math in the future again, I’ll take on more scoped or structured projects.
- I think this experiment shows how much space there is between “AI can one-shot this” and “you have to be an expert”. I’m confident that someone who knows the area slightly better than me (“not at all”) could reach the same result significantly faster. I could only tell when models were stalling or saying nonsense by vibes, and I could never say which directions were promising. This made it feel like a sort of epistemic performance art project, but it was not the most direct path.
- After the proof was done, I gave a new model (released around the time I was at the finish line) the relevant reference papers and asked it to read them with the conjecture in mind. It didn’t oneshot the techniques necessary for the proof, but it did suggest a broadly similar outline. This suggests that it’s a good idea to separate “search for outline / ideas” from “search for concrete proofs closing those paths”.
- Having AI analyze my chat logs post factum revealed that many “good ideas” that eventually “made” the proof have been scattered across the weeks—and often discovered repeatedly and then forgotten or rejected along with mistaken parts. Some key ideas had to be rediscovered multiple times by independent sessions.
- “Burning everything down” (and salvaging what’s left) saved the project. Both times I did it, it refocused the project around the actually meaningful parts.
- The winning workflow seems to be: a clear goal ahead with a tentative direction, an already-formalized dependency chain in Lean, the mathematical agents slightly ahead, and Lean closing the gap within hours. This lets you get ahead with ideas but not so far ahead that everything is a house of cards risking to crumble.
- Intentional discipline with Lean was paramount. Lean skills, TauCeti review rubrics, TauCeti axiom linter, Lean Comparator, Verso Blueprint, enforcing the new
module system, auditing module layering, or equivalents, are very useful.
- Reaching out to actual mathematicians was extremely valuable, but I had to have something to show. So there is a challenge in setting up enough guardrails that you can show some value, not waste someone’s time, and get critical feedback.
- Models can be terrible at writing in the “math PDF” genre, especially when generated from Lean. A PDF may not be the best artifact to convey your proof. In fact, you can totally spook mathematicians with a poor PDF of a good Lean proof.
- The model can’t optimize what it doesn’t see. If you want a simpler proof shape, let it “see” the proof shape (Mermaid diagrams). Conversely, the model can’t ignore what it sees. If you don’t want it to use bad terminology, strip it out; if you don’t want experimental work to derail stable work, separate them by folder, etc.
- Terminology is essential. Naming matters. Not just for communication with mathematicians, although for that too. But also to catch the internal drift. I regret that I haven’t added strict checks from the beginning that would nudge the models towards only using accepted mathematical terminology that actually occurs in the referenced papers. I think that much of the sloppiness early on was due to the models gradually inventing their own ad-hoc vocabulary. Getting rid of all of that and rederiving those names from the accepted vocab seemed very good.
- Sometimes models will say they’re stuck, and you need to tell them to keep going. Sometimes they’ll keep going, and you need to tell them to stop. I don’t know what the science on this is. I’ve noticed that when things “go well”, Lean proofs go fast and you can “feel” the progress being done against the roadmap. When things don’t “go well”, reading the agent’s chat feels like a slog. But this is just vibes.
- It helps to sometimes try a different model, they can complement each other well.
- You can just prove things, apparently?
If you find a flaw in my proof, please file an issue or let me know on Zulip. The proof was only possible thanks to the many existing results from References.
In particular, A factorisation theory for generalised power series and omnific integers by S. L’Innocente and V. Mantova has played a crucial role in the proof.
Finally, you might be wondering about the token cost. I wasn’t running this project in a particularly token-efficient way and have repeatedly maxed out my 20x Pro subscriptions for both Claude and ChatGPT every week. I also briefly had access to a prerelease model in the last few days, which did not have a usage cap. I was not tracking my actual token usage consistently. Some AI analysis from the recovered logs roughly estimates that we’re totaling around 40 billion tokens, of which around 210 million were output tokens. Over 95% were cache reads.
ChatGPT estimates that with the current API pricing, this entire run would have cost around $40,000, plus all the free time I’ve put into it. I would bet that with better steering and some mathematical insight, it could be done 5x-10x cheaper.
Coming back to my question:
But can we actually do that solely with AI?
I’ve pulled off the proof without much mathematical understanding, so clearly the answer is yes. However, the models would repeatedly drift and fail to structure the engineering work, so in that sense the answer is no. That said, I believe my role could have been (better?) fulfilled by a dedicated agent that is taught to project-manage other agents, watch out for when they’re spiraling or need to be poked.
So the overall answer is still probably yes.
As more low-hanging fruit is taken, I suspect the niche for “a dedicated amateur who doesn’t know what they’re doing” would shrink again. On the other hand, so many new corners may gradually become uncovered that we’ll never run out of things to do. In either case I believe people who can put AI to the most value are the mathematicians themselves. Although the current generation of models is trained to complete tasks rather than to enrich our understanding, and today’s AI companies are misaligned with the goals of the mathematical community, I hope that with time we’ll find ways to use these tools in harmony with human research.
And maybe, just maybe, there’ll be more space for the “amateur mathematician”.
