In the last six month, autoformalization of research-level mathematics has gone from possible to easy. Back in early April, when Julia Kempe and I finished formalizing De Giorgi-Nash-Moser theory in lean, I did not expect autoformalization to progress quickly enough that I would be formalizing my own research articles by the end of the year. I certainly not expect it would be so easy.
In the last 2-3 weeks, with collaborators, I’ve lean formalized the following:
- the Caffarelli-Kohn-Nirenberg theorem on partial regularity of solutions of three-dimensional incompressible Navier–Stokes equations, and existence of Leray weak solutions (roughly 48 hours, 414k lines of code, link)
- the Escauriaza-Seregin-Šverák and Ladyzhenskaya-Prodi-Serrin regularity theorems for the three-dimensional incompressible Navier–Stokes equations (roughly 48 hours, 208k loc, link)
- my (145-page) paper with Vicol on anomalous diffusion by iterated quantitative homogenization (about 36 hours, 270k loc, link)
- the paper by Constantin, Ignatova and Vicol on the regularity of asymptotically axisymmetric solutions to the 3D Navier-Stokes equations with analytic forcing, which shows that the blow-up solution with smooth forcing term constructed by OpenAI cannot be improved to analytic forcing (1-2 days, 87k loc, link)
- the last paper we posted, joint with Kuusi and Loher, on high contrast homogenization (381k loc, link)
- two more papers of mine that will hit arxiv soon, both fully formalized (one of which is more than 1 million loc)
In the words of a great American president, I chose to do these things not because they were hard, but because they were easy.
Recently I have found that the more that I insist that lean autoformalization is easy, the more disbelief I encounter. People think I’m humblebragging or something. So in this post I want to explain exactly how to do it. First of all, I am making my skill files public. These are the files which tell your LLM agents exactly what to do, including how to organize a large lean project. I have put them in a github repo which I will keep updated as things evolve: find it here.
The Setup. You will need either a Claude Pro Max or a ChatGPT Pro account — the $200 version, and ideally both. You will need a github account. Start a new directory and github repository and dump the written mathematical material you want to formalize in there. Make sure the repository is set to “private” and describe to your agent what you want to do. Let’s assume you want to formalize a particular theorem. Then it’s best to have several different accounts of the theorem, for instance several papers and/or books, if available. You will need to install your CLI (i.e. claude code and/or codex), which I would suggest running from a terminal window. You should open it and ask your agent to install my lean skills files (just point it to my repo) the latest version of lean compiler and mathlib. This may take some time.
Tmux. I also strongly suggest using tmux to control your sessions and running them on a server. This allows you to login to your server (using ssh from your laptop) and launch agents, and then close the laptop and walk away and the agents keep running. Again, ask your agent to set this up for you and explain how to use it. I promise it is worth it, but if you don’t want to use this, then you can use the codex or claude code apps, or vscode, or whatever.
Launching the lean project. What I do is open claude code in the directory of my new github repo (inside a tmux shell as usual). I set the agent to Opus 5.5 (xhigh effort). Then I usually make a prompt like this:
You are the orchestrator for our lean formalization project – the goal is to formalize the Caffarelli-Kohn-Nirenberg theorem on the partial regularity of suitable weak solutions of 3d Navier-Stokes. We have placed several papers in the /sources subfolder, these have different proofs of the theorem. We need to find the best proof which is most convenient for formalization. This will be your first task. Please launch a codex agent in yolo mode (in a tmux shell) set to GPT 6 Astra (high effort) to do the research on the best proof, and audit its findings with an Opus (high) subagent. Then have a Sonnet subagent write a detailed MD file (“blueprint.md”) with the proof outline. Let me know if you have questions before you begin.
If you already have a decent paper you want to just formalize and don’t need “blueprint” you can skip this prompt and go to the following.
You are the orchestrator for our lean formalization project – the goal is to formalize the Caffarelli-Kohn-Nirenberg theorem on the partial regularity of suitable weak solutions of 3d Navier-Stokes (based on blueprint.md). You will use up to four Sonnet subagents at a time, and can use up to eight GPT 6 Luna (xhigh effort) subagents in a tmux shell. Audits should be done by Sonnet, and if there is a particularly difficult part of the code to write you can upgrade up to 2 of the Lunas at any time to GPT 6.1 Sol (medium). Please read the lean orchestration skill files and the dependency graph skill files and make a detailed plan to launch the project. Let me know if you have any questions before you begin.
I have both Claude and ChatGPT subscriptions (several in fact), and I use them both to do lean formalization, as you can see. The models are changing very rapidly and which ones I am using for the “subagent/minions” are also changing all the time, but as of this writing I am using mostly Luna and Sonnet for writing the lean. This is because Luna is very cheap, and about 80% of it gets done correctly. The smarter Sonnet and Sol will clean up the problems they cause by being too stupid.
(In this setup I’m allowing up to 12 subagents. To use so many agents you need to make sure you have enough CPUs and memory, because these agents will start compiling lean in parallel. I do this on a server with many CPUs and even one or two large lean projects can throttle the machine. So keep an eye on this, or else use fewer agents.)
What will happen now is the orchestrating agent will manage the construction of a dependency graph – a map of the whole proof in which statements are nodes and ordered edges are the implications. The most important statements – not just the main theorems but also the key propositions and lemmas – will be written in lean and “frozen” so that the agents cannot change it again. This freezing process is elaborate, but automated (the orchestrator will do it, following the skill files). The statements have to be written in lean, audited several times by different agents, etc, and finally they will be presented to you the human for approval. In practice, the agents almost never mess up the statements, but this is the point where you can ensure they do not. Once you “approve” the agents will freeze the statements, and put a “sorry” for the proof. Things are set up in such a way that the only permitted sorries are the ones for the frozen statements.
Once the dependency graph is done and polished, the orchestrating agent will launch subagents to write the proofs. When the proof of a frozen statement is finished, there is a two-key system whereby first the frozen node is “sealed” (the orchestrator checks that yes, replacing the sorry with the proof actually compiles and yes, the theorem statement has not changed) and then re-audited (an agent checks the statement against the latex or md files one more time to makes sure it is faithful) and upgraded to PROVED.
Now you wait until your agent says that all nodes are PROVED. Check back from time to time to make sure nothing is going wrong, and because the orchestrating agent may have questions for you. For instance, maybe they found an error in the informal writing (latex file) and need to tell you that they need to rewrite one of the frozen nodes. You will have to re-approve that. Other things can also go wrong: maybe there is a really hard lemma and a Luna has been struggling for five hours and every other agent is blocked waiting for that, and you should intervene and say “hey please put a Sol (or Sonnet) on that important lemma.”
That’s basically it. Depending on how hard your formalization project is, you now just have to wait. The models are good enough that with the detailed skill files, and the whole project should more or less take care of itself without too much intervention.
Finishing up. Once the agents report that “all frozen nodes are PROVED” you are not yet done. Because most likely your repo is a mess, 20% of the code is “dead code” (meaning it’s lemma fragments that are not used anywhere, the agents do not delete their abortive attempts) and some of your lean code will not compile nicely. So then you have to send the following prompt to clean things up:
Excellent news! Congratulations on finishing the theorem formalization. Now we turn to code cleanup. I want you to first do a dead code sweep, followed by an elaboration test (use the elab test skill file), and elaboration cleanup pass (use the skill file) followed by a second elaboration test. Give me a detailed before/after reports on elaboration.
If you plan to publish your repository, then I would suggest making a public mirror of your private repo and telling the agent that you want to prepare for public release (there is a skill file for that) and your agent will take care of it. There is even a skill file for prepping your repo for Palomar submission.
If I have left anything out, please tell me in the comments!
hi Scott! this is of great help. how does your dependency graph or the blueprint.md file you mention in the first prompt relate to the built-in lean blueprint function that creates a blueprint out of a latex file?
No the blueprint.md is just an example, often I have a latex file for my paper instead. I don’t use the built-in lean blueprint
Hello!
While I agree with much of the sentiment (if not all the details), I think it would be interesting if you could quantify “without too much intervention”. For example, are two questions that I think a python script should be able to answer, for each of the projects above:
1) How many times did the agent ask you for feedback?
2) For each of those, how long (in characters) was a) the agent’s question and b) your response.
It depends on the project. For some projects, the lean proof is based on a very shaky manuscript we are not sure is correct. Then there is a lot of intervention. Caffarelli-Kohn-Nirenberg is a good “model” case. Here is the suggested reply my agent drafted for me to send you in reply to me copy-pasting your comment to him:
Of my 170 messages, 27 concerned the mathematics or the Lean. Only 9 (about 1,500–2,200 characters in all) were my own input that changed direction: adding a force term, pointing to Kukavica’s proof and Lemarié-Rieusset’s book (which fixed the proof route), correcting the agent on Calderón–Zygmund and the parabolic metric, asking for a non-vacuity witness, and having the paper rewritten to follow the formalized route. Ten more just approved its proposals (median 41 characters). Another 26 were about README and publication, and the rest were status checks and agent management.
I attended a PDE talk by Mitchell Taylor at Imperial last week and their message was pretty much exactly the same as yours, except Mitchell did not make his instruction file public, instead he instructed people to email him if they wanted it 🙂 I think I prefer your approach!
Mitchell’s main messages in the talk were: (1) if you have a paper of <= 50 pages with all assumptions being in Lean's maths library then it is easy to formalize in 1 day. (2) For more than 50 pages it's still possible but slightly more convoluted. (3) You do not necessarily need the top models to do everything — e.g. he said that Luna was already good enough to do the formalization. This is really striking because it means that probably in a years' time the open weight models will be good enough to do what you're doing. Are you and Mitchell using the same framework? I'm assuming you're in contact with him!
After the talk I asked Mitchell what the point of formalizing PDE papers was. His answer was that a typical PDE paper might be 10 pages of explaining how the argument works and then 100 pages of gory details checking all the inequalities, and the experts just read the 10 pages and ignore the 100 pages. He envisaged a future where the 10 pages remain but the 100 pages are replaced with "here's the Lean proof, if you want to see the details then just translate it back into English using an LLM (or learn Lean ;-)" ).
He also told me the following interesting story: he was experimenting with an LLM and it claimed to resolve a big problem in the field. Mitchell could see that the argument was never going to work but the machine was confident. Mitchell then piped everything into his framework and Lean isolated the problem as an error in one of the references which the LLM had quoted.
Interesting – I don’t know Mitchell Taylor! I think his vision about 10 pages and gory details hidden in a lean repo is exactly what I think and hope will happen.
I have a recent story kind of like what you describe in your last paragraph. One of the projects I describe above (the one with Amélie and Tuomo) was one in which lean formalization was not something that came after we found the proof. The proof was first written in lean, and the actual paper was written after. So here it was really used a research tool. I will maybe write a longer post about that if I can find the time. The very brief version is that we had a much longer and more convoluted proof attempt that we’d been working on for more than a year, which wasn’t quite right. Attempts to lean formalize it helped us narrow down where the gap was in each of the five different versions of the proof attempt we had. By comparing these, we somehow really understood more fundamentally what we were missing. Then we were able to close to gap, using a combination of human brains and agents – and we only really knew we had it when the lean was finished. Then we spent more than a month improving the proof and writing the paper (and then eventually redoing the proof in lean to follow the simplified proof).
Actually — correction — I do know Mitchell Taylor! Upon googling and seeing his picture, I realize I recently met him at Courant and we talked about lean together with Jaume de Dios Pont. (I apparently did not properly learn his surname when I met him.)
Yes, we are absolutely on the same page.
One potentially very useful thing to mention is GPT’s new Dots feature for pro users. In this month., it appears to provide roughly 2 billion tokens per day, which changes the scale of what is practical. You can essentially give it a substantial formalization task and let it work through it over a long run(Although human checking become impractical.). At that budget, it seems entirely realistic to formalize a whole textbook within a month, rather than just individual theorems or short papers. Based on my own experiments, this looks like a genuinely different workflow for large-scale formalization.
Thank you very much for sharing, this post motivated me to start learning Lean and to autoformalize. I’m amazed at how easy it is!