Founder Note
The Depreciation of Intelligence
How an $87,000 frontier AI model job fell to $1,000
I was about to run a data extraction job through Claude Sonnet when the estimate came back - $87,000. I almost had a heart attack.
The job sounds simple. We take financial text and extract who said what, what they actually claimed, and how certain the source was. But those extractions become facts inside the system. One quiet mistake does not remain one bad model answer. Everything downstream begins treating it as true.
So we could not solve the $87,000 problem by simply choosing a worse model. Sonnet was good at the job. It was also a general-purpose intelligence capable of doing an enormous number of things we would never ask it to do. Our job kept asking for the same narrow judgment, over and over again.
Could we pay only for the intelligence we used?
Our first answer was distillation. We would use Sonnet as the teacher, train a much smaller student for our specific task, and run that student at a fraction of the inference cost. The student only needed to learn this one job under this one quality standard. Everything else Sonnet could do was outside the assignment.
The modeled inference cost fell from $87,000 to roughly $2,000. That number came with a different bill attached. We would have to build the dataset, train the model, serve it, evaluate it, retrain it when the task changed, and maintain the surrounding infrastructure ourselves.
We built the dataset and trained the first baseline adapter. Then we hit the first hard limit of distillation. A student trained only on the teacher's answers will inherit the teacher's mistakes. We designed a separate corrective layer so that the student could eventually learn the cases Sonnet itself got wrong.
Then we ran six models against the same job. DeepSeek came surprisingly close to Sonnet. It was less complete on several difficult cases, but it still preserved the underlying meaning. On the failures that could quietly corrupt the knowledge layer, DeepSeek and Sonnet initially failed in the same places.
The first full-job estimate for DeepSeek was roughly $3,500. That was dramatically cheaper than Sonnet and still more expensive than the modeled $2,000 distilled route. Price alone could not settle the decision. DeepSeek appeared to fail the cases we cared about most.
At that point, the evidence appeared to strengthen the case for distillation. If no rented model could make the judgment correctly, we would have to change the weights ourselves. We prepared the next training run.
Before we fired it, we asked a cheaper question. Had we actually proven that the failure lived in the model weights?
We had tested one version of the instructions and mistaken that result for the model's capability. We wrote several new versions of the rubric. Some added examples. Others made the governing rule more explicit. Then we ran them against held-out failures and a separate set of clear cases.
One revision that made the rule explicit recovered most of the failed cases. We tested it again on fresh held-out cases. The clear cases remained intact. Adding examples produced almost no additional improvement.
We stopped the training run.
The cognition was in the rubric. DeepSeek had been capable of making the judgment. We had failed to specify that judgment precisely enough to reach it.
That changed the economic decision. The difference between DeepSeek and the model we thought we had to build was sitting inside our rubric. Changing the rubric was cheaper than retraining a model.
Only then did the operational advantage become decisive. We could rent DeepSeek directly and avoid the engineering required to train, serve, evaluate, and maintain another model.
Then we optimized the route itself. We removed an unnecessary reasoning tax, selected the right provider path, and priced the same full job again. The estimate fell from roughly $3,500 to roughly $1,000.
The numbers now described four different decisions. $87,000 bought Sonnet for every call. Roughly $2,000 bought a distilled student and left us to train, serve, evaluate, and maintain it. Roughly $3,500 bought DeepSeek directly before we corrected the rubric and route. Roughly $1,000 bought the same full job after we did.
We parked distillation. Scale, ownership, sovereignty, or a future capability gap may bring it back. For the workload in front of us, it had not earned the engineering cost.
Before we fine-tune or distill another model to repair a failure, we will ask a different question. Have we proven that the failure lives in the weights? Vary the rubric. Isolate the judgment. Test the change on fresh failures and clear controls. Retrain only after the rubric space fails.
Provider choice still includes data residency, reliability, batch execution, rate limits, and operational support. Once a cheaper model clears those constraints and the quality bar, the frontier model has to justify its price on the work that remains.
A frontier model can be more capable across almost every benchmark and still lose the workload. Customers purchase a result under a cost, quality, and operating constraint. The cheaper model gets the bulk. The frontier model gets the hard residual.
From the customer's side, intelligence depreciates as the next substitute becomes capable enough. A task that required frontier pricing can move into a cheaper model. A failure that appeared to require new weights can disappear after a better instruction. A workload that justified a large API bill can move down the cost curve without changing the required result.
We began with an $87,000 estimate and a plan to train our own model. We ended with the same quality requirement, a roughly $1,000 estimate, and no second training run. The $87,000 estimate had changed meaning. We had priced broad general-purpose intelligence on every call for a job that repeatedly used one narrow judgment.
No weights changed.
We changed the instructions.
