For fifteen years, software had a magic property that every other business envied: once the product was built, serving one more customer cost almost nothing. That is the whole reason classic SaaS commanded 80 to 90 percent gross margins and the valuations that come with them. The code was written once. Each new signup was almost pure margin. Cost of goods sold, the boring line that dominates a manufacturing business, barely registered.
AI products do not get to inherit that property, and pretending otherwise is how a promising company discovers, a year in, that its unit economics do not work. When your product calls a model on every request, you have a real, recurring cost of goods sold again. The AI SaaS gross margin that results is structurally lower than the software margins your investors and your instincts were trained on.
Every request costs money now
The uncomfortable fact underneath this is simple. Serving a cached web page costs effectively nothing. Running a language model over a user's request costs real money, every single time, because tokens are metered and inference is compute you are paying for on the spot.
That changes the shape of the business. In classic SaaS, cost was mostly fixed: build the thing, run some servers, and margin improves as you add customers. In an AI product, a meaningful chunk of cost is variable and scales directly with usage. The heaviest users, the ones you are most excited to land, are also the ones running up the largest inference bill. Growth and cost rise together in a way software people are not used to.
Public analyses of AI companies keep landing in the same range: gross margins around 50 to 60 percent, against the 80 to 90 percent that defined the SaaS era. The exact figure moves with model prices and product design, so treat it as a shape rather than a number and verify current rates for your own stack. But the direction is not in dispute. Inference is a cost of goods sold, and it does not disappear as you scale. If anything, it grows as a share of spend as usage deepens.
Why this matters more than it sounds
A lower gross margin is not just a smaller number on a slide. It reshapes three decisions.
Pricing. Flat, unlimited, all-you-can-eat pricing is dangerous when every unit of use costs you money. A single power user can turn a profitable plan into a loss. Pricing has to cover the variable cost of the usage it permits, which is why so much AI pricing has moved toward usage components rather than pure flat seats. If you want the full treatment, see how to price an AI product.
Fundraising. Investors have caught on. The question is no longer just growth. It is gross margin and the path to improving it. A pitch that assumes software-grade margins on an inference-heavy product will get the polite version of a no. Knowing your cost per unit of value delivered, and having a credible story for driving it down, is now table stakes.
What you build. When COGS is real, engineering effort spent lowering cost per request is not overhead. It is margin. The team that treats inference cost as a first-class metric ships a more valuable company than the one that treats it as someone else's problem.
The levers that move the margin
The good news is that AI COGS is not fixed the way a raw material cost is. It is a design variable, and there are well understood ways to move it.
Model routing. Not every request needs your most expensive model. Sending routine work to a cheaper model and reserving the premium one for genuinely hard work can cut a large share of cost while keeping quality where it matters. This is often the single biggest lever, covered in model routing.
Caching. When a large, stable chunk of context repeats across calls, paying full price for it every time is waste. Prompt caching charges the repeated part at a steep discount, which is close to free money for any product with a large fixed context.
Batching. Work that does not need an instant answer can run through a batch path at a substantial discount. Anything analytical or offline is a candidate.
Bring your own model. The most direct move of all is to not carry the inference cost yourself. In a bring-your-own-key arrangement, the customer pays their model provider directly and you never touch the token bill. That takes your variable inference cost close to zero and changes the margin conversation entirely, which is the subject of bring your own model. It is not free of trade-offs, but it is the cleanest answer to the COGS problem when it fits your product.
Stack these together and the 50 to 60 percent starting point can improve materially. The companies that take margin seriously are not accepting inference cost as fate. They are engineering it down.
The honest limitation
Margin is not the only thing that matters, and optimizing it too early can be its own mistake. Early on, you often want to eat some COGS to learn fast and win customers, and that is a reasonable choice made deliberately rather than by accident. Bring-your-own-key shifts the cost off your books but also changes your value proposition and your control over the experience, so it is not a universal answer. Caching and routing carry real engineering cost, and a router that sends work to a model that is not quite good enough trades a visible bill for an invisible quality tax. And every number here moves with model prices, which have only trended down, so today's math is not tomorrow's.
The point is not that AI businesses are bad businesses. It is that they are a different kind of business than the SaaS playbook assumes, and the founders who internalize that early make better pricing, hiring, and architecture decisions than the ones who discover it in a board meeting.
This is a stance we built into Loopsfinity from the start: the customer brings their own model, so inference is billed to them directly and we do not carry it or mark it up. We would rather have an honest, defensible margin than a story that falls apart the first time someone runs the numbers. Whatever you are building, the discipline is the same. Treat inference as the cost of goods it is, measure it, and design it down.