The question now comes up in roughly every second conversation: could we not run this ourselves? What people mean are the open language models you can download and run on your own hardware instead of sending every request off to a provider. The answer is uncomfortable, because it starts with a calculation rather than a conviction.
What self-hosting actually involves
Running an open model is no longer a feat of engineering. You need a machine with suitable graphics hardware, either in your own server room or rented from a data centre, plus a runtime and an interface your applications can talk to. A first attempt fits into one afternoon. That is exactly the trap. Setting it up is the cheap part.
What follows is operations. Hardware fails. Drivers want updating. A new model appears and your prompts suddenly behave differently. Someone has to notice when the service stops answering at three in the morning while a batch job is hammering it. None of this is hard. All of it is work, and the work never ends. Anyone whose first conversation is only about model quality and never about who is on call has not touched the expensive part yet.
The case in favour
The strongest argument is data sovereignty. If your engineering drawings, patient records or draft contracts never leave the building, an entire category of questions disappears from your conversations with clients, works councils and auditors. Not because the risk drops to zero, but because it sits with you.
The second argument is cost control at high, steady volume. If you classify millions of text fragments or push large document batches through a model every day, a provider charges you for every single request. Your own machine costs money once and electricity thereafter. Above a certain load the maths flips, and that point sits lower than most people assume.
The third argument gets overlooked: independence. A provider can change prices, retire a model or adjust its behaviour without asking you. If a core business process hangs on it, that is concentration risk in its purest form. A model sitting on your hardware behaves tomorrow the way it behaves today.
The case against
The strongest models right now are not the open ones. The gap is narrowing, but it exists, and you feel it the moment a task gets genuinely difficult. If you need the best available quality for demanding reasoning, you will find it at a provider and not on your own card.
Then there is utilisation. Graphics hardware is expensive and it does not get cheaper while it sits idle. A model that works two hours a day and waits for the other twenty two is a very elegant way to burn money. Providers bill per request, and for uneven load that is simply the better structure.
And there is the factor nobody puts in the quote: the person who looks after it. If the knowledge about your model setup lives in one colleague's head, you do not have an infrastructure problem, you have a staffing problem, and it will surface precisely when that colleague is on holiday.
A usable rule of thumb
If your usage is occasional and irregular, you are better off with a provider. Full stop. The effort for operations, maintenance and failover bears no relation to a few hundred requests a month. If you process sensitive data continuously and in volume, then it is worth doing the maths honestly: hardware, power, redundancy and person days per month all belong in the calculation, not just the purchase price.
In practice, the middle route is often the best one. Sensitive tasks run locally, uncritical ones go to a provider, and the line between them is drawn cleanly and written down. Draft contracts stay in house, the phrasing of a newsletter may travel. This does assume your application was built with interchangeable models from the start. Nail yourself to one vendor interface and you no longer have the choice later. That makes it a question of architecture rather than procurement.
An expensive misunderstanding
Self-hosting is not automatically compliant with data protection law. I hear that equation often and it is wrong. If you feed a model in your own building with personal data, you still need a legal basis, a deletion concept, access controls and an answer to the question of what ends up in your log files. Running the model yourself moves the responsibility onto you. It does not dissolve it. It removes a processor from the chain, which is a relief and not an absolution.
Frequently asked questions
What hardware do we need to start?
That depends on the model size and the response time you want. For first experiments with smaller open models, a single graphics card in a rented server is often enough. Before you buy anything, rent for a week and measure how many requests you really generate.
Can we switch between self-hosting and a provider later?
Yes, if the application allows for it. Model access belongs behind an interface of your own, so that swapping stays a matter of configuration rather than a rebuild.
Does this pay off for a mid-sized company at all?
Sometimes. It depends on volume and on how sensitive the data is. Run the numbers with figures from your own business rather than the example calculations from conference talks, and include the ongoing care and feeding.
If you would like to walk through that calculation with someone who knows both routes and has no reason to sell you either, get in touch, with no obligation and no sales pressure.
This article is part of our knowledge hub Custom software development.