The demo answered one question beautifully. Then the whole team logged in.
That is where local LLM capacity stops being a hardware conversation and becomes a business decision. A machine that impresses one person might leave everyone else watching a spinning circle. Your receptionist has customers waiting. Your sales team has quotes to finish.
At UltiMedia, we recommend sizing around the work people must complete. Start with the task, measure the busiest realistic period and agree what happens when demand exceeds supply. The graphics card comes later.
This practical article uses Ollama as a concrete example for South African teams considering shared local inference. We checked its official documentation on 27 October 2026. The test plan and fictional scenarios below are recommendations, not client results or hardware benchmarks.
Prefer to watch? See the three tests before buying a local AI server.
Watch the full local AI series
What does local LLM capacity mean?
Local LLM capacity means how much useful work your own model server can handle within agreed waiting times. Assess the model, document length, simultaneous requests, available memory and recovery behaviour together. Measure complete tasks under realistic demand before buying hardware or promising staff a dependable shared service.
1. Count work arriving together, not employee seats
Twenty employees do not necessarily mean twenty simultaneous requests. Equally, three colleagues running document-heavy jobs could create your busiest period. Headcount gives you a guest list. It does not tell you how crowded the kitchen gets.
Build your local LLM capacity brief from a small sample of ordinary work. Record when requests arrive, what each person submits, how long the answer needs to be and whether the task can wait. Use synthetic examples where real material contains sensitive information.
- Separate quick lookups from long summaries.
- Include the morning rush, when several colleagues may open the same tool before customer calls.
- Count automated jobs too.
- Identify which tasks can run after hours and which need an answer while somebody waits at the counter.
For local LLM capacity, write down a service target for each task. A proposed target might require a short internal answer within a chosen number of seconds. That number should come from your workflow. We offer no universal response-time promise.
2. Make document length part of the specification
A short question can carry a long payload. The application might attach conversation history, retrieved documents and instructions before the model starts answering. Ask the builder to show the complete input rather than only the sentence visible in the chat box.
Ollama defines context length as the token capacity available to the model and explains that increasing it requires more memory. Its context-length documentation also shows how to inspect allocated context and processor placement.
For local LLM capacity planning, keep three representative inputs: short, ordinary and unusually long. Pair each with a realistic answer length. Test the awkward case where a colleague requests a lengthy summary just as another needs a quick product answer.
Do not copy a default context value from an old installation note. The FAQ and dedicated context page describe defaults differently. Record the actual allocation on your installed version, then test that configuration. A setting you can inspect beats a number you merely assumed.
3. Verify the hardware doing the work
A local LLM capacity quote should identify the exact machine configuration, operating system, runtime version and model version. Ask the supplier to demonstrate the intended setup before you treat a successful installation as a performance result.
Ollama publishes hardware support by platform and GPU family, including driver requirements. Check the proposed configuration against its current hardware support documentation. A supported device still needs a workload test; compatibility alone establishes no business throughput.
Your local LLM capacity record should capture what runs during the test. Ollama’s running-models endpoint returns loaded models, with model identity, memory information and context details in its documented response. Save those details alongside the results.
Then ask a plain question: did we test the machine we intend to buy? A demonstration on a supplier’s larger workstation does not answer that. Neither does a screenshot of a model responding on somebody’s laptop. Keep the raw results with the quote.
4. Separate a cold start from steady service
First impressions can mislead in both directions. A cold model may spend time loading. A model already sitting in memory may answer differently from the same model after an idle period. Your local LLM capacity test should describe which condition produced each measurement.
Ollama exposes a keep_alive parameter for controlling model residency. Its FAQ also describes preloading with an empty request. These controls affect the conditions you test; they do not guarantee a particular response time. See Ollama’s model-loading and concurrency FAQ.
Measure local LLM capacity after a fresh start, during repeated use and after the idle period your configuration permits. Include a switch between models if your application uses more than one. Record the sequence so another operator can repeat it.
Give staff an honest interface. If the model needs to start, say so. If a request waits, show that state. We recommend a visible acknowledgement and a clear cancellation route instead of a silent screen that encourages repeated clicks.
5. Measure the whole wait, then diagnose its parts
The useful clock starts when the colleague submits a task and stops when they can act on the result. Model generation measures only part of that journey. Your application may also search documents, prepare the request and render the answer.
Ollama’s generation API documents total duration, model-load duration, prompt processing and output generation counts and durations. Use these fields to investigate the model-serving portion, alongside timing from the surrounding application.
For local LLM capacity, keep the measurements separate. Record end-to-end completion time, time until useful output appears, failures and whether the answer meets the task. A fast incomplete summary does not count as successful work.
Review the slow cases. An average can make an unpleasant queue look tidy, especially when many short requests hide a few very long waits. Keep individual results and explain which request types caused trouble. Do not turn a tiny pilot into a statistical certainty.
6. Test a busy period without hiding the queue
Run the same task mix with progressively more requests arriving together. Start gently in an isolated test environment. Keep the model, settings and input set fixed while you change demand, otherwise you will struggle to explain the result.
Ollama documents memory-dependent concurrent processing and notes that parallel requests increase context-related memory needs. It also describes queue limits and an overload response with HTTP status 503. Read the concurrency and queue sections before changing those settings.
A local LLM capacity test should show accepted, waiting, completed, cancelled and failed requests separately. Increasing a queue limit does not prove the machine completes more useful work. It may simply allow more people to wait.
Agree an overload policy before launch. We recommend a clear busy message, a controlled retry option and a route to the original manual process. Keep repeated automated attempts bounded. A struggling server does not need enthusiastic software hammering its door.
7. Turn measurements into a purchase decision
Consider a fictional Cape Town distributor. Staff need brief product lookups during calls, while an administrator prepares long supplier summaries. The proposed pilot tests both jobs together, then repeats the exercise with summaries outside the busy period.
This local LLM capacity comparison asks whether scheduling solves the bottleneck before the business buys more equipment. It does not assume that it will. If lookup performance remains unacceptable, test a revised configuration against the same inputs and agreed quality checks.
Compare local LLM capacity costs in rand using actual quotes. Include equipment, installation, support, electricity assumptions, replacement planning and staff review time. State the measurement period and workload. Avoid claiming savings merely because a local runtime does not meter every request like a hosted API.
Keep answer quality as a separate gate. Our AI agent supervisor article explains the ownership role. For this decision, name the person who approves task usefulness and the person who maintains the server.
8. Leave the operator a repeatable handover
Write a short local LLM capacity operating sheet. Include the approved workload, configuration, expected waiting behaviour, escalation contact and retest triggers. Record how staff report a slow answer without copying confidential content into an unrestricted support channel.
When performance changes, inspect evidence before changing settings. Ollama’s troubleshooting documentation identifies log locations for different platforms and describes GPU discovery problems. Capture the relevant diagnostic information for the person maintaining your deployment.
Recheck local LLM capacity after a model change, runtime update, major document-length change or increase in simultaneous demand. Use the same baseline tasks first. Then add the new workload, so you can distinguish a regression from an expanded requirement.
Also rehearse a restart and a planned interruption. For a South African office, include the actual network and power arrangements in that exercise. Treat resilience as something to demonstrate on your installation, not something the word local automatically provides.
Local LLM capacity FAQs
How many users can one local server support?
There is no reliable answer from employee count alone. Measure simultaneous requests, input size, output length and the exact model configuration. Agree acceptable waiting times for each task, then test realistic busy periods. Report the tested workload and conditions instead of promising an unlimited number of users.
Will a bigger GPU fix every slow answer?
No. First measure where the time goes. A request may wait for document search, model loading, generation or another application component. Inspect the running configuration and compare repeatable tests before buying hardware. Faster generation cannot repair every delay elsewhere in the workflow or guarantee a useful answer.
Should we increase the queue limit?
Only after deciding how long people can reasonably wait and what happens when demand exceeds service. A larger queue changes waiting capacity, not proof of completed work. Test the user experience, cancellation behaviour and retry policy together, then choose settings that match the business task and support plan.
When should we repeat the capacity test?
Repeat it after changes that could alter the workload or execution conditions, including model updates, longer inputs and more simultaneous requests. Keep an unchanged baseline for comparison. Retest failed cases, record the configuration and let the task owner decide whether the new results still meet the agreed service target.
Buy evidence before you buy a bigger box
Good local LLM capacity planning gives you a defensible decision: a tested workload, visible limits and an owner who knows what to do when demand grows. The goal is useful work at an acceptable cost, not a server specification that looks impressive in a proposal.
UltiMedia is a Cape Town-based AI-first digital agency offering local and private LLM deployment and integration. Bring one real workflow, its busiest period and your waiting-time expectations. Discuss a local LLM deployment with UltiMedia and build the pilot around that evidence.