Technical Report
Date:
This article was written by
Consultant, Offensive Security
Consultant, Offensive Security
AI models, particularly so-called Large Language Models (LLMs), have attracted significant attention in recent years through providers such as OpenAI and Anthropic. Today, they are used in numerous application areas, for example as tools for text processing or automation. However, two key challenges stand in the way of their practical use: ongoing usage costs associated with proprietary providers (e.g., OpenAI or Anthropic) and data protection concerns when integrating such external AI services. One possible solution to these challenges is to self-host AI services (LLM instances).
This insight provides an introductory overview of what is required to operate an LLM locally and highlights essential aspects that should be taken into consideration.
If a company decides to use LLMs, the first decision that must be made is which AI model to use (Model Selection). One option is the use of proprietary models, in which case self-hosting is not possible. Proprietary LLMs are exclusively owned by their respective developers. Neither the training data nor model properties such as model weights, which determine how a model processes inputs and generates outputs, are publicly accessible. Since developers of proprietary models often have significantly greater resources available for development and training, these LLMs generally offer higher performance than open-source models. Proprietary LLMs are typically provided as Software as a Service (SaaS). They are operated by the provider within its cloud infrastructure, meaning that access is exclusively provided through the respective provider's commercial API.
For the user, this means that both the operation of the model and the processing of the transmitted data take place outside their own infrastructure. This results in high usage costs and data protection restrictions, as all data submitted to such a model could potentially be reused by the provider for training new models or for other purposes that may compromise data privacy.
An alternative to this is self-hosting LLMs. In this approach, the model is run on one’s own infrastructure, which not only eliminates dependence on a single provider but also allows for significantly greater flexibility, since models can be adapted later for specific use cases (fine-tuning).
Models that can be used for self-hosting are available either as open-source or open-weight models. Unlike open-weight models, open-source models provide (additionally to the model weights) the training code and the dataset used for training, or complete details about the composition of the dataset if the dataset itself cannot be made available. This increases the transparency and comprehensibility of the model and makes it easier to assess its functionality and potential bias.
For the practical use of an LLM, not only the characteristics and development of the model are important, but also its deployment. A trained LLM can only be used in practice if it is deployed appropriately.
But what exactly does it mean to deploy an LLM?
The execution of an LLM is also referred to as “inference” in the technical literature. The term describes an LLM’s ability to process user inputs, known as prompts, and subsequently generate a response. Put simply, it refers to the moment when the trained model applies its knowledge. Inference plays a central role for LLMs, as factors such as response speed, memory requirements, and computational power depend to a large extent on how efficiently this process is implemented.
To run an LLM locally, appropriate inference software is required to receive requests and send them to the model. This software typically provides server-like functionality that allows the LLM to be managed via an API. This includes, for example, API endpoints designed for inference tasks such as text generation and tokenization. Some of the most popular projects include “vLLM” and “llama.cpp”. For simple local use, the “Ollama” project is also frequently used. Ollama is based on llama.cpp and wraps its functionality, providing not only model management features but also a simple initial setup and straightforward operation.
Although it may seem that the model and inference software cover all the necessary components for running an LLM locally, one fundamental component has so far been missing from the picture: the hardware. Without suitable computational resources, the selected LLM may either be impossible to run or may only operate with limited performance.
In principle, an LLM can be run in two different ways:
With the CPU-based approach, the LLM is loaded into the computer’s main memory at runtime, and the subsequent inference is performed entirely by the CPU. The main advantage of this approach is its lower hardware cost, as no dedicated GPU is required and RAM is generally less expensive than comparable VRAM. However, CPUs are not optimized for the highly parallel computations involved in processing LLMs. LLMs rely heavily on matrix operations, which can be parallelized very efficiently. GPUs are specifically designed for this type of computation, which is why GPU-based inference can provide significantly higher inference performance in many use cases.
Using a GPU therefore offers significant performance advantages but requires additional investment in suitable hardware. A particularly important factor is the amount of available VRAM. If it is insufficient to store the entire model, parts of the model must be continuously transferred between the GPU and RAM. This process significantly reduces inference speed.
Model size is typically measured by the number of parameters in a model. Parameters are the values learned during training (weights) that determine how a model processes inputs and generates outputs. Modern Large Language Models usually have billions of parameters. As the number of parameters increases, both memory requirements and, generally, model performance tend to increase. Larger models can therefore often handle more demanding tasks and capture more complex relationships. However, due to their increased memory requirements, they are often less suitable for local operation on conventional hardware. Moreover, a high number of parameters does not automatically guarantee better model performance, as this also depends on factors such as the quality of the training data, the model architecture, and the quantization used.
Hardware requirements depend on the LLM being used. In particular, the model size determines how much system memory or video memory is required for inference. Since LLMs are evolving at an extremely rapid pace, listing specific models at this point would not be practical. Instead, various open-source and open-weight model families have become established in recent years, including:
Model Family Name | Developer |
Gemma | |
LLaMA | Meta |
Nemotron | NVIDIA |
Qwen | Alibaba Cloud |
GLM | Z.ai |
Kimi | Moonshot AI |
In practice, it is advisable to additionally plan for a memory reserve of around 10–15% so that temporary data structures and runtime data can fit into the RAM/VRAM alongside the model weights.
A system with 16 GB of RAM and 16 GB of VRAM is available. The current model in the Gemma model family, Gemma 4, is to be run entirely on the GPU using the commonly used “Q4_K_M” quantization. With this configuration, Gemma 4 is primarily available in the following parameter sizes: 12B, 26B, and 31B (“B” ≙ “billion”). Depending on the model size, the VRAM size which is needed is approximately 7.6 GB, 18 GB, and 20 GB, respectively. Therefore, only the 12B variant is suitable for the system under consideration, as it is the only one that can be loaded entirely into the available VRAM.
Self-hosting LLMs essentially requires three components:
Outlook
With the fundamentals presented in this Insight, it should now be clear which components are required for self-hosting a Large Language Model and what role they play within the overall system. We will demonstrate what such a self-hosting setup can look like in practice in a separate Insight. There, we will explain step by step how an LLM can be set up and run locally using a suitable LLM backend and made available for your own applications.