Self-Host AI Models Locally with Ollama and vLLM
Self-Hosting AI Models Locally is an advanced-level course of 5 lessons (about 1 h 40 min), with 15 exercises and 15 quiz questions. Prerequisites: Docker and Deployment. The course needs a full project environment, planned for a later phase of the platform.
- Generative AI
- Advanced
- 5 lessons
- 1 h 40 min
- 15 exercises
- Full project environment (phase 2)
The course in brief
This course teaches you to serve open language models on your own infrastructure. You run and query Ollama, place vLLM in context, choose a model and its quantization, estimate GPU memory and measure throughput, then design a "local first, cloud as a fallback" gateway and secure the server. Exercises run in a real Linux terminal, with a few calculations in Python.
What you will be able to do
- Run, query, and configure a model with Ollama, and place vLLM in context
- Choose an open model based on the task, language, license, and quantization
- Estimate the VRAM required (weights, attention cache, headroom) and measure throughput
- Design a local-first gateway with controlled escalation, a spending cap, and fallback
- Secure a model server: network exposure, authentication, model provenance
Syllabus: 5 lessons
- Explain why and when to host a model yourself
- Download, run, and query a model with Ollama
- Place vLLM relative to Ollama and know its OpenAI-compatible API
- 02
Choosing a Model and Its Quantization
- List the criteria for choosing an open model
- Explain quantization and its effect on size and quality
- Estimate the size of a model's weights based on its precision
- 03
VRAM and Throughput
- Estimate the GPU memory required: weights, attention cache, and margin
- Measure throughput in tokens per second from the server's responses
- Relate context, simultaneous requests, and memory
GPU memory (VRAM)Attention cache (KV cache)Throughput and latency
- 04
A "Local First, Cloud as a Fallback" Gateway
- Design a single gateway for all model calls
- Write a local-first routing policy with controlled escalation
- Plan for anonymization, a spending cap, fallback, and logging
- 05
Securing a Model Server
- Limit the network exposure of a model server
- Control the provenance and format of the models you load
- Protect the service against abuse and data leaks
Final project
Deploy a local model behind a secure gateway
Use Docker Compose to deploy an Ollama server and a small gateway (Python or PHP) that exposes a single chat route. The model server is reachable only by the gateway. The gateway applies the "local first" policy (with a cloud provider simulated by a dummy service), a spending cap in fictitious units, anonymization of outputs to the cloud, an explicit fallback, and logging of every call. Include a VRAM sizing note for the model targeted in production.
Deliverables
- The compose.yml file (internal network, no model port published) and the versioned Modelfile
- The gateway code: routing, health check, anonymization, cap, fallback
- A call log showing at least one local call, one escalation, and one fallback
- A script that measures throughput from the server's responses, with its results on your machine
- A sizing note: calculation of the weights, the attention cache, and the margin for the production target
Assessment criteria
- The model server does not listen on a public interface (checked with ss or docker port)
- Tasks marked local are never escalated and the cap is respected
- No secret is present in the repository
- Throughput measurements come from your own run, with the calculation method explained
- The VRAM calculation details weights, cache, and margin and states its assumptions
Frequently asked questions
Do I need a GPU to take this course?
No. The exercises use a small model that runs on CPU in the online terminal. The course explains how to size a GPU server, but you can practice everything without a graphics card.
Does the course provide model performance benchmarks?
No. Performance depends on the hardware, the model, the quantization, and the load. The course teaches you to measure it yourself from the server's responses.
Ollama or vLLM: which one should I learn?
Both. Ollama is the simplest way to get started and suits small workloads; vLLM is designed to serve many simultaneous requests on a GPU. Thanks to their OpenAI-compatible API, you can switch from one to the other without rewriting the application.
What are the prerequisites?
Being comfortable with the terminal and Docker (course B4). Basic Python helps for the memory calculations.
This course, told for your job
Say what you want to be able to do: “Self-Hosting AI Models Locally” takes its place in your path, with examples from your field. The concepts stay those of the course.