From an empty AWS account to a resident's question answered by our own model, in nine steps. Written from a real attempt, including the two things that stopped it.
This takes you from an AWS account with nothing in it to a resident's question answered by a model on hardware we control, with the application falling back to Gemini whenever that hardware is off. It is one ordered list. Follow it top to bottom.
Every step below was carried out for real in the client's own account on 30 August 2026. Where something failed, the failure is written down here rather than smoothed over, because two of them will happen to anybody who repeats this.
The single most important sentence in this document: Ollama has no authentication of any kind. Anything that can reach port 11434 can use the model, read what is being asked and run the bill up. Section 3 is not optional and it is not last.
What you need in front of you:
| Thing | Why |
|---|---|
| An AWS account on a paid plan | Section 1 explains why the Free Tier plan blocks this outright |
| An access key with EC2 rights | To make the machine and, later, to start and stop it |
| The public addresses of the application machines | They are the only addresses allowed near the model |
| Python with boto3 | The scripts under deploy/gpu/ use it |
Both of these are account level, both take time, and neither can be worked around by anybody except the account owner. Start them first and do the rest while they clear.
A new AWS account is on the Free Tier plan, which is not the same thing as the free tier allowance. On that plan, launching anything outside a small list of types is refused outright:
InvalidParameterCombination: The specified instance type is not
eligible for Free Tier.On the account this was tried in, the only launchable types were t3.micro, t3.small, t4g.micro, t4g.small, c7i-flex.large and m7i-flex.large. No GPU type is on that list, so no quota increase can help until the account is moved to a paid plan in Billing. This surprises people because the error names the instance type, and the instance type is not the problem.
Separately, a new account has a quota of 0 running on-demand G instances. A g6.xlarge needs 4 vCPUs of it. Request the increase in Service Quotas, in the region you will actually use, against "Running On-Demand G and VT instances". Approval is usually hours and can be two days.
$ .venv/bin/python deploy/gpu/quota.py
current 0
requested 4
status PENDINGBoth must clear. Clearing one and not the other looks identical from the command line, which is why they are listed as two steps rather than one.
It does not have to be in the same account as the applications, and in this case it is not: the applications live in one account and the GPU in another. That rules out private networking between them, so the model is reached over the public internet and the firewall in section 3 is the only thing in front of it.
Before the model is installed, not after. A machine with Ollama running and an open 11434 is a free GPU for whoever scans for it, and they scan continuously.
| Port | From | Never |
|---|---|---|
| 11434 | The application addresses, as /32 | 0.0.0.0/0 |
| 22 | Your own address, as /32 | 0.0.0.0/0 |
| Anything else | Nobody | Ever |
security group sg-06ad9397831595b43 ollama-gpu
11434 from 35.91.251.211/32, 54.188.207.85/32, 54.254.25.0/32
22 from the operator's own address
and nothing elseThree addresses because there are three agents and each one asks the model directly. Adding a fourth agent means adding a fourth rule, which is a small piece of friction worth keeping.
If the user data ran, this is already done and the check at the end of the section confirms it. By hand:
curl -fsSL https://ollama.com/install.sh | sh
systemctl enable --now ollamaThen pull the model through the HTTP API, not through the command line:
curl -s http://127.0.0.1:11434/api/pull \
-d '{"name": "llama3.1:8b"}'This is the bug that cost an afternoon. Running ollama pull from cloud-init dies with panic: $HOME is not defined: the CLI reads $HOME to find its model directory and cloud-init has none. The first unattended build finished with Ollama running and no model in it, which from the outside is indistinguishable from a download that failed. The HTTP API has no such dependency. The idle check in section 5 had the same problem and was moved to the API for the same reason.
The model must be the one the application is configured for:
$ curl -s localhost:11434/api/tags
{"models":[{"name":"llama3.1:8b", ...}]}Two mechanisms, because they fail differently. This is the whole difference between roughly $580 a month and roughly $8.
| Timer on the machine | Lambda outside it | |
|---|---|---|
| Knows when the last question was | Yes, from Ollama's journal | No, only CPU |
| Survives the machine wedging | No | Yes |
| Fires at | 20 idle minutes | 45 minutes running under 5% CPU |
The on-box timer does the day to day work and is precise. The Lambda catches the machine that has hung, or that somebody started from the console and forgot, and that is the failure that actually costs money, because a timer on a hung machine does not run.
systemctl list-timers ollama-idle-off.timer
sudo /usr/local/bin/ollama-idle-off.sh # says why it is or is not stoppingFour modules, and they are deliberately small. Nothing else in the application knows a GPU exists.
| Module | Its one job |
|---|---|
| ai_runtime | Which engine is switched on. A file, not a database row, so it survives a restart and is readable by eye |
| gpu_instance | Where the machine is and whether it is answering. Starts and stops it |
| ollama_service | Talks to the model |
| llm | The router. Everything that wants a sentence written asks this and nothing else |
The routing is three lines and the third one is the design:
switch says gemini -> Gemini
switch says gpu, and the GPU is ready -> Ollama, on our own hardware
switch says gpu, and it is not -> Gemini, and the panel says soA resident does not care whose hardware answered and must never see a broken assistant because a machine was still booting. What matters is that the panel tells the truth, so nobody demonstrates "our own GPU" to a room while Gemini is quietly doing the work. The panel therefore reports two separate facts: what is switched on, and what actually answered the last question.
By instance id, which is the normal one. The public address changes every time the machine stops and starts, so it is read from the AWS API rather than written down:
GPU_INSTANCE_ID=i-...
AWS_REGION=us-west-2
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...By hand, which is for a model at a fixed address, such as one on a tunnel or a laptop:
OLLAMA_URL=http://10.8.0.1:11434OLLAMA_URL did not work until 30 August 2026. The readiness check returned "not configured" whenever AWS credentials were absent, so a perfectly good model at a hand written address was never asked, and every question quietly went to Gemini. Both the state and the health check now short-circuit on OLLAMA_URL before they look at AWS at all, with four tests holding it there.
Set one or the other, never both. Then restart the application.
In this order. Each one can fail on its own and each failure means something different.
| What you see | What it is |
|---|---|
| Launch refused, naming the instance type | Almost certainly the Free Tier plan, not the type and not the quota. Section 1 |
| Ollama running, no model in it | The $HOME panic in cloud-init. Pull through the HTTP API. Section 4 |
| Everything answered by Gemini, no error anywhere | The application does not think the GPU is ready. Check the switch, then the health check, then the firewall from the application machine itself |
| Answers time out then arrive from Gemini | Working as designed. If it happens on a real GPU rather than a stand-in, the model is too big for the card |
| The machine is gone rather than stopped | Shutdown behaviour was left at terminate. It cannot be undone. Section 2 |
| A bill nobody expected | A machine started from the console and forgotten, with the timer wedged. That is what the Lambda is for. Section 5 |
| State | Roughly |
|---|---|
| Running | $0.80 an hour |
| Stopped | $8 a month, for the disk |
| Left running all month | $580 |
The entire gap between the second row and the third is the auto-off, which is why it has two mechanisms rather than one. Compute is not billed while an instance is stopped; the volume is, whether it runs or not.
Written by Abad Naseer. Every measured number in this document is cited from the running system or from the code, and every target says that it is a target. Where the two disagree, the document says so rather than choosing the flattering one.