How I self-hosted an AI chat on my portfolio (and why I stopped)

There is a chat on this site called Ask Tungi. You can ask it about my projects, how I work, what I think about product problems. Under the hood, a retrieval pipeline pulls in project context, and a language model responds in my voice.

Most people would call an API and be done. I did that too, eventually. But first I spent a few weeks running my own model on a dedicated GPU, and the detour taught me more than the destination did.

No per-token billing: you pay for the compute you provision, however much or little it produces. Full control over the model. Open weights, so you can fine-tune later. And your data stays on your infrastructure. After years of shipping under GDPR and KRITIS, that last part matters to me.

I found a GPU platform that wraps dedicated machines behind a standard chat completions API. You pick a model, pick a machine, deploy, and you get an endpoint. I had a working chat streaming responses within an hour.

Then the problems started.

Your model size decides your GPU. A 7B parameter model runs fine on an A10G with 24GB VRAM. Go bigger (13B, 70B) and you need A100s or H100s, which cost serious money per hour.

The platform had serverless GPUs that scale to zero when nobody is chatting. Sounds ideal for a portfolio site. But when the first message comes in after idle, the model has to load back into VRAM, and that takes ten to thirty seconds.

My site gets single-digit visitors a day, so every first message hits a cold GPU. At that traffic, the cold start is the whole experience.

Keeping a GPU always on fixes that and runs fifty to two hundred dollars a month, which I could not justify for a portfolio project.

In this setup you get two of the three.

A good model on a fast GPU gives you quality and speed, and costs more than my domain and hosting combined. A good model on a serverless GPU gives you quality and low cost, plus fifteen-second cold starts; the visitor stares at a spinner and some close the tab. A small model on a serverless GPU is fast and cheap, and the answers get worse: shorter, more generic, bad at following a detailed system prompt.

I sat in the middle one for a while, quality and cheap, and every session started with a fifteen-second wait. I had built the feature to show how I think about products, and it was making a poor case.

Part of the reason I self-hosted was fine-tuning. The plan: take an open-weight model, train it on how I write, and get a chat that sounds like me without needing a giant system prompt. I started putting together the training data. Q&A pairs, tone examples, guardrails.

Then I did the maths on maintenance.

Training a model from scratch is not realistic for one person. You need compute, data pipelines, months of work. Fine-tuning is more accessible, but it ages badly. Models get replaced every six to twelve months, and when the next generation comes out, your fine-tuned weights sit on old architecture. Every cycle you retrain, re-evaluate and redeploy.

A good system prompt works on any model. A retrieval pipeline that pulls in the right project context doesn't care what generated the response. And a solid eval suite tells you within minutes whether a new model still holds up; mine runs sixty-plus tests for voice, guardrails and injection resistance.

So I built the eval suite instead of the fine-tune. When I need to swap models, I change an environment variable, run the tests, and ship. I would rather invest in that than in weights I will throw away in a year.

There are cases where fine-tuning is the right call. Specialised reasoning, unusual output formats, edge cases that no prompt covers. For a conversational chat with a clear personality, prompting and retrieval got me there without the overhead.

When I moved off the GPU, every security layer came with me unchanged. Prompt injection detection for jailbreak attempts. Rate limiting through Redis, twenty requests per hour per IP. CSRF origin checks. Input sanitisation with length limits. A stateless edge function: the rate limiter keys on the IP, the request log keeps a truncated one, no message text is stored on the server, and the conversation itself lives in your browser tab until you close it.

All of that sits upstream of the model call. It works the same whether the model lives on a GPU I rent or behind a third-party API, because the security boundary belongs to the application, and the provider is a detail behind it.

I moved to a hosted API with a small, fast model: sub-second time to first token, no cold starts, and better errors. I could now tell apart a bad key, a rate limit, an upstream outage and a timeout, where before I got one generic "sleeping" message for everything.

At my traffic, under a hundred messages a month, API costs are less than a dollar. The GPU setup was free when idle, but ran fifteen to twenty dollars once anyone used it. Plus the cold start on every session.

If I had thousands of conversations a day, or needed strict data residency, the maths would go the other way. Self-hosting wins when token costs pile up and you need control the API does not offer. I was not in that situation.

Once the infrastructure decision was made, the interesting work started.

The retrieval pipeline grew into the part I'm proudest of. I moved from simple keyword matching to TF-IDF vectors with cosine similarity: each project and case study chunk gets vectorised at build time, and incoming queries are matched against them. I added query expansion so synonyms and related terms don't miss relevant chunks. The similarity threshold took several rounds of tuning: too aggressive and starter questions like "what was your hardest project?" returned nothing; too loose and unrelated chunks polluted the context.

The voice work was harder. I rewrote the system prompt several times before the tone felt right: confident without boasting, specific to my own projects, and able to stay in character across a long conversation without slipping into generic consultant speak. The eval suite caught every regression when I changed the prompt.

The feature I'm most glad I added is document upload. You can drop in a job description or a project brief, and the chat will map my experience against it: specific projects, relevant skills, a match score if it's a JD. That only works because the retrieval is grounded in the case studies themselves, and the vector index is what makes it specific.

Start with the API and ship the feature. The hard problems in an AI chat are elsewhere: making retrieval good enough that responses are grounded, writing a prompt that holds a consistent voice, building tests that catch when quality drops, and keeping the security boundary tight between user input and model output.

Self-host when you have a concrete reason: a token bill that hurts, a data-residency requirement, a model the APIs don't offer.

The GPU was a rental. The retrieval pipeline, the system prompt and the eval suite are what I still use, and they run on any backend.