↓ Skip to main content
wiki.wompmacho.com

Self-hosted AI

·307 words·2 mins·
Ai Ollama Continue.dev Open-Webui Self-Hosted Gemma
Michael
Author
Michael
some dude that works on datacenters, plays guitar, streams, has a lot of side projects and unhealthy addiction to ow

Homelab AI Infrastructure Overview
#

This document outlines the current self-hosted Artificial Intelligence infrastructure, detailing how models are hosted, accessed, and utilized across different interfaces within the homelab environment.

Core Inference Engine: Ollama
#

The backbone of the AI setup is driven by Ollama, which handles the actual model inference and API routing.

  • Host Environment: Dedicated Gaming PC. This machine provides the necessary GPU compute power and VRAM to run large language models efficiently.
  • Network Address: http://10.0.0.109:11434
  • Active Models:
    • gemma4:26b (Heavy): The primary model used for complex reasoning, comprehensive chat, and applying structural code edits.
    • gemma4:e4b (Fast): A smaller, optimized model specifically dedicated to low-latency tasks like real-time code autocomplete.

Developer Integration: Continue.dev
#

For software development, the AI is integrated directly into the coding environment, turning the IDE into an AI-powered workspace.

  • Environment: VS Code running via code-server on the Linux host.
  • Extension: Continue.dev
  • Routing: The extension is configured via ~/.continue/config.yaml to offload all inference to the remote Ollama instance on the Gaming PC.
  • Autonomous Capabilities:
    • The setup is enhanced with the Model Context Protocol (MCP).
    • A local shell-mcp-server runs on the code-server host, exposing the execute_shell_command tool.
    • Combined with aggressive system prompting, this allows the gemma4:26b model to autonomously execute Linux commands, read outputs, and debug the local workspace without requiring the developer to manually run commands.

General User Interface: Open WebUI
#

For general-purpose queries, document analysis, and conversational AI outside of the IDE, a dedicated web interface is utilized.

  • Hosting: Deployed as a Docker container within the homelab server infrastructure (data stored in /srv/open-webui).
  • Integration: Natively connected to the Ollama API on the Gaming PC over the local network (10.0.0.109).
  • Features:
    • Provides a polished, ChatGPT-like experience.
    • Allows users to interact seamlessly with the Gemma models.
    • Supports persistent chat histories and file uploads (via the /srv/open-webui/uploads and vector_db volumes) for Retrieval-Augmented Generation (RAG) capabilities.