Models

Overview

• The Docker image ships with zero models — only code and manifests
• A Docker volume (sirene-models) persists models between restarts
• Models are downloaded on demand from the web interface
• The Python service downloads into the volume and lazy-loads into GPU memory

Manifest System

Each model has a JSON manifest embedded in the manifests/ directory that describes its files, download URLs, and variants.

Example manifest:

{
  "name": "qwen3-tts",
  "description": "Qwen3-TTS by Alibaba",
  "backend": "qwen",
  "license": "Apache-2.0",
  "variants": {
    "1.7B": {
      "files": ["model.safetensors", "config.json", "tokenizer.json"],
      "url": "https://huggingface.co/Qwen/Qwen3-TTS/resolve/main/",
      "size": 3400000000
    },
    "0.6B": {
      "files": ["model.safetensors", "config.json", "tokenizer.json"],
      "url": "https://huggingface.co/Qwen/Qwen3-TTS-0.6B/resolve/main/",
      "size": 1200000000
    }
  },
  "defaults": {
    "variant": "1.7B",
    "sample_rate": 24000
  }
}

Download Flow

1. User clicks [Install] on a model in the web UI
2. Client opens an SSE connection to GET /api/models/:id/pull
3. Hono server:
   a. Stores state in memory (activeDownloads Map)
   b. Downloads files from HuggingFace into /data/models/
   c. Sends progress via SSE to the client (progress, complete, error)
4. Client updates the React Query cache with progress
5. fs.watch() detects changes and notifies via GET /api/models/events (SSE)

Storage Layout

/data/models/
├── kokoro/
│   └── v1.0/
│       ├── model.onnx
│       └── config.json
├── qwen3-tts/
│   └── 1.7B/
│       ├── model.safetensors
│       ├── config.json
│       └── tokenizer.json
├── f5tts/
│   └── default/
│       ├── model.safetensors
│       └── vocab.txt
└── piper/
    └── fr_FR-siwis-medium/
        ├── model.onnx
        └── model.onnx.json

Memory Management

The inference service manages GPU memory with:

• Lazy loading — Models are loaded only when first used
• LRU eviction — When INFERENCE_MAX_LOADED_MODELS is reached, the least recently used model is unloaded
• Manual unload — Models can be explicitly unloaded via POST /models/unload