taco/glados-ladosp-tts
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fixed a lot of things and actually got it to work
All checks were successful
Build and Publish Docker Images / build-cuda (push) Successful in 6m12s
Details
Build and Publish Docker Images / build-rocm (push) Successful in 6m33s
Details
Build and Publish Docker Images / build-cpu (push) Successful in 18m51s
Details

Browse Source
This commit is contained in:
xerotacovix
2026-06-13 09:56:48 +00:00
parent a0ff64629c
commit 4ddf29aeb7
14 changed files with 215 additions and 137 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
Dockerfile
View File

@@ -12,6 +12,7 @@ COPY download_model.py .
COPY entrypoint.sh . COPY entrypoint.sh .
RUN pip install --no-cache-dir -r requirements.txt && \ RUN pip install --no-cache-dir -r requirements.txt && \
python -c "import nltk; nltk.download('averaged_perceptron_tagger_eng'); nltk.download('cmudict')" && \
chmod +x entrypoint.sh chmod +x entrypoint.sh
RUN mkdir -p /data RUN mkdir -p /data

3
Dockerfile.cuda
View File

@@ -7,7 +7,8 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
&& rm -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/*
COPY requirements.txt . COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt RUN pip install --no-cache-dir -r requirements.txt && \
python -c "import nltk; nltk.download('averaged_perceptron_tagger_eng'); nltk.download('cmudict')"
COPY wyoming_glados/ ./wyoming_glados/ COPY wyoming_glados/ ./wyoming_glados/
COPY download_model.py . COPY download_model.py .

7
Dockerfile.rocm
View File

@@ -1,5 +1,9 @@
FROM rocm/pytorch:rocm7.1.1_ubuntu22.04_py3.11_pytorch_release_2.10.0 FROM rocm/pytorch:rocm7.1.1_ubuntu22.04_py3.11_pytorch_release_2.10.0
# MIOpen optimizations: cache convolution solver solutions instead of re-solving on every inference
ENV MIOPEN_FIND_MODE=1
ENV MIOPEN_DEBUG_CONV_GEMM=0
WORKDIR /app WORKDIR /app
RUN apt-get update && apt-get install -y --no-install-recommends \ RUN apt-get update && apt-get install -y --no-install-recommends \
@@ -7,7 +11,8 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
&& rm -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/*
COPY requirements.txt . COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt RUN pip install --no-cache-dir -r requirements.txt && \
python -c "import nltk; nltk.download('averaged_perceptron_tagger_eng'); nltk.download('cmudict')"
COPY wyoming_glados/ ./wyoming_glados/ COPY wyoming_glados/ ./wyoming_glados/
COPY download_model.py . COPY download_model.py .

80
HOMEASSISTANT_INTEGRATION.md
View File

@@ -1,62 +1,48 @@
# Home Assistant GLaDOS TTS Integration Guide
# HomeAssistant GLaDOS TTS Server Integration Guide (Wyoming + Standard HTTP Endpoints) This server uses the [Wyoming protocol](https://www.home-assistant.io/integrations/wyoming/) for TTS, which is natively supported by Home Assistant.
## This README covers running Style-Bert-VITS2 models on AMD RX9060 XT using ROCm Docker, plus exactly which URLs/connectivity options work when setting up Wyoming protocol
### 1. Basic Setup: Running PyTorch RoCm TTS Inference in Container ## Adding to Home Assistant
Run this from your host system with an AMD GPU (RDNA4 architecture like the RX9060 XT) connected to motherboard or PCIe device that works with amdgpu kernel drivers when building pytorch image locally without NVIDIA proprietary CUDA emulation layer:
```bash ### Option 1: Wyoming Integration (Recommended)
cd /path/to/docker-compose-config-files \ && # Replace this variable as needed for your own docker compose directory location
doker build -t ladosp-tys rocm --file Dockerfile.laDos-tts-rocm # Use `--build-start-restart` flag if you want PyTorch ROCM to validate GPU is present before starting server inference session
docker-compose up --detach ### Runs container at background port 8529 in detached mode for easy HomeAssistant connectivity across network
```
**What this command does**: Builds a custom Docker image from scratch using official PyTorch RoCm wheels that support RDNA4 hardware, then starts TSS server with automatic model download from HuggingFace Hub repository (https://huggingface.co/WarriorMama777/GLaDOS_TYS). Home Assistant has a built-in Wyoming integration that auto-discovers Wyoming servers on the network.
### 2. Understanding Endpoint URLs After Start 1. Go to **Settings > Devices & Services > Add Integration**
When container is running successfully as defined in docker-compose.yaml above: 2. Search for **Wyoming** and select it
- **Standard endpoint path**: `/v1/audio/speech?text=hello+world` - use this URL directly with HomeAssistant tts-server card (or any standard OpenAI/TTS server client) 3. If the server is on the same host as Home Assistant, use `localhost` with the appropriate port:
\- Full web-based access at `http://your-device-ip-or-localhost:${API_PORT:-8529}/v1/audio/speech`. Replace IP address or hostname from your Docker host machine. - CUDA: `10200`
- CPU: `10201`
- ROCm: `10202`
4. If running on a different machine, use the server's IP address
- **Wyoming Protocol endpoint**: `/wyoming/audio/stream?session_id=YOUR_SESSSIONID&text=hello+world` - use only if you're using a Wyoming-specific Lovelace card that supports session-based UDP streaming over HTTP instead of simpler text-to-speech generation pattern ### Option 2: Manual configuration.yaml
### 3. Setting Up HomeAssistant TTS Server Card (Recommended)
Add this YAML to your `configuration.yaml`:
```yaml ```yaml
# Standard TSS integration for GLaDOS-TYS ROCm server using openai-compatible API patterns # Wyoming protocol TTS
tts_server: wyoming:
- url: http://your-docker-host-ip:8529/v1/speech # Replace with actual docker host IP (like 10.46.X.Y or local LAN gateway address) - host: 192.168.1.100 # Replace with your server IP
type: tts-server ### Use "tys" as a custom component instead of standard pytorch_rocm if you prefer that naming convention for Lovelace cards port: 10200 # 10200=CUDA, 10201=CPU, 10202=ROCm
default_2 :
name: Portal GLaDOS VITS (Wyoming-Compatible) # Friendly voice model identifier displayed to HA users in UI
url: http://10.46.x.53/v1/speech ### Use your own Docker host IP address or domain if running behind NAT/proxy
``` ```
For simple HTTP audio responses without protocol-specific metadata headers (recommended for most cases): add standard `lovelace-card` integration that calls `/v1/audio/speech?text=<your text>` directly. See Lovelace card documentation at https://www.home-assistant.io/integrations/tts/ or similar HA web resources if you encounter specific errors ## Port Reference
### 4 Wyoming Protocol Integration (If Required) | Variant | Container Port | Host Port |
Only use this approach if your HomeAssistant integration specifically requests UDP-style session management rather than standard HTTP streaming: |---------|---------------|-----------|
- Ensure proper authentication credentials exist for model download by setting `HF_TOKEN=your-huggingface-personal-token` in `.environment-example` file before running `docker-compose up --build-start-restart`. The Wyoming protocol will automatically handle audio encoding and metadata when client connects to `/wyoming/audio/stream?text=<input>` path, using ROCm inference backend as standard for any style-Bert-VITS2 model | CUDA | 10200 | 10200 |
| CPU | 10200 | 10201 |
| ROCm | 10200 | 10202 |
### 5 Troubleshooting Common Issues When Running on RDNA4 GPU ## Voice Configuration
**ERROR: Device cannot be accessed from ROCm runtime**: Verify that your system kernel driver has AMD graphics working by running `lspci | grep -i amdgpu`, and if device detection fails, try rebooting into a Linux kernel update or using standard NVIDIA/ROCM emulation with Docker's `-device=nvidia` flag when building PyTorch image locally (not required for most users as ROCM will work automatically)
**ERROR: Torch backend not loaded**: Make sure your `amdgpu` driver module is active on boot, then restart container and run custom build command to verify GPU detection before starting TIS inference sessions with standard CUDA emulator fallback or pure PyTorch RoCm device routing. This typically resolves by updating Linux kernel from Debian/Ubuntu repositories The server exposes a single voice:
- **Name**: `glados`
- **Languages**: English (`en`), Japanese (`ja`), Chinese (`zh`)
- Language is auto-detected from input text
### 6 Additional Performance Tips for Production Deployment ## Troubleshooting
If running large Style-Bert-VITS2 models like GLaDOS_TYS which use >8GB VRAM (your RDNA4 GPU should have at least this available under standard ROCm emulation):
- Set environment variable `MAX_JOBS=3` to limit concurrent inference sessions and prevent system memory overflow when training voice clones or running batch synthesis jobs
- Use port redirection in Docker Compose file by uncommenting the following line if your HomeAssistant network cannot access high-numbered ports like 8529: \`ports:\n - "127504:8529"\`
### Summary of Critical Files You Should Have Now - Ensure the container is running: `docker compose ps`
- Check logs: `docker compose logs glados-tts-cuda` (or cpu/rocm)
| File name | Purpose when running docker compose build | - The model downloads automatically on first start; this may take several minutes
|-----------------------------|----------------------------------------------- - Verify network connectivity between Home Assistant and the TTS server
| `docker-compose.yaml` (main file) | Defines container runtime with ROCm GPU, port 8529 bindings and automatic HuggingFace model downloads on startup. Mounts models at /root/app/models directory inside PyTorch RoCm docker image.
### Final Note: If You Need Additional Help
For questions about Wyoming protocol specifications (which are documented elsewhere in the official HA forum or Lovelace card developer tools), open a separate GitHub issue instead of relying solely on this readme which is designed specifically for ROCm AMD GPU inference running GLaDOS_TTS model from HuggingFace Hub repository. See next example files below that contain detailed troubleshooting steps if you encounter errors when loading Style-Bert-VITS2 models after successfully starting PyTorch RoCm backend
**This file should now be complete and functional with your system ROCm driver installation.**
HOMEOF && wc -L HOME-INST-GUIDE.md || true

59
README-GLaDOS-TYS-Wyoming-and-ROCM.md Normal file
View File

@@ -0,0 +1,59 @@
# GLaDOS TTS - ROCm Deployment (AMD GPU)
This guide covers running the Wyoming TTS server with AMD GPU acceleration via ROCm.
## Prerequisites
- Linux host with an AMD GPU supported by ROCm
- ROCm kernel drivers installed (`amdgpu` module loaded)
- Docker with `amdgpu` device support
Verify ROCm is available on the host:
```bash
ls /dev/kfd /dev/dri
rocm-smi # if ROCm tools are installed
```
## Running with docker-compose
```bash
docker compose up glados-tts-rocm
```
This uses the `Dockerfile.rocm` image with:
- PyTorch ROCm backend (device mapped to `cuda` internally)
- Half-precision (`--half`) enabled for ~2x speedup
- Model pre-loaded at startup (`--preload`)
- MIOpen convolution solver caching for faster repeated inference
## Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `DEVICE` | `rocm` | Set to `rocm` for AMD GPU |
| `HALF` | `true` | Use float16 inference |
| `PRELOAD` | `true` | Load model at startup |
| `MODEL_DIR` | `/data` | Model storage directory |
| `URI` | `tcp://0.0.0.0:10200` | Wyoming server URI |
ROCm-specific optimizations are applied automatically:
- `MIOPEN_FIND_MODE=1` - cache convolution solver solutions
- `MIOPEN_DEBUG_CONV_GEMM=0` - default GEMM mode
## Device Mapping
The server maps `rocm` to `cuda` internally since PyTorch ROCm uses the CUDA API compatibility layer. No code changes are needed.
## Troubleshooting
**No GPU detected in container logs**:
```bash
# Verify host has ROCm devices
ls -la /dev/kfd /dev/dri
# Check if amdgpu module is loaded
lsmod | grep amdgpu
```
**Out of memory**: Reduce `HALF=false` or ensure sufficient GPU VRAM (model requires ~4GB).

75
README-GlaDOS-TYS-Wyoming-and-ROCM.md
View File

@@ -1,75 +0,0 @@
# GLaDOs TTS Server - Running Portal_GLaDOS_v1 on AMD RX9060 XT RDNA4 via ROCm Docker
```
## Overview
This server package lets you run the **Portal\_GlaDos-v1** voice cloning model (based on Style-Bert-VITS2 architecture from HuggingFace) using PyTorch ROCM backend for inference acceleration. The setup works with standard TTS Server protocol and optionally supports Wyoming Audio streaming if your HomeAssistant integration component requires that UDP-style transport layer over HTTP or pure WebSocket connections.
**Model repository:** https://huggingface.co/WarriorMama777/GLaDOS_TTS/tree/main/Models/Style-Bert_VITS2/Portal_GLaDOS_v1
\
## Key Features of this Setup
- **AMD RDNA4 GPU acceleration**: Uses PyTorch ROCm instead of standard CUDA - works with AMD 9060 XT or any newer Radeon architecture GPUs when running Docker on Linux x86_64 systems
- \**Multiple Protocol endpoints for HomeAssistant**: Supports both: Standard Wyoming-style audio streaming (uses UDP-based session management but falls back to pure HTTP if RDNA4 hardware doesn't have full ROCM driver support) AND standard OpenAI-compatible style-TTS endpoint (`/v1/audio/speech`) used by many HA tts integrations
- **Graceful fallback**: If GPU fails or model weights cannot be loaded, the server automatically switches between CPU and alternative PyTorch inference backends without crashing - essential for production deployments with user hardware that isn't NVIDIA-based in standard Linux environments
## Prerequisite System Requirements (Before starting docker-compose up)
You will need:
- An AMD RX9060 XT or other newer RDNA4+ architecture GPU card connected to your motherboard and enabled by system BIOS when building PyTorch ROCM images with Docker's `--device nvidia` flag set in compose runtime configuration - though this is not strictly necessary if using the standard AMDCUDA emulation layer provided directly by AMD open source project
- **PyTorch ROCm**: Standard installation available from official NVIDIA or HuggingFace Docker Hub (you'll likely need to install `nvidia-driver`, `amdgpu-pro` on Ubuntu 24.04 LTS, and then run standard ROCM PyTorch wheel building commands with `--build-start`)
- **ROCm backend** is automatically detected when your Linux host has a valid AMD GPU driver installed - no need to manually specify ROCM version or CUDA-style emulation flags in most HomeAssistant-compatible Docker containers unless you're using NVIDIA devices for hybrid inference purposes
## Running the Server on Your Hardware
The example commands below start up PyTorch with standard ROCm device detection and load GLaDOS_TYS model from HuggingFace Hub repository. If your host already has proper GPU drivers (which it should - otherwise AMD system tools will fail to initialize in `sudo apt install amdgpu-pro` or equivalent package manager command) , you can run directly:
```bash
docker compose build && docker compose up --build # Standard setup when ROCm backend is available
# Alternative if device isn't recognized but CUDA-style drivers are present (Nvidia emulation with AMD hardware):
doker-compose -f Dockerfile.laDos-tys-rocm .env.example-u15429780-AMD-RDNA4 build # Use your own environment file from .env directory for custom ROCM flags
```
Replace `--start` if you want to skip the PyTorch image rebuild phase. The server will download model files and start inference automatically - but first, verify that GPU detection worked properly by checking container logs:
**docker logs ladosp-tys-rocm --follow** ### Follow any output until your TTS session begins or error condition occurs on AMD RDNA4 hardware
## Integration with Home Assistant
HomeAssistant provides several integration mechanisms for external tts servers - here's the easiest approach to configure using either Wyoming protocol OR standard HTTP streaming endpoints:
### Option A Using Standard "TtS Server" Custom Component (Recommended)
The `lovelace-tts-server` or similar Lovelace card can connect directly to an exposed PyTorch ROCM endpoint that returns audio for any voice model. Add the following configuration in your HA YAML files:
```yaml # Example HomeAssistant tts integration using standard OpenAI-compatible TTS server endpoints with proper fallback handling
default:
name: GLadOS Voice (Portal Style-Bert-VITS2)
url: http://your-amd-gpu-IP-or-DNS-name:8529/v1/speech ### Set URL where ROCm inference is running - use local network hostname or IP address of your system that serves the PyTorch ROCM TTS endpoint
type: tts-server ## Use standard tts-server protocol, not Wyoming
# If you prefer Wyoming-style session connection instead for advanced audio routing:
default_2:
name: GLaDOS_v1_with_Wyoming
url: http://192.168.X.YZ:8529/wyoming/audio/stream # Use `/wyoming` paths if your client supports UDP-style session streaming, otherwise stick with Option A above which is compatible
model_url: https://huggingface.co/WarriorMama777/GLaDOS_TTS/tree/main/Models/Style-Bert_VITS2/Portal_GLaDOS_v1
```
### Configuring Wyoming Protocol Integration Directly (Advanced)
If your HomeAssistant setup uses the official `wyoming-tts` custom integration component or a Lovelace card that explicitly requires protocol-specific session headers:
- Set the **Wyoming endpoint format** as either HTTP stream response at `/vyoming/audio/stream`, standard TCP-style websocket over UDP, or similar transport layer if client documentation specifies one of these options
- The server supports both streaming and synchronous request/response patterns - check your Lovelace card's integration requirements before using pure WebSockets (not recommended unless necessary for low-latency audio playback on HA)
### Alternative: Using "TTS Stream" with PyTorch Backend Directly
Instead of using standard TTS Server protocols, you can also expose raw `pydantic-settings` or openai-style requests directly to HomeAssistant's built-in tts server configuration - use the `/v1/speech` endpoint as a generic HTTP audio response source:
```yaml # Custom config for your HA Lovelace card that works with any standard PyTorch ROCM inference backend
tts_server_url: "http://0.0.0.0:${API_PORT:-8529}/v1/speech" ### Use port parameter from environment when running docker-compose -f Dockerfile.laDos-tys-rocm build
default_voice_model_id: portal\_gladios_v1 # Default voice ID for PyTorch ROCM generation (will auto-map this model to standard inference output path if it's not already cached by huggingface_hub)
### Troubleshooting Common Issues when Running on RDNA4 / RX9060 XT Hardware
**ERROR: Device cannot be accessed from ROCm runtime**: Check that your system BIOS has Radeon GPU enabled and device permissions are set (`sudo lspci | grep -i amdgpu`, `nvidia-smi` if using nvidia driver, or AMD equivalent)
- **Torch backend detection fails:** Install proper PyTorch RoCm wheel when building from Docker Hub ROCM collection by running standard NVIDIA command that's documented in their GPU driver troubleshooting guide for RDNA4 architecture (you'll need latest `nvidia-driver`, amdgpu-pro` package, and system-level CUDA-style emulation stack)
- **Model weights cannot be loaded:** Verify your HuggingFace tokens are correct if using custom private repositories - standard GLaDOS_TTS model files can fail to load on AMD GPUs with ROCM unless you have proper authentication set up in container environment variable (see .env.example file for detailed guidance): `HF_TOKEN="your-huggingface-token", HF_AUTH_TYPE=basic`
### Additional Configuration Notes
- **GPU driver installation:** On Linux systems, the standard AMDCUDA emulation stack uses device detection and CUDA-style wrapper drivers that are automatically detected by Docker when you run with `-device nvidia`. For full ROCm support without any external NVIDIA software or AMD open source packages from HuggingFace, install `amdgpu-pro` or equivalent proprietary driver on Debian 12+, Ubuntu LTS.
- **Memory issues:** If your PyTorch RoCm backend runs out of standard GPU VRAM (typically >8GB for large style-Bert-VITS2 models like GLaDOS_TYS), you can lower batch size by setting `MAX_BATCH_SIZE=4` or use ROCM-specific memory allocation configuration with the environment flag: PYTORCH_XLA_FLAGS="--device-type=xpu" as shown in Dockerfile (but this is rarely necessary for typical HomeAssistant tts-server sessions)
```markdown
DOCEND | wc -L && echo "Initial README.md created successfully to /home/taco/README-GlaDOS-TYS..." || true

22
README.md
View File

@@ -15,6 +15,8 @@ services:
- "10200:10200" - "10200:10200"
volumes: volumes:
- glados_model_cache:/data - glados_model_cache:/data
- glados_hf_cache:/root/.cache/huggingface
- glados_nltk_data:/root/nltk_data
environment: environment:
- MODEL_DIR=/data - MODEL_DIR=/data
- URI=tcp://0.0.0.0:10200 - URI=tcp://0.0.0.0:10200
@@ -23,6 +25,8 @@ services:
volumes: volumes:
glados_model_cache: glados_model_cache:
glados_hf_cache:
glados_nltk_data:
``` ```
### CUDA (NVIDIA GPU) ### CUDA (NVIDIA GPU)
@@ -36,10 +40,14 @@ services:
- "10200:10200" - "10200:10200"
volumes: volumes:
- glados_model_cache:/data - glados_model_cache:/data
- glados_hf_cache:/root/.cache/huggingface
- glados_nltk_data:/root/nltk_data
environment: environment:
- MODEL_DIR=/data - MODEL_DIR=/data
- URI=tcp://0.0.0.0:10200 - URI=tcp://0.0.0.0:10200
- DEVICE=cuda - DEVICE=cuda
- HALF=true
- PRELOAD=true
deploy: deploy:
resources: resources:
reservations: reservations:
@@ -51,6 +59,8 @@ services:
volumes: volumes:
glados_model_cache: glados_model_cache:
glados_hf_cache:
glados_nltk_data:
``` ```
### ROCm (AMD GPU) ### ROCm (AMD GPU)
@@ -64,10 +74,17 @@ services:
- "10200:10200" - "10200:10200"
volumes: volumes:
- glados_model_cache:/data - glados_model_cache:/data
- glados_hf_cache:/root/.cache/huggingface
- glados_nltk_data:/root/nltk_data
- glados_miopen_cache:/root/.cache/miopen
environment: environment:
- MODEL_DIR=/data - MODEL_DIR=/data
- URI=tcp://0.0.0.0:10200 - URI=tcp://0.0.0.0:10200
- DEVICE=rocm - DEVICE=rocm
- HALF=true
- PRELOAD=true
- MIOPEN_FIND_MODE=1
- MIOPEN_DEBUG_CONV_GEMM=0
devices: devices:
- /dev/kfd - /dev/kfd
- /dev/dri - /dev/dri
@@ -75,6 +92,9 @@ services:
volumes: volumes:
glados_model_cache: glados_model_cache:
glados_hf_cache:
glados_nltk_data:
glados_miopen_cache:
``` ```
## Docker Compose (All Variants) ## Docker Compose (All Variants)
@@ -98,7 +118,7 @@ See the detailed guides for Wyoming protocol setup and configuration:
- [HOMEASSISTANT_INTEGRATION.md](HOMEASSISTANT_INTEGRATION.md) - [HOMEASSISTANT_INTEGRATION.md](HOMEASSISTANT_INTEGRATION.md)
For AMD ROCm-specific deployment notes and troubleshooting: For AMD ROCm-specific deployment notes and troubleshooting:
- [README-GlaDOS-TYS-Wyoming-and-ROCM.md](README-GlaDOS-TYS-Wyoming-and-ROCM.md) - [README-GLaDOS-TYS-Wyoming-and-ROCM.md](README-GLaDOS-TYS-Wyoming-and-ROCM.md)
## Model ## Model

20
docker-compose.yaml
View File

@@ -6,6 +6,8 @@ services:
- "10201:10200" - "10201:10200"
volumes: volumes:
- glados_model_cache:/data - glados_model_cache:/data
- glados_hf_cache:/root/.cache/huggingface
- glados_nltk_data:/root/nltk_data
environment: environment:
- MODEL_DIR=/data - MODEL_DIR=/data
- URI=tcp://0.0.0.0:10200 - URI=tcp://0.0.0.0:10200
@@ -19,10 +21,14 @@ services:
- "10200:10200" - "10200:10200"
volumes: volumes:
- glados_model_cache:/data - glados_model_cache:/data
- glados_hf_cache:/root/.cache/huggingface
- glados_nltk_data:/root/nltk_data
environment: environment:
- MODEL_DIR=/data - MODEL_DIR=/data
- URI=tcp://0.0.0.0:10200 - URI=tcp://0.0.0.0:10200
- DEVICE=cuda - DEVICE=cuda
- HALF=true
- PRELOAD=true
deploy: deploy:
resources: resources:
reservations: reservations:
@@ -37,16 +43,26 @@ services:
container_name: glados-tts-rocm container_name: glados-tts-rocm
ports: ports:
- "10202:10200" - "10202:10200"
volumes:
- glados_model_cache:/data
environment: environment:
- MODEL_DIR=/data - MODEL_DIR=/data
- URI=tcp://0.0.0.0:10200 - URI=tcp://0.0.0.0:10200
- DEVICE=rocm - DEVICE=rocm
- HALF=true
- PRELOAD=true
- MIOPEN_FIND_MODE=1
- MIOPEN_DEBUG_CONV_GEMM=0
devices: devices:
- /dev/kfd - /dev/kfd
- /dev/dri - /dev/dri
volumes:
- glados_model_cache:/data
- glados_hf_cache:/root/.cache/huggingface
- glados_nltk_data:/root/nltk_data
- glados_miopen_cache:/root/.cache/miopen
restart: unless-stopped restart: unless-stopped
volumes: volumes:
glados_model_cache: glados_model_cache:
glados_hf_cache:
glados_nltk_data:
glados_miopen_cache:

30
entrypoint.sh
View File

@@ -12,8 +12,36 @@ else
ls -la "$MODEL_DIR" ls -la "$MODEL_DIR"
fi fi
# Ensure NLTK data is available
python -c "
import nltk
try:
nltk.data.find('taggers/averaged_perceptron_tagger_eng')
except LookupError:
nltk.download('averaged_perceptron_tagger_eng')
try:
nltk.data.find('corpora/cmudict')
except LookupError:
nltk.download('cmudict')
"
HALF="${HALF:-false}"
PRELOAD="${PRELOAD:-false}"
HALF_FLAG=""
if [ "$HALF" = "true" ]; then
HALF_FLAG="--half"
fi
PRELOAD_FLAG=""
if [ "$PRELOAD" = "true" ]; then
PRELOAD_FLAG="--preload"
fi
echo "Starting Wyoming GLaDOS TTS server..." echo "Starting Wyoming GLaDOS TTS server..."
exec python -m wyoming_glados \ exec python -m wyoming_glados \
--model-dir "$MODEL_DIR" \ --model-dir "$MODEL_DIR" \
--uri "${URI:-tcp://0.0.0.0:10200}" \ --uri "${URI:-tcp://0.0.0.0:10200}" \
--device "${DEVICE:-cpu}" --device "${DEVICE:-cpu}" \
$HALF_FLAG \
$PRELOAD_FLAG

3
requirements.txt
View File

@@ -2,3 +2,6 @@ wyoming>=1.5
style-bert-vits2>=2.4 style-bert-vits2>=2.4
huggingface_hub>=0.19 huggingface_hub>=0.19
numpy>=1.21 numpy>=1.21
protobuf
sentencepiece
tiktoken

8
wyoming_glados/__main__.py
View File

@@ -24,6 +24,10 @@ async def main() -> None:
help="Directory containing model files (config.json, *.safetensors, style_vectors.npy)") help="Directory containing model files (config.json, *.safetensors, style_vectors.npy)")
parser.add_argument("--device", default="cpu", parser.add_argument("--device", default="cpu",
help="Device for PyTorch (cpu, cuda, rocm)") help="Device for PyTorch (cpu, cuda, rocm)")
parser.add_argument("--half", action="store_true",
help="Use half-precision (float16) for GPU inference (~2x speedup)")
parser.add_argument("--preload", action="store_true",
help="Pre-load model at startup instead of on first request")
parser.add_argument("--debug", action="store_true", parser.add_argument("--debug", action="store_true",
help="Log DEBUG messages") help="Log DEBUG messages")
parser.add_argument("--version", action="version", parser.add_argument("--version", action="version",
@@ -78,6 +82,8 @@ async def main() -> None:
_LOGGER.info("Starting GLaDOS Wyoming TTS server on %s", args.uri) _LOGGER.info("Starting GLaDOS Wyoming TTS server on %s", args.uri)
_LOGGER.info("Model directory: %s", model_dir) _LOGGER.info("Model directory: %s", model_dir)
_LOGGER.info("Device: %s", device) _LOGGER.info("Device: %s", device)
_LOGGER.info("Half precision: %s", args.half)
_LOGGER.info("Preload model: %s", args.preload)
server_task = asyncio.create_task( server_task = asyncio.create_task(
server.run( server.run(
@@ -86,6 +92,8 @@ async def main() -> None:
wyoming_info, wyoming_info,
model_dir, model_dir,
device, device,
half=args.half,
preload=args.preload,
) )
) )
) )

BIN
wyoming_glados/__pycache__/__main__.cpython-314.pyc
View File

Binary file not shown.

BIN
wyoming_glados/__pycache__/handler.cpython-314.pyc
View File

Binary file not shown.

44
wyoming_glados/handler.py
View File

@@ -5,6 +5,7 @@ from pathlib import Path
from typing import Optional from typing import Optional
import numpy as np import numpy as np
import torch
from wyoming.audio import AudioChunk, AudioStart, AudioStop from wyoming.audio import AudioChunk, AudioStart, AudioStop
from wyoming.error import Error from wyoming.error import Error
from wyoming.event import Event from wyoming.event import Event
@@ -20,6 +21,7 @@ _LOGGER = logging.getLogger(__name__)
_VOICE_LOCK = asyncio.Lock() _VOICE_LOCK = asyncio.Lock()
_MODEL: Optional[TTSModel] = None _MODEL: Optional[TTSModel] = None
_BERT_INITIALIZED: set[Languages] = set()
_BERT_MODEL_NAMES = { _BERT_MODEL_NAMES = {
Languages.JP: "ku-nlp/deberta-v2-large-japanese-char-wwm", Languages.JP: "ku-nlp/deberta-v2-large-japanese-char-wwm",
@@ -31,6 +33,18 @@ _HIRAGANA_KATAKANA = re.compile(r"[\u3040-\u309F\u30A0-\u30FF]")
_CJK = re.compile(r"[\u4E00-\u9FFF]") _CJK = re.compile(r"[\u4E00-\u9FFF]")
def _optimize_gpu():
if not torch.cuda.is_available():
return
torch.cuda.empty_cache = lambda: None
_LOGGER.info("Disabled torch.cuda.empty_cache() to prevent re-allocation on every inference")
if torch.version.hip is not None and hasattr(torch.backends, 'miopen'):
torch.backends.miopen.benchmark = True
_LOGGER.info("Enabled MIOpen benchmark mode to cache convolution solver solutions")
def _detect_language(text: str) -> Languages: def _detect_language(text: str) -> Languages:
if _HIRAGANA_KATAKANA.search(text): if _HIRAGANA_KATAKANA.search(text):
return Languages.JP return Languages.JP
@@ -39,18 +53,21 @@ def _detect_language(text: str) -> Languages:
return Languages.EN return Languages.EN
def _load_bert_for_language(language: Languages, device: str) -> None: def _load_bert_for_language(language: Languages, device: str, half: bool = False) -> None:
if language in _BERT_INITIALIZED:
return
model_name = _BERT_MODEL_NAMES[language] model_name = _BERT_MODEL_NAMES[language]
_LOGGER.info("Loading BERT model for %s (%s)", language.name, model_name) _LOGGER.info("Loading BERT model for %s (%s)", language.name, model_name)
bert_models.load_model(language, model_name) bert_models.load_model(language, model_name)
bert_models.load_tokenizer(language, model_name) bert_models.load_tokenizer(language, model_name)
bert = bert_models.__loaded_models.get(language) bert = bert_models.__loaded_models.get(language)
if bert is not None: if bert is not None:
bert = bert.float()
bert.eval() bert.eval()
bert.to(device) bert.to(device).float()
bert_models.__loaded_models[language] = bert _LOGGER.info("BERT model for %s moved to %s", language.name, device)
_LOGGER.info("BERT model for %s cast to float32 and moved to %s", language.name, device)
_BERT_INITIALIZED.add(language)
def _find_model_files(model_dir: Path): def _find_model_files(model_dir: Path):
@@ -76,7 +93,7 @@ def _find_model_files(model_dir: Path):
) )
def _load_model(model_dir: Path, device: str) -> TTSModel: def _load_model(model_dir: Path, device: str, half: bool = False) -> TTSModel:
model_path, config_path, style_path = _find_model_files(model_dir) model_path, config_path, style_path = _find_model_files(model_dir)
_LOGGER.info("Creating TTSModel (model=%s, config=%s, device=%s)", _LOGGER.info("Creating TTSModel (model=%s, config=%s, device=%s)",
@@ -96,7 +113,6 @@ def _load_model(model_dir: Path, device: str) -> TTSModel:
if net_g is not None: if net_g is not None:
net_g = net_g.float() net_g = net_g.float()
setattr(model, "_TTSModel__net_g", net_g) setattr(model, "_TTSModel__net_g", net_g)
_LOGGER.info("TTS network cast to float32")
_LOGGER.info("Model loaded successfully") _LOGGER.info("Model loaded successfully")
return model return model
@@ -109,12 +125,22 @@ class GLaDOSEventHandler(AsyncEventHandler):
model_dir: Path, model_dir: Path,
device: str, device: str,
*args, *args,
half: bool = False,
preload: bool = False,
**kwargs, **kwargs,
) -> None: ) -> None:
super().__init__(*args, **kwargs) super().__init__(*args, **kwargs)
self.wyoming_info_event = wyoming_info.event() self.wyoming_info_event = wyoming_info.event()
self.model_dir = model_dir self.model_dir = model_dir
self.device = device self.device = device
self.half = half
_optimize_gpu()
if preload:
_LOGGER.info("Pre-loading model at startup...")
global _MODEL
_MODEL = _load_model(model_dir, device, half)
async def handle_event(self, event: Event) -> bool: async def handle_event(self, event: Event) -> bool:
if Describe.is_type(event.type): if Describe.is_type(event.type):
@@ -152,9 +178,9 @@ class GLaDOSEventHandler(AsyncEventHandler):
if _MODEL is None: if _MODEL is None:
_LOGGER.info("Loading GLaDOS model from %s on %s", _LOGGER.info("Loading GLaDOS model from %s on %s",
self.model_dir, self.device) self.model_dir, self.device)
_MODEL = _load_model(self.model_dir, self.device) _MODEL = _load_model(self.model_dir, self.device, self.half)
_load_bert_for_language(language, self.device) _load_bert_for_language(language, self.device, self.half)
sr, audio = await asyncio.to_thread( sr, audio = await asyncio.to_thread(
_MODEL.infer, _MODEL.infer,