Release NVIDIA WakeWord Trainer v18

This commit is contained in:
MasterPhooey
2026-07-26 09:07:22 -05:00
parent 2eee70cb34
commit 6ee228e8d3
8 changed files with 360 additions and 76 deletions

View File

@@ -171,7 +171,7 @@ Starting a new session does not clear samples. Use the clear buttons in `Samples
For each new wake-trigger clip sent to the trainer: For each new wake-trigger clip sent to the trainer:
1. Faster Whisper transcribes the audio locally. 1. The selected local STT engine transcribes the audio.
2. If the transcript contains the configured wake phrase, the clip stays in `Captured Audio` for manual review by default. 2. If the transcript contains the configured wake phrase, the clip stays in `Captured Audio` for manual review by default.
3. If speech was transcribed but the wake phrase is absent, the clip moves to `/data/negative_samples/` as an auto-reviewed hard negative. 3. If speech was transcribed but the wake phrase is absent, the clip moves to `/data/negative_samples/` as an auto-reviewed hard negative.
4. Empty transcripts, VAD-blocked captures, and captures for another wake word stay out of the automatic negative path. 4. Empty transcripts, VAD-blocked captures, and captures for another wake word stay out of the automatic negative path.
@@ -183,7 +183,7 @@ Two optional cleanup rules are available:
A close miss with an empty transcript or without the configured phrase stays in `Captured Audio`; it is never turned into a negative automatically. Saving Auto Training settings also scans existing eligible captures. Enabling close-miss promotion reviews previous unreviewed close misses, while enabling cleanup removes previously confirmed good wakes without transcribing them a second time. A close miss with an empty transcript or without the configured phrase stays in `Captured Audio`; it is never turned into a negative automatically. Saving Auto Training settings also scans existing eligible captures. Enabling close-miss promotion reviews previous unreviewed close misses, while enabling cleanup removes previously confirmed good wakes without transcribing them a second time.
The default `small.en` model uses CUDA with `float16` when CTranslate2 can see an NVIDIA GPU, and falls back to CPU with `int8`. Choose a multilingual Faster Whisper model such as `small` when the wake phrase is not English. Downloaded STT models are cached in `/data/auto_train_models/`. Auto Training exposes only an engine selector. Faster Whisper is the recommended default and uses CUDA with `float16` when CTranslate2 can see an NVIDIA GPU, with a CPU `int8` fallback. The trainer manages `small.en` for English and `small` for other languages. Parakeet ONNX uses the managed INT8 `nemo-parakeet-tdt-0.6b-v3` model with CUDA and CPU fallback. Downloaded STT models are cached in `/data/auto_train_models/`.
Scheduled training runs only after the configured number of new automatic negatives has accumulated. A successful run securely publishes the trained wake-word name and JSON URL to the linked Tater instance. Tater saves it as the global satellite wake word and pushes the updated setting to every connected satellite, so no satellite firmware change is required. Scheduled training runs only after the configured number of new automatic negatives has accumulated. A successful run securely publishes the trained wake-word name and JSON URL to the linked Tater instance. Tater saves it as the global satellite wake word and pushes the updated setting to every connected satellite, so no satellite firmware change is required.

View File

@@ -1 +1 @@
17 18

View File

@@ -1,2 +1,3 @@
- Fixed the training-status endpoint crashing after a training log was created because its log-tail limits were missing. - Added Parakeet ONNX as a second local Auto Training STT engine alongside Faster Whisper.
- Restored bounded, incremental training-log updates so the UI can continue showing live progress without repeatedly reading the entire log. - Replaced manual model, device, and compute fields with a simple engine selector and managed language-aware models.
- Added CUDA-enabled ONNX Runtime with CPU fallback, runtime reporting, and model-cache cleanup when switching engines.

9
run.sh
View File

@@ -33,8 +33,11 @@ install_ui_deps() {
"silero-vad>=5.0.0" \ "silero-vad>=5.0.0" \
"numpy>=1.24.0" \ "numpy>=1.24.0" \
"faster-whisper>=1.0.0" \ "faster-whisper>=1.0.0" \
"onnx-asr[hub]>=0.12.0" \
"nvidia-cublas-cu12" \ "nvidia-cublas-cu12" \
"nvidia-cudnn-cu12==9.*" "nvidia-cudnn-cu12==9.*"
${PIP} uninstall -y onnxruntime
${PIP} install "onnxruntime-gpu[cuda,cudnn]<1.27"
} }
# ----------------------------- # -----------------------------
@@ -82,11 +85,13 @@ minimum = {
"silero-vad": "5.0.0", "silero-vad": "5.0.0",
"numpy": "1.24.0", "numpy": "1.24.0",
"faster-whisper": "1.0.0", "faster-whisper": "1.0.0",
"onnx-asr": "0.12.0",
"nvidia-cudnn-cu12": "9.0.0", "nvidia-cudnn-cu12": "9.0.0",
} }
present = ( present = (
"torch", "torch",
"nvidia-cublas-cu12", "nvidia-cublas-cu12",
"onnxruntime-gpu",
) )
for package, expected in exact.items(): for package, expected in exact.items():
@@ -97,6 +102,10 @@ for package, minimum_version in minimum.items():
raise SystemExit(1) raise SystemExit(1)
for package in present: for package in present:
md.version(package) md.version(package)
import onnxruntime as ort
if "CUDAExecutionProvider" not in ort.get_available_providers():
raise SystemExit(1)
PY PY
then then
echo "UI dependencies missing or stale; installing recorder dependencies" echo "UI dependencies missing or stale; installing recorder dependencies"

View File

@@ -1348,7 +1348,7 @@
</div> </div>
<label class="checkField"> <label class="checkField">
<input id="autoEnabled" type="checkbox" /> <input id="autoEnabled" type="checkbox" />
<span><strong>Enable Auto Training</strong>Eligible wake triggers will be queued for local Faster Whisper transcription.</span> <span><strong>Enable Auto Training</strong>Eligible wake triggers will be queued for transcription with the selected local STT engine.</span>
</label> </label>
<label class="checkField"> <label class="checkField">
<input id="autoDeleteConfirmedWakes" type="checkbox" /> <input id="autoDeleteConfirmedWakes" type="checkbox" />
@@ -1368,26 +1368,10 @@
<input id="autoLanguage" type="text" value="en" placeholder="en" /> <input id="autoLanguage" type="text" value="en" placeholder="en" />
</label> </label>
<label class="field wide"> <label class="field wide">
<strong>Faster Whisper model</strong> <strong>STT engine</strong>
<input id="autoSttModel" type="text" value="small.en" placeholder="small.en" /> <select id="autoSttEngine">
</label> <option value="faster_whisper" selected>Faster Whisper (recommended)</option>
<label class="field"> <option value="parakeet_onnx">Parakeet ONNX</option>
<strong>STT device</strong>
<select id="autoSttDevice">
<option value="auto" selected>Auto (prefer CUDA)</option>
<option value="cuda">CUDA</option>
<option value="cpu">CPU</option>
</select>
</label>
<label class="field">
<strong>Compute type</strong>
<select id="autoSttComputeType">
<option value="auto" selected>Auto (float16 CUDA / int8 CPU)</option>
<option value="float16">float16</option>
<option value="int8_float16">int8_float16</option>
<option value="int8">int8</option>
<option value="float32">float32</option>
<option value="default">CTranslate2 default</option>
</select> </select>
</label> </label>
<label class="field"> <label class="field">
@@ -2091,9 +2075,7 @@
$("autoEnabled").checked = Boolean(config.enabled); $("autoEnabled").checked = Boolean(config.enabled);
$("autoWakePhrase").value = config.wake_phrase || uiState.session?.raw_phrase || ""; $("autoWakePhrase").value = config.wake_phrase || uiState.session?.raw_phrase || "";
$("autoLanguage").value = config.language || uiState.session?.language || "en"; $("autoLanguage").value = config.language || uiState.session?.language || "en";
$("autoSttModel").value = config.stt_model || "small.en"; $("autoSttEngine").value = config.stt_engine || "faster_whisper";
$("autoSttDevice").value = config.stt_device || "auto";
$("autoSttComputeType").value = config.stt_compute_type || "auto";
$("autoMinimumChars").value = String(config.minimum_transcript_chars ?? 2); $("autoMinimumChars").value = String(config.minimum_transcript_chars ?? 2);
$("autoDeleteConfirmedWakes").checked = Boolean(config.delete_confirmed_wakes); $("autoDeleteConfirmedWakes").checked = Boolean(config.delete_confirmed_wakes);
$("autoPromoteCloseMisses").checked = Boolean(config.promote_close_misses); $("autoPromoteCloseMisses").checked = Boolean(config.promote_close_misses);
@@ -2138,7 +2120,10 @@
if (state.last_review_file) audit.push(state.last_review_file); if (state.last_review_file) audit.push(state.last_review_file);
if (state.last_review_transcript) audit.push(`STT: “${state.last_review_transcript}`); if (state.last_review_transcript) audit.push(`STT: “${state.last_review_transcript}`);
if (state.last_review_error) audit.push(`Error: ${state.last_review_error}`); if (state.last_review_error) audit.push(`Error: ${state.last_review_error}`);
if (state.last_stt_device) audit.push(`STT runtime: ${state.last_stt_device} / ${state.last_stt_compute_type || "default"}`); if (state.last_stt_engine) {
const runtimeLabel = [state.last_stt_device, state.last_stt_compute_type].filter(Boolean).join(" / ");
audit.push(`STT engine: ${String(state.last_stt_engine).replaceAll("_", " ")}${runtimeLabel ? ` · ${runtimeLabel}` : ""}`);
}
if (state.last_notify_at) { if (state.last_notify_at) {
audit.push(state.last_notify_error audit.push(state.last_notify_error
? `Wake-word publish failed: ${state.last_notify_error}` ? `Wake-word publish failed: ${state.last_notify_error}`
@@ -2159,9 +2144,7 @@
enabled: $("autoEnabled").checked, enabled: $("autoEnabled").checked,
wake_phrase: ($("autoWakePhrase").value || "").trim(), wake_phrase: ($("autoWakePhrase").value || "").trim(),
language: ($("autoLanguage").value || "en").trim(), language: ($("autoLanguage").value || "en").trim(),
stt_model: ($("autoSttModel").value || "").trim(), stt_engine: $("autoSttEngine").value || "faster_whisper",
stt_device: $("autoSttDevice").value || "auto",
stt_compute_type: $("autoSttComputeType").value || "auto",
minimum_transcript_chars: Number($("autoMinimumChars").value || 2), minimum_transcript_chars: Number($("autoMinimumChars").value || 2),
delete_confirmed_wakes: $("autoDeleteConfirmedWakes").checked, delete_confirmed_wakes: $("autoDeleteConfirmedWakes").checked,
promote_close_misses: $("autoPromoteCloseMisses").checked, promote_close_misses: $("autoPromoteCloseMisses").checked,

View File

@@ -62,8 +62,6 @@ class AutoTrainTests(unittest.TestCase):
"wake_phrase": "hey tater", "wake_phrase": "hey tater",
"language": "en", "language": "en",
"tater_url": "http://127.0.0.1:8501", "tater_url": "http://127.0.0.1:8501",
"stt_device": "auto",
"stt_compute_type": "auto",
} }
) )
) )
@@ -110,9 +108,90 @@ class AutoTrainTests(unittest.TestCase):
self.assertTrue(trainer._transcript_contains_wake_phrase("Okay, HEY TATER!", "hey_tater")) self.assertTrue(trainer._transcript_contains_wake_phrase("Okay, HEY TATER!", "hey_tater"))
self.assertFalse(trainer._transcript_contains_wake_phrase("Turn on the television", "hey tater")) self.assertFalse(trainer._transcript_contains_wake_phrase("Turn on the television", "hey tater"))
def test_stt_engine_selection_uses_managed_models(self):
config = trainer._normalize_auto_train_config(
{
"stt_engine": "parakeet-onnx",
"stt_model": "user/should-not-be-used",
"stt_device": "cpu",
"stt_compute_type": "float32",
}
)
self.assertEqual(config["stt_engine"], trainer.STT_ENGINE_PARAKEET_ONNX)
self.assertNotIn("stt_model", config)
self.assertNotIn("stt_device", config)
self.assertNotIn("stt_compute_type", config)
self.assertEqual(
trainer._managed_stt_model(config["stt_engine"], "en"),
trainer.DEFAULT_PARAKEET_ONNX_MODEL,
)
self.assertEqual(
trainer._managed_stt_model(trainer.STT_ENGINE_FASTER_WHISPER, "de"),
trainer.DEFAULT_FASTER_WHISPER_MULTILINGUAL_MODEL,
)
def test_stt_router_supports_both_nvidia_engines(self):
audio_path = Path("wake.wav")
with (
patch.object(trainer, "_transcribe_capture_with_faster_whisper", return_value="faster") as faster,
patch.object(trainer, "_transcribe_capture_with_parakeet", return_value="parakeet") as parakeet,
):
self.assertEqual(
trainer._transcribe_capture(
audio_path,
engine=trainer.STT_ENGINE_FASTER_WHISPER,
language="en",
),
"faster",
)
self.assertEqual(
trainer._transcribe_capture(
audio_path,
engine=trainer.STT_ENGINE_PARAKEET_ONNX,
language="en",
),
"parakeet",
)
faster.assert_called_once()
parakeet.assert_called_once()
def test_parakeet_loader_prefers_cuda_then_cpu(self):
fake_model = object()
fake_onnx_asr = SimpleNamespace(load_model=Mock(return_value=fake_model))
with (
patch.dict(sys.modules, {"onnx_asr": fake_onnx_asr}),
patch.object(
trainer,
"_parakeet_onnx_providers",
return_value=["CUDAExecutionProvider", "CPUExecutionProvider"],
),
):
with trainer.PARAKEET_ONNX_MODEL_LOCK:
trainer.PARAKEET_ONNX_MODEL_CACHE.clear()
loaded = trainer._load_parakeet_onnx_model()
self.assertIs(loaded, fake_model)
fake_onnx_asr.load_model.assert_called_once_with(
trainer.DEFAULT_PARAKEET_ONNX_MODEL,
str(trainer.AUTO_TRAIN_MODEL_DIR),
quantization="int8",
providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)
def test_ui_exposes_engine_selector_without_manual_runtime_fields(self):
source = (Path(__file__).resolve().parents[1] / "static" / "index.html").read_text(
encoding="utf-8"
)
self.assertIn('id="autoSttEngine"', source)
self.assertNotIn('id="autoSttModel"', source)
self.assertNotIn('id="autoSttDevice"', source)
self.assertNotIn('id="autoSttComputeType"', source)
def test_phrase_miss_moves_wake_trigger_to_negative_samples(self): def test_phrase_miss_moves_wake_trigger_to_negative_samples(self):
self.add_capture() self.add_capture()
with patch.object(trainer, "_transcribe_capture_with_faster_whisper", return_value="turn on the kitchen lights"): with patch.object(trainer, "_transcribe_capture", return_value="turn on the kitchen lights"):
trainer._auto_review_capture("wake.wav") trainer._auto_review_capture("wake.wav")
self.assertFalse((trainer.CAPTURED_DIR / "wake.wav").exists()) self.assertFalse((trainer.CAPTURED_DIR / "wake.wav").exists())
@@ -122,11 +201,13 @@ class AutoTrainTests(unittest.TestCase):
self.assertTrue(metadata["auto_negative"]) self.assertTrue(metadata["auto_negative"])
self.assertEqual(metadata["review_status"], "auto_approved_negative") self.assertEqual(metadata["review_status"], "auto_approved_negative")
self.assertEqual(metadata["transcript"], "turn on the kitchen lights") self.assertEqual(metadata["transcript"], "turn on the kitchen lights")
self.assertEqual(metadata["auto_review_stt_engine"], "faster_whisper")
self.assertEqual(metadata["auto_review_stt_model"], "small.en")
self.assertEqual(trainer.AUTO_TRAIN_STATE["pending_negative_count"], 1) self.assertEqual(trainer.AUTO_TRAIN_STATE["pending_negative_count"], 1)
def test_matching_phrase_stays_in_manual_review_inbox(self): def test_matching_phrase_stays_in_manual_review_inbox(self):
audio_path = self.add_capture() audio_path = self.add_capture()
with patch.object(trainer, "_transcribe_capture_with_faster_whisper", return_value="hey tater turn on the lights"): with patch.object(trainer, "_transcribe_capture", return_value="hey tater turn on the lights"):
trainer._auto_review_capture("wake.wav") trainer._auto_review_capture("wake.wav")
self.assertTrue(audio_path.exists()) self.assertTrue(audio_path.exists())
@@ -140,7 +221,7 @@ class AutoTrainTests(unittest.TestCase):
trainer.AUTO_TRAIN_CONFIG["delete_confirmed_wakes"] = True trainer.AUTO_TRAIN_CONFIG["delete_confirmed_wakes"] = True
with patch.object( with patch.object(
trainer, trainer,
"_transcribe_capture_with_faster_whisper", "_transcribe_capture",
return_value="hey tater turn on the lights", return_value="hey tater turn on the lights",
): ):
trainer._auto_review_capture("wake.wav") trainer._auto_review_capture("wake.wav")
@@ -164,7 +245,7 @@ class AutoTrainTests(unittest.TestCase):
trainer.AUTO_TRAIN_CONFIG["delete_confirmed_wakes"] = True trainer.AUTO_TRAIN_CONFIG["delete_confirmed_wakes"] = True
self.assertEqual(trainer._queue_pending_auto_reviews(), 1) self.assertEqual(trainer._queue_pending_auto_reviews(), 1)
with patch.object(trainer, "_transcribe_capture_with_faster_whisper") as transcribe: with patch.object(trainer, "_transcribe_capture") as transcribe:
trainer._auto_review_capture("wake.wav") trainer._auto_review_capture("wake.wav")
transcribe.assert_not_called() transcribe.assert_not_called()
@@ -173,7 +254,7 @@ class AutoTrainTests(unittest.TestCase):
def test_close_miss_is_not_transcribed_by_default(self): def test_close_miss_is_not_transcribed_by_default(self):
audio_path = self.add_capture(event_type="close_miss") audio_path = self.add_capture(event_type="close_miss")
with patch.object(trainer, "_transcribe_capture_with_faster_whisper") as transcribe: with patch.object(trainer, "_transcribe_capture") as transcribe:
trainer._auto_review_capture("wake.wav") trainer._auto_review_capture("wake.wav")
transcribe.assert_not_called() transcribe.assert_not_called()
@@ -190,7 +271,7 @@ class AutoTrainTests(unittest.TestCase):
def test_close_miss_with_phrase_is_promoted_when_enabled(self): def test_close_miss_with_phrase_is_promoted_when_enabled(self):
self.add_capture(event_type="close_miss") self.add_capture(event_type="close_miss")
trainer.AUTO_TRAIN_CONFIG["promote_close_misses"] = True trainer.AUTO_TRAIN_CONFIG["promote_close_misses"] = True
with patch.object(trainer, "_transcribe_capture_with_faster_whisper", return_value="hey tater"): with patch.object(trainer, "_transcribe_capture", return_value="hey tater"):
trainer._auto_review_capture("wake.wav") trainer._auto_review_capture("wake.wav")
self.assertFalse((trainer.CAPTURED_DIR / "wake.wav").exists()) self.assertFalse((trainer.CAPTURED_DIR / "wake.wav").exists())
@@ -208,7 +289,7 @@ class AutoTrainTests(unittest.TestCase):
trainer.AUTO_TRAIN_CONFIG["promote_close_misses"] = True trainer.AUTO_TRAIN_CONFIG["promote_close_misses"] = True
with patch.object( with patch.object(
trainer, trainer,
"_transcribe_capture_with_faster_whisper", "_transcribe_capture",
return_value="turn on the lights", return_value="turn on the lights",
): ):
trainer._auto_review_capture("wake.wav") trainer._auto_review_capture("wake.wav")
@@ -222,7 +303,7 @@ class AutoTrainTests(unittest.TestCase):
def test_vad_blocked_close_miss_is_never_transcribed(self): def test_vad_blocked_close_miss_is_never_transcribed(self):
audio_path = self.add_capture(event_type="close_miss", blocked_by_vad=True) audio_path = self.add_capture(event_type="close_miss", blocked_by_vad=True)
trainer.AUTO_TRAIN_CONFIG["promote_close_misses"] = True trainer.AUTO_TRAIN_CONFIG["promote_close_misses"] = True
with patch.object(trainer, "_transcribe_capture_with_faster_whisper") as transcribe: with patch.object(trainer, "_transcribe_capture") as transcribe:
trainer._auto_review_capture("wake.wav") trainer._auto_review_capture("wake.wav")
transcribe.assert_not_called() transcribe.assert_not_called()
@@ -231,7 +312,7 @@ class AutoTrainTests(unittest.TestCase):
def test_capture_for_another_wake_word_is_not_transcribed(self): def test_capture_for_another_wake_word_is_not_transcribed(self):
audio_path = self.add_capture(wake_word="computer") audio_path = self.add_capture(wake_word="computer")
with patch.object(trainer, "_transcribe_capture_with_faster_whisper") as transcribe: with patch.object(trainer, "_transcribe_capture") as transcribe:
trainer._auto_review_capture("wake.wav") trainer._auto_review_capture("wake.wav")
transcribe.assert_not_called() transcribe.assert_not_called()

View File

@@ -61,6 +61,13 @@ class RunShCudaLibraryPathTests(unittest.TestCase):
self.assertEqual(result.returncode, 0, result.stderr) self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(result.stdout.strip(), "") self.assertEqual(result.stdout.strip(), "")
def test_parakeet_uses_cuda_onnxruntime_package(self) -> None:
source = RUN_SH.read_text(encoding="utf-8")
self.assertIn('"onnx-asr[hub]>=0.12.0"', source)
self.assertIn('"onnxruntime-gpu[cuda,cudnn]<1.27"', source)
self.assertIn('"CUDAExecutionProvider"', source)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()

View File

@@ -2,6 +2,7 @@
# trainer_server.py # trainer_server.py
import contextlib import contextlib
import gc
import io import io
import os import os
import queue import queue
@@ -88,15 +89,37 @@ TARGET_SAMPLE_RATE = 16000
TARGET_CHANNELS = 1 TARGET_CHANNELS = 1
TARGET_SAMPLE_WIDTH_BYTES = 2 TARGET_SAMPLE_WIDTH_BYTES = 2
CAPTURE_GAIN_PROFILE = "capture_rms_v1" CAPTURE_GAIN_PROFILE = "capture_rms_v1"
DEFAULT_FASTER_WHISPER_MODEL = os.environ.get("AUTO_TRAIN_STT_MODEL", "small.en") STT_ENGINE_FASTER_WHISPER = "faster_whisper"
STT_ENGINE_PARAKEET_ONNX = "parakeet_onnx"
SUPPORTED_STT_ENGINES = {
STT_ENGINE_FASTER_WHISPER,
STT_ENGINE_PARAKEET_ONNX,
}
DEFAULT_STT_ENGINE = os.environ.get(
"AUTO_TRAIN_STT_ENGINE",
STT_ENGINE_FASTER_WHISPER,
).strip().lower().replace("-", "_")
if DEFAULT_STT_ENGINE not in SUPPORTED_STT_ENGINES:
DEFAULT_STT_ENGINE = STT_ENGINE_FASTER_WHISPER
DEFAULT_FASTER_WHISPER_EN_MODEL = os.environ.get(
"AUTO_TRAIN_FASTER_WHISPER_EN_MODEL",
"small.en",
)
DEFAULT_FASTER_WHISPER_MULTILINGUAL_MODEL = os.environ.get(
"AUTO_TRAIN_FASTER_WHISPER_MULTILINGUAL_MODEL",
"small",
)
DEFAULT_PARAKEET_ONNX_MODEL = os.environ.get(
"AUTO_TRAIN_PARAKEET_ONNX_MODEL",
"nemo-parakeet-tdt-0.6b-v3",
)
DEFAULT_PARAKEET_ONNX_QUANTIZATION = "int8"
AUTO_TRAIN_DEFAULT_CONFIG: Dict[str, Any] = { AUTO_TRAIN_DEFAULT_CONFIG: Dict[str, Any] = {
"enabled": False, "enabled": False,
"wake_phrase": "", "wake_phrase": "",
"language": DEFAULT_LANGUAGE, "language": DEFAULT_LANGUAGE,
"stt_model": DEFAULT_FASTER_WHISPER_MODEL, "stt_engine": DEFAULT_STT_ENGINE,
"stt_device": "auto",
"stt_compute_type": "auto",
"minimum_transcript_chars": 2, "minimum_transcript_chars": 2,
"delete_confirmed_wakes": False, "delete_confirmed_wakes": False,
"promote_close_misses": False, "promote_close_misses": False,
@@ -119,6 +142,8 @@ AUTO_TRAIN_DEFAULT_STATE: Dict[str, Any] = {
"last_review_transcript": "", "last_review_transcript": "",
"last_review_result": "", "last_review_result": "",
"last_review_error": "", "last_review_error": "",
"last_stt_engine": "",
"last_stt_model": "",
"last_stt_device": "", "last_stt_device": "",
"last_stt_compute_type": "", "last_stt_compute_type": "",
"last_train_started_at": "", "last_train_started_at": "",
@@ -186,6 +211,10 @@ AUTO_TRAIN_RUNTIME: Dict[str, Any] = {
LAN_ADDRESS_CACHE: Dict[str, Any] = {"value": "", "fetched_at": 0.0} LAN_ADDRESS_CACHE: Dict[str, Any] = {"value": "", "fetched_at": 0.0}
FASTER_WHISPER_MODEL_LOCK = threading.RLock() FASTER_WHISPER_MODEL_LOCK = threading.RLock()
FASTER_WHISPER_MODEL_CACHE: Dict[Tuple[str, str, str], Any] = {} FASTER_WHISPER_MODEL_CACHE: Dict[Tuple[str, str, str], Any] = {}
FASTER_WHISPER_TRANSCRIBE_LOCK = threading.RLock()
PARAKEET_ONNX_MODEL_LOCK = threading.RLock()
PARAKEET_ONNX_MODEL_CACHE: Dict[Tuple[str, str, Tuple[str, ...]], Any] = {}
PARAKEET_ONNX_TRANSCRIBE_LOCK = threading.RLock()
PIPER_CATALOG_CACHE: Dict[str, Any] = { PIPER_CATALOG_CACHE: Dict[str, Any] = {
"fetched_at": 0.0, "fetched_at": 0.0,
"entries": None, "entries": None,
@@ -461,25 +490,60 @@ def _normalize_http_base_url(value: Any, *, allow_empty: bool = True) -> str:
return token return token
def _normalize_stt_engine(value: Any) -> str:
token = str(value or DEFAULT_STT_ENGINE).strip().lower().replace("-", "_")
aliases = {
"faster": STT_ENGINE_FASTER_WHISPER,
"fasterwhisper": STT_ENGINE_FASTER_WHISPER,
"parakeet": STT_ENGINE_PARAKEET_ONNX,
"onnx_parakeet": STT_ENGINE_PARAKEET_ONNX,
}
token = aliases.get(token, token)
if token not in SUPPORTED_STT_ENGINES:
raise ValueError("STT engine must be Faster Whisper or Parakeet ONNX.")
return token
def _managed_stt_model(engine: Any, language: Any = DEFAULT_LANGUAGE) -> str:
token = _normalize_stt_engine(engine)
language_token = str(language or DEFAULT_LANGUAGE).strip().lower().replace("-", "_")
english = language_token == "en" or language_token.startswith("en_")
if token == STT_ENGINE_FASTER_WHISPER:
return (
DEFAULT_FASTER_WHISPER_EN_MODEL
if english
else DEFAULT_FASTER_WHISPER_MULTILINGUAL_MODEL
)
return DEFAULT_PARAKEET_ONNX_MODEL
def _stt_engine_catalog(language: Any = DEFAULT_LANGUAGE) -> List[Dict[str, Any]]:
return [
{
"value": STT_ENGINE_FASTER_WHISPER,
"label": "Faster Whisper",
"model": _managed_stt_model(STT_ENGINE_FASTER_WHISPER, language),
"recommended": True,
},
{
"value": STT_ENGINE_PARAKEET_ONNX,
"label": "Parakeet ONNX",
"model": _managed_stt_model(STT_ENGINE_PARAKEET_ONNX, language),
},
]
def _normalize_auto_train_config(values: Dict[str, Any] | None, *, base: Dict[str, Any] | None = None) -> Dict[str, Any]: def _normalize_auto_train_config(values: Dict[str, Any] | None, *, base: Dict[str, Any] | None = None) -> Dict[str, Any]:
incoming = values if isinstance(values, dict) else {} incoming = values if isinstance(values, dict) else {}
source = {**AUTO_TRAIN_DEFAULT_CONFIG, **(base or {}), **incoming} source = {**AUTO_TRAIN_DEFAULT_CONFIG, **(base or {}), **incoming}
schedule_hours = _bounded_int(source.get("schedule_hours"), 24, 0, 24 * 30) schedule_hours = _bounded_int(source.get("schedule_hours"), 24, 0, 24 * 30)
language = str(source.get("language") or DEFAULT_LANGUAGE).strip().lower().replace("-", "_") language = str(source.get("language") or DEFAULT_LANGUAGE).strip().lower().replace("-", "_")
language = re.sub(r"[^a-z0-9_]", "", language) or DEFAULT_LANGUAGE language = re.sub(r"[^a-z0-9_]", "", language) or DEFAULT_LANGUAGE
stt_device = str(source.get("stt_device") or "auto").strip().lower()
if stt_device not in {"auto", "cuda", "cpu"}:
raise ValueError("Faster Whisper device must be auto, cuda, or cpu.")
stt_compute_type = str(source.get("stt_compute_type") or "auto").strip().lower()
if stt_compute_type not in {"auto", "default", "float16", "float32", "int8", "int8_float16"}:
raise ValueError("Unsupported Faster Whisper compute type.")
return { return {
"enabled": _config_bool(source.get("enabled")), "enabled": _config_bool(source.get("enabled")),
"wake_phrase": str(source.get("wake_phrase") or "").strip(), "wake_phrase": str(source.get("wake_phrase") or "").strip(),
"language": language, "language": language,
"stt_model": str(source.get("stt_model") or DEFAULT_FASTER_WHISPER_MODEL).strip() or DEFAULT_FASTER_WHISPER_MODEL, "stt_engine": _normalize_stt_engine(source.get("stt_engine")),
"stt_device": stt_device,
"stt_compute_type": stt_compute_type,
"minimum_transcript_chars": _bounded_int(source.get("minimum_transcript_chars"), 2, 1, 100), "minimum_transcript_chars": _bounded_int(source.get("minimum_transcript_chars"), 2, 1, 100),
"delete_confirmed_wakes": _config_bool(source.get("delete_confirmed_wakes")), "delete_confirmed_wakes": _config_bool(source.get("delete_confirmed_wakes")),
"promote_close_misses": _config_bool(source.get("promote_close_misses")), "promote_close_misses": _config_bool(source.get("promote_close_misses")),
@@ -555,10 +619,12 @@ def _public_auto_train_config() -> Dict[str, Any]:
def _auto_train_status_payload() -> Dict[str, Any]: def _auto_train_status_payload() -> Dict[str, Any]:
with AUTO_TRAIN_LOCK: with AUTO_TRAIN_LOCK:
language = AUTO_TRAIN_CONFIG.get("language") or DEFAULT_LANGUAGE
return { return {
"config": _public_auto_train_config(), "config": _public_auto_train_config(),
"state": dict(AUTO_TRAIN_STATE), "state": dict(AUTO_TRAIN_STATE),
"runtime": dict(AUTO_TRAIN_RUNTIME), "runtime": dict(AUTO_TRAIN_RUNTIME),
"stt_engines": _stt_engine_catalog(language),
"advertised_base_url": _advertised_base_url(), "advertised_base_url": _advertised_base_url(),
"trainer_link": _tater_link_public_status(), "trainer_link": _tater_link_public_status(),
} }
@@ -828,15 +894,13 @@ def _load_faster_whisper_model(*, model_name: str, device: str, compute_type: st
def _transcribe_capture_with_faster_whisper(audio_path: Path, *, model: str, language: str) -> str: def _transcribe_capture_with_faster_whisper(audio_path: Path, *, model: str, language: str) -> str:
with AUTO_TRAIN_LOCK: device, compute_type = _resolve_faster_whisper_runtime("auto", "auto")
device_value = AUTO_TRAIN_CONFIG.get("stt_device")
compute_value = AUTO_TRAIN_CONFIG.get("stt_compute_type")
device, compute_type = _resolve_faster_whisper_runtime(device_value, compute_value)
whisper_model = _load_faster_whisper_model( whisper_model = _load_faster_whisper_model(
model_name=model, model_name=model,
device=device, device=device,
compute_type=compute_type, compute_type=compute_type,
) )
with FASTER_WHISPER_TRANSCRIBE_LOCK:
segments, _info = whisper_model.transcribe( segments, _info = whisper_model.transcribe(
str(audio_path), str(audio_path),
language=language or None, language=language or None,
@@ -849,12 +913,144 @@ def _transcribe_capture_with_faster_whisper(audio_path: Path, *, model: str, lan
" ".join(str(segment.text or "").strip() for segment in segments), " ".join(str(segment.text or "").strip() for segment in segments),
).strip() ).strip()
with AUTO_TRAIN_LOCK: with AUTO_TRAIN_LOCK:
AUTO_TRAIN_STATE["last_stt_engine"] = STT_ENGINE_FASTER_WHISPER
AUTO_TRAIN_STATE["last_stt_model"] = model
AUTO_TRAIN_STATE["last_stt_device"] = device AUTO_TRAIN_STATE["last_stt_device"] = device
AUTO_TRAIN_STATE["last_stt_compute_type"] = compute_type AUTO_TRAIN_STATE["last_stt_compute_type"] = compute_type
_save_auto_train_state_locked() _save_auto_train_state_locked()
return transcript return transcript
def _parakeet_onnx_providers() -> List[str]:
try:
import onnxruntime as ort
except Exception as exc:
raise RuntimeError(f"onnxruntime is unavailable: {exc}") from exc
available = [str(value) for value in ort.get_available_providers()]
preferred = [
"CUDAExecutionProvider",
"CPUExecutionProvider",
]
resolved = [provider for provider in preferred if provider in set(available)]
if not resolved:
raise RuntimeError("ONNX Runtime has no usable CUDA or CPU execution provider.")
return resolved
def _load_parakeet_onnx_model():
try:
import onnx_asr
except Exception as exc:
raise RuntimeError(f"onnx-asr is unavailable: {exc}") from exc
providers = tuple(_parakeet_onnx_providers())
cache_key = (
DEFAULT_PARAKEET_ONNX_MODEL,
DEFAULT_PARAKEET_ONNX_QUANTIZATION,
providers,
)
with PARAKEET_ONNX_MODEL_LOCK:
cached = PARAKEET_ONNX_MODEL_CACHE.get(cache_key)
if cached is not None:
return cached
AUTO_TRAIN_MODEL_DIR.mkdir(parents=True, exist_ok=True)
previous = {
key: os.environ.get(key)
for key in ("HF_HOME", "HF_HUB_CACHE", "HUGGINGFACE_HUB_CACHE")
}
os.environ["HF_HOME"] = str(AUTO_TRAIN_MODEL_DIR)
os.environ["HF_HUB_CACHE"] = str(AUTO_TRAIN_MODEL_DIR / "hub")
os.environ["HUGGINGFACE_HUB_CACHE"] = str(AUTO_TRAIN_MODEL_DIR / "hub")
try:
model = onnx_asr.load_model(
DEFAULT_PARAKEET_ONNX_MODEL,
str(AUTO_TRAIN_MODEL_DIR),
quantization=DEFAULT_PARAKEET_ONNX_QUANTIZATION,
providers=list(providers),
)
finally:
for key, value in previous.items():
if value is None:
os.environ.pop(key, None)
else:
os.environ[key] = value
PARAKEET_ONNX_MODEL_CACHE.clear()
PARAKEET_ONNX_MODEL_CACHE[cache_key] = model
return model
def _normalized_wav_float32(audio_path: Path):
import numpy as np
with wave.open(str(audio_path), "rb") as wav_file:
channels = wav_file.getnchannels()
sample_width = wav_file.getsampwidth()
sample_rate = wav_file.getframerate()
frames = wav_file.readframes(wav_file.getnframes())
if sample_width != 2 or sample_rate != TARGET_SAMPLE_RATE or channels < 1:
raise RuntimeError("STT input must be 16 kHz, 16-bit PCM WAV audio.")
samples = np.frombuffer(frames, dtype=np.int16).astype(np.float32)
if channels > 1:
samples = samples.reshape((-1, channels)).mean(axis=1)
return samples / 32768.0
def _transcribe_capture_with_parakeet(audio_path: Path, *, model: str, language: str) -> str:
parakeet_model = _load_parakeet_onnx_model()
kwargs: Dict[str, Any] = {
"sample_rate": TARGET_SAMPLE_RATE,
"channel": "mean",
}
if language:
kwargs["language"] = language
with PARAKEET_ONNX_TRANSCRIBE_LOCK:
result = parakeet_model.recognize(
_normalized_wav_float32(audio_path),
**kwargs,
)
providers = _parakeet_onnx_providers()
with AUTO_TRAIN_LOCK:
AUTO_TRAIN_STATE["last_stt_engine"] = STT_ENGINE_PARAKEET_ONNX
AUTO_TRAIN_STATE["last_stt_model"] = model
AUTO_TRAIN_STATE["last_stt_device"] = providers[0]
AUTO_TRAIN_STATE["last_stt_compute_type"] = DEFAULT_PARAKEET_ONNX_QUANTIZATION
_save_auto_train_state_locked()
return re.sub(r"\s+", " ", str(result or "")).strip()
def _transcribe_capture(audio_path: Path, *, engine: str, language: str) -> str:
token = _normalize_stt_engine(engine)
model = _managed_stt_model(token, language)
if token == STT_ENGINE_PARAKEET_ONNX:
return _transcribe_capture_with_parakeet(
audio_path,
model=model,
language=language,
)
return _transcribe_capture_with_faster_whisper(
audio_path,
model=model,
language=language,
)
def _clear_stt_model_caches(*, keep_engine: str) -> None:
token = _normalize_stt_engine(keep_engine)
cleared = False
if token != STT_ENGINE_FASTER_WHISPER:
with FASTER_WHISPER_TRANSCRIBE_LOCK:
with FASTER_WHISPER_MODEL_LOCK:
cleared = bool(FASTER_WHISPER_MODEL_CACHE) or cleared
FASTER_WHISPER_MODEL_CACHE.clear()
if token != STT_ENGINE_PARAKEET_ONNX:
with PARAKEET_ONNX_TRANSCRIBE_LOCK:
with PARAKEET_ONNX_MODEL_LOCK:
cleared = bool(PARAKEET_ONNX_MODEL_CACHE) or cleared
PARAKEET_ONNX_MODEL_CACHE.clear()
if cleared:
gc.collect()
def _queue_auto_review(file_name: str) -> bool: def _queue_auto_review(file_name: str) -> bool:
safe_file_name = Path(str(file_name or "")).name safe_file_name = Path(str(file_name or "")).name
if not safe_file_name: if not safe_file_name:
@@ -960,12 +1156,17 @@ def _auto_review_capture(file_name: str) -> None:
metadata["auto_review_status"] = "transcribing" metadata["auto_review_status"] = "transcribing"
metadata["auto_reviewed_at"] = _iso_now() metadata["auto_reviewed_at"] = _iso_now()
metadata["auto_review_wake_phrase"] = wake_phrase metadata["auto_review_wake_phrase"] = wake_phrase
metadata["auto_review_stt_model"] = config["stt_model"] stt_engine = _normalize_stt_engine(config.get("stt_engine"))
metadata["auto_review_stt_engine"] = stt_engine
metadata["auto_review_stt_model"] = _managed_stt_model(
stt_engine,
config.get("language"),
)
_write_sidecar_json(audio_path, metadata) _write_sidecar_json(audio_path, metadata)
transcript = _transcribe_capture_with_faster_whisper( transcript = _transcribe_capture(
audio_path, audio_path,
model=str(config["stt_model"]), engine=stt_engine,
language=str(config.get("language") or DEFAULT_LANGUAGE), language=str(config.get("language") or DEFAULT_LANGUAGE),
) )
normalized = _normalize_transcript_text(transcript) normalized = _normalize_transcript_text(transcript)
@@ -2386,7 +2587,7 @@ def auto_train_status(request: Request):
payload = _auto_train_status_payload() payload = _auto_train_status_payload()
payload["ok"] = True payload["ok"] = True
payload["advertised_base_url"] = _advertised_base_url(request) payload["advertised_base_url"] = _advertised_base_url(request)
payload["stt_backend"] = "faster-whisper" payload["stt_backend"] = payload["config"].get("stt_engine")
return payload return payload
@@ -2420,6 +2621,8 @@ def update_auto_train(payload: Dict[str, Any] = None):
) )
if schedule_changed or not AUTO_TRAIN_STATE.get("next_run_at"): if schedule_changed or not AUTO_TRAIN_STATE.get("next_run_at"):
_schedule_next_auto_run_locked() _schedule_next_auto_run_locked()
if previous.get("stt_engine") != normalized.get("stt_engine"):
_clear_stt_model_caches(keep_engine=normalized["stt_engine"])
if normalized["enabled"]: if normalized["enabled"]:
queued = _queue_pending_auto_reviews() queued = _queue_pending_auto_reviews()
AUTO_TRAIN_WAKE_EVENT.set() AUTO_TRAIN_WAKE_EVENT.set()