Release NVIDIA WakeWord Trainer v13

README.md

@@ -22,7 +22,7 @@ docker pull ghcr.io/tatertotterson/microwakeword:latest
 Tagged releases also publish matching immutable image tags:
 
 ```bash
-docker pull ghcr.io/tatertotterson/microwakeword:v12
+docker pull ghcr.io/tatertotterson/microwakeword:v13
 ```
 
 The release tag must match `VERSION`. Update `WHATS_NEW.md` before tagging; the Docker workflow prepends it to GitHub's automatically generated release notes.
@@ -32,7 +32,7 @@ Python 3.13 TensorFlow build for `sm_120`:
 
 ```bash
 docker pull ghcr.io/tatertotterson/microwakeword:blackwell
-docker pull ghcr.io/tatertotterson/microwakeword:v12-blackwell
+docker pull ghcr.io/tatertotterson/microwakeword:v13-blackwell
 ```
 
 Use the Blackwell image only for RTX 50-series cards. It includes the
@@ -53,9 +53,9 @@ docker run -d \
   ghcr.io/tatertotterson/microwakeword:latest
 ```
 
-Use a version tag such as `ghcr.io/tatertotterson/microwakeword:v12` when you want to pin a known release instead of tracking `latest`.
+Use a version tag such as `ghcr.io/tatertotterson/microwakeword:v13` when you want to pin a known release instead of tracking `latest`.
 For RTX 50-series cards, use `ghcr.io/tatertotterson/microwakeword:blackwell`
-or a pinned tag such as `ghcr.io/tatertotterson/microwakeword:v12-blackwell`
+or a pinned tag such as `ghcr.io/tatertotterson/microwakeword:v13-blackwell`
 in the same `docker run` command.
 
 The flags:
@@ -167,14 +167,21 @@ Starting a new session does not clear samples. Use the clear buttons in `Samples
 
 ## Auto Training
 
-`Auto Training` is an opt-in false-positive loop. It is disabled until you enter the exact wake phrase and enable it.
+`Auto Training` is an opt-in sample-review and retraining loop. It is disabled until you enter the exact wake phrase and enable it.
 
 For each new wake-trigger clip sent to the trainer:
 
 1. Faster Whisper transcribes the audio locally.
-2. If the transcript contains the configured wake phrase, the clip stays in `Captured Audio` for manual positive review.
+2. If the transcript contains the configured wake phrase, the clip stays in `Captured Audio` for manual review by default.
 3. If speech was transcribed but the wake phrase is absent, the clip moves to `/data/negative_samples/` as an auto-reviewed hard negative.
-4. Empty transcripts, close misses, VAD-blocked captures, and captures for another wake word stay out of the automatic negative path.
+4. Empty transcripts, VAD-blocked captures, and captures for another wake word stay out of the automatic negative path.
+
+Two optional cleanup rules are available:
+
+- `Delete confirmed good wakes` removes normal wake-trigger clips after STT confirms the configured phrase.
+- `Promote confirmed close misses` checks close misses that passed VAD and moves them to the personal positive samples only when STT confirms the configured phrase.
+
+A close miss with an empty transcript or without the configured phrase stays in `Captured Audio`; it is never turned into a negative automatically. Saving Auto Training settings also scans existing eligible captures. Enabling close-miss promotion reviews previous unreviewed close misses, while enabling cleanup removes previously confirmed good wakes without transcribing them a second time.
 
 The default `small.en` model uses CUDA with `float16` when CTranslate2 can see an NVIDIA GPU, and falls back to CPU with `int8`. Choose a multilingual Faster Whisper model such as `small` when the wake phrase is not English. Downloaded STT models are cached in `/data/auto_train_models/`.
 

VERSION

@@ -1 +1 @@
-12
+13

WHATS_NEW.md

@@ -1,5 +1,6 @@
 - Added opt-in Auto Training for false-positive wake triggers, using Faster Whisper with automatic CUDA/float16 selection and CPU/int8 fallback.
-- Wake triggers whose transcripts do not contain the configured phrase can now become hard negatives automatically; close misses, empty transcripts, and phrase matches remain available for manual review.
+- Wake triggers whose transcripts do not contain the configured phrase can now become hard negatives automatically; empty transcripts and phrase matches remain available for manual review unless optional cleanup is enabled.
+- Added optional confirmed-good-wake cleanup and conservative close-miss recovery that promotes only VAD-approved, STT-confirmed phrase matches to positive samples.
 - Added scheduled retraining with a minimum-new-negatives threshold and automatic Tater Native satellite refresh after a successful model build.
 - Wake-word download links now advertise a LAN-reachable trainer URL instead of `127.0.0.1`.
 - Tightened detector calibration defaults to favor fewer ambient false accepts while preserving candidates within 0.5 percentage points of the best recall.

static/index.html

@@ -1288,7 +1288,7 @@
           <div>
             <div class="studioKicker">False-Positive Loop</div>
             <h3>Auto Training</h3>
-            <p>Transcribe real wake triggers, turn confirmed phrase-misses into hard negatives, retrain on your schedule, and ask Tater to refresh connected satellites.</p>
+            <p>Sort false wakes into negatives, recover spoken close misses as positives, clean up confirmed wakes, retrain on your schedule, and refresh connected satellites.</p>
             <div class="studioSteps" aria-label="Auto Training steps">
               <span class="studioStepChip"><b>1</b> Transcribe wakes</span>
               <span class="studioStepChip"><b>2</b> Collect negatives</span>
@@ -1305,13 +1305,21 @@
             <span class="studioStepBadge">1</span>
             <div>
               <h3>Review Rules</h3>
-              <p>Only wake-trigger clips are reviewed automatically. Close misses, empty STT results, and clips containing the wake phrase remain in the inbox.</p>
+              <p>False-wake sorting is always conservative. Cleanup and close-miss promotion remain optional, and empty STT results stay in the inbox.</p>
             </div>
           </div>
         </div>
         <label class="checkField">
           <input id="autoEnabled" type="checkbox" />
-          <span><strong>Enable Auto Training</strong>New wake triggers will be queued for local Faster Whisper transcription.</span>
+          <span><strong>Enable Auto Training</strong>Eligible wake triggers will be queued for local Faster Whisper transcription.</span>
+        </label>
+        <label class="checkField">
+          <input id="autoDeleteConfirmedWakes" type="checkbox" />
+          <span><strong>Delete confirmed good wakes</strong>When a normal wake trigger contains the configured phrase, remove it from Captured Audio instead of keeping it for manual review.</span>
+        </label>
+        <label class="checkField">
+          <input id="autoPromoteCloseMisses" type="checkbox" />
+          <span><strong>Promote confirmed close misses</strong>Transcribe close misses that passed VAD and move them to Positive samples only when STT finds the configured phrase.</span>
         </label>
         <div class="autoGrid">
           <label class="field">
@@ -2040,6 +2048,8 @@
       $("autoSttDevice").value = config.stt_device || "auto";
       $("autoSttComputeType").value = config.stt_compute_type || "auto";
       $("autoMinimumChars").value = String(config.minimum_transcript_chars ?? 2);
+      $("autoDeleteConfirmedWakes").checked = Boolean(config.delete_confirmed_wakes);
+      $("autoPromoteCloseMisses").checked = Boolean(config.promote_close_misses);
       $("autoScheduleHours").value = String(config.schedule_hours ?? 24);
       $("autoMinimumNegatives").value = String(config.minimum_new_negatives ?? 3);
       $("autoAdvertisedUrl").value = config.advertised_base_url || "";
@@ -2102,6 +2112,8 @@
       stt_device: $("autoSttDevice").value || "auto",
       stt_compute_type: $("autoSttComputeType").value || "auto",
       minimum_transcript_chars: Number($("autoMinimumChars").value || 2),
+      delete_confirmed_wakes: $("autoDeleteConfirmedWakes").checked,
+      promote_close_misses: $("autoPromoteCloseMisses").checked,
       schedule_hours: Number($("autoScheduleHours").value || 0),
       minimum_new_negatives: Number($("autoMinimumNegatives").value || 3),
       advertised_base_url: ($("autoAdvertisedUrl").value || "").trim(),
@@ -2280,6 +2292,7 @@
     if (when) subtitleParts.push(`Saved ${when}`);
     if (item.message) subtitleParts.push(item.message);
     if (item.auto_negative) subtitleParts.push("Auto-reviewed false positive");
+    if (item.auto_positive) subtitleParts.push("Auto-promoted close miss");
     let revertBtn = '';
     if (item.trimmed) {
         revertBtn = `<button type="button" data-sample-revert="${escapeAttr(item.saved_as)}" data-bucket="${escapeAttr(bucket)}">Revert</button>`;

tests/test_auto_train.py

@@ -1,5 +1,6 @@
 import io
 import json
+import queue
 import sys
 import tempfile
 import unittest
@@ -22,7 +23,18 @@ def silent_wav_bytes(duration_s: float = 0.25) -> bytes:
 
 
 class AutoTrainTests(unittest.TestCase):
+    def clear_review_queue(self):
+        while True:
+            try:
+                trainer.AUTO_TRAIN_REVIEW_QUEUE.get_nowait()
+            except queue.Empty:
+                break
+            else:
+                trainer.AUTO_TRAIN_REVIEW_QUEUE.task_done()
+        trainer.AUTO_TRAIN_QUEUED_FILES.clear()
+
     def setUp(self):
+        self.clear_review_queue()
         self.tempdir = tempfile.TemporaryDirectory()
         root = Path(self.tempdir.name)
         self.original_paths = (
@@ -70,9 +82,16 @@ class AutoTrainTests(unittest.TestCase):
         trainer.AUTO_TRAIN_CONFIG.update(self.original_config)
         trainer.AUTO_TRAIN_STATE.clear()
         trainer.AUTO_TRAIN_STATE.update(self.original_state)
+        self.clear_review_queue()
         self.tempdir.cleanup()
 
-    def add_capture(self, name: str = "wake.wav", wake_word: str = "hey_tater") -> Path:
+    def add_capture(
+        self,
+        name: str = "wake.wav",
+        wake_word: str = "hey_tater",
+        event_type: str = "wake_detected",
+        blocked_by_vad: bool = False,
+    ) -> Path:
         audio_path = trainer.CAPTURED_DIR / name
         audio_path.write_bytes(silent_wav_bytes())
         trainer._write_sidecar_json(
@@ -80,7 +99,8 @@ class AutoTrainTests(unittest.TestCase):
             {
                 "original_name": name,
                 "wake_word": wake_word,
-                "event_type": "wake_detected",
+                "event_type": event_type,
+                "blocked_by_vad": blocked_by_vad,
                 "review_status": "pending",
             },
         )
@@ -115,6 +135,100 @@ class AutoTrainTests(unittest.TestCase):
         self.assertEqual(metadata["auto_review_status"], "wake_phrase_detected")
         self.assertEqual(trainer.AUTO_TRAIN_STATE["pending_negative_count"], 0)
 
+    def test_matching_phrase_is_deleted_when_cleanup_is_enabled(self):
+        audio_path = self.add_capture()
+        trainer.AUTO_TRAIN_CONFIG["delete_confirmed_wakes"] = True
+        with patch.object(
+            trainer,
+            "_transcribe_capture_with_faster_whisper",
+            return_value="hey tater turn on the lights",
+        ):
+            trainer._auto_review_capture("wake.wav")
+
+        self.assertFalse(audio_path.exists())
+        self.assertFalse(audio_path.with_suffix(".json").exists())
+        self.assertFalse(list(trainer.PERSONAL_DIR.glob("*.wav")))
+        self.assertFalse(list(trainer.NEGATIVE_DIR.glob("*.wav")))
+        self.assertEqual(trainer.AUTO_TRAIN_STATE["last_review_result"], "deleted_confirmed_wake")
+
+    def test_cleanup_processes_previously_confirmed_wake_without_retranscribing(self):
+        audio_path = self.add_capture()
+        metadata = trainer._load_sidecar_json(audio_path)
+        metadata.update(
+            {
+                "auto_review_status": "wake_phrase_detected",
+                "transcript": "hey tater",
+            }
+        )
+        trainer._write_sidecar_json(audio_path, metadata)
+        trainer.AUTO_TRAIN_CONFIG["delete_confirmed_wakes"] = True
+
+        self.assertEqual(trainer._queue_pending_auto_reviews(), 1)
+        with patch.object(trainer, "_transcribe_capture_with_faster_whisper") as transcribe:
+            trainer._auto_review_capture("wake.wav")
+
+        transcribe.assert_not_called()
+        self.assertFalse(audio_path.exists())
+        self.assertEqual(trainer.AUTO_TRAIN_STATE["last_review_transcript"], "hey tater")
+
+    def test_close_miss_is_not_transcribed_by_default(self):
+        audio_path = self.add_capture(event_type="close_miss")
+        with patch.object(trainer, "_transcribe_capture_with_faster_whisper") as transcribe:
+            trainer._auto_review_capture("wake.wav")
+
+        transcribe.assert_not_called()
+        self.assertTrue(audio_path.exists())
+        self.assertFalse(trainer._load_sidecar_json(audio_path).get("auto_review_status"))
+
+    def test_existing_close_miss_is_queued_when_promotion_is_enabled(self):
+        self.add_capture(event_type="close_miss")
+        self.assertEqual(trainer._queue_pending_auto_reviews(), 0)
+
+        trainer.AUTO_TRAIN_CONFIG["promote_close_misses"] = True
+        self.assertEqual(trainer._queue_pending_auto_reviews(), 1)
+
+    def test_close_miss_with_phrase_is_promoted_when_enabled(self):
+        self.add_capture(event_type="close_miss")
+        trainer.AUTO_TRAIN_CONFIG["promote_close_misses"] = True
+        with patch.object(trainer, "_transcribe_capture_with_faster_whisper", return_value="hey tater"):
+            trainer._auto_review_capture("wake.wav")
+
+        self.assertFalse((trainer.CAPTURED_DIR / "wake.wav").exists())
+        positives = list(trainer.PERSONAL_DIR.glob("*.wav"))
+        self.assertEqual(len(positives), 1)
+        metadata = trainer._load_sidecar_json(positives[0])
+        self.assertTrue(metadata["auto_positive"])
+        self.assertEqual(metadata["review_status"], "auto_approved_personal")
+        self.assertEqual(metadata["transcript"], "hey tater")
+        self.assertFalse(list(trainer.NEGATIVE_DIR.glob("*.wav")))
+        self.assertEqual(trainer.AUTO_TRAIN_STATE["pending_negative_count"], 0)
+
+    def test_close_miss_without_phrase_stays_in_inbox(self):
+        audio_path = self.add_capture(event_type="close_miss")
+        trainer.AUTO_TRAIN_CONFIG["promote_close_misses"] = True
+        with patch.object(
+            trainer,
+            "_transcribe_capture_with_faster_whisper",
+            return_value="turn on the lights",
+        ):
+            trainer._auto_review_capture("wake.wav")
+
+        self.assertTrue(audio_path.exists())
+        self.assertFalse(list(trainer.PERSONAL_DIR.glob("*.wav")))
+        self.assertFalse(list(trainer.NEGATIVE_DIR.glob("*.wav")))
+        metadata = trainer._load_sidecar_json(audio_path)
+        self.assertEqual(metadata["auto_review_status"], "close_miss_phrase_not_detected")
+
+    def test_vad_blocked_close_miss_is_never_transcribed(self):
+        audio_path = self.add_capture(event_type="close_miss", blocked_by_vad=True)
+        trainer.AUTO_TRAIN_CONFIG["promote_close_misses"] = True
+        with patch.object(trainer, "_transcribe_capture_with_faster_whisper") as transcribe:
+            trainer._auto_review_capture("wake.wav")
+
+        transcribe.assert_not_called()
+        self.assertTrue(audio_path.exists())
+        self.assertFalse(trainer._load_sidecar_json(audio_path).get("auto_review_status"))
+
     def test_capture_for_another_wake_word_is_not_transcribed(self):
         audio_path = self.add_capture(wake_word="computer")
         with patch.object(trainer, "_transcribe_capture_with_faster_whisper") as transcribe:

trainer_server.py

@@ -94,6 +94,8 @@ AUTO_TRAIN_DEFAULT_CONFIG: Dict[str, Any] = {
     "stt_device": "auto",
     "stt_compute_type": "auto",
     "minimum_transcript_chars": 2,
+    "delete_confirmed_wakes": False,
+    "promote_close_misses": False,
     "schedule_hours": 24,
     "minimum_new_negatives": 3,
     "advertised_base_url": "",
@@ -473,6 +475,8 @@ def _normalize_auto_train_config(values: Dict[str, Any] | None, *, base: Dict[st
         "stt_device": stt_device,
         "stt_compute_type": stt_compute_type,
         "minimum_transcript_chars": _bounded_int(source.get("minimum_transcript_chars"), 2, 1, 100),
+        "delete_confirmed_wakes": _config_bool(source.get("delete_confirmed_wakes")),
+        "promote_close_misses": _config_bool(source.get("promote_close_misses")),
         "schedule_hours": schedule_hours,
         "minimum_new_negatives": _bounded_int(source.get("minimum_new_negatives"), 3, 1, 10000),
         "advertised_base_url": _normalize_http_base_url(source.get("advertised_base_url")),
@@ -636,12 +640,20 @@ def _transcript_contains_wake_phrase(transcript: Any, wake_phrase: Any) -> bool:
     return f" {normalized_phrase} " in f" {normalized_transcript} "
 
 
-def _captured_event_is_auto_reviewable(metadata: Dict[str, Any]) -> bool:
+def _captured_event_is_close_miss(metadata: Dict[str, Any]) -> bool:
+    event_type = str(metadata.get("event_type") or "captured").strip().lower()
+    return "close" in event_type
+
+
+def _captured_event_is_auto_reviewable(
+    metadata: Dict[str, Any],
+    config: Dict[str, Any] | None = None,
+) -> bool:
     if _parse_bool(metadata.get("blocked_by_vad")):
         return False
     event_type = str(metadata.get("event_type") or "captured").strip().lower()
     if "close" in event_type:
-        return False
+        return bool((config or {}).get("promote_close_misses"))
     return event_type in {"captured", "trigger", "false_trigger"} or "wake" in event_type or "detect" in event_type
 
 
@@ -733,10 +745,14 @@ def _queue_auto_review(file_name: str) -> bool:
 
 def _queue_pending_auto_reviews(*, force: bool = False) -> int:
     queued = 0
+    with AUTO_TRAIN_LOCK:
+        config = dict(AUTO_TRAIN_CONFIG)
+    if not config.get("enabled"):
+        return queued
     CAPTURED_DIR.mkdir(parents=True, exist_ok=True)
     for audio_path in sorted(CAPTURED_DIR.glob("*.wav")):
         metadata = _load_sidecar_json(audio_path)
-        if not _captured_event_is_auto_reviewable(metadata):
+        if not _captured_event_is_auto_reviewable(metadata, config):
             continue
         status = str(metadata.get("auto_review_status") or "").strip()
         if status == "transcribing":
@@ -747,6 +763,12 @@ def _queue_pending_auto_reviews(*, force: bool = False) -> int:
             metadata.pop("auto_review_status", None)
             _write_sidecar_json(audio_path, metadata)
             status = ""
+        if (
+            status == "wake_phrase_detected"
+            and config.get("delete_confirmed_wakes")
+            and not _captured_event_is_close_miss(metadata)
+        ):
+            status = ""
         if status:
             continue
         if _queue_auto_review(audio_path.name):
@@ -782,7 +804,22 @@ def _auto_review_capture(file_name: str) -> None:
         except FileNotFoundError:
             return
         metadata = _load_sidecar_json(audio_path)
-        if metadata.get("auto_review_status") or not _captured_event_is_auto_reviewable(metadata):
+        is_close_miss = _captured_event_is_close_miss(metadata)
+        status = str(metadata.get("auto_review_status") or "").strip()
+        if (
+            status == "wake_phrase_detected"
+            and config.get("delete_confirmed_wakes")
+            and not is_close_miss
+        ):
+            transcript = str(metadata.get("transcript") or "")
+            _remove_audio_with_sidecar(audio_path)
+            _record_auto_review_result(
+                file_name=file_name,
+                transcript=transcript,
+                result="deleted_confirmed_wake",
+            )
+            return
+        if status or not _captured_event_is_auto_reviewable(metadata, config):
             return
         captured_wake_phrase = str(metadata.get("wake_word") or "").strip()
         if captured_wake_phrase and _normalize_transcript_text(captured_wake_phrase) != _normalize_transcript_text(wake_phrase):
@@ -819,12 +856,52 @@ def _auto_review_capture(file_name: str) -> None:
             return
 
         if _transcript_contains_wake_phrase(transcript, wake_phrase):
+            if is_close_miss:
+                metadata["auto_review_status"] = "approved_positive"
+                metadata["auto_review_reason"] = (
+                    "Close miss contained the configured wake phrase and was promoted to a positive sample."
+                )
+                metadata["auto_positive"] = True
+                _write_sidecar_json(audio_path, metadata)
+                _move_captured_audio(
+                    file_name,
+                    PERSONAL_DIR,
+                    target_prefix="sample",
+                    review_status="auto_approved_personal",
+                )
+                _record_auto_review_result(
+                    file_name=file_name,
+                    transcript=transcript,
+                    result="promoted_close_miss",
+                )
+                return
+            if config.get("delete_confirmed_wakes"):
+                _remove_audio_with_sidecar(audio_path)
+                _record_auto_review_result(
+                    file_name=file_name,
+                    transcript=transcript,
+                    result="deleted_confirmed_wake",
+                )
+                return
             metadata["auto_review_status"] = "wake_phrase_detected"
             metadata["auto_review_reason"] = "Wake phrase found in transcript; left for manual positive review."
             _write_sidecar_json(audio_path, metadata)
             _record_auto_review_result(file_name=file_name, transcript=transcript, result="wake_phrase_detected")
             return
 
+        if is_close_miss:
+            metadata["auto_review_status"] = "close_miss_phrase_not_detected"
+            metadata["auto_review_reason"] = (
+                "Close miss did not contain the configured wake phrase; left for manual review."
+            )
+            _write_sidecar_json(audio_path, metadata)
+            _record_auto_review_result(
+                file_name=file_name,
+                transcript=transcript,
+                result="close_miss_phrase_not_detected",
+            )
+            return
+
         metadata["auto_review_status"] = "approved_negative"
         metadata["auto_review_reason"] = "Wake phrase was not found in the STT transcript."
         metadata["auto_negative"] = True
@@ -1719,6 +1796,7 @@ def _sample_item_from_path(audio_path: Path, bucket: str) -> Dict[str, Any]:
         "transcript": meta.get("transcript") or "",
         "transcribed_at": meta.get("transcribed_at") or "",
         "auto_negative": bool(meta.get("auto_negative")),
+        "auto_positive": bool(meta.get("auto_positive")),
         "auto_review_reason": meta.get("auto_review_reason") or "",
         "size_bytes": stat.st_size,
         "audio_url": f"/api/audio/{bucket}/{audio_path.name}",
@@ -2401,8 +2479,8 @@ async def upload_captured_audio(
     }
     _write_sidecar_json(audio_path, sidecar)
     with AUTO_TRAIN_LOCK:
-        auto_review_enabled = bool(AUTO_TRAIN_CONFIG.get("enabled"))
+        auto_review_config = dict(AUTO_TRAIN_CONFIG)
-    if auto_review_enabled and _captured_event_is_auto_reviewable(sidecar):
+    if auto_review_config.get("enabled") and _captured_event_is_auto_reviewable(sidecar, auto_review_config):
         _queue_auto_review(audio_path.name)
 
     return {
@@ -2487,8 +2565,8 @@ async def upload_captured_audio_raw(
     }
     _write_sidecar_json(audio_path, sidecar)
     with AUTO_TRAIN_LOCK:
-        auto_review_enabled = bool(AUTO_TRAIN_CONFIG.get("enabled"))
+        auto_review_config = dict(AUTO_TRAIN_CONFIG)
-    if auto_review_enabled and _captured_event_is_auto_reviewable(sidecar):
+    if auto_review_config.get("enabled") and _captured_event_is_auto_reviewable(sidecar, auto_review_config):
         _queue_auto_review(audio_path.name)
 
     return {