mirror of
https://github.com/TaterTotterson/microWakeWord-Trainer-Nvidia-Docker.git
synced 2026-08-12 07:55:33 -06:00
Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7d77f71dc3 |
18
.github/workflows/docker-publish.yml
vendored
18
.github/workflows/docker-publish.yml
vendored
@@ -26,6 +26,18 @@ jobs:
|
||||
- name: Check out repository
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Validate tag matches trainer version
|
||||
if: startsWith(github.ref, 'refs/tags/')
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
version="$(tr -d '[:space:]' < VERSION)"
|
||||
expected_tag="v${version#v}"
|
||||
if [[ "${GITHUB_REF_NAME}" != "${expected_tag}" ]]; then
|
||||
echo "Tag ${GITHUB_REF_NAME} does not match trainer version ${expected_tag}." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
@@ -94,6 +106,7 @@ jobs:
|
||||
title="microWakeWord Nvidia Trainer ${TAG_NAME}"
|
||||
generated_notes="$(mktemp)"
|
||||
release_notes="$(mktemp)"
|
||||
test -s WHATS_NEW.md
|
||||
|
||||
gh api "repos/${REPO}/releases/generate-notes" \
|
||||
-f tag_name="${TAG_NAME}" \
|
||||
@@ -101,6 +114,11 @@ jobs:
|
||||
--jq '.body' > "${generated_notes}"
|
||||
|
||||
{
|
||||
echo "## What's New"
|
||||
echo
|
||||
cat WHATS_NEW.md
|
||||
echo
|
||||
echo
|
||||
echo "## Docker Images"
|
||||
echo
|
||||
echo "- \`ghcr.io/tatertotterson/microwakeword:${TAG_NAME}\`"
|
||||
|
||||
42
README.md
42
README.md
@@ -22,15 +22,17 @@ docker pull ghcr.io/tatertotterson/microwakeword:latest
|
||||
Tagged releases also publish matching immutable image tags:
|
||||
|
||||
```bash
|
||||
docker pull ghcr.io/tatertotterson/microwakeword:v11
|
||||
docker pull ghcr.io/tatertotterson/microwakeword:v12
|
||||
```
|
||||
|
||||
The release tag must match `VERSION`. Update `WHATS_NEW.md` before tagging; the Docker workflow prepends it to GitHub's automatically generated release notes.
|
||||
|
||||
RTX 50-series / Blackwell GPUs use a separate image with CUDA 12.8 and a
|
||||
Python 3.13 TensorFlow build for `sm_120`:
|
||||
|
||||
```bash
|
||||
docker pull ghcr.io/tatertotterson/microwakeword:blackwell
|
||||
docker pull ghcr.io/tatertotterson/microwakeword:v11-blackwell
|
||||
docker pull ghcr.io/tatertotterson/microwakeword:v12-blackwell
|
||||
```
|
||||
|
||||
Use the Blackwell image only for RTX 50-series cards. It includes the
|
||||
@@ -51,9 +53,9 @@ docker run -d \
|
||||
ghcr.io/tatertotterson/microwakeword:latest
|
||||
```
|
||||
|
||||
Use a version tag such as `ghcr.io/tatertotterson/microwakeword:v11` when you want to pin a known release instead of tracking `latest`.
|
||||
Use a version tag such as `ghcr.io/tatertotterson/microwakeword:v12` when you want to pin a known release instead of tracking `latest`.
|
||||
For RTX 50-series cards, use `ghcr.io/tatertotterson/microwakeword:blackwell`
|
||||
or a pinned tag such as `ghcr.io/tatertotterson/microwakeword:v11-blackwell`
|
||||
or a pinned tag such as `ghcr.io/tatertotterson/microwakeword:v12-blackwell`
|
||||
in the same `docker run` command.
|
||||
|
||||
The flags:
|
||||
@@ -78,6 +80,7 @@ If you change `REC_PORT`, open that port instead and use the same port in the sa
|
||||
## What The UI Does
|
||||
|
||||
- `Trainer` starts a wake-word session, shows positive/negative sample counts, and launches training.
|
||||
- `Auto Training` transcribes real wake triggers, promotes phrase-misses to hard negatives, schedules retraining, and refreshes Tater Native satellites.
|
||||
- `Captured Audio` reviews clips sent by Tater Native or ESPHome sats, including wake hits, close misses, and false wakes.
|
||||
- `Samples` plays, removes, clears, and manually imports personal or negative samples.
|
||||
- `Wake Words` lists locally trained JSON/model links for live wake-word switching in Tater.
|
||||
@@ -162,6 +165,27 @@ Starting a new session does not clear samples. Use the clear buttons in `Samples
|
||||
|
||||
---
|
||||
|
||||
## Auto Training
|
||||
|
||||
`Auto Training` is an opt-in false-positive loop. It is disabled until you enter the exact wake phrase and enable it.
|
||||
|
||||
For each new wake-trigger clip sent to the trainer:
|
||||
|
||||
1. Faster Whisper transcribes the audio locally.
|
||||
2. If the transcript contains the configured wake phrase, the clip stays in `Captured Audio` for manual positive review.
|
||||
3. If speech was transcribed but the wake phrase is absent, the clip moves to `/data/negative_samples/` as an auto-reviewed hard negative.
|
||||
4. Empty transcripts, close misses, VAD-blocked captures, and captures for another wake word stay out of the automatic negative path.
|
||||
|
||||
The default `small.en` model uses CUDA with `float16` when CTranslate2 can see an NVIDIA GPU, and falls back to CPU with `int8`. Choose a multilingual Faster Whisper model such as `small` when the wake phrase is not English. Downloaded STT models are cached in `/data/auto_train_models/`.
|
||||
|
||||
Scheduled training runs only after the configured number of new automatic negatives has accumulated. A successful run publishes the replacement model at the same wake-word URL and can call Tater's native satellite settings API to make connected satellites pull it again. This refresh uses the existing Tater Native update path, so no satellite firmware change is required.
|
||||
|
||||
The `Trainer public URL` must be reachable from the satellites. With the documented `--network host` command, the trainer can normally use the LAN address from the browser request or host network. If you open the UI as `http://localhost:8789`, enter a value such as `http://192.168.1.50:8789`, or start the container with `REC_PUBLIC_BASE_URL` set to that value. When using Docker bridge networking, always set this URL to the published host address; a container bridge address is not satellite-reachable.
|
||||
|
||||
The default Tater URL, `http://127.0.0.1:8501`, assumes the documented host networking. Change it to a container-reachable Tater address if you use another Docker network. The optional API token is stored in `/data/auto_train_config.json` with owner-only permissions.
|
||||
|
||||
---
|
||||
|
||||
## Training Flow
|
||||
|
||||
1. Enter the wake phrase in `Trainer`.
|
||||
@@ -212,6 +236,7 @@ After those assets are prepared, later runs reuse the local copies unless the mo
|
||||
The `Wake Words` tab lists locally trained wake-word packages from `/data/trained_wake_words/`.
|
||||
|
||||
- Copy the JSON URL into the Tater Native satellite settings to switch wake words live.
|
||||
- Links use the configured public trainer URL, a non-loopback browser host, or the detected LAN address instead of advertising `127.0.0.1` to satellites.
|
||||
- Open the JSON or model links directly for quick inspection.
|
||||
- The JSON includes the matching model path plus Tater tuning metadata.
|
||||
- No firmware flashing happens from this trainer app anymore.
|
||||
@@ -244,7 +269,7 @@ The JSON keeps the standard microWakeWord fields for compatibility:
|
||||
{
|
||||
"micro": {
|
||||
"probability_cutoff": 0.97,
|
||||
"sliding_window_size": 5
|
||||
"sliding_window_size": 6
|
||||
}
|
||||
}
|
||||
```
|
||||
@@ -259,8 +284,8 @@ It also includes Tater Native metadata used by newer satellites and the Tater se
|
||||
"tater_native": {
|
||||
"format_version": 1,
|
||||
"wake_threshold": 0.97,
|
||||
"wake_sliding_window": 5,
|
||||
"close_miss_threshold": 0.78,
|
||||
"wake_sliding_window": 6,
|
||||
"close_miss_threshold": 0.80,
|
||||
"frontend": {
|
||||
"name": "tflm_microfrontend",
|
||||
"sample_rate": 16000,
|
||||
@@ -273,6 +298,7 @@ It also includes Tater Native metadata used by newer satellites and the Tater se
|
||||
```
|
||||
|
||||
Calibration metrics are included under `calibration` so false accepts/hour and recall can be surfaced in the UI.
|
||||
Calibration evaluates thresholds from `0.95` through `1.00` with sliding windows of `5`, `6`, and `7`. Among candidates within 0.5 percentage points of the best recall, it prefers the lowest measured ambient false-accept rate. If calibration cannot complete, packaging uses the conservative `0.97` threshold and a window of `6`.
|
||||
|
||||
---
|
||||
|
||||
@@ -289,6 +315,7 @@ That removes:
|
||||
- cached datasets
|
||||
- training environments
|
||||
- trained models
|
||||
- Auto Training settings, state, transcripts, and cached Faster Whisper models
|
||||
|
||||
---
|
||||
|
||||
@@ -296,6 +323,7 @@ That removes:
|
||||
|
||||
- Personal samples are optional.
|
||||
- Negative samples are optional but useful for reducing false wakes.
|
||||
- Auto Training is disabled by default and only classifies actual wake triggers automatically.
|
||||
- The UI server is `trainer_server.py`.
|
||||
- The launcher is `run.sh`.
|
||||
- Trainer capture settings live in Tater for Tater Native satellites, and on device entities for older ESPHome satellites.
|
||||
|
||||
5
WHATS_NEW.md
Normal file
5
WHATS_NEW.md
Normal file
@@ -0,0 +1,5 @@
|
||||
- Added opt-in Auto Training for false-positive wake triggers, using Faster Whisper with automatic CUDA/float16 selection and CPU/int8 fallback.
|
||||
- Wake triggers whose transcripts do not contain the configured phrase can now become hard negatives automatically; close misses, empty transcripts, and phrase matches remain available for manual review.
|
||||
- Added scheduled retraining with a minimum-new-negatives threshold and automatic Tater Native satellite refresh after a successful model build.
|
||||
- Wake-word download links now advertise a LAN-reachable trainer URL instead of `127.0.0.1`.
|
||||
- Tightened detector calibration defaults to favor fewer ambient false accepts while preserving candidates within 0.5 percentage points of the best recall.
|
||||
@@ -9,24 +9,22 @@ import math
|
||||
import os
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Iterable, Sequence
|
||||
from typing import Any, Iterable, Sequence
|
||||
|
||||
import numpy as np
|
||||
import yaml
|
||||
|
||||
from microwakeword.data import FeatureHandler
|
||||
from microwakeword.inference import Model
|
||||
|
||||
|
||||
DEFAULT_WINDOW_SIZES = [4, 5, 6, 7]
|
||||
DEFAULT_WINDOW_SIZES = [5, 6, 7]
|
||||
DEFAULT_TARGET_FAPH = float(os.environ.get("MWW_CALIBRATION_TARGET_FAPH", "0.25"))
|
||||
DEFAULT_COOLDOWN_SLICES = int(os.environ.get("MWW_CALIBRATION_COOLDOWN_SLICES", "25"))
|
||||
DEFAULT_POSITIVE_SKIP_SLICES = int(
|
||||
os.environ.get("MWW_CALIBRATION_POSITIVE_SKIP_SLICES", "25")
|
||||
)
|
||||
DEFAULT_CUTOFF_STEP = float(os.environ.get("MWW_CALIBRATION_CUTOFF_STEP", "0.01"))
|
||||
DEFAULT_CUTOFF_MIN = float(os.environ.get("MWW_CALIBRATION_CUTOFF_MIN", "0.85"))
|
||||
DEFAULT_CUTOFF_MIN = float(os.environ.get("MWW_CALIBRATION_CUTOFF_MIN", "0.95"))
|
||||
DEFAULT_CUTOFF_MAX = float(os.environ.get("MWW_CALIBRATION_CUTOFF_MAX", "1.00"))
|
||||
DEFAULT_RECALL_MARGIN = float(os.environ.get("MWW_CALIBRATION_RECALL_MARGIN", "0.005"))
|
||||
PREFERRED_WINDOW_SIZE = 6
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
@@ -65,6 +63,15 @@ def parse_args() -> argparse.Namespace:
|
||||
default=DEFAULT_TARGET_FAPH,
|
||||
help="Target ambient false accepts per hour for the selected operating point.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--recall-margin",
|
||||
type=float,
|
||||
default=DEFAULT_RECALL_MARGIN,
|
||||
help=(
|
||||
"Maximum recall loss allowed when preferring a candidate with fewer "
|
||||
"ambient false accepts (0.005 means 0.5 percentage points)."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--cooldown-slices",
|
||||
type=int,
|
||||
@@ -159,7 +166,13 @@ def _compute_false_accepts_per_hour(
|
||||
def _select_best_candidate(
|
||||
candidates: list[dict[str, float]],
|
||||
target_faph: float,
|
||||
recall_margin: float = DEFAULT_RECALL_MARGIN,
|
||||
) -> tuple[dict[str, float], float]:
|
||||
if not candidates:
|
||||
raise ValueError("at least one calibration candidate is required")
|
||||
if recall_margin < 0:
|
||||
raise ValueError("recall margin must be >= 0")
|
||||
|
||||
fallback_limits = [
|
||||
target_faph,
|
||||
max(target_faph * 2.0, target_faph + 0.5),
|
||||
@@ -172,13 +185,27 @@ def _select_best_candidate(
|
||||
return index
|
||||
return len(fallback_limits)
|
||||
|
||||
# Stay in the strictest false-accept tier that has a viable candidate. Within
|
||||
# that tier, keep candidates close to the best recall, then spend the allowed
|
||||
# recall margin on the lowest measured false-accept rate.
|
||||
best_tier = min(tier(candidate) for candidate in candidates)
|
||||
tier_candidates = [
|
||||
candidate for candidate in candidates if tier(candidate) == best_tier
|
||||
]
|
||||
best_recall = max(candidate["recall"] for candidate in tier_candidates)
|
||||
recall_floor = best_recall - recall_margin
|
||||
viable_candidates = [
|
||||
candidate
|
||||
for candidate in tier_candidates
|
||||
if candidate["recall"] >= recall_floor - 1e-12
|
||||
]
|
||||
|
||||
best = min(
|
||||
candidates,
|
||||
viable_candidates,
|
||||
key=lambda candidate: (
|
||||
tier(candidate),
|
||||
-candidate["recall"],
|
||||
candidate["false_accepts_per_hour"],
|
||||
abs(candidate["sliding_window_size"] - 5),
|
||||
-candidate["recall"],
|
||||
abs(candidate["sliding_window_size"] - PREFERRED_WINDOW_SIZE),
|
||||
-candidate["probability_cutoff"],
|
||||
),
|
||||
)
|
||||
@@ -195,7 +222,7 @@ def _load_config(config_path: Path) -> dict:
|
||||
|
||||
|
||||
def _load_eval_sets(
|
||||
handler: FeatureHandler,
|
||||
handler: Any,
|
||||
config: dict,
|
||||
) -> tuple[str, str, list[np.ndarray], list[np.ndarray]]:
|
||||
for positive_mode, ambient_mode in (
|
||||
@@ -228,7 +255,7 @@ def _load_eval_sets(
|
||||
|
||||
|
||||
def _predict_tracks(
|
||||
model: Model,
|
||||
model: Any,
|
||||
tracks: Sequence[np.ndarray],
|
||||
label: str,
|
||||
) -> list[np.ndarray]:
|
||||
@@ -244,8 +271,13 @@ def _predict_tracks(
|
||||
|
||||
|
||||
def main() -> int:
|
||||
from microwakeword.data import FeatureHandler
|
||||
from microwakeword.inference import Model
|
||||
|
||||
args = parse_args()
|
||||
window_sizes = _parse_window_sizes(args.window_sizes)
|
||||
if args.recall_margin < 0 or args.recall_margin > 1:
|
||||
raise ValueError("recall-margin must be between 0 and 1")
|
||||
if args.cutoff_step <= 0:
|
||||
raise ValueError("cutoff-step must be > 0")
|
||||
if args.cutoff_max < args.cutoff_min:
|
||||
@@ -276,6 +308,10 @@ def main() -> int:
|
||||
f"→ Evaluating window sizes {window_sizes} with target <= "
|
||||
f"{args.target_faph:.2f} false accepts/hour"
|
||||
)
|
||||
print(
|
||||
f"→ Favoring lower false accepts within "
|
||||
f"{args.recall_margin:.2%} of the best recall"
|
||||
)
|
||||
|
||||
config = _load_config(config_path)
|
||||
config["flags"] = config.get("flags", {})
|
||||
@@ -338,7 +374,11 @@ def main() -> int:
|
||||
candidates.append(candidate)
|
||||
window_candidates.append(candidate)
|
||||
|
||||
best_window, _ = _select_best_candidate(window_candidates, args.target_faph)
|
||||
best_window, _ = _select_best_candidate(
|
||||
window_candidates,
|
||||
args.target_faph,
|
||||
args.recall_margin,
|
||||
)
|
||||
best_by_window.append(best_window)
|
||||
print(
|
||||
" window={window}: cutoff={cutoff:.2f}; recall={recall:.2%}; "
|
||||
@@ -350,7 +390,11 @@ def main() -> int:
|
||||
)
|
||||
)
|
||||
|
||||
best, selected_limit = _select_best_candidate(candidates, args.target_faph)
|
||||
best, selected_limit = _select_best_candidate(
|
||||
candidates,
|
||||
args.target_faph,
|
||||
args.recall_margin,
|
||||
)
|
||||
if best["false_accepts_per_hour"] > args.target_faph + 1e-9:
|
||||
print(
|
||||
"⚠️ No candidate met the target false accepts/hour budget; "
|
||||
@@ -390,6 +434,8 @@ def main() -> int:
|
||||
"cutoff_min": round(float(cutoffs[0]), 4),
|
||||
"cutoff_max": round(float(cutoffs[-1]), 4),
|
||||
"cutoff_step": float(args.cutoff_step),
|
||||
"recall_margin": float(args.recall_margin),
|
||||
"preferred_window_size": PREFERRED_WINDOW_SIZE,
|
||||
},
|
||||
"per_window_best": best_by_window,
|
||||
"generated_at": datetime.now(timezone.utc).isoformat(),
|
||||
|
||||
@@ -467,7 +467,12 @@ echo "🎯 Calibrating detector settings for on-device use…"
|
||||
if "${PYTHON_BIN:-python}" "${PROGDIR}/calibrate_detector.py" \
|
||||
--training-config "${WORK_DIR}/trained_models/wakeword/training_config.yaml" \
|
||||
--model "${source_path}" \
|
||||
--output "${calibration_path}"; then
|
||||
--output "${calibration_path}" \
|
||||
--target-faph "${MWW_CALIBRATION_TARGET_FAPH:-0.25}" \
|
||||
--recall-margin "${MWW_CALIBRATION_RECALL_MARGIN:-0.005}" \
|
||||
--window-sizes "${MWW_CALIBRATION_WINDOW_SIZES:-5,6,7}" \
|
||||
--cutoff-min "${MWW_CALIBRATION_CUTOFF_MIN:-0.95}" \
|
||||
--cutoff-max "${MWW_CALIBRATION_CUTOFF_MAX:-1.00}"; then
|
||||
echo "✅ Detector calibration complete."
|
||||
else
|
||||
echo "⚠️ Detector calibration failed; packaging with default detector settings."
|
||||
@@ -496,8 +501,8 @@ from pathlib import Path
|
||||
json_path = Path(os.environ["JSON_PATH"])
|
||||
calibration_path = Path(os.environ.get("CALIBRATION_PATH", ""))
|
||||
language = (os.environ.get("LANGUAGE", "en") or "en").strip().lower()
|
||||
probability_cutoff = 0.85
|
||||
sliding_window_size = 4
|
||||
probability_cutoff = 0.97
|
||||
sliding_window_size = 6
|
||||
strict_min_close_miss_threshold = 0.68
|
||||
calibration = {}
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@ ENV DEBIAN_FRONTEND=noninteractive
|
||||
# System deps
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
python3.12 python3.12-venv python3.12-dev python3-pip python-is-python3 \
|
||||
git wget curl unzip patch ninja-build ca-certificates nano less \
|
||||
git wget curl unzip patch ninja-build ca-certificates nano less libgomp1 \
|
||||
&& rm -rf /var/lib/apt/lists/* \
|
||||
&& mkdir -p /data
|
||||
|
||||
|
||||
@@ -13,7 +13,7 @@ ENV MWW_BLACKWELL_TF_WHEEL_URL=https://github.com/chivitiH/tensorflow-blackwell-
|
||||
# Python 3.13 is used only for the Blackwell TensorFlow training step.
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
software-properties-common ca-certificates curl git wget unzip patch \
|
||||
ninja-build nano less \
|
||||
ninja-build nano less libgomp1 \
|
||||
&& add-apt-repository -y ppa:deadsnakes/ppa \
|
||||
&& apt-get update \
|
||||
&& apt-get install -y --no-install-recommends \
|
||||
|
||||
34
run.sh
34
run.sh
@@ -31,7 +31,10 @@ install_ui_deps() {
|
||||
"uvicorn[standard]==${UVICORN_VERSION}" \
|
||||
"python-multipart==${PY_MULTIPART_VERSION}" \
|
||||
"silero-vad>=5.0.0" \
|
||||
"numpy>=1.24.0"
|
||||
"numpy>=1.24.0" \
|
||||
"faster-whisper>=1.0.0" \
|
||||
"nvidia-cublas-cu12" \
|
||||
"nvidia-cudnn-cu12==9.*"
|
||||
}
|
||||
|
||||
# -----------------------------
|
||||
@@ -78,8 +81,13 @@ exact = {
|
||||
minimum = {
|
||||
"silero-vad": "5.0.0",
|
||||
"numpy": "1.24.0",
|
||||
"faster-whisper": "1.0.0",
|
||||
"nvidia-cudnn-cu12": "9.0.0",
|
||||
}
|
||||
present = ("torch",)
|
||||
present = (
|
||||
"torch",
|
||||
"nvidia-cublas-cu12",
|
||||
)
|
||||
|
||||
for package, expected in exact.items():
|
||||
if md.version(package) != expected:
|
||||
@@ -95,6 +103,28 @@ PY
|
||||
install_ui_deps
|
||||
fi
|
||||
fi
|
||||
|
||||
# Faster Whisper/CTranslate2 loads these CUDA libraries before Python starts.
|
||||
# They live in the persistent UI venv so both Docker image variants can use GPU STT.
|
||||
WHISPER_CUDA_LIBRARY_PATH="$("${PY}" - <<'PY'
|
||||
import os
|
||||
|
||||
try:
|
||||
import nvidia.cublas.lib
|
||||
import nvidia.cudnn.lib
|
||||
except ImportError:
|
||||
print("")
|
||||
else:
|
||||
print(
|
||||
os.path.dirname(nvidia.cublas.lib.__file__)
|
||||
+ ":"
|
||||
+ os.path.dirname(nvidia.cudnn.lib.__file__)
|
||||
)
|
||||
PY
|
||||
)"
|
||||
if [[ -n "${WHISPER_CUDA_LIBRARY_PATH}" ]]; then
|
||||
export LD_LIBRARY_PATH="${WHISPER_CUDA_LIBRARY_PATH}${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}"
|
||||
fi
|
||||
# -----------------------------
|
||||
# Trainer server env
|
||||
# -----------------------------
|
||||
|
||||
@@ -210,6 +210,35 @@
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
.autoGrid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(2, minmax(0, 1fr));
|
||||
gap: 14px;
|
||||
}
|
||||
.autoGrid .wide { grid-column: 1 / -1; }
|
||||
.checkField {
|
||||
display: flex;
|
||||
align-items: flex-start;
|
||||
gap: 10px;
|
||||
padding: 12px;
|
||||
border-radius: 12px;
|
||||
border: 1px solid rgba(255,255,255,0.09);
|
||||
background: rgba(255,255,255,0.035);
|
||||
color: var(--muted);
|
||||
line-height: 1.4;
|
||||
}
|
||||
.checkField input { margin-top: 3px; }
|
||||
.checkField[hidden] { display: none !important; }
|
||||
.checkField strong { color: var(--text); display: block; margin-bottom: 3px; }
|
||||
.autoActions { display: flex; flex-wrap: wrap; gap: 10px; }
|
||||
.autoAudit {
|
||||
padding: 14px;
|
||||
border-radius: 14px;
|
||||
border: 1px solid rgba(255,255,255,0.08);
|
||||
background: rgba(0,0,0,0.22);
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
|
||||
.firmwareGrid {
|
||||
display: grid;
|
||||
grid-template-columns: minmax(260px, 1fr) minmax(160px, 220px) minmax(220px, 280px);
|
||||
@@ -1074,6 +1103,8 @@
|
||||
input[type="text"] { width: 100%; }
|
||||
.fileItem { align-items: flex-start; flex-direction: column; }
|
||||
.firmwareGrid { grid-template-columns: 1fr; }
|
||||
.autoGrid { grid-template-columns: 1fr; }
|
||||
.autoGrid .wide { grid-column: auto; }
|
||||
.firmwareLayout,
|
||||
.firmwareTargetGrid,
|
||||
.firmwareActionsPanel {
|
||||
@@ -1164,6 +1195,7 @@
|
||||
|
||||
<div class="tabs">
|
||||
<button id="tabTrainer" class="tabBtn active" type="button">Trainer</button>
|
||||
<button id="tabAuto" class="tabBtn" type="button">Auto Training</button>
|
||||
<button id="tabFirmware" class="tabBtn" type="button">Wake Words</button>
|
||||
<button id="tabCaptured" class="tabBtn" type="button">Captured Audio</button>
|
||||
<button id="tabSamples" class="tabBtn" type="button">Samples</button>
|
||||
@@ -1250,6 +1282,160 @@
|
||||
</section>
|
||||
</div>
|
||||
|
||||
<div id="autoView" class="viewStack stack" hidden>
|
||||
<div class="card studioHero trainerHero">
|
||||
<div class="row space">
|
||||
<div>
|
||||
<div class="studioKicker">False-Positive Loop</div>
|
||||
<h3>Auto Training</h3>
|
||||
<p>Transcribe real wake triggers, turn confirmed phrase-misses into hard negatives, retrain on your schedule, and ask Tater to refresh connected satellites.</p>
|
||||
<div class="studioSteps" aria-label="Auto Training steps">
|
||||
<span class="studioStepChip"><b>1</b> Transcribe wakes</span>
|
||||
<span class="studioStepChip"><b>2</b> Collect negatives</span>
|
||||
<span class="studioStepChip"><b>3</b> Retrain + refresh</span>
|
||||
</div>
|
||||
</div>
|
||||
<span id="autoStatus" class="pill">Disabled</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<section class="card studioPanel stack">
|
||||
<div class="studioPanelHeader">
|
||||
<div class="studioPanelTitle">
|
||||
<span class="studioStepBadge">1</span>
|
||||
<div>
|
||||
<h3>Review Rules</h3>
|
||||
<p>Only wake-trigger clips are reviewed automatically. Close misses, empty STT results, and clips containing the wake phrase remain in the inbox.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<label class="checkField">
|
||||
<input id="autoEnabled" type="checkbox" />
|
||||
<span><strong>Enable Auto Training</strong>New wake triggers will be queued for local Faster Whisper transcription.</span>
|
||||
</label>
|
||||
<div class="autoGrid">
|
||||
<label class="field">
|
||||
<strong>Wake phrase</strong>
|
||||
<input id="autoWakePhrase" type="text" placeholder='e.g. "hey tater"' />
|
||||
</label>
|
||||
<label class="field">
|
||||
<strong>STT language</strong>
|
||||
<input id="autoLanguage" type="text" value="en" placeholder="en" />
|
||||
</label>
|
||||
<label class="field wide">
|
||||
<strong>Faster Whisper model</strong>
|
||||
<input id="autoSttModel" type="text" value="small.en" placeholder="small.en" />
|
||||
</label>
|
||||
<label class="field">
|
||||
<strong>STT device</strong>
|
||||
<select id="autoSttDevice">
|
||||
<option value="auto" selected>Auto (prefer CUDA)</option>
|
||||
<option value="cuda">CUDA</option>
|
||||
<option value="cpu">CPU</option>
|
||||
</select>
|
||||
</label>
|
||||
<label class="field">
|
||||
<strong>Compute type</strong>
|
||||
<select id="autoSttComputeType">
|
||||
<option value="auto" selected>Auto (float16 CUDA / int8 CPU)</option>
|
||||
<option value="float16">float16</option>
|
||||
<option value="int8_float16">int8_float16</option>
|
||||
<option value="int8">int8</option>
|
||||
<option value="float32">float32</option>
|
||||
<option value="default">CTranslate2 default</option>
|
||||
</select>
|
||||
</label>
|
||||
<label class="field">
|
||||
<strong>Minimum transcript characters</strong>
|
||||
<input id="autoMinimumChars" type="number" min="1" max="100" value="2" />
|
||||
</label>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="card studioPanel stack">
|
||||
<div class="studioPanelHeader">
|
||||
<div class="studioPanelTitle">
|
||||
<span class="studioStepBadge">2</span>
|
||||
<div>
|
||||
<h3>Training Schedule</h3>
|
||||
<p>A scheduled run starts only after enough new auto-reviewed negatives have accumulated.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="autoGrid">
|
||||
<label class="field">
|
||||
<strong>Run training</strong>
|
||||
<select id="autoScheduleHours">
|
||||
<option value="0">Manually only</option>
|
||||
<option value="6">Every 6 hours</option>
|
||||
<option value="12">Every 12 hours</option>
|
||||
<option value="24" selected>Every day</option>
|
||||
<option value="48">Every 2 days</option>
|
||||
<option value="168">Every week</option>
|
||||
</select>
|
||||
</label>
|
||||
<label class="field">
|
||||
<strong>Minimum new negatives</strong>
|
||||
<input id="autoMinimumNegatives" type="number" min="1" max="10000" value="3" />
|
||||
</label>
|
||||
</div>
|
||||
<div class="statGrid">
|
||||
<div class="stat"><span class="label">Pending Negatives</span><span class="value" id="autoPendingNegatives">0</span></div>
|
||||
<div class="stat"><span class="label">Next Scheduled Check</span><span class="value" id="autoNextRun" style="font-size:16px;">Manual</span></div>
|
||||
<div class="stat"><span class="label">Last Training</span><span class="value" id="autoLastTraining" style="font-size:16px;">Never</span></div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="card studioPanel stack">
|
||||
<div class="studioPanelHeader">
|
||||
<div class="studioPanelTitle">
|
||||
<span class="studioStepBadge">3</span>
|
||||
<div>
|
||||
<h3>Publish + Satellite Refresh</h3>
|
||||
<p>The trainer publishes a LAN-reachable model URL, then asks Tater to re-push live settings so the firmware downloads the updated model at the same URL.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="autoGrid">
|
||||
<label class="field wide">
|
||||
<strong>Trainer public URL</strong>
|
||||
<input id="autoAdvertisedUrl" type="text" placeholder="Auto-detect this host's LAN IP" />
|
||||
<span id="autoDetectedUrl" class="muted">Auto-detected when left blank.</span>
|
||||
</label>
|
||||
<label class="field wide">
|
||||
<strong>Tater URL</strong>
|
||||
<input id="autoTaterUrl" type="text" value="http://127.0.0.1:8501" />
|
||||
</label>
|
||||
<label class="field">
|
||||
<strong>Satellite selector (optional)</strong>
|
||||
<input id="autoTaterSelector" type="text" placeholder="Blank refreshes all connected sats" />
|
||||
</label>
|
||||
<label class="field">
|
||||
<strong>Tater API token (if enabled)</strong>
|
||||
<input id="autoTaterToken" type="password" placeholder="Not configured" autocomplete="off" />
|
||||
</label>
|
||||
</div>
|
||||
<label class="checkField">
|
||||
<input id="autoNotifySatellites" type="checkbox" checked />
|
||||
<span><strong>Refresh satellites after successful training</strong>Uses Tater's existing native satellite settings API.</span>
|
||||
</label>
|
||||
<label id="autoClearTokenRow" class="checkField" hidden>
|
||||
<input id="autoClearTaterToken" type="checkbox" />
|
||||
<span><strong>Clear the saved Tater token</strong>The token is otherwise preserved when the password field is blank.</span>
|
||||
</label>
|
||||
</section>
|
||||
|
||||
<section class="card studioPanel stack">
|
||||
<div class="autoActions">
|
||||
<button id="autoSaveBtn" class="primary" type="button">Save Auto Training</button>
|
||||
<button id="autoReviewNowBtn" type="button">Review inbox now</button>
|
||||
<button id="autoTrainNowBtn" type="button">Train now</button>
|
||||
<button id="autoNotifyNowBtn" type="button">Refresh satellites now</button>
|
||||
</div>
|
||||
<div id="autoAudit" class="autoAudit muted">No automatic review has run yet.</div>
|
||||
</section>
|
||||
</div>
|
||||
|
||||
<div id="capturedView" class="viewStack stack" hidden>
|
||||
<div class="card studioHero captureHero">
|
||||
<div class="row space">
|
||||
@@ -1498,9 +1684,11 @@
|
||||
captured: { items: [], captured_count: 0, negative_count: 0, personal_count: 0 },
|
||||
samples: { personal: [], negative: [], personal_count: 0, negative_count: 0, activeBucket: "personal", pages: { personal: 0, negative: 0 } },
|
||||
firmware: { wakeWords: [] },
|
||||
autoTrain: null,
|
||||
uploadBusy: false,
|
||||
reviewBusy: false,
|
||||
firmwareBusy: false,
|
||||
autoBusy: false,
|
||||
trainingPoller: null,
|
||||
activeView: "trainer",
|
||||
};
|
||||
@@ -1774,12 +1962,14 @@
|
||||
}
|
||||
|
||||
function setActiveView(view) {
|
||||
uiState.activeView = ["captured", "samples", "firmware"].includes(view) ? view : "trainer";
|
||||
uiState.activeView = ["auto", "captured", "samples", "firmware"].includes(view) ? view : "trainer";
|
||||
$("trainerView").hidden = uiState.activeView !== "trainer";
|
||||
$("autoView").hidden = uiState.activeView !== "auto";
|
||||
$("capturedView").hidden = uiState.activeView !== "captured";
|
||||
$("samplesView").hidden = uiState.activeView !== "samples";
|
||||
$("firmwareView").hidden = uiState.activeView !== "firmware";
|
||||
$("tabTrainer").classList.toggle("active", uiState.activeView === "trainer");
|
||||
$("tabAuto").classList.toggle("active", uiState.activeView === "auto");
|
||||
$("tabCaptured").classList.toggle("active", uiState.activeView === "captured");
|
||||
$("tabSamples").classList.toggle("active", uiState.activeView === "samples");
|
||||
$("tabFirmware").classList.toggle("active", uiState.activeView === "firmware");
|
||||
@@ -1835,6 +2025,143 @@
|
||||
return Number.isNaN(parsed.getTime()) ? String(value) : parsed.toLocaleString();
|
||||
}
|
||||
|
||||
function renderAutoTrain(payload, populateForm = true) {
|
||||
const data = payload || {};
|
||||
const config = data.config || {};
|
||||
const state = data.state || {};
|
||||
const runtime = data.runtime || {};
|
||||
uiState.autoTrain = data;
|
||||
|
||||
if (populateForm) {
|
||||
$("autoEnabled").checked = Boolean(config.enabled);
|
||||
$("autoWakePhrase").value = config.wake_phrase || uiState.session?.raw_phrase || "";
|
||||
$("autoLanguage").value = config.language || uiState.session?.language || "en";
|
||||
$("autoSttModel").value = config.stt_model || "small.en";
|
||||
$("autoSttDevice").value = config.stt_device || "auto";
|
||||
$("autoSttComputeType").value = config.stt_compute_type || "auto";
|
||||
$("autoMinimumChars").value = String(config.minimum_transcript_chars ?? 2);
|
||||
$("autoScheduleHours").value = String(config.schedule_hours ?? 24);
|
||||
$("autoMinimumNegatives").value = String(config.minimum_new_negatives ?? 3);
|
||||
$("autoAdvertisedUrl").value = config.advertised_base_url || "";
|
||||
$("autoTaterUrl").value = config.tater_url || "http://127.0.0.1:8501";
|
||||
$("autoTaterSelector").value = config.tater_selector || "";
|
||||
$("autoTaterToken").value = "";
|
||||
$("autoTaterToken").placeholder = config.tater_api_token_configured ? "Saved token (leave blank to keep)" : "Not configured";
|
||||
$("autoNotifySatellites").checked = config.notify_satellites !== false;
|
||||
$("autoClearTokenRow").hidden = !config.tater_api_token_configured;
|
||||
$("autoClearTaterToken").checked = false;
|
||||
}
|
||||
|
||||
$("autoDetectedUrl").textContent = config.advertised_base_url
|
||||
? `Using configured URL: ${config.advertised_base_url}`
|
||||
: `Auto-detected URL: ${data.advertised_base_url || "unavailable"}`;
|
||||
$("autoPendingNegatives").textContent = String(Number(state.pending_negative_count || 0));
|
||||
$("autoNextRun").textContent = state.next_run_at ? formatTimestamp(state.next_run_at) : "Manual";
|
||||
const lastExit = state.last_train_exit_code;
|
||||
$("autoLastTraining").textContent = state.last_train_finished_at
|
||||
? `${formatTimestamp(state.last_train_finished_at)}${lastExit === null || lastExit === undefined ? "" : ` · exit ${lastExit}`}`
|
||||
: "Never";
|
||||
|
||||
if (runtime.review_running) {
|
||||
setPill($("autoStatus"), `Transcribing ${runtime.review_file || "wake"}`, "warn");
|
||||
} else if (uiState.training?.running && config.enabled) {
|
||||
setPill($("autoStatus"), "Training running", "warn");
|
||||
} else if (config.enabled) {
|
||||
setPill($("autoStatus"), "Enabled", "ok");
|
||||
} else {
|
||||
setPill($("autoStatus"), "Disabled", "");
|
||||
}
|
||||
|
||||
const audit = [];
|
||||
if (state.last_review_result) audit.push(`Last review: ${String(state.last_review_result).replaceAll("_", " ")}`);
|
||||
if (state.last_review_file) audit.push(state.last_review_file);
|
||||
if (state.last_review_transcript) audit.push(`STT: “${state.last_review_transcript}”`);
|
||||
if (state.last_review_error) audit.push(`Error: ${state.last_review_error}`);
|
||||
if (state.last_stt_device) audit.push(`STT runtime: ${state.last_stt_device} / ${state.last_stt_compute_type || "default"}`);
|
||||
if (state.last_notify_at) {
|
||||
audit.push(state.last_notify_error
|
||||
? `Satellite refresh failed: ${state.last_notify_error}`
|
||||
: `Satellite refresh: ${state.last_notify_count ?? "requested"} connected at ${formatTimestamp(state.last_notify_at)}`);
|
||||
}
|
||||
$("autoAudit").textContent = audit.join(" · ") || "No automatic review has run yet.";
|
||||
syncButtons();
|
||||
}
|
||||
|
||||
async function refreshAutoTrain(populateForm = true) {
|
||||
const data = await api("/api/auto_train", { method: "GET" });
|
||||
renderAutoTrain(data, populateForm);
|
||||
return data;
|
||||
}
|
||||
|
||||
function autoTrainFormPayload() {
|
||||
const payload = {
|
||||
enabled: $("autoEnabled").checked,
|
||||
wake_phrase: ($("autoWakePhrase").value || "").trim(),
|
||||
language: ($("autoLanguage").value || "en").trim(),
|
||||
stt_model: ($("autoSttModel").value || "").trim(),
|
||||
stt_device: $("autoSttDevice").value || "auto",
|
||||
stt_compute_type: $("autoSttComputeType").value || "auto",
|
||||
minimum_transcript_chars: Number($("autoMinimumChars").value || 2),
|
||||
schedule_hours: Number($("autoScheduleHours").value || 0),
|
||||
minimum_new_negatives: Number($("autoMinimumNegatives").value || 3),
|
||||
advertised_base_url: ($("autoAdvertisedUrl").value || "").trim(),
|
||||
tater_url: ($("autoTaterUrl").value || "").trim(),
|
||||
tater_selector: ($("autoTaterSelector").value || "").trim(),
|
||||
notify_satellites: $("autoNotifySatellites").checked,
|
||||
clear_tater_api_token: $("autoClearTaterToken").checked,
|
||||
};
|
||||
const token = ($("autoTaterToken").value || "").trim();
|
||||
if (token) payload.tater_api_token = token;
|
||||
return payload;
|
||||
}
|
||||
|
||||
async function saveAutoTrain() {
|
||||
uiState.autoBusy = true;
|
||||
syncButtons();
|
||||
setPill($("autoStatus"), "Saving...", "warn");
|
||||
try {
|
||||
const data = await api("/api/auto_train", {
|
||||
method: "PUT",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify(autoTrainFormPayload()),
|
||||
});
|
||||
renderAutoTrain(data, true);
|
||||
setPill($("autoStatus"), data.config?.enabled ? "Saved + enabled" : "Saved + disabled", data.config?.enabled ? "ok" : "");
|
||||
await refreshTrainedWakeWords().catch(() => {});
|
||||
return data;
|
||||
} finally {
|
||||
uiState.autoBusy = false;
|
||||
syncButtons();
|
||||
}
|
||||
}
|
||||
|
||||
async function runAutoTrainAction(action) {
|
||||
uiState.autoBusy = true;
|
||||
syncButtons();
|
||||
setPill($("autoStatus"), "Working...", "warn");
|
||||
try {
|
||||
const data = await api("/api/auto_train/action", {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ action }),
|
||||
});
|
||||
renderAutoTrain(data, false);
|
||||
if (action === "review_now") {
|
||||
setPill($("autoStatus"), `${Number(data.queued || 0)} clip${Number(data.queued || 0) === 1 ? "" : "s"} queued`, "ok");
|
||||
} else if (action === "train_now") {
|
||||
setPill($("autoStatus"), "Training started", "warn");
|
||||
await refreshSession();
|
||||
pollTraining();
|
||||
} else {
|
||||
setPill($("autoStatus"), `Satellite refresh requested${data.count === null || data.count === undefined ? "" : ` for ${data.count}`}`, "ok");
|
||||
}
|
||||
return data;
|
||||
} finally {
|
||||
uiState.autoBusy = false;
|
||||
syncButtons();
|
||||
}
|
||||
}
|
||||
|
||||
function captureBadge(item) {
|
||||
if (item.blocked_by_vad) return { label: "Blocked by VAD", cls: "warn" };
|
||||
const eventType = String(item?.event_type || "").toLowerCase();
|
||||
@@ -1879,6 +2206,7 @@
|
||||
if (item.max_probability !== null && item.max_probability !== undefined) meta.push(`<span class="pill">max ${escapeHtml(item.max_probability)}</span>`);
|
||||
if (item.average_probability !== null && item.average_probability !== undefined) meta.push(`<span class="pill">avg ${escapeHtml(item.average_probability)}</span>`);
|
||||
if (item.detection_profile) meta.push(`<span class="pill">profile ${escapeHtml(formatDetectionProfile(item.detection_profile))}</span>`);
|
||||
if (item.auto_review_status) meta.push(`<span class="pill ${item.auto_review_status === "error" ? "err" : "warn"}">auto ${escapeHtml(String(item.auto_review_status).replaceAll("_", " "))}</span>`);
|
||||
if (item.peak_probability_cutoff !== null && item.peak_probability_cutoff !== undefined) meta.push(`<span class="pill">peak cutoff ${escapeHtml(item.peak_probability_cutoff)}</span>`);
|
||||
if (item.probability_cutoff !== null && item.probability_cutoff !== undefined) meta.push(`<span class="pill">avg cutoff ${escapeHtml(item.probability_cutoff)}</span>`);
|
||||
if (item.active_window_count !== null && item.active_window_count !== undefined && item.min_active_windows !== null && item.min_active_windows !== undefined) {
|
||||
@@ -1906,6 +2234,8 @@
|
||||
<span class="pill ${badge.cls}">${escapeHtml(badge.label)}</span>
|
||||
</div>
|
||||
<div class="fileMeta">${meta.join("") || `<span class="muted">No metadata attached</span>`}</div>
|
||||
${item.transcript ? `<div class="autoAudit"><strong>STT transcript</strong><br>${escapeHtml(item.transcript)}</div>` : ""}
|
||||
${item.auto_review_error ? `<div class="muted">Auto review error: ${escapeHtml(item.auto_review_error)}</div>` : ""}
|
||||
<audio class="audioPlayer" controls preload="none" src="${escapeHtml(item.audio_url || `/api/audio/captured/${encodeURIComponent(item.saved_as)}`)}"></audio>
|
||||
<div class="muted">Stored as ${escapeHtml(item.saved_as)} · ${escapeHtml(formatSummary)}</div>
|
||||
<div class="captureActions">
|
||||
@@ -1949,6 +2279,7 @@
|
||||
if (item.original_name && item.original_name !== item.saved_as) subtitleParts.push(`From ${item.original_name}`);
|
||||
if (when) subtitleParts.push(`Saved ${when}`);
|
||||
if (item.message) subtitleParts.push(item.message);
|
||||
if (item.auto_negative) subtitleParts.push("Auto-reviewed false positive");
|
||||
let revertBtn = '';
|
||||
if (item.trimmed) {
|
||||
revertBtn = `<button type="button" data-sample-revert="${escapeAttr(item.saved_as)}" data-bucket="${escapeAttr(bucket)}">Revert</button>`;
|
||||
@@ -1963,6 +2294,7 @@
|
||||
<span class="pill ${badge.cls}">${badge.label}</span>
|
||||
${trimBadgeHtml}
|
||||
</div>
|
||||
${item.transcript ? `<div class="autoAudit"><strong>STT transcript</strong><br>${escapeHtml(item.transcript)}</div>` : ""}
|
||||
<audio class="audioPlayer" controls preload="none" src="${escapeAttr(item.audio_url || `/api/audio/${bucket}/${encodeURIComponent(item.saved_as)}`)}?t=${encodeURIComponent(item.created_at || '')}"></audio>
|
||||
<div class="muted">Stored in ${bucket === "negative" ? "negative_samples" : "personal_samples"} · ${escapeHtml(formatSummary)}</div>
|
||||
<div class="captureActions">
|
||||
@@ -2250,6 +2582,10 @@
|
||||
if (refreshWakeWordsBtn) {
|
||||
refreshWakeWordsBtn.disabled = uiState.firmwareBusy;
|
||||
}
|
||||
for (const id of ["autoSaveBtn", "autoReviewNowBtn", "autoTrainNowBtn", "autoNotifyNowBtn"]) {
|
||||
const button = $(id);
|
||||
if (button) button.disabled = uiState.autoBusy || (id === "autoTrainNowBtn" && Boolean(training.running));
|
||||
}
|
||||
}
|
||||
|
||||
function refreshSessionUI(session) {
|
||||
@@ -2506,6 +2842,13 @@
|
||||
|
||||
$("phrase").addEventListener("input", syncButtons);
|
||||
$("tabTrainer").addEventListener("click", () => setActiveView("trainer"));
|
||||
$("tabAuto").addEventListener("click", () => {
|
||||
setActiveView("auto");
|
||||
refreshAutoTrain(true).catch((error) => {
|
||||
setPill($("autoStatus"), "Status failed", "err");
|
||||
alert("Auto Training status failed: " + error.message);
|
||||
});
|
||||
});
|
||||
$("tabCaptured").addEventListener("click", () => setActiveView("captured"));
|
||||
$("tabSamples").addEventListener("click", () => {
|
||||
setActiveView("samples");
|
||||
@@ -2556,6 +2899,39 @@
|
||||
}
|
||||
});
|
||||
|
||||
$("autoSaveBtn").addEventListener("click", async () => {
|
||||
try {
|
||||
await saveAutoTrain();
|
||||
} catch (error) {
|
||||
setPill($("autoStatus"), "Save failed", "err");
|
||||
alert("Auto Training save failed: " + error.message);
|
||||
}
|
||||
});
|
||||
$("autoReviewNowBtn").addEventListener("click", async () => {
|
||||
try {
|
||||
await runAutoTrainAction("review_now");
|
||||
} catch (error) {
|
||||
setPill($("autoStatus"), "Review failed", "err");
|
||||
alert("Auto review failed: " + error.message);
|
||||
}
|
||||
});
|
||||
$("autoTrainNowBtn").addEventListener("click", async () => {
|
||||
try {
|
||||
await runAutoTrainAction("train_now");
|
||||
} catch (error) {
|
||||
setPill($("autoStatus"), "Training failed", "err");
|
||||
alert("Auto training failed: " + error.message);
|
||||
}
|
||||
});
|
||||
$("autoNotifyNowBtn").addEventListener("click", async () => {
|
||||
try {
|
||||
await runAutoTrainAction("notify_now");
|
||||
} catch (error) {
|
||||
setPill($("autoStatus"), "Refresh failed", "err");
|
||||
alert("Satellite refresh failed: " + error.message);
|
||||
}
|
||||
});
|
||||
|
||||
$("openConsoleBtn").addEventListener("click", () => {
|
||||
setConsoleLogAutoScroll($("trainLog"), (uiState.training?.log_lines || []).join("\n") || "(no training started)");
|
||||
openConsole(true, "Training Console", "Live training output appears here with color-coded console styling.");
|
||||
@@ -2774,6 +3150,7 @@
|
||||
await refreshSession();
|
||||
await refreshSamples();
|
||||
await refreshCapturedAudio();
|
||||
await refreshAutoTrain(true);
|
||||
} catch (_) {}
|
||||
|
||||
try {
|
||||
@@ -2864,6 +3241,10 @@
|
||||
});
|
||||
|
||||
bootstrap();
|
||||
setInterval(() => {
|
||||
if (uiState.activeView !== "auto" || uiState.autoBusy) return;
|
||||
refreshAutoTrain(false).catch(() => {});
|
||||
}, 2500);
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
|
||||
238
tests/test_auto_train.py
Normal file
238
tests/test_auto_train.py
Normal file
@@ -0,0 +1,238 @@
|
||||
import io
|
||||
import json
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import wave
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import Mock, patch
|
||||
|
||||
import trainer_server as trainer
|
||||
|
||||
|
||||
def silent_wav_bytes(duration_s: float = 0.25) -> bytes:
|
||||
output = io.BytesIO()
|
||||
with wave.open(output, "wb") as wav_file:
|
||||
wav_file.setnchannels(1)
|
||||
wav_file.setsampwidth(2)
|
||||
wav_file.setframerate(16000)
|
||||
wav_file.writeframes(b"\x00\x00" * int(16000 * duration_s))
|
||||
return output.getvalue()
|
||||
|
||||
|
||||
class AutoTrainTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tempdir = tempfile.TemporaryDirectory()
|
||||
root = Path(self.tempdir.name)
|
||||
self.original_paths = (
|
||||
trainer.CAPTURED_DIR,
|
||||
trainer.NEGATIVE_DIR,
|
||||
trainer.PERSONAL_DIR,
|
||||
trainer.AUTO_TRAIN_CONFIG_FILE,
|
||||
trainer.AUTO_TRAIN_STATE_FILE,
|
||||
)
|
||||
trainer.CAPTURED_DIR = root / "captured_audio"
|
||||
trainer.NEGATIVE_DIR = root / "negative_samples"
|
||||
trainer.PERSONAL_DIR = root / "personal_samples"
|
||||
trainer.AUTO_TRAIN_CONFIG_FILE = root / "auto_train_config.json"
|
||||
trainer.AUTO_TRAIN_STATE_FILE = root / "auto_train_state.json"
|
||||
for directory in (trainer.CAPTURED_DIR, trainer.NEGATIVE_DIR, trainer.PERSONAL_DIR):
|
||||
directory.mkdir(parents=True)
|
||||
|
||||
self.original_config = dict(trainer.AUTO_TRAIN_CONFIG)
|
||||
self.original_state = dict(trainer.AUTO_TRAIN_STATE)
|
||||
trainer.AUTO_TRAIN_CONFIG.clear()
|
||||
trainer.AUTO_TRAIN_CONFIG.update(
|
||||
trainer._normalize_auto_train_config(
|
||||
{
|
||||
"enabled": True,
|
||||
"wake_phrase": "hey tater",
|
||||
"language": "en",
|
||||
"tater_url": "http://127.0.0.1:8501",
|
||||
"stt_device": "auto",
|
||||
"stt_compute_type": "auto",
|
||||
}
|
||||
)
|
||||
)
|
||||
trainer.AUTO_TRAIN_STATE.clear()
|
||||
trainer.AUTO_TRAIN_STATE.update(trainer.AUTO_TRAIN_DEFAULT_STATE)
|
||||
|
||||
def tearDown(self):
|
||||
(
|
||||
trainer.CAPTURED_DIR,
|
||||
trainer.NEGATIVE_DIR,
|
||||
trainer.PERSONAL_DIR,
|
||||
trainer.AUTO_TRAIN_CONFIG_FILE,
|
||||
trainer.AUTO_TRAIN_STATE_FILE,
|
||||
) = self.original_paths
|
||||
trainer.AUTO_TRAIN_CONFIG.clear()
|
||||
trainer.AUTO_TRAIN_CONFIG.update(self.original_config)
|
||||
trainer.AUTO_TRAIN_STATE.clear()
|
||||
trainer.AUTO_TRAIN_STATE.update(self.original_state)
|
||||
self.tempdir.cleanup()
|
||||
|
||||
def add_capture(self, name: str = "wake.wav", wake_word: str = "hey_tater") -> Path:
|
||||
audio_path = trainer.CAPTURED_DIR / name
|
||||
audio_path.write_bytes(silent_wav_bytes())
|
||||
trainer._write_sidecar_json(
|
||||
audio_path,
|
||||
{
|
||||
"original_name": name,
|
||||
"wake_word": wake_word,
|
||||
"event_type": "wake_detected",
|
||||
"review_status": "pending",
|
||||
},
|
||||
)
|
||||
return audio_path
|
||||
|
||||
def test_phrase_matching_normalizes_case_punctuation_and_underscores(self):
|
||||
self.assertTrue(trainer._transcript_contains_wake_phrase("Okay, HEY TATER!", "hey_tater"))
|
||||
self.assertFalse(trainer._transcript_contains_wake_phrase("Turn on the television", "hey tater"))
|
||||
|
||||
def test_phrase_miss_moves_wake_trigger_to_negative_samples(self):
|
||||
self.add_capture()
|
||||
with patch.object(trainer, "_transcribe_capture_with_faster_whisper", return_value="turn on the kitchen lights"):
|
||||
trainer._auto_review_capture("wake.wav")
|
||||
|
||||
self.assertFalse((trainer.CAPTURED_DIR / "wake.wav").exists())
|
||||
negatives = list(trainer.NEGATIVE_DIR.glob("*.wav"))
|
||||
self.assertEqual(len(negatives), 1)
|
||||
metadata = trainer._load_sidecar_json(negatives[0])
|
||||
self.assertTrue(metadata["auto_negative"])
|
||||
self.assertEqual(metadata["review_status"], "auto_approved_negative")
|
||||
self.assertEqual(metadata["transcript"], "turn on the kitchen lights")
|
||||
self.assertEqual(trainer.AUTO_TRAIN_STATE["pending_negative_count"], 1)
|
||||
|
||||
def test_matching_phrase_stays_in_manual_review_inbox(self):
|
||||
audio_path = self.add_capture()
|
||||
with patch.object(trainer, "_transcribe_capture_with_faster_whisper", return_value="hey tater turn on the lights"):
|
||||
trainer._auto_review_capture("wake.wav")
|
||||
|
||||
self.assertTrue(audio_path.exists())
|
||||
self.assertFalse(list(trainer.NEGATIVE_DIR.glob("*.wav")))
|
||||
metadata = trainer._load_sidecar_json(audio_path)
|
||||
self.assertEqual(metadata["auto_review_status"], "wake_phrase_detected")
|
||||
self.assertEqual(trainer.AUTO_TRAIN_STATE["pending_negative_count"], 0)
|
||||
|
||||
def test_capture_for_another_wake_word_is_not_transcribed(self):
|
||||
audio_path = self.add_capture(wake_word="computer")
|
||||
with patch.object(trainer, "_transcribe_capture_with_faster_whisper") as transcribe:
|
||||
trainer._auto_review_capture("wake.wav")
|
||||
|
||||
transcribe.assert_not_called()
|
||||
self.assertTrue(audio_path.exists())
|
||||
metadata = trainer._load_sidecar_json(audio_path)
|
||||
self.assertEqual(metadata["auto_review_status"], "different_wake_phrase")
|
||||
|
||||
def test_due_schedule_starts_training_after_minimum_negatives(self):
|
||||
trainer.AUTO_TRAIN_CONFIG["schedule_hours"] = 24
|
||||
trainer.AUTO_TRAIN_CONFIG["minimum_new_negatives"] = 3
|
||||
trainer.AUTO_TRAIN_STATE["pending_negative_count"] = 3
|
||||
trainer.AUTO_TRAIN_STATE["next_run_at"] = "2000-01-01T00:00:00+00:00"
|
||||
with patch.object(trainer, "_start_auto_training", return_value={"ok": True, "started": True}) as start:
|
||||
trainer._maybe_run_scheduled_auto_training()
|
||||
|
||||
start.assert_called_once_with()
|
||||
self.assertTrue(trainer.AUTO_TRAIN_STATE["next_run_at"])
|
||||
|
||||
def test_tater_refresh_repushes_settings_with_selector_and_token(self):
|
||||
trainer.AUTO_TRAIN_CONFIG.update(
|
||||
{
|
||||
"notify_satellites": True,
|
||||
"tater_url": "http://127.0.0.1:8501",
|
||||
"tater_selector": "kitchen-sat",
|
||||
"tater_api_token": "secret-token",
|
||||
}
|
||||
)
|
||||
|
||||
class Response:
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *_args):
|
||||
return False
|
||||
|
||||
def read(self):
|
||||
return b'{"push":{"count":2}}'
|
||||
|
||||
with patch.object(trainer, "urlopen", return_value=Response()) as open_url:
|
||||
result = trainer._notify_tater_satellites()
|
||||
|
||||
self.assertTrue(result["ok"])
|
||||
self.assertEqual(result["count"], 2)
|
||||
request = open_url.call_args.args[0]
|
||||
self.assertEqual(request.full_url, "http://127.0.0.1:8501/api/tater/satellite/v1/settings")
|
||||
self.assertEqual(request.get_header("X-tater-token"), "secret-token")
|
||||
self.assertEqual(json.loads(request.data), {"selector": "kitchen-sat", "settings": {}})
|
||||
|
||||
def test_advertised_url_uses_non_loopback_browser_host(self):
|
||||
request = SimpleNamespace(
|
||||
base_url="http://192.168.1.50:8789/",
|
||||
url=SimpleNamespace(hostname="192.168.1.50", scheme="http", port=8789),
|
||||
)
|
||||
self.assertEqual(trainer._advertised_base_url(request), "http://192.168.1.50:8789")
|
||||
|
||||
def test_advertised_url_replaces_localhost_with_discovered_lan_host(self):
|
||||
request = SimpleNamespace(
|
||||
base_url="http://127.0.0.1:8789/",
|
||||
url=SimpleNamespace(hostname="127.0.0.1", scheme="http", port=8789),
|
||||
)
|
||||
with patch.object(trainer, "_discover_lan_ipv4", return_value="192.168.1.60"):
|
||||
self.assertEqual(trainer._advertised_base_url(request), "http://192.168.1.60:8789")
|
||||
|
||||
def test_configured_public_url_takes_precedence(self):
|
||||
trainer.AUTO_TRAIN_CONFIG["advertised_base_url"] = "http://trainer.local:8789"
|
||||
request = SimpleNamespace(
|
||||
base_url="http://127.0.0.1:8789/",
|
||||
url=SimpleNamespace(hostname="127.0.0.1", scheme="http", port=8789),
|
||||
)
|
||||
self.assertEqual(trainer._advertised_base_url(request), "http://trainer.local:8789")
|
||||
|
||||
def test_faster_whisper_auto_runtime_prefers_cuda_and_float16(self):
|
||||
fake_ctranslate2 = SimpleNamespace(get_cuda_device_count=lambda: 1)
|
||||
with patch.dict(sys.modules, {"ctranslate2": fake_ctranslate2}):
|
||||
self.assertEqual(
|
||||
trainer._resolve_faster_whisper_runtime("auto", "auto"),
|
||||
("cuda", "float16"),
|
||||
)
|
||||
|
||||
def test_faster_whisper_auto_runtime_falls_back_to_cpu_int8(self):
|
||||
fake_ctranslate2 = SimpleNamespace(get_cuda_device_count=lambda: 0)
|
||||
with patch.dict(sys.modules, {"ctranslate2": fake_ctranslate2}):
|
||||
self.assertEqual(
|
||||
trainer._resolve_faster_whisper_runtime("auto", "auto"),
|
||||
("cpu", "int8"),
|
||||
)
|
||||
|
||||
def test_faster_whisper_transcription_joins_segments_and_records_runtime(self):
|
||||
fake_model = SimpleNamespace()
|
||||
fake_model.transcribe = Mock(
|
||||
return_value=(
|
||||
iter([SimpleNamespace(text=" turn on "), SimpleNamespace(text="the lights ")]),
|
||||
SimpleNamespace(),
|
||||
)
|
||||
)
|
||||
with (
|
||||
patch.object(trainer, "_resolve_faster_whisper_runtime", return_value=("cuda", "float16")),
|
||||
patch.object(trainer, "_load_faster_whisper_model", return_value=fake_model),
|
||||
):
|
||||
transcript = trainer._transcribe_capture_with_faster_whisper(
|
||||
Path("wake.wav"),
|
||||
model="small.en",
|
||||
language="en",
|
||||
)
|
||||
|
||||
self.assertEqual(transcript, "turn on the lights")
|
||||
fake_model.transcribe.assert_called_once_with(
|
||||
"wake.wav",
|
||||
language="en",
|
||||
beam_size=1,
|
||||
condition_on_previous_text=False,
|
||||
)
|
||||
self.assertEqual(trainer.AUTO_TRAIN_STATE["last_stt_device"], "cuda")
|
||||
self.assertEqual(trainer.AUTO_TRAIN_STATE["last_stt_compute_type"], "float16")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
88
tests/test_calibrate_detector.py
Normal file
88
tests/test_calibrate_detector.py
Normal file
@@ -0,0 +1,88 @@
|
||||
import importlib.util
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
SCRIPT_PATH = (
|
||||
Path(__file__).resolve().parents[1]
|
||||
/ "cli"
|
||||
/ "calibrate_detector.py"
|
||||
)
|
||||
SPEC = importlib.util.spec_from_file_location("calibrate_detector", SCRIPT_PATH)
|
||||
calibrate_detector = importlib.util.module_from_spec(SPEC)
|
||||
assert SPEC.loader is not None
|
||||
SPEC.loader.exec_module(calibrate_detector)
|
||||
|
||||
|
||||
def candidate(cutoff, window, recall, false_accepts_per_hour):
|
||||
return {
|
||||
"probability_cutoff": cutoff,
|
||||
"sliding_window_size": window,
|
||||
"recall": recall,
|
||||
"false_accepts_per_hour": false_accepts_per_hour,
|
||||
}
|
||||
|
||||
|
||||
class CalibrationSelectionTests(unittest.TestCase):
|
||||
def test_defaults_are_conservative(self):
|
||||
self.assertEqual(calibrate_detector.DEFAULT_WINDOW_SIZES, [5, 6, 7])
|
||||
self.assertEqual(calibrate_detector.DEFAULT_CUTOFF_MIN, 0.95)
|
||||
self.assertEqual(calibrate_detector.DEFAULT_RECALL_MARGIN, 0.005)
|
||||
|
||||
def test_prefers_zero_false_accepts_within_recall_margin(self):
|
||||
candidates = [
|
||||
candidate(0.95, 5, 0.99894, 0.103408),
|
||||
candidate(0.95, 6, 0.99744, 0.0),
|
||||
candidate(0.95, 7, 0.99554, 0.0),
|
||||
]
|
||||
|
||||
best, selected_limit = calibrate_detector._select_best_candidate(
|
||||
candidates,
|
||||
target_faph=0.25,
|
||||
recall_margin=0.005,
|
||||
)
|
||||
|
||||
self.assertEqual(best["sliding_window_size"], 6)
|
||||
self.assertEqual(best["false_accepts_per_hour"], 0.0)
|
||||
self.assertEqual(selected_limit, 0.25)
|
||||
|
||||
def test_does_not_trade_away_recall_beyond_margin(self):
|
||||
candidates = [
|
||||
candidate(0.95, 5, 0.99, 0.1),
|
||||
candidate(0.99, 6, 0.90, 0.0),
|
||||
]
|
||||
|
||||
best, _ = calibrate_detector._select_best_candidate(
|
||||
candidates,
|
||||
target_faph=0.25,
|
||||
recall_margin=0.005,
|
||||
)
|
||||
|
||||
self.assertEqual(best["sliding_window_size"], 5)
|
||||
|
||||
def test_uses_strictest_available_false_accept_tier(self):
|
||||
candidates = [
|
||||
candidate(0.95, 5, 0.99, 0.6),
|
||||
candidate(0.99, 6, 0.99, 1.5),
|
||||
]
|
||||
|
||||
best, selected_limit = calibrate_detector._select_best_candidate(
|
||||
candidates,
|
||||
target_faph=0.25,
|
||||
recall_margin=0.005,
|
||||
)
|
||||
|
||||
self.assertEqual(best["false_accepts_per_hour"], 0.6)
|
||||
self.assertEqual(selected_limit, 0.75)
|
||||
|
||||
def test_rejects_negative_recall_margin(self):
|
||||
with self.assertRaises(ValueError):
|
||||
calibrate_detector._select_best_candidate(
|
||||
[candidate(0.95, 6, 0.99, 0.0)],
|
||||
target_faph=0.25,
|
||||
recall_margin=-0.001,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -4,17 +4,20 @@
|
||||
import contextlib
|
||||
import io
|
||||
import os
|
||||
import queue
|
||||
import re
|
||||
import json
|
||||
import socket
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import unicodedata
|
||||
import wave
|
||||
from array import array
|
||||
from datetime import datetime, timezone
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from math import isfinite, log10
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any, List, Callable, Optional, Tuple
|
||||
@@ -38,6 +41,15 @@ TRIM_HISTORY_DIR.mkdir(parents=True, exist_ok=True)
|
||||
TRAINED_WAKE_WORDS_DIR = Path(
|
||||
os.environ.get("TRAINED_WAKE_WORDS_DIR", str(DATA_DIR / "trained_wake_words"))
|
||||
).resolve()
|
||||
AUTO_TRAIN_CONFIG_FILE = Path(
|
||||
os.environ.get("AUTO_TRAIN_CONFIG_FILE", str(DATA_DIR / "auto_train_config.json"))
|
||||
).resolve()
|
||||
AUTO_TRAIN_STATE_FILE = Path(
|
||||
os.environ.get("AUTO_TRAIN_STATE_FILE", str(DATA_DIR / "auto_train_state.json"))
|
||||
).resolve()
|
||||
AUTO_TRAIN_MODEL_DIR = Path(
|
||||
os.environ.get("AUTO_TRAIN_MODEL_DIR", str(DATA_DIR / "auto_train_models"))
|
||||
).resolve()
|
||||
CLI_DIR = Path(os.environ.get("CLI_DIR", str(ROOT_DIR / "cli"))).resolve()
|
||||
PIPER_ROOT = DATA_DIR / "tools" / "piper-sample-generator"
|
||||
PIPER_VOICES_DIR = PIPER_ROOT / "voices"
|
||||
@@ -72,6 +84,42 @@ TARGET_SAMPLE_RATE = 16000
|
||||
TARGET_CHANNELS = 1
|
||||
TARGET_SAMPLE_WIDTH_BYTES = 2
|
||||
CAPTURE_GAIN_PROFILE = "capture_rms_v1"
|
||||
DEFAULT_FASTER_WHISPER_MODEL = os.environ.get("AUTO_TRAIN_STT_MODEL", "small.en")
|
||||
|
||||
AUTO_TRAIN_DEFAULT_CONFIG: Dict[str, Any] = {
|
||||
"enabled": False,
|
||||
"wake_phrase": "",
|
||||
"language": DEFAULT_LANGUAGE,
|
||||
"stt_model": DEFAULT_FASTER_WHISPER_MODEL,
|
||||
"stt_device": "auto",
|
||||
"stt_compute_type": "auto",
|
||||
"minimum_transcript_chars": 2,
|
||||
"schedule_hours": 24,
|
||||
"minimum_new_negatives": 3,
|
||||
"advertised_base_url": "",
|
||||
"tater_url": "http://127.0.0.1:8501",
|
||||
"tater_selector": "",
|
||||
"tater_api_token": "",
|
||||
"notify_satellites": True,
|
||||
}
|
||||
|
||||
AUTO_TRAIN_DEFAULT_STATE: Dict[str, Any] = {
|
||||
"pending_negative_count": 0,
|
||||
"next_run_at": "",
|
||||
"last_review_at": "",
|
||||
"last_review_file": "",
|
||||
"last_review_transcript": "",
|
||||
"last_review_result": "",
|
||||
"last_review_error": "",
|
||||
"last_stt_device": "",
|
||||
"last_stt_compute_type": "",
|
||||
"last_train_started_at": "",
|
||||
"last_train_finished_at": "",
|
||||
"last_train_exit_code": None,
|
||||
"last_notify_at": "",
|
||||
"last_notify_count": None,
|
||||
"last_notify_error": "",
|
||||
}
|
||||
|
||||
|
||||
app = FastAPI(title="microWakeWord Personal Samples")
|
||||
@@ -115,6 +163,21 @@ STATE: Dict[str, Any] = {
|
||||
STATE_LOCK = threading.Lock()
|
||||
SAMPLES_LOCK = threading.Lock()
|
||||
PIPER_CATALOG_LOCK = threading.Lock()
|
||||
AUTO_TRAIN_LOCK = threading.RLock()
|
||||
AUTO_TRAIN_WAKE_EVENT = threading.Event()
|
||||
AUTO_TRAIN_STOP_EVENT = threading.Event()
|
||||
AUTO_TRAIN_REVIEW_QUEUE: queue.Queue[str] = queue.Queue()
|
||||
AUTO_TRAIN_QUEUED_FILES: set[str] = set()
|
||||
AUTO_TRAIN_WORKER: threading.Thread | None = None
|
||||
AUTO_TRAIN_RUNTIME: Dict[str, Any] = {
|
||||
"review_running": False,
|
||||
"review_file": "",
|
||||
"scheduler_running": False,
|
||||
"training_pending_consumed": 0,
|
||||
}
|
||||
LAN_ADDRESS_CACHE: Dict[str, Any] = {"value": "", "fetched_at": 0.0}
|
||||
FASTER_WHISPER_MODEL_LOCK = threading.RLock()
|
||||
FASTER_WHISPER_MODEL_CACHE: Dict[Tuple[str, str, str], Any] = {}
|
||||
PIPER_CATALOG_CACHE: Dict[str, Any] = {
|
||||
"fetched_at": 0.0,
|
||||
"entries": None,
|
||||
@@ -334,6 +397,603 @@ def _request_base_url(request: Request) -> str:
|
||||
return str(request.base_url).rstrip("/")
|
||||
|
||||
|
||||
def _utc_now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def _iso_now() -> str:
|
||||
return _utc_now().isoformat()
|
||||
|
||||
|
||||
def _read_json_object(path: Path) -> Dict[str, Any]:
|
||||
try:
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
return data if isinstance(data, dict) else {}
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
|
||||
def _write_json_object(path: Path, payload: Dict[str, Any]) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
temp_path = path.with_name(f".{path.name}.tmp")
|
||||
temp_path.write_text(
|
||||
json.dumps(payload, ensure_ascii=False, indent=2, sort_keys=True) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
temp_path.replace(path)
|
||||
with contextlib.suppress(Exception):
|
||||
path.chmod(0o600)
|
||||
|
||||
|
||||
def _bounded_int(value: Any, default: int, minimum: int, maximum: int) -> int:
|
||||
try:
|
||||
parsed = int(value)
|
||||
except (TypeError, ValueError):
|
||||
parsed = default
|
||||
return max(minimum, min(maximum, parsed))
|
||||
|
||||
|
||||
def _config_bool(value: Any, default: bool = False) -> bool:
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
token = str(value or "").strip().lower()
|
||||
if token in {"1", "true", "yes", "on", "enabled"}:
|
||||
return True
|
||||
if token in {"0", "false", "no", "off", "disabled"}:
|
||||
return False
|
||||
return bool(default)
|
||||
|
||||
|
||||
def _normalize_http_base_url(value: Any, *, allow_empty: bool = True) -> str:
|
||||
token = str(value or "").strip().rstrip("/")
|
||||
if not token and allow_empty:
|
||||
return ""
|
||||
if not token.startswith(("http://", "https://")):
|
||||
raise ValueError("URL must start with http:// or https://")
|
||||
return token
|
||||
|
||||
|
||||
def _normalize_auto_train_config(values: Dict[str, Any] | None, *, base: Dict[str, Any] | None = None) -> Dict[str, Any]:
|
||||
incoming = values if isinstance(values, dict) else {}
|
||||
source = {**AUTO_TRAIN_DEFAULT_CONFIG, **(base or {}), **incoming}
|
||||
schedule_hours = _bounded_int(source.get("schedule_hours"), 24, 0, 24 * 30)
|
||||
language = str(source.get("language") or DEFAULT_LANGUAGE).strip().lower().replace("-", "_")
|
||||
language = re.sub(r"[^a-z0-9_]", "", language) or DEFAULT_LANGUAGE
|
||||
stt_device = str(source.get("stt_device") or "auto").strip().lower()
|
||||
if stt_device not in {"auto", "cuda", "cpu"}:
|
||||
raise ValueError("Faster Whisper device must be auto, cuda, or cpu.")
|
||||
stt_compute_type = str(source.get("stt_compute_type") or "auto").strip().lower()
|
||||
if stt_compute_type not in {"auto", "default", "float16", "float32", "int8", "int8_float16"}:
|
||||
raise ValueError("Unsupported Faster Whisper compute type.")
|
||||
return {
|
||||
"enabled": _config_bool(source.get("enabled")),
|
||||
"wake_phrase": str(source.get("wake_phrase") or "").strip(),
|
||||
"language": language,
|
||||
"stt_model": str(source.get("stt_model") or DEFAULT_FASTER_WHISPER_MODEL).strip() or DEFAULT_FASTER_WHISPER_MODEL,
|
||||
"stt_device": stt_device,
|
||||
"stt_compute_type": stt_compute_type,
|
||||
"minimum_transcript_chars": _bounded_int(source.get("minimum_transcript_chars"), 2, 1, 100),
|
||||
"schedule_hours": schedule_hours,
|
||||
"minimum_new_negatives": _bounded_int(source.get("minimum_new_negatives"), 3, 1, 10000),
|
||||
"advertised_base_url": _normalize_http_base_url(source.get("advertised_base_url")),
|
||||
"tater_url": _normalize_http_base_url(source.get("tater_url"), allow_empty=False),
|
||||
"tater_selector": str(source.get("tater_selector") or "").strip(),
|
||||
"tater_api_token": str(source.get("tater_api_token") or "").strip(),
|
||||
"notify_satellites": _config_bool(source.get("notify_satellites"), True),
|
||||
}
|
||||
|
||||
|
||||
try:
|
||||
AUTO_TRAIN_CONFIG: Dict[str, Any] = _normalize_auto_train_config(
|
||||
_read_json_object(AUTO_TRAIN_CONFIG_FILE)
|
||||
)
|
||||
except ValueError:
|
||||
AUTO_TRAIN_CONFIG = dict(AUTO_TRAIN_DEFAULT_CONFIG)
|
||||
AUTO_TRAIN_STATE: Dict[str, Any] = {
|
||||
**AUTO_TRAIN_DEFAULT_STATE,
|
||||
**_read_json_object(AUTO_TRAIN_STATE_FILE),
|
||||
}
|
||||
|
||||
|
||||
def _save_auto_train_config_locked() -> None:
|
||||
_write_json_object(AUTO_TRAIN_CONFIG_FILE, AUTO_TRAIN_CONFIG)
|
||||
|
||||
|
||||
def _save_auto_train_state_locked() -> None:
|
||||
persisted = {key: AUTO_TRAIN_STATE.get(key) for key in AUTO_TRAIN_DEFAULT_STATE}
|
||||
_write_json_object(AUTO_TRAIN_STATE_FILE, persisted)
|
||||
|
||||
|
||||
def _parse_iso_datetime(value: Any) -> datetime | None:
|
||||
token = str(value or "").strip()
|
||||
if not token:
|
||||
return None
|
||||
try:
|
||||
parsed = datetime.fromisoformat(token.replace("Z", "+00:00"))
|
||||
except ValueError:
|
||||
return None
|
||||
if parsed.tzinfo is None:
|
||||
parsed = parsed.replace(tzinfo=timezone.utc)
|
||||
return parsed.astimezone(timezone.utc)
|
||||
|
||||
|
||||
def _schedule_next_auto_run_locked(*, from_time: datetime | None = None) -> None:
|
||||
hours = int(AUTO_TRAIN_CONFIG.get("schedule_hours") or 0)
|
||||
if hours <= 0 or not AUTO_TRAIN_CONFIG.get("enabled"):
|
||||
AUTO_TRAIN_STATE["next_run_at"] = ""
|
||||
else:
|
||||
base = from_time or _utc_now()
|
||||
AUTO_TRAIN_STATE["next_run_at"] = (base + timedelta(hours=hours)).isoformat()
|
||||
_save_auto_train_state_locked()
|
||||
|
||||
|
||||
def _public_auto_train_config() -> Dict[str, Any]:
|
||||
with AUTO_TRAIN_LOCK:
|
||||
config = {key: value for key, value in AUTO_TRAIN_CONFIG.items() if key != "tater_api_token"}
|
||||
config["tater_api_token_configured"] = bool(AUTO_TRAIN_CONFIG.get("tater_api_token"))
|
||||
return config
|
||||
|
||||
|
||||
def _auto_train_status_payload() -> Dict[str, Any]:
|
||||
with AUTO_TRAIN_LOCK:
|
||||
return {
|
||||
"config": _public_auto_train_config(),
|
||||
"state": dict(AUTO_TRAIN_STATE),
|
||||
"runtime": dict(AUTO_TRAIN_RUNTIME),
|
||||
"advertised_base_url": _advertised_base_url(),
|
||||
}
|
||||
|
||||
|
||||
def _discover_lan_ipv4() -> str:
|
||||
override = str(os.environ.get("REC_ADVERTISED_HOST") or "").strip()
|
||||
if override and override not in {"0.0.0.0", "127.0.0.1", "localhost", "::1"}:
|
||||
return override
|
||||
|
||||
now = time.time()
|
||||
cached_value = str(LAN_ADDRESS_CACHE.get("value") or "")
|
||||
if cached_value and (now - float(LAN_ADDRESS_CACHE.get("fetched_at") or 0.0)) < 30:
|
||||
return cached_value
|
||||
|
||||
candidates: List[str] = []
|
||||
with contextlib.suppress(Exception):
|
||||
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
|
||||
try:
|
||||
sock.connect(("192.0.2.1", 9))
|
||||
candidates.append(str(sock.getsockname()[0]))
|
||||
finally:
|
||||
sock.close()
|
||||
with contextlib.suppress(Exception):
|
||||
for row in socket.getaddrinfo(socket.gethostname(), None, socket.AF_INET, socket.SOCK_DGRAM):
|
||||
candidates.append(str(row[4][0]))
|
||||
|
||||
with contextlib.suppress(Exception):
|
||||
proc = subprocess.run(
|
||||
["/sbin/ifconfig"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=3,
|
||||
check=False,
|
||||
)
|
||||
blocks = re.split(r"(?m)(?=^[^\s].*?: flags=)", proc.stdout or "")
|
||||
interface_rows: List[tuple[int, str]] = []
|
||||
for block in blocks:
|
||||
name_match = re.match(r"^([^:]+):", block)
|
||||
address_match = re.search(r"(?m)^\s+inet\s+(\d+(?:\.\d+){3})\b", block)
|
||||
if not name_match or not address_match or "status: active" not in block:
|
||||
continue
|
||||
name = name_match.group(1)
|
||||
if name == "lo0" or name.startswith(("utun", "awdl", "llw", "ap")):
|
||||
continue
|
||||
priority = 0 if name == "en0" else 1 if name == "en1" else 10
|
||||
interface_rows.append((priority, address_match.group(1)))
|
||||
candidates.extend(address for _priority, address in sorted(interface_rows))
|
||||
|
||||
for candidate in candidates:
|
||||
if candidate and not candidate.startswith("127.") and candidate != "0.0.0.0":
|
||||
LAN_ADDRESS_CACHE["value"] = candidate
|
||||
LAN_ADDRESS_CACHE["fetched_at"] = now
|
||||
return candidate
|
||||
LAN_ADDRESS_CACHE["fetched_at"] = now
|
||||
return ""
|
||||
|
||||
|
||||
def _advertised_base_url(request: Request | None = None) -> str:
|
||||
env_url = str(os.environ.get("REC_PUBLIC_BASE_URL") or "").strip().rstrip("/")
|
||||
with AUTO_TRAIN_LOCK:
|
||||
configured_url = str(AUTO_TRAIN_CONFIG.get("advertised_base_url") or "").strip().rstrip("/")
|
||||
if configured_url:
|
||||
return configured_url
|
||||
if env_url:
|
||||
return env_url
|
||||
|
||||
request_url = _request_base_url(request) if request is not None else ""
|
||||
request_host = str(request.url.hostname or "").lower() if request is not None else ""
|
||||
if request_url and request_host not in {"127.0.0.1", "localhost", "::1", "0.0.0.0"}:
|
||||
return request_url
|
||||
|
||||
host = _discover_lan_ipv4()
|
||||
if not host:
|
||||
return request_url
|
||||
scheme = str(request.url.scheme or "http") if request is not None else "http"
|
||||
port = request.url.port if request is not None else None
|
||||
if port is None:
|
||||
port = _bounded_int(os.environ.get("REC_PORT"), 8789, 1, 65535)
|
||||
default_port = (scheme == "http" and port == 80) or (scheme == "https" and port == 443)
|
||||
return f"{scheme}://{host}{'' if default_port else f':{port}'}"
|
||||
|
||||
|
||||
def _normalize_transcript_text(value: Any) -> str:
|
||||
text = unicodedata.normalize("NFKC", str(value or "")).casefold().replace("_", " ")
|
||||
text = re.sub(r"[^\w]+", " ", text, flags=re.UNICODE)
|
||||
return re.sub(r"\s+", " ", text).strip()
|
||||
|
||||
|
||||
def _transcript_contains_wake_phrase(transcript: Any, wake_phrase: Any) -> bool:
|
||||
normalized_transcript = _normalize_transcript_text(transcript)
|
||||
normalized_phrase = _normalize_transcript_text(wake_phrase)
|
||||
if not normalized_transcript or not normalized_phrase:
|
||||
return False
|
||||
return f" {normalized_phrase} " in f" {normalized_transcript} "
|
||||
|
||||
|
||||
def _captured_event_is_auto_reviewable(metadata: Dict[str, Any]) -> bool:
|
||||
if _parse_bool(metadata.get("blocked_by_vad")):
|
||||
return False
|
||||
event_type = str(metadata.get("event_type") or "captured").strip().lower()
|
||||
if "close" in event_type:
|
||||
return False
|
||||
return event_type in {"captured", "trigger", "false_trigger"} or "wake" in event_type or "detect" in event_type
|
||||
|
||||
|
||||
def _resolve_faster_whisper_runtime(device_value: Any, compute_value: Any) -> Tuple[str, str]:
|
||||
requested_device = str(device_value or "auto").strip().lower()
|
||||
if requested_device not in {"auto", "cuda", "cpu"}:
|
||||
raise ValueError("Faster Whisper device must be auto, cuda, or cpu.")
|
||||
|
||||
cuda_devices = 0
|
||||
with contextlib.suppress(Exception):
|
||||
import ctranslate2
|
||||
cuda_devices = int(ctranslate2.get_cuda_device_count())
|
||||
|
||||
if requested_device == "cuda" and cuda_devices <= 0:
|
||||
raise RuntimeError("CUDA was selected for Faster Whisper, but CTranslate2 cannot see an NVIDIA GPU.")
|
||||
device = "cuda" if requested_device == "cuda" or (requested_device == "auto" and cuda_devices > 0) else "cpu"
|
||||
|
||||
requested_compute = str(compute_value or "auto").strip().lower()
|
||||
allowed_compute = {"auto", "default", "float16", "float32", "int8", "int8_float16"}
|
||||
if requested_compute not in allowed_compute:
|
||||
raise ValueError("Unsupported Faster Whisper compute type.")
|
||||
compute_type = ("float16" if device == "cuda" else "int8") if requested_compute == "auto" else requested_compute
|
||||
return device, compute_type
|
||||
|
||||
|
||||
def _load_faster_whisper_model(*, model_name: str, device: str, compute_type: str):
|
||||
cache_key = (model_name, device, compute_type)
|
||||
with FASTER_WHISPER_MODEL_LOCK:
|
||||
cached = FASTER_WHISPER_MODEL_CACHE.get(cache_key)
|
||||
if cached is not None:
|
||||
return cached
|
||||
try:
|
||||
from faster_whisper import WhisperModel
|
||||
except Exception as exc:
|
||||
raise RuntimeError(f"faster-whisper is unavailable: {exc}") from exc
|
||||
|
||||
AUTO_TRAIN_MODEL_DIR.mkdir(parents=True, exist_ok=True)
|
||||
model = WhisperModel(
|
||||
model_name,
|
||||
device=device,
|
||||
compute_type=compute_type,
|
||||
download_root=str(AUTO_TRAIN_MODEL_DIR),
|
||||
)
|
||||
FASTER_WHISPER_MODEL_CACHE.clear()
|
||||
FASTER_WHISPER_MODEL_CACHE[cache_key] = model
|
||||
return model
|
||||
|
||||
|
||||
def _transcribe_capture_with_faster_whisper(audio_path: Path, *, model: str, language: str) -> str:
|
||||
with AUTO_TRAIN_LOCK:
|
||||
device_value = AUTO_TRAIN_CONFIG.get("stt_device")
|
||||
compute_value = AUTO_TRAIN_CONFIG.get("stt_compute_type")
|
||||
device, compute_type = _resolve_faster_whisper_runtime(device_value, compute_value)
|
||||
whisper_model = _load_faster_whisper_model(
|
||||
model_name=model,
|
||||
device=device,
|
||||
compute_type=compute_type,
|
||||
)
|
||||
segments, _info = whisper_model.transcribe(
|
||||
str(audio_path),
|
||||
language=language or None,
|
||||
beam_size=1,
|
||||
condition_on_previous_text=False,
|
||||
)
|
||||
transcript = re.sub(
|
||||
r"\s+",
|
||||
" ",
|
||||
" ".join(str(segment.text or "").strip() for segment in segments),
|
||||
).strip()
|
||||
with AUTO_TRAIN_LOCK:
|
||||
AUTO_TRAIN_STATE["last_stt_device"] = device
|
||||
AUTO_TRAIN_STATE["last_stt_compute_type"] = compute_type
|
||||
_save_auto_train_state_locked()
|
||||
return transcript
|
||||
|
||||
|
||||
def _queue_auto_review(file_name: str) -> bool:
|
||||
safe_file_name = Path(str(file_name or "")).name
|
||||
if not safe_file_name:
|
||||
return False
|
||||
with AUTO_TRAIN_LOCK:
|
||||
if safe_file_name in AUTO_TRAIN_QUEUED_FILES:
|
||||
return False
|
||||
AUTO_TRAIN_QUEUED_FILES.add(safe_file_name)
|
||||
AUTO_TRAIN_REVIEW_QUEUE.put(safe_file_name)
|
||||
AUTO_TRAIN_WAKE_EVENT.set()
|
||||
return True
|
||||
|
||||
|
||||
def _queue_pending_auto_reviews(*, force: bool = False) -> int:
|
||||
queued = 0
|
||||
CAPTURED_DIR.mkdir(parents=True, exist_ok=True)
|
||||
for audio_path in sorted(CAPTURED_DIR.glob("*.wav")):
|
||||
metadata = _load_sidecar_json(audio_path)
|
||||
if not _captured_event_is_auto_reviewable(metadata):
|
||||
continue
|
||||
status = str(metadata.get("auto_review_status") or "").strip()
|
||||
if status == "transcribing":
|
||||
metadata.pop("auto_review_status", None)
|
||||
_write_sidecar_json(audio_path, metadata)
|
||||
status = ""
|
||||
if force and status in {"error", "no_speech"}:
|
||||
metadata.pop("auto_review_status", None)
|
||||
_write_sidecar_json(audio_path, metadata)
|
||||
status = ""
|
||||
if status:
|
||||
continue
|
||||
if _queue_auto_review(audio_path.name):
|
||||
queued += 1
|
||||
return queued
|
||||
|
||||
|
||||
def _record_auto_review_result(*, file_name: str, transcript: str = "", result: str = "", error: str = "") -> None:
|
||||
with AUTO_TRAIN_LOCK:
|
||||
AUTO_TRAIN_STATE["last_review_at"] = _iso_now()
|
||||
AUTO_TRAIN_STATE["last_review_file"] = file_name
|
||||
AUTO_TRAIN_STATE["last_review_transcript"] = transcript
|
||||
AUTO_TRAIN_STATE["last_review_result"] = result
|
||||
AUTO_TRAIN_STATE["last_review_error"] = error
|
||||
_save_auto_train_state_locked()
|
||||
|
||||
|
||||
def _auto_review_capture(file_name: str) -> None:
|
||||
try:
|
||||
with AUTO_TRAIN_LOCK:
|
||||
config = dict(AUTO_TRAIN_CONFIG)
|
||||
AUTO_TRAIN_RUNTIME["review_running"] = True
|
||||
AUTO_TRAIN_RUNTIME["review_file"] = file_name
|
||||
if not config.get("enabled"):
|
||||
return
|
||||
wake_phrase = str(config.get("wake_phrase") or "").strip()
|
||||
if not wake_phrase:
|
||||
_record_auto_review_result(file_name=file_name, result="waiting_for_wake_phrase")
|
||||
return
|
||||
|
||||
try:
|
||||
audio_path = _resolve_audio_path(CAPTURED_DIR, file_name)
|
||||
except FileNotFoundError:
|
||||
return
|
||||
metadata = _load_sidecar_json(audio_path)
|
||||
if metadata.get("auto_review_status") or not _captured_event_is_auto_reviewable(metadata):
|
||||
return
|
||||
captured_wake_phrase = str(metadata.get("wake_word") or "").strip()
|
||||
if captured_wake_phrase and _normalize_transcript_text(captured_wake_phrase) != _normalize_transcript_text(wake_phrase):
|
||||
metadata["auto_review_status"] = "different_wake_phrase"
|
||||
metadata["auto_review_reason"] = (
|
||||
f"Capture is for '{captured_wake_phrase}', not configured phrase '{wake_phrase}'; left for manual review."
|
||||
)
|
||||
metadata["auto_reviewed_at"] = _iso_now()
|
||||
_write_sidecar_json(audio_path, metadata)
|
||||
_record_auto_review_result(file_name=file_name, result="different_wake_phrase")
|
||||
return
|
||||
|
||||
metadata["auto_review_status"] = "transcribing"
|
||||
metadata["auto_reviewed_at"] = _iso_now()
|
||||
metadata["auto_review_wake_phrase"] = wake_phrase
|
||||
metadata["auto_review_stt_model"] = config["stt_model"]
|
||||
_write_sidecar_json(audio_path, metadata)
|
||||
|
||||
transcript = _transcribe_capture_with_faster_whisper(
|
||||
audio_path,
|
||||
model=str(config["stt_model"]),
|
||||
language=str(config.get("language") or DEFAULT_LANGUAGE),
|
||||
)
|
||||
normalized = _normalize_transcript_text(transcript)
|
||||
metadata = _load_sidecar_json(audio_path)
|
||||
metadata["transcript"] = transcript
|
||||
metadata["transcribed_at"] = _iso_now()
|
||||
|
||||
if len(normalized) < int(config.get("minimum_transcript_chars") or 2):
|
||||
metadata["auto_review_status"] = "no_speech"
|
||||
metadata["auto_review_reason"] = "STT did not return enough text; left for manual review."
|
||||
_write_sidecar_json(audio_path, metadata)
|
||||
_record_auto_review_result(file_name=file_name, transcript=transcript, result="no_speech")
|
||||
return
|
||||
|
||||
if _transcript_contains_wake_phrase(transcript, wake_phrase):
|
||||
metadata["auto_review_status"] = "wake_phrase_detected"
|
||||
metadata["auto_review_reason"] = "Wake phrase found in transcript; left for manual positive review."
|
||||
_write_sidecar_json(audio_path, metadata)
|
||||
_record_auto_review_result(file_name=file_name, transcript=transcript, result="wake_phrase_detected")
|
||||
return
|
||||
|
||||
metadata["auto_review_status"] = "approved_negative"
|
||||
metadata["auto_review_reason"] = "Wake phrase was not found in the STT transcript."
|
||||
metadata["auto_negative"] = True
|
||||
_write_sidecar_json(audio_path, metadata)
|
||||
_move_captured_audio(
|
||||
file_name,
|
||||
NEGATIVE_DIR,
|
||||
target_prefix="negative",
|
||||
review_status="auto_approved_negative",
|
||||
)
|
||||
with AUTO_TRAIN_LOCK:
|
||||
AUTO_TRAIN_STATE["pending_negative_count"] = int(AUTO_TRAIN_STATE.get("pending_negative_count") or 0) + 1
|
||||
_save_auto_train_state_locked()
|
||||
_record_auto_review_result(file_name=file_name, transcript=transcript, result="approved_negative")
|
||||
except Exception as exc:
|
||||
error = str(exc)
|
||||
with contextlib.suppress(Exception):
|
||||
audio_path = _resolve_audio_path(CAPTURED_DIR, file_name)
|
||||
metadata = _load_sidecar_json(audio_path)
|
||||
metadata["auto_review_status"] = "error"
|
||||
metadata["auto_review_error"] = error
|
||||
metadata["auto_reviewed_at"] = _iso_now()
|
||||
_write_sidecar_json(audio_path, metadata)
|
||||
_record_auto_review_result(file_name=file_name, result="error", error=error)
|
||||
finally:
|
||||
with AUTO_TRAIN_LOCK:
|
||||
AUTO_TRAIN_RUNTIME["review_running"] = False
|
||||
AUTO_TRAIN_RUNTIME["review_file"] = ""
|
||||
|
||||
|
||||
def _notify_tater_satellites() -> Dict[str, Any]:
|
||||
with AUTO_TRAIN_LOCK:
|
||||
config = dict(AUTO_TRAIN_CONFIG)
|
||||
if not config.get("notify_satellites"):
|
||||
return {"ok": True, "skipped": True, "message": "Satellite notification is disabled."}
|
||||
|
||||
endpoint = f"{str(config.get('tater_url') or '').rstrip('/')}/api/tater/satellite/v1/settings"
|
||||
body = json.dumps(
|
||||
{
|
||||
"selector": str(config.get("tater_selector") or ""),
|
||||
"settings": {},
|
||||
}
|
||||
).encode("utf-8")
|
||||
headers = {"Content-Type": "application/json", "User-Agent": "microWakeWord-Trainer/auto-train"}
|
||||
token = str(config.get("tater_api_token") or "").strip()
|
||||
if token:
|
||||
headers["X-Tater-Token"] = token
|
||||
|
||||
try:
|
||||
req = URLRequest(endpoint, data=body, headers=headers, method="POST")
|
||||
with urlopen(req, timeout=15) as response:
|
||||
payload = json.loads(response.read().decode("utf-8"))
|
||||
push = payload.get("push") if isinstance(payload, dict) and isinstance(payload.get("push"), dict) else {}
|
||||
count = push.get("count")
|
||||
with AUTO_TRAIN_LOCK:
|
||||
AUTO_TRAIN_STATE["last_notify_at"] = _iso_now()
|
||||
AUTO_TRAIN_STATE["last_notify_count"] = count
|
||||
AUTO_TRAIN_STATE["last_notify_error"] = ""
|
||||
_save_auto_train_state_locked()
|
||||
return {"ok": True, "count": count, "response": payload}
|
||||
except Exception as exc:
|
||||
with AUTO_TRAIN_LOCK:
|
||||
AUTO_TRAIN_STATE["last_notify_at"] = _iso_now()
|
||||
AUTO_TRAIN_STATE["last_notify_count"] = None
|
||||
AUTO_TRAIN_STATE["last_notify_error"] = str(exc)
|
||||
_save_auto_train_state_locked()
|
||||
return {"ok": False, "error": str(exc)}
|
||||
|
||||
|
||||
def _start_auto_training() -> Dict[str, Any]:
|
||||
with AUTO_TRAIN_LOCK:
|
||||
config = dict(AUTO_TRAIN_CONFIG)
|
||||
wake_phrase = str(config.get("wake_phrase") or "").strip()
|
||||
if not wake_phrase:
|
||||
return {"ok": False, "error": "Auto Training needs a wake phrase."}
|
||||
safe_word = safe_name(wake_phrase)
|
||||
language = str(config.get("language") or DEFAULT_LANGUAGE)
|
||||
with STATE_LOCK:
|
||||
if STATE["training"]["running"]:
|
||||
return {"ok": False, "error": "Training already running."}
|
||||
STATE["raw_phrase"] = wake_phrase
|
||||
STATE["safe_word"] = safe_word
|
||||
STATE["language"] = language
|
||||
STATE["training"]["running"] = True
|
||||
with AUTO_TRAIN_LOCK:
|
||||
AUTO_TRAIN_STATE["last_train_started_at"] = _iso_now()
|
||||
AUTO_TRAIN_RUNTIME["training_pending_consumed"] = int(AUTO_TRAIN_STATE.get("pending_negative_count") or 0)
|
||||
_save_auto_train_state_locked()
|
||||
threading.Thread(
|
||||
target=_run_training_background,
|
||||
args=(safe_word, language, True, True),
|
||||
daemon=True,
|
||||
).start()
|
||||
return {"ok": True, "started": True, "safe_word": safe_word, "language": language}
|
||||
|
||||
|
||||
def _maybe_run_scheduled_auto_training() -> None:
|
||||
with AUTO_TRAIN_LOCK:
|
||||
if not AUTO_TRAIN_CONFIG.get("enabled"):
|
||||
return
|
||||
schedule_hours = int(AUTO_TRAIN_CONFIG.get("schedule_hours") or 0)
|
||||
if schedule_hours <= 0:
|
||||
return
|
||||
next_run = _parse_iso_datetime(AUTO_TRAIN_STATE.get("next_run_at"))
|
||||
if next_run is None:
|
||||
_schedule_next_auto_run_locked()
|
||||
return
|
||||
now = _utc_now()
|
||||
if now < next_run:
|
||||
return
|
||||
pending = int(AUTO_TRAIN_STATE.get("pending_negative_count") or 0)
|
||||
minimum = int(AUTO_TRAIN_CONFIG.get("minimum_new_negatives") or 1)
|
||||
if pending < minimum:
|
||||
_schedule_next_auto_run_locked(from_time=now)
|
||||
return
|
||||
result = _start_auto_training()
|
||||
with AUTO_TRAIN_LOCK:
|
||||
if result.get("started"):
|
||||
_schedule_next_auto_run_locked()
|
||||
else:
|
||||
AUTO_TRAIN_STATE["next_run_at"] = (_utc_now() + timedelta(minutes=10)).isoformat()
|
||||
_save_auto_train_state_locked()
|
||||
|
||||
|
||||
def _auto_train_worker_loop() -> None:
|
||||
with AUTO_TRAIN_LOCK:
|
||||
AUTO_TRAIN_RUNTIME["scheduler_running"] = True
|
||||
_queue_pending_auto_reviews()
|
||||
try:
|
||||
while not AUTO_TRAIN_STOP_EVENT.is_set():
|
||||
try:
|
||||
file_name = AUTO_TRAIN_REVIEW_QUEUE.get_nowait()
|
||||
except queue.Empty:
|
||||
file_name = ""
|
||||
if file_name:
|
||||
try:
|
||||
_auto_review_capture(file_name)
|
||||
finally:
|
||||
with AUTO_TRAIN_LOCK:
|
||||
AUTO_TRAIN_QUEUED_FILES.discard(file_name)
|
||||
AUTO_TRAIN_REVIEW_QUEUE.task_done()
|
||||
_maybe_run_scheduled_auto_training()
|
||||
AUTO_TRAIN_WAKE_EVENT.wait(1.0)
|
||||
AUTO_TRAIN_WAKE_EVENT.clear()
|
||||
finally:
|
||||
with AUTO_TRAIN_LOCK:
|
||||
AUTO_TRAIN_RUNTIME["scheduler_running"] = False
|
||||
|
||||
|
||||
def _start_auto_train_worker() -> None:
|
||||
global AUTO_TRAIN_WORKER
|
||||
with AUTO_TRAIN_LOCK:
|
||||
if AUTO_TRAIN_WORKER is not None and AUTO_TRAIN_WORKER.is_alive():
|
||||
return
|
||||
AUTO_TRAIN_STOP_EVENT.clear()
|
||||
AUTO_TRAIN_WORKER = threading.Thread(
|
||||
target=_auto_train_worker_loop,
|
||||
name="auto-train-worker",
|
||||
daemon=True,
|
||||
)
|
||||
AUTO_TRAIN_WORKER.start()
|
||||
|
||||
|
||||
def _stop_auto_train_worker() -> None:
|
||||
AUTO_TRAIN_STOP_EVENT.set()
|
||||
AUTO_TRAIN_WAKE_EVENT.set()
|
||||
|
||||
|
||||
|
||||
def _sync_personal_samples_state() -> List[str]:
|
||||
takes = _list_personal_samples()
|
||||
with STATE_LOCK:
|
||||
@@ -1016,6 +1676,11 @@ def _captured_item_from_path(audio_path: Path) -> Dict[str, Any]:
|
||||
"message": meta.get("message") or "",
|
||||
"notes": meta.get("notes") or "",
|
||||
"review_status": meta.get("review_status") or "pending",
|
||||
"transcript": meta.get("transcript") or "",
|
||||
"transcribed_at": meta.get("transcribed_at") or "",
|
||||
"auto_review_status": meta.get("auto_review_status") or "",
|
||||
"auto_review_reason": meta.get("auto_review_reason") or "",
|
||||
"auto_review_error": meta.get("auto_review_error") or "",
|
||||
"size_bytes": stat.st_size,
|
||||
"audio_url": f"/api/audio/captured/{audio_path.name}",
|
||||
}
|
||||
@@ -1051,6 +1716,10 @@ def _sample_item_from_path(audio_path: Path, bucket: str) -> Dict[str, Any]:
|
||||
"source_file": meta.get("source_file") or "",
|
||||
"final_format": final_format,
|
||||
"message": meta.get("message") or "",
|
||||
"transcript": meta.get("transcript") or "",
|
||||
"transcribed_at": meta.get("transcribed_at") or "",
|
||||
"auto_negative": bool(meta.get("auto_negative")),
|
||||
"auto_review_reason": meta.get("auto_review_reason") or "",
|
||||
"size_bytes": stat.st_size,
|
||||
"audio_url": f"/api/audio/{bucket}/{audio_path.name}",
|
||||
}
|
||||
@@ -1345,8 +2014,14 @@ def _normalize_output_artifacts(safe_word: str, log_path: Path) -> None:
|
||||
_append_train_log(f"✅ Trained wake words synced to {TRAINED_WAKE_WORDS_DIR}")
|
||||
|
||||
|
||||
def _run_training_background(safe_word: str, language: str, allow_no_personal: bool):
|
||||
def _run_training_background(
|
||||
safe_word: str,
|
||||
language: str,
|
||||
allow_no_personal: bool,
|
||||
auto_run: bool = False,
|
||||
):
|
||||
language = (language or DEFAULT_LANGUAGE).strip().lower() or DEFAULT_LANGUAGE
|
||||
rc = 999
|
||||
with STATE_LOCK:
|
||||
raw_phrase = STATE.get("raw_phrase") or ""
|
||||
|
||||
@@ -1418,6 +2093,7 @@ def _run_training_background(safe_word: str, language: str, allow_no_personal: b
|
||||
_normalize_output_artifacts(safe_word, log_path)
|
||||
|
||||
except Exception as e:
|
||||
rc = 999
|
||||
_append_train_log(f"✗ Training crashed: {e!r}")
|
||||
with STATE_LOCK:
|
||||
STATE["training"]["exit_code"] = 999
|
||||
@@ -1426,8 +2102,111 @@ def _run_training_background(safe_word: str, language: str, allow_no_personal: b
|
||||
with STATE_LOCK:
|
||||
STATE["training"]["running"] = False
|
||||
|
||||
if auto_run:
|
||||
with AUTO_TRAIN_LOCK:
|
||||
AUTO_TRAIN_STATE["last_train_finished_at"] = _iso_now()
|
||||
AUTO_TRAIN_STATE["last_train_exit_code"] = rc
|
||||
if rc == 0:
|
||||
consumed = int(AUTO_TRAIN_RUNTIME.get("training_pending_consumed") or 0)
|
||||
AUTO_TRAIN_STATE["pending_negative_count"] = max(
|
||||
0,
|
||||
int(AUTO_TRAIN_STATE.get("pending_negative_count") or 0) - consumed,
|
||||
)
|
||||
AUTO_TRAIN_RUNTIME["training_pending_consumed"] = 0
|
||||
_save_auto_train_state_locked()
|
||||
if rc == 0:
|
||||
_append_train_log("→ Asking Tater to refresh the active wake model on connected satellites")
|
||||
notify_result = _notify_tater_satellites()
|
||||
if notify_result.get("ok"):
|
||||
if notify_result.get("skipped"):
|
||||
_append_train_log("→ Satellite refresh skipped (disabled in Auto Training)")
|
||||
else:
|
||||
count = notify_result.get("count")
|
||||
suffix = f" ({count} connected)" if count is not None else ""
|
||||
_append_train_log(f"✓ Tater satellite refresh requested{suffix}")
|
||||
else:
|
||||
_append_train_log(f"✗ Tater satellite refresh failed: {notify_result.get('error')}")
|
||||
|
||||
|
||||
# -------------------- Routes --------------------
|
||||
@app.on_event("startup")
|
||||
def start_auto_train_worker_event():
|
||||
_start_auto_train_worker()
|
||||
|
||||
|
||||
@app.on_event("shutdown")
|
||||
def stop_auto_train_worker_event():
|
||||
_stop_auto_train_worker()
|
||||
|
||||
|
||||
@app.get("/api/auto_train")
|
||||
def auto_train_status(request: Request):
|
||||
payload = _auto_train_status_payload()
|
||||
payload["ok"] = True
|
||||
payload["advertised_base_url"] = _advertised_base_url(request)
|
||||
payload["stt_backend"] = "faster-whisper"
|
||||
return payload
|
||||
|
||||
|
||||
@app.put("/api/auto_train")
|
||||
def update_auto_train(payload: Dict[str, Any] = None):
|
||||
incoming = dict(payload or {})
|
||||
with AUTO_TRAIN_LOCK:
|
||||
previous = dict(AUTO_TRAIN_CONFIG)
|
||||
if incoming.pop("clear_tater_api_token", False):
|
||||
incoming["tater_api_token"] = ""
|
||||
elif not str(incoming.get("tater_api_token") or "").strip():
|
||||
incoming.pop("tater_api_token", None)
|
||||
try:
|
||||
normalized = _normalize_auto_train_config(incoming, base=previous)
|
||||
except ValueError as exc:
|
||||
return JSONResponse({"ok": False, "error": str(exc)}, status_code=400)
|
||||
if normalized["enabled"] and not normalized["wake_phrase"]:
|
||||
return JSONResponse(
|
||||
{"ok": False, "error": "Enter the wake phrase before enabling Auto Training."},
|
||||
status_code=400,
|
||||
)
|
||||
AUTO_TRAIN_CONFIG.clear()
|
||||
AUTO_TRAIN_CONFIG.update(normalized)
|
||||
_save_auto_train_config_locked()
|
||||
schedule_changed = (
|
||||
previous.get("enabled") != normalized.get("enabled")
|
||||
or previous.get("schedule_hours") != normalized.get("schedule_hours")
|
||||
)
|
||||
if schedule_changed or not AUTO_TRAIN_STATE.get("next_run_at"):
|
||||
_schedule_next_auto_run_locked()
|
||||
if normalized["enabled"]:
|
||||
queued = _queue_pending_auto_reviews()
|
||||
AUTO_TRAIN_WAKE_EVENT.set()
|
||||
else:
|
||||
queued = 0
|
||||
return {"ok": True, "queued": queued, **_auto_train_status_payload()}
|
||||
|
||||
|
||||
@app.post("/api/auto_train/action")
|
||||
def auto_train_action(payload: Dict[str, Any] = None):
|
||||
action = str((payload or {}).get("action") or "").strip().lower()
|
||||
if action == "review_now":
|
||||
with AUTO_TRAIN_LOCK:
|
||||
if not AUTO_TRAIN_CONFIG.get("enabled"):
|
||||
return JSONResponse({"ok": False, "error": "Enable Auto Training first."}, status_code=400)
|
||||
queued = _queue_pending_auto_reviews(force=True)
|
||||
AUTO_TRAIN_WAKE_EVENT.set()
|
||||
return {"ok": True, "queued": queued, **_auto_train_status_payload()}
|
||||
if action == "train_now":
|
||||
result = _start_auto_training()
|
||||
if not result.get("ok"):
|
||||
return JSONResponse(result, status_code=400)
|
||||
return {**result, **_auto_train_status_payload()}
|
||||
if action == "notify_now":
|
||||
result = _notify_tater_satellites()
|
||||
if not result.get("ok"):
|
||||
return JSONResponse(result, status_code=502)
|
||||
return {**result, **_auto_train_status_payload()}
|
||||
return JSONResponse({"ok": False, "error": "Unknown Auto Training action."}, status_code=400)
|
||||
|
||||
|
||||
|
||||
@app.get("/", response_class=HTMLResponse)
|
||||
def index():
|
||||
html_path = STATIC_DIR / "index.html"
|
||||
@@ -1621,6 +2400,10 @@ async def upload_captured_audio(
|
||||
"review_status": "pending",
|
||||
}
|
||||
_write_sidecar_json(audio_path, sidecar)
|
||||
with AUTO_TRAIN_LOCK:
|
||||
auto_review_enabled = bool(AUTO_TRAIN_CONFIG.get("enabled"))
|
||||
if auto_review_enabled and _captured_event_is_auto_reviewable(sidecar):
|
||||
_queue_auto_review(audio_path.name)
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
@@ -1703,6 +2486,10 @@ async def upload_captured_audio_raw(
|
||||
"review_status": "pending",
|
||||
}
|
||||
_write_sidecar_json(audio_path, sidecar)
|
||||
with AUTO_TRAIN_LOCK:
|
||||
auto_review_enabled = bool(AUTO_TRAIN_CONFIG.get("enabled"))
|
||||
if auto_review_enabled and _captured_event_is_auto_reviewable(sidecar):
|
||||
_queue_auto_review(audio_path.name)
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
@@ -1760,9 +2547,17 @@ def delete_sample(bucket: str, file_name: str):
|
||||
return JSONResponse({"ok": False, "error": "Unknown sample bucket."}, status_code=404)
|
||||
try:
|
||||
path = _resolve_audio_path(directory, file_name)
|
||||
metadata = _load_sidecar_json(path)
|
||||
_remove_audio_with_sidecar(path)
|
||||
except FileNotFoundError as e:
|
||||
return JSONResponse({"ok": False, "error": str(e)}, status_code=404)
|
||||
if bucket == "negative" and metadata.get("auto_negative"):
|
||||
with AUTO_TRAIN_LOCK:
|
||||
AUTO_TRAIN_STATE["pending_negative_count"] = max(
|
||||
0,
|
||||
int(AUTO_TRAIN_STATE.get("pending_negative_count") or 0) - 1,
|
||||
)
|
||||
_save_auto_train_state_locked()
|
||||
return {"ok": True, "deleted_bucket": bucket, "deleted_file": file_name, "message": f"Deleted {file_name}"}
|
||||
|
||||
|
||||
@@ -1936,7 +2731,11 @@ def discard_captured_audio(file_name: str):
|
||||
|
||||
@app.get("/api/trained_wake_words/catalog")
|
||||
def trained_wake_words_catalog(request: Request):
|
||||
return {"ok": True, "wake_words": _list_trained_wake_words(_request_base_url(request))}
|
||||
return {
|
||||
"ok": True,
|
||||
"base_url": _advertised_base_url(request),
|
||||
"wake_words": _list_trained_wake_words(_advertised_base_url(request)),
|
||||
}
|
||||
|
||||
|
||||
@app.get("/api/trained_wake_words/{filename}")
|
||||
@@ -1985,7 +2784,13 @@ def train_now(payload: Dict[str, Any] = None):
|
||||
status_code=400,
|
||||
)
|
||||
|
||||
t = threading.Thread(target=_run_training_background, args=(safe_word, language, allow_no_personal), daemon=True)
|
||||
with STATE_LOCK:
|
||||
STATE["training"]["running"] = True
|
||||
t = threading.Thread(
|
||||
target=_run_training_background,
|
||||
args=(safe_word, language, allow_no_personal, False),
|
||||
daemon=True,
|
||||
)
|
||||
t.start()
|
||||
|
||||
return {
|
||||
@@ -2043,4 +2848,7 @@ def reset_recordings():
|
||||
@app.post("/api/reset_negative_samples")
|
||||
def reset_negative_samples():
|
||||
_reset_audio_dir(NEGATIVE_DIR)
|
||||
with AUTO_TRAIN_LOCK:
|
||||
AUTO_TRAIN_STATE["pending_negative_count"] = 0
|
||||
_save_auto_train_state_locked()
|
||||
return {"ok": True, "negative_count": len(_list_negative_samples())}
|
||||
|
||||
Reference in New Issue
Block a user