The files in the `cli` directory allow you to train wake words
from the command line without needing to use the Jupyter notebook
or a web browser. Basically, the logic from the notebook has been
placed in separate shell scripts and python files wrapped by 3 high-level
scripts that do the following:
* setup_python_venv: Creates a Python virtual environment with all the
packages needed to train. The venv is created in the container's /data
directory and is therefore stored on the host, not in the container's root
docker volume.
* setup_training_datasets: Downloads, extracts and converts the MIT RIR,
FMA, Audioset and Negative training reference datasets. Also stored in /data.
* train_wake_word: Generates the wake word samples, augments them with the
audio from the training datasets, and finally runs the microwakeword training.
The resulting model tflite and json files are placed in the /data/output
directory.
See the README.md file for much more information.
Two issues:
* The notebook cell that actually runs model_train_eval was running it in a
subprocess so while it inherited environment variables from the running
python kernel, it couldn't inherit the tensorflow environment from it.
This resulted in the `set_memory_growth(g, True)` and
`mixed_precision.set_global_policy("mixed_float16")` calls in the previous
cell to be lost.
* TFlite doesn't support "mixed_float16" anyway and causes the model export to
fail spectacularly so it's kind of a good thing it wasn't being applied.
So..
* The tensorflow environment variable and memory_growth setting code was moved
from the notebook cell that also wrote the config yaml to the next cell
which does the train and test. This leaves the "config" cell to just write
the yaml. This is really just a cosmetic change to group functionality
better.
* The code that tried to set "mixed_float16" has been removed but since setting
memory_growth to true is a good thing, the model_train_eval is now run using
runpy instead in a subprocess. This way it's run in the same python kernel
instance and tensorflow environment as the rest of the notebook and inherits
the memory_growth setting.
Resolves: #14
Since this is a pure Python environment, the CUDA toolkit isn't really
necessary. The various Python packages that can use CUDA will download
and install the CUDA dependencies they need. This shaves off at least
8gb from the final image.
The Python package install order needed to be tweaked to ensure onnxruntime,
tensorflow and torch are installed in that order. Any other order results
in dependent cuda package clashes.
Resolves: #12
