to execute or view/download this notebook on GitHub
HyperPyYAML Tutorial
An essential aspect of any deep learning pipeline is the definition of hyperparameters and other metadata. These hyperparameters, in conjunction with the deep learning algorithms, govern various aspects of the pipeline, including model architecture, training, and decoding.
In SpeechBrain, we emphasize a clear distinction between hyperparameters and learning algorithms in the structure of our toolkit. To achieve this, we separate our recipes into two primary files: train.py
and train.yaml
.
The train.yaml
file follows a format developed by SpeechBrain, known as “HyperPyYAML.” We chose to extend YAML due to its highly readable nature for data serialization. By building upon this already user-friendly format, we have created an extended definition of hyperparameters, ensuring that our experimental code remains concise and easily readable.
Here’s a brief example using PyTorch code to illustrate the use of HyperPyYAML. It’s important to note that PyTorch is not a requirement for utilizing HyperPyYAML:
%%capture
!pip install torch
!pip install hyperpyyaml
import torch
from hyperpyyaml import load_hyperpyyaml
example_hyperparams = """
base_channels: 32
kernel_size: 11
padding: !ref <kernel_size> // 2
layer1: !new:torch.nn.Conv1d
in_channels: 1
out_channels: !ref <base_channels>
kernel_size: !ref <kernel_size>
padding: !ref <padding>
layer2: !new:torch.nn.Conv1d
in_channels: !ref <base_channels>
out_channels: !ref <base_channels> * 2
kernel_size: !ref <kernel_size>
padding: !ref <padding>
layer3: !new:torch.nn.Conv1d
in_channels: !ref <base_channels> * 2
out_channels: 1
kernel_size: !ref <kernel_size>
padding: !ref <padding>
model: !new:torch.nn.Sequential
- !ref <layer1>
- !new:torch.nn.LeakyReLU
- !ref <layer2>
- !new:torch.nn.LeakyReLU
- !ref <layer3>
"""
# Create model directly by loading the YAML
loaded_hparams = load_hyperpyyaml(example_hyperparams)
model = loaded_hparams["model"]
# Transform a 2-second audio clip
input_audio = torch.rand(1, 1, 32000)
transformed_audio = model(input_audio)
print(transformed_audio.shape)
# Try a different hyperparameter value by overriding the padding value
loaded_hparams = load_hyperpyyaml(example_hyperparams, {"padding": 0})
model = loaded_hparams["model"]
transformed_audio = model(input_audio)
print(transformed_audio.shape)
As this example shows, HyperPyYAML allows for complex hyperparameter definitions with compositions. In addition, any value can be overridden for hyperparameter tuning. To grasp how all of this works, let’s first briefly look at the basics of YAML.
Basic YAML syntax
Enough prelude: lets talk YAML! Here’s a brief example of a yaml snippet and what it would look like once loaded to python:
import yaml
yaml_string = """
foo: 1
bar:
- item1
- item2
baz:
item1: 3.4
item2: True
"""
yaml.safe_load(yaml_string)
As you can see, YAML has built-in support for a variety of data types, including string, int, float, bool, list, and dictionary. Our HyperPyYAML format keeps all of this functionality.
from hyperpyyaml import load_hyperpyyaml
load_hyperpyyaml(yaml_string)
Our primary additions to yaml format are added with YAML tags. Tags are added before an item definition, and are prefixed with !
. For the purpose of illustrating how tags are used, here is an example with a minor addition that we’ve made, the !tuple
tag:
yaml_string = """
foo: !tuple (3, 4)
"""
load_hyperpyyaml(yaml_string)
Now you know the YAML basics, time to move on to our additions!
Overrides
In order to run experiments with various values for a hyperparameter, we have a system for overriding the values that are listed in the yaml file.
overrides = {"foo": 7}
fake_file = """
foo: 2
bar: 5
"""
load_hyperpyyaml(fake_file, overrides)
As shown in this example, overrides can take an ordinary python dictionary. However, this form does not support python objects. To override a python object, overrides can also take a yaml-formatted string with the HyperPyYAML syntax.
load_hyperpyyaml(fake_file, "foo: !new:collections.Counter")
Conclusion
We are proud to present our HyperPyYAML syntax, and we think that it provides a readable and concise way to structure hyperparameter definitions. In addition, it removes unnecessary complexity from experiment files, allowing the algorithms to become clear. As is evident in the first example, overrides are easy, making hyperparameter tuning a cinch. Overall, we have found this package to be a valuable tool for deep learning!
Citing SpeechBrain
If you use SpeechBrain in your research or business, please cite it using the following BibTeX entry:
@misc{speechbrainV1,
title={Open-Source Conversational AI with {SpeechBrain} 1.0},
author={Mirco Ravanelli and Titouan Parcollet and Adel Moumen and Sylvain de Langen and Cem Subakan and Peter Plantinga and Yingzhi Wang and Pooneh Mousavi and Luca Della Libera and Artem Ploujnikov and Francesco Paissan and Davide Borra and Salah Zaiem and Zeyu Zhao and Shucong Zhang and Georgios Karakasidis and Sung-Lin Yeh and Pierre Champion and Aku Rouhe and Rudolf Braun and Florian Mai and Juan Zuluaga-Gomez and Seyed Mahed Mousavi and Andreas Nautsch and Xuechen Liu and Sangeet Sagar and Jarod Duret and Salima Mdhaffar and Gaelle Laperriere and Mickael Rouvier and Renato De Mori and Yannick Esteve},
year={2024},
eprint={2407.00463},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2407.00463},
}
@misc{speechbrain,
title={{SpeechBrain}: A General-Purpose Speech Toolkit},
author={Mirco Ravanelli and Titouan Parcollet and Peter Plantinga and Aku Rouhe and Samuele Cornell and Loren Lugosch and Cem Subakan and Nauman Dawalatabad and Abdelwahab Heba and Jianyuan Zhong and Ju-Chieh Chou and Sung-Lin Yeh and Szu-Wei Fu and Chien-Feng Liao and Elena Rastorgueva and François Grondin and William Aris and Hwidong Na and Yan Gao and Renato De Mori and Yoshua Bengio},
year={2021},
eprint={2106.04624},
archivePrefix={arXiv},
primaryClass={eess.AS},
note={arXiv:2106.04624}
}