{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sb_auto_header",
    "tags": [
     "sb_auto_header"
    ]
   },
   "source": [
    "<!-- This cell is automatically updated by tools/tutorial-cell-updater.py -->\n",
    "<!-- The contents are initialized from tutorials/notebook-header.md -->\n",
    "\n",
    "[<img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>](https://colab.research.google.com/github/speechbrain/speechbrain/blob/develop/docs/tutorials/basics/hyperpyyaml.ipynb)\n",
    "to execute or view/download this notebook on\n",
    "[GitHub](https://github.com/speechbrain/speechbrain/tree/develop/docs/tutorials/basics/hyperpyyaml.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jr2jH1sDZcml"
   },
   "source": [
    "# HyperPyYAML Tutorial\n",
    "\n",
    "An essential aspect of any deep learning pipeline is the definition of hyperparameters and other metadata. These hyperparameters, in conjunction with the deep learning algorithms, govern various aspects of the pipeline, including model architecture, training, and decoding.\n",
    "\n",
    "In SpeechBrain, we emphasize a clear distinction between hyperparameters and learning algorithms in the structure of our toolkit. To achieve this, we separate our recipes into two primary files: `train.py` and `train.yaml`.\n",
    "\n",
    "The `train.yaml` file follows a format developed by SpeechBrain, known as \"HyperPyYAML.\" We chose to extend YAML due to its highly readable nature for data serialization. By building upon this already user-friendly format, we have created an extended definition of hyperparameters, ensuring that our experimental code remains concise and easily readable.\n",
    "\n",
    "Here's a brief example using PyTorch code to illustrate the use of HyperPyYAML. It's important to note that PyTorch is not a requirement for utilizing HyperPyYAML:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "sahuT6WdbeAy"
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "!pip install torch\n",
    "!pip install hyperpyyaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "OYuI_vzLbhJz"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from hyperpyyaml import load_hyperpyyaml\n",
    "\n",
    "example_hyperparams = \"\"\"\n",
    "base_channels: 32\n",
    "kernel_size: 11\n",
    "padding: !ref <kernel_size> // 2\n",
    "\n",
    "layer1: !new:torch.nn.Conv1d\n",
    "  in_channels: 1\n",
    "  out_channels: !ref <base_channels>\n",
    "  kernel_size: !ref <kernel_size>\n",
    "  padding: !ref <padding>\n",
    "\n",
    "layer2: !new:torch.nn.Conv1d\n",
    "  in_channels: !ref <base_channels>\n",
    "  out_channels: !ref <base_channels> * 2\n",
    "  kernel_size: !ref <kernel_size>\n",
    "  padding: !ref <padding>\n",
    "\n",
    "layer3: !new:torch.nn.Conv1d\n",
    "  in_channels: !ref <base_channels> * 2\n",
    "  out_channels: 1\n",
    "  kernel_size: !ref <kernel_size>\n",
    "  padding: !ref <padding>\n",
    "\n",
    "model: !new:torch.nn.Sequential\n",
    "  - !ref <layer1>\n",
    "  - !new:torch.nn.LeakyReLU\n",
    "  - !ref <layer2>\n",
    "  - !new:torch.nn.LeakyReLU\n",
    "  - !ref <layer3>\n",
    "\"\"\"\n",
    "\n",
    "# Create model directly by loading the YAML\n",
    "loaded_hparams = load_hyperpyyaml(example_hyperparams)\n",
    "model = loaded_hparams[\"model\"]\n",
    "\n",
    "# Transform a 2-second audio clip\n",
    "input_audio = torch.rand(1, 1, 32000)\n",
    "transformed_audio = model(input_audio)\n",
    "print(transformed_audio.shape)\n",
    "\n",
    "# Try a different hyperparameter value by overriding the padding value\n",
    "loaded_hparams = load_hyperpyyaml(example_hyperparams, {\"padding\": 0})\n",
    "model = loaded_hparams[\"model\"]\n",
    "transformed_audio = model(input_audio)\n",
    "print(transformed_audio.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jYsOid7mE673"
   },
   "source": [
    "As this example shows, HyperPyYAML allows for complex hyperparameter definitions with compositions. In addition, any value can be overridden for hyperparameter tuning. To grasp how all of this works, let's first briefly look at the basics of YAML."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DnfiIsycbqIs"
   },
   "source": [
    "## Basic YAML syntax\n",
    "\n",
    "Enough prelude: lets talk YAML! Here's a brief example of a yaml snippet and what it would look like once loaded to python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8cARbf0cZYCT"
   },
   "outputs": [],
   "source": [
    "import yaml\n",
    "yaml_string = \"\"\"\n",
    "foo: 1\n",
    "bar:\n",
    "  - item1\n",
    "  - item2\n",
    "baz:\n",
    "  item1: 3.4\n",
    "  item2: True\n",
    "\"\"\"\n",
    "yaml.safe_load(yaml_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pf7vC0x5d5vg"
   },
   "source": [
    "As you can see, YAML has built-in support for a variety of data types, including string, int, float, bool, list, and dictionary. Our HyperPyYAML format keeps all of this functionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "xMkgyuexc_3M"
   },
   "outputs": [],
   "source": [
    "from hyperpyyaml import load_hyperpyyaml\n",
    "load_hyperpyyaml(yaml_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WmTHoVgjfNEv"
   },
   "source": [
    "Our primary additions to yaml format are added with YAML tags. Tags are added before an item definition, and are prefixed with `!`. For the purpose of illustrating how tags are used, here is an example with a minor addition that we've made, the `!tuple` tag:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "XRNSh8PCej15"
   },
   "outputs": [],
   "source": [
    "yaml_string = \"\"\"\n",
    "foo: !tuple (3, 4)\n",
    "\"\"\"\n",
    "load_hyperpyyaml(yaml_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7Qwa5t-OkL4b"
   },
   "source": [
    "Now you know the YAML basics, time to move on to our additions!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KmLsqpZUgi87"
   },
   "source": [
    "## Tags `!new:` and `!name:`\n",
    "YAML tags can contain a suffix to more specifically define what type of tag it is. We use this to define a tag that is able to create any python object, not just a basic type. This tag starts with `!new:` and contains the type of the object. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "lSlv5ootgW-g"
   },
   "outputs": [],
   "source": [
    "yaml_string = \"\"\"\n",
    "foo: !new:collections.Counter\n",
    "\"\"\"\n",
    "loaded_yaml = load_hyperpyyaml(yaml_string)\n",
    "loaded_yaml[\"foo\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "uVxoxhIDhoBD"
   },
   "outputs": [],
   "source": [
    "loaded_yaml[\"foo\"].update({\"a\": 3, \"b\": 5})\n",
    "loaded_yaml[\"foo\"][\"a\"] += 1\n",
    "loaded_yaml[\"foo\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NtJ7bt2FmLpQ"
   },
   "source": [
    "Of course many python objects take arguments during creation. These arguments can be passed with a list for positional arguments or a dictionary for keyword arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jBi0dTUzl_Eq"
   },
   "outputs": [],
   "source": [
    "yaml_string = \"\"\"\n",
    "foo: !new:collections.Counter\n",
    "  - [a, b, r, a, c, a, d, a, b, r, a]\n",
    "bar: !new:collections.Counter\n",
    "  a: 2\n",
    "  b: 1\n",
    "  c: 5\n",
    "\"\"\"\n",
    "load_hyperpyyaml(yaml_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2xlFj5Te7A9f"
   },
   "source": [
    "Another python object that is useful to create is a function object. In HyperPyYAML this can be done with the `!name:` tag. Behind the scenes, this tag uses `functools.partial` to create a new function definition with the default arguments provided. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "60wI82NRm8M8"
   },
   "outputs": [],
   "source": [
    "yaml_string = \"\"\"\n",
    "foo: !name:collections.Counter\n",
    "  a: 2\n",
    "\"\"\"\n",
    "loaded_yaml = load_hyperpyyaml(yaml_string)\n",
    "loaded_yaml[\"foo\"](b=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qfBKfFDB8ds8"
   },
   "source": [
    "The default arguments can be overridden, just as a normal python function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "P2SKGzIE8MQ5"
   },
   "outputs": [],
   "source": [
    "loaded_yaml[\"foo\"](a=3, b=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "o8Dh9KB18rn7"
   },
   "source": [
    "## Tags `!ref` and `!copy`\n",
    "\n",
    "Of course some hyperparameters get used in multiple places, so we added a mechanism for referring to another item called `!ref`. The node that this tag is applied to must be a string that contains the location of the node to copy. Sub-nodes can be accessed with square brackets, same as in Python. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BkpqtyKR8mqa"
   },
   "outputs": [],
   "source": [
    "yaml_string = \"\"\"\n",
    "foo:\n",
    "  a: 3\n",
    "  b: 4\n",
    "bar:\n",
    "  c: !ref <foo>\n",
    "  d: !ref <foo[b]>\n",
    "\"\"\"\n",
    "load_hyperpyyaml(yaml_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3sgZ3MNa-uGH"
   },
   "source": [
    "The `!ref` tag can support simple arithmetic and string concatenation for basic hyperparameter combinations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BT0xjRzp-qym"
   },
   "outputs": [],
   "source": [
    "yaml_string = \"\"\"\n",
    "folder1: abc/def\n",
    "folder2: ghi/jkl\n",
    "folder3: !ref <folder1>/<folder2>\n",
    "\n",
    "foo: 1024\n",
    "bar: 512\n",
    "baz: !ref <foo> // <bar> + 1\n",
    "\"\"\"\n",
    "load_hyperpyyaml(yaml_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vp3a9jsnGWBV"
   },
   "source": [
    "The `!ref` tag can also refer to objects, in which case it makes a reference to the same object, rather than a copy. If you'd prefer to make a copy instead, use the `!copy` tag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Wy0NDnrO_Vb0"
   },
   "outputs": [],
   "source": [
    "yaml_string = \"\"\"\n",
    "foo: !new:collections.Counter\n",
    "  a: 4\n",
    "bar: !ref <foo>\n",
    "baz: !copy <foo>\n",
    "\"\"\"\n",
    "loaded_yaml = load_hyperpyyaml(yaml_string)\n",
    "loaded_yaml[\"foo\"].update({\"b\": 10})\n",
    "print(loaded_yaml[\"bar\"])\n",
    "print(loaded_yaml[\"baz\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d8NBOA9WHQIG"
   },
   "source": [
    "## Other tags\n",
    "\n",
    "We introduced a variety of other tags as well:\n",
    "* `!tuple` to create python tuples. Note this is implicitly resolved, so you do not need to explicitly write out the tuple tag, just use parentheses as you would in Python.\n",
    "* `!include` to insert other yaml files directly\n",
    "* `!apply` to load and execute a python function, storing the result\n",
    "\n",
    "We use `!apply` to set the random seed at the beginning of loading the yaml, so that the models have the same parameters each run. The result is not stored, because it starts with `__`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Q9ftz9wMGp9L"
   },
   "outputs": [],
   "source": [
    "yaml_string = \"\"\"\n",
    "sum: !apply:sum\n",
    "  - [1, 2]\n",
    "__set_seed: !apply:torch.manual_seed [1234]\n",
    "\"\"\"\n",
    "load_hyperpyyaml(yaml_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0jzO7UWLJ_iP"
   },
   "source": [
    "## Overrides\n",
    "\n",
    "In order to run experiments with various values for a hyperparameter, we have a system for overriding the values that are listed in the yaml file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "TuZ9s7mBJI0B"
   },
   "outputs": [],
   "source": [
    "overrides = {\"foo\": 7}\n",
    "fake_file = \"\"\"\n",
    "foo: 2\n",
    "bar: 5\n",
    "\"\"\"\n",
    "load_hyperpyyaml(fake_file, overrides)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TRseBBcVL4Hv"
   },
   "source": [
    "As shown in this example, overrides can take an ordinary python dictionary. However, this form does not support python objects. To override a python object, overrides can also take a yaml-formatted string with the HyperPyYAML syntax."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "R05TG8UzLQNj"
   },
   "outputs": [],
   "source": [
    "load_hyperpyyaml(fake_file, \"foo: !new:collections.Counter\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nsWb8t-NMxOo"
   },
   "source": [
    "## Conclusion\n",
    "\n",
    "We are proud to present our HyperPyYAML syntax, and we think that it provides a readable and concise way to structure hyperparameter definitions. In addition, it removes unnecessary complexity from experiment files, allowing the algorithms to become clear. As is evident in the first example, overrides are easy, making hyperparameter tuning a cinch. Overall, we have found this package to be a valuable tool for deep learning!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sb_auto_footer",
    "tags": [
     "sb_auto_footer"
    ]
   },
   "source": [
    "## Citing SpeechBrain\n",
    "\n",
    "If you use SpeechBrain in your research or business, please cite it using the following BibTeX entry:\n",
    "\n",
    "```bibtex\n",
    "@misc{speechbrainV1,\n",
    "  title={Open-Source Conversational AI with {SpeechBrain} 1.0},\n",
    "  author={Mirco Ravanelli and Titouan Parcollet and Adel Moumen and Sylvain de Langen and Cem Subakan and Peter Plantinga and Yingzhi Wang and Pooneh Mousavi and Luca Della Libera and Artem Ploujnikov and Francesco Paissan and Davide Borra and Salah Zaiem and Zeyu Zhao and Shucong Zhang and Georgios Karakasidis and Sung-Lin Yeh and Pierre Champion and Aku Rouhe and Rudolf Braun and Florian Mai and Juan Zuluaga-Gomez and Seyed Mahed Mousavi and Andreas Nautsch and Xuechen Liu and Sangeet Sagar and Jarod Duret and Salima Mdhaffar and Gaelle Laperriere and Mickael Rouvier and Renato De Mori and Yannick Esteve},\n",
    "  year={2024},\n",
    "  eprint={2407.00463},\n",
    "  archivePrefix={arXiv},\n",
    "  primaryClass={cs.LG},\n",
    "  url={https://arxiv.org/abs/2407.00463},\n",
    "}\n",
    "@misc{speechbrain,\n",
    "  title={{SpeechBrain}: A General-Purpose Speech Toolkit},\n",
    "  author={Mirco Ravanelli and Titouan Parcollet and Peter Plantinga and Aku Rouhe and Samuele Cornell and Loren Lugosch and Cem Subakan and Nauman Dawalatabad and Abdelwahab Heba and Jianyuan Zhong and Ju-Chieh Chou and Sung-Lin Yeh and Szu-Wei Fu and Chien-Feng Liao and Elena Rastorgueva and François Grondin and William Aris and Hwidong Na and Yan Gao and Renato De Mori and Yoshua Bengio},\n",
    "  year={2021},\n",
    "  eprint={2106.04624},\n",
    "  archivePrefix={arXiv},\n",
    "  primaryClass={eess.AS},\n",
    "  note={arXiv:2106.04624}\n",
    "}\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": [
    {
     "file_id": "10jZah2QHZ7xuajv9M1yIwRQdePxPV97U",
     "timestamp": 1612452207452
    }
   ]
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}