\n",
"
speechbrain.dataio.batch.PaddedBatch
def __init__(examples, padded_keys=None, device_prep_keys=None, padding_func=batch_pad_right, padding_kwargs=None, per_key_padding_kwargs=None, apply_default_convert=True, nonpadded_stack=True)
/usr/local/lib/python3.12/dist-packages/speechbrain/dataio/batch.pyCollate_fn when examples are dicts and have variable-length sequences.\n",
"\n",
"Different elements in the examples get matched by key.\n",
"All numpy tensors get converted to Torch (PyTorch default_convert)\n",
"Then, by default, all torch.Tensor valued elements get padded and support\n",
"collective pin_memory() and to() calls.\n",
"Regular Python data types are just collected in a list.\n",
"\n",
"Arguments\n",
"---------\n",
"examples : list\n",
" List of example dicts, as produced by Dataloader.\n",
"padded_keys : list, None\n",
" (Optional) List of keys to pad on. If None, pad all torch.Tensors\n",
"device_prep_keys : list, None\n",
" (Optional) Only these keys participate in collective memory pinning and moving with\n",
" to().\n",
" If None, defaults to all items with torch.Tensor values.\n",
"padding_func : callable, optional\n",
" Called with a list of tensors to be padded together. Needs to return\n",
" two tensors: the padded data, and another tensor for the data lengths.\n",
"padding_kwargs : dict, None\n",
" (Optional) Extra kwargs to pass to padding_func. E.G. mode, value\n",
" This is used as the default padding configuration for all keys.\n",
"per_key_padding_kwargs : dict, None\n",
" (Optional) Per-key padding configuration. Keys in this dict should match\n",
" the keys in the examples. Each value should be a dict with padding parameters\n",
" (e.g., {'value': -100, 'mode': 'constant'}). If a key is not in this dict,\n",
" the global padding_kwargs will be used.\n",
"apply_default_convert : bool\n",
" Whether to apply PyTorch default_convert (numpy to torch recursively,\n",
" etc.) on all data. Default:True, usually does the right thing.\n",
"nonpadded_stack : bool\n",
" Whether to apply PyTorch-default_collate-like stacking on values that\n",
" didn't get padded. This stacks if it can, but doesn't error out if it\n",
" cannot. Default:True, usually does the right thing.\n",
"\n",
"Example\n",
"-------\n",
">>> batch = PaddedBatch(\n",
"... [\n",
"... {"id": "ex1", "foo": torch.Tensor([1.0])},\n",
"... {"id": "ex2", "foo": torch.Tensor([2.0, 1.0])},\n",
"... ]\n",
"... )\n",
">>> # Attribute or key-based access:\n",
">>> batch.id\n",
"['ex1', 'ex2']\n",
">>> batch["id"]\n",
"['ex1', 'ex2']\n",
">>> # torch.Tensors get padded\n",
">>> type(batch.foo)\n",
"<class 'speechbrain.dataio.batch.PaddedData'>\n",
">>> batch.foo.data\n",
"tensor([[1., 0.],\n",
" [2., 1.]])\n",
">>> batch.foo.lengths\n",
"tensor([0.5000, 1.0000])\n",
">>> # Batch supports collective operations:\n",
">>> _ = batch.to(dtype=torch.half)\n",
">>> batch.foo.data\n",
"tensor([[1., 0.],\n",
" [2., 1.]], dtype=torch.float16)\n",
">>> batch.foo.lengths\n",
"tensor([0.5000, 1.0000], dtype=torch.float16)\n",
">>> # Numpy tensors get converted to torch and padded as well:\n",
">>> import numpy as np\n",
">>> batch = PaddedBatch(\n",
"... [{"wav": np.asarray([1, 2, 3, 4])}, {"wav": np.asarray([1, 2, 3])}]\n",
"... )\n",
">>> batch.wav # +ELLIPSIS\n",
"PaddedData(data=tensor([[1, 2,...\n",
">>> # Basic stacking collation deals with non padded data:\n",
">>> batch = PaddedBatch(\n",
"... [\n",
"... {\n",
"... "spk_id": torch.tensor([1]),\n",
"... "wav": torch.tensor([0.1, 0.0, 0.3]),\n",
"... },\n",
"... {\n",
"... "spk_id": torch.tensor([2]),\n",
"... "wav": torch.tensor([0.2, 0.3, -0.1]),\n",
"... },\n",
"... ],\n",
"... padded_keys=["wav"],\n",
"... )\n",
">>> batch.spk_id\n",
"tensor([[1],\n",
" [2]])\n",
">>> # And some data is left alone:\n",
">>> batch = PaddedBatch(\n",
"... [{"text": ["Hello"]}, {"text": ["How", "are", "you?"]}]\n",
"... )\n",
">>> batch.text\n",
"[['Hello'], ['How', 'are', 'you?']]\n",
">>> # Per-key padding configuration:\n",
">>> batch = PaddedBatch(\n",
"... [\n",
"... {\n",
"... "wav": torch.tensor([1, 2, 3]),\n",
"... "labels": torch.tensor([1, 2]),\n",
"... },\n",
"... {"wav": torch.tensor([4, 5]), "labels": torch.tensor([3])},\n",
"... ],\n",
"... per_key_padding_kwargs={\n",
"... "wav": {"value": 0},\n",
"... "labels": {"value": -100},\n",
"... },\n",
"... )\n",
">>> batch.wav.data\n",
"tensor([[1, 2, 3],\n",
" [4, 5, 0]])\n",
">>> batch.labels.data\n",
"tensor([[ 1, 2],\n",
" [ 3, -100]])\n",
" \n",
"
![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
](https://colab.research.google.com/github/speechbrain/speechbrain/blob/develop/docs/tutorials/basics/data-loading-pipeline.ipynb)\n",
"to execute or view/download this notebook on\n",
"[GitHub](https://github.com/speechbrain/speechbrain/tree/develop/docs/tutorials/basics/data-loading-pipeline.ipynb)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5ZZTjoafZw49"
},
"source": [
"# Data Loading\n",
"\n",
"Handling data consumes 90% of the active work time in many machine learning projects.\n",
"\n",
"SpeechBrain complements standard PyTorch data loading in handling variable-length sequences, large datasets, and complex data transformation pipelines. These are typical challenges when working with speech, but SpeechBrain tries not to make assumptions about your data."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EnirXev8XUyt"
},
"source": [
"## Install dependencies\n"
]
},
{
"cell_type": "code",
"execution_count": 71,
"metadata": {
"id": "5spDEaL1yddy"
},
"outputs": [],
"source": [
"%%capture\n",
"# Installing SpeechBrain via pip\n",
"BRANCH = 'speechllm_librispeech'\n",
"!python -m pip install git+https://github.com/speechbrain/speechbrain.git@$BRANCH"
]
},
{
"cell_type": "code",
"execution_count": 72,
"metadata": {
"id": "DPX-4BBbX5L9"
},
"outputs": [],
"source": [
"import speechbrain\n",
"import torch"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HL_oCiDyoP8Y"
},
"source": [
"In this tutorial we will use MiniLibriSpeech from https://www.openslr.org/resources/31: we download the validation set in the following two cells as well as images and scripts."
]
},
{
"cell_type": "code",
"execution_count": 73,
"metadata": {
"id": "vtSWx0KAX5L9"
},
"outputs": [],
"source": [
"%%capture\n",
"# downloading mini_librispeech dev data\n",
"!wget https://www.openslr.org/resources/31/dev-clean-2.tar.gz\n",
"!tar -xvzf dev-clean-2.tar.gz"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HBlTW2Mq5S35"
},
"source": [
"## Preface: PyTorch data loading pipeline\n",
"\n",
"SpeechBrain data-IO follows and extends [PyTorch data loading](https://pytorch.org/docs/stable/data.html). This preface section recaps PyTorch data loading and does not yet consider the SpeechBrain data loading extensions.\n",
"\n",
"### Overview\n",
"PyTorch data loading can run in many configurations,\n",
"but a typical approach has these basic elements:\n",
"- a [Dataset](https://pytorch.org/docs/stable/data.html#torch.utils.data.Dataset), which loads data points one-at-a-time.\n",
"- a [collation function](https://pytorch.org/docs/stable/data.html#dataloader-collate-fn), or ``collate_fn`` for short, which takes a list of data points and forms a batch.\n",
"- a [Sampler](https://pytorch.org/docs/stable/data.html#data-loading-order-and-sampler), which determines the order in which the Dataset is iterated.\n",
"- a [DataLoader](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader), which combines the elements above (and has defaults for ``collate_fn`` and Sampler), and orchestrates the whole pipeline.\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qHNJPq6PfInp"
},
"source": [
"### Dataset\n",
"The role of the Dataset is to produce single data points. Typically they are loaded off the disk, but they could also come from some more complex source or in some cases just from RAM. You can write your own Dataset subclass or sometimes you can use a standardized class. The training, validation, and test subsets get their own Dataset instances.\n",
"\n",
"The Dataset interface is simple; it implements\n",
"`__getitem__` and usually also `__len__`. Usually, \"map-style\" Datasets are used, but it's worth noting that PyTorch also has a notion of [IterableDataset](https://pytorch.org/docs/stable/data.html#iterable-style-datasets)s.\n",
"\n",
"`__getitem__` can return _anything_ because data can look like _anything_. Often, though, a data point consists of multiple associated things (e.g. an image and a label, or a speech waveform and its transcription). The Dataset should return all of these associated things.\n",
"\n",
"It is also relatively common for the Dataset to somehow transform the data on the fly, on the CPU.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "huVMufrKo9uf"
},
"source": [
"### Collation function\n",
"\n",
"The ``collate_fn`` just converts a list of examples into a PyTorch tensor batch. If the data has variable-length sequences, ``collate_fn`` usually needs to implement padding."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZmGerYfHpBxu"
},
"source": [
"### Sampler\n",
"Typically users don't need to create their own sampler; the two default options are to iterate in the original Dataset order or to iterate in random order.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fpuhtBUEpECO"
},
"source": [
"### DataLoader\n",
"\n",
"The DataLoader takes the other elements above and many other arguments such as batch size. The DataLoader has basic defaults for all arguments (except the Dataset, of course), but it's worthwhile to understand the [args](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader).\n",
"\n",
"The DataLoader object is iterated in the training loop, and every Dataset instance (e.g. training, validation, test) gets its own DataLoader.\n",
"```python\n",
"train_loader = DataLoader(train_data, collate_fn=PaddedBatch, batch_size=32, num_workers=2)\n",
"for batch in train_loader:\n",
" pred = model(batch.signal)\n",
" ...\n",
"```\n",
"\n",
"The iterator that DataLoader returns can either load batches in the same process that it is created in (`num_workers=0`), or it can start a new process (`num_workers=1`) or multiple new processes (`num_workers>1`). Because of the Global Interpreter Lock, Python cannot work on two tasks at the same time in a single process. Using at least one background worker process to load data while simultaneously running training is often essential for taking full advantage of GPU compute resources."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4rXqNQYb_Taw"
},
"source": [
"## SpeechBrain Basic dataIO\n",
"\n",
"The basic dataIO pipeline is organized around three \"key\" blocks: **DynamicItemDataset**, **Dynamic Items Pipelines (DIPs)** and **CategoricalEncoder** which are tightly connected.\n",
"\n",
"**DynamicItemDataset** inherits from `torch.utils.data.Dataset` and has been built to work together with **Dynamic Items Pipelines** to provide a straightforward and flexible way to fetch and transform data and labels from the raw dataset stored in your disk.\n",
"\n",
"**DIPs** consists of user-defined functions in which the user specifies operations applied to metadata and data contained in the dataset. E.g. reading and augmenting an audio file or encoding a sequence of words using SentencePiece tokenizer.\n",
"These functions are called inside **DynamicItemDataset** `__getitem__` method and are run in parallel on the CPU.\n",
"\n",
"The **CategoricalEncoder** is a convenient abstraction we provide for multi-class classification problems and it is sub-classed into **TextEncoder** and **CTCTextEncoder** which instead can be used for sequence-to-sequence applications related to text, such as ASR.\n",
"\n",
"\n",
"Thanks to these abstractions, most of the work necessary to set up the data IO pipeline is parsing the dataset into suitable annotation supported by **DynamicItemDataset** (SpeechBrain supports both CSV and JSON formats).\n",
"\n",
"\n",
"Once this annotation is ready, a flexible and efficient pipeline can be created in few lines of code, as SpeechBrain will take care under the hood of padding and other operations."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kOmWMuv_XDg4"
},
"source": [
"In the following tutorial, we will explain in detail how these blocks work.\n",
"We will start from the required CSV or JSON annotation whose purpose is to represent and describe the information contained in the dataset.\n",
"For example:\n",
"\n",
"* Paths to audio files, pre-extracted features et cetera.\n",
"* Metadata such as words spoken in the audio files, Signal-to-Noise-Ratio, sound event tags, speaker identities et cetera.\n",
"\n",
"Basically any information required to train your algorithm."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "T9VSd39Cmqxm"
},
"source": [
"### Dataset Annotation\n",
"\n",
"SpeechBrain offers native support for JSON and CSV formats for **describing a dataset** and, in fact, in official recipes (such as LibriSpeech ASR recipes) we provide parsing scripts to obtain such formats. "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZFaNums8qj1J"
},
"source": [
"We can take a glimpse to how these files can be structured using the downloaded Mini-LibriSpeech example.\n",
"\n",
"Each file in Mini-LibriSpeech is an utterance from a single speaker, thus either JSON and CSV formats can be used to contain the absolute path to that file, the speaker identity and the words the speaker utters. This is enough to build an Automatic Speech Recognition (ASR) system.\n",
"\n",
"Creating those files should be rather easy for most datasets as Python offers many tools for manipulating CSV and JSON files (such as [pandas](https://pandas.pydata.org/)). In fact parsing the data to JSON, for example, can be done in few lines of codes:"
]
},
{
"cell_type": "code",
"execution_count": 74,
"metadata": {
"id": "nzgmi6IyNWMb"
},
"outputs": [],
"source": [
"%%capture\n",
"!pip install soundfile"
]
},
{
"cell_type": "code",
"execution_count": 75,
"metadata": {
"id": "N-UPJi6v93w_",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "b03f60e3-62da-4c9a-a6c8-a98b30cfce73"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Number of flac audio files 1089\n",
"Number of transcriptions 1089\n"
]
}
],
"source": [
"import soundfile\n",
"from pathlib import Path\n",
"\n",
"# Load all the audio files into one list\n",
"dev_clean_root = Path(\"./LibriSpeech/dev-clean-2\")\n",
"flac_files = list(dev_clean_root.glob(\"**/*.flac\"))\n",
"print(\"Number of flac audio files {}\".format(len(flac_files)))\n",
"\n",
"# we build a dictionary to map utterance id to words for each utterance\n",
"# each row in the text file is simply:\n",
"#