{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "sb_auto_header",
"tags": [
"sb_auto_header"
]
},
"source": [
"\n",
"\n",
"\n",
"[![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\")
](https://colab.research.google.com/github/speechbrain/speechbrain/blob/develop/docs/tutorials/preprocessing/speech-features.ipynb)\n",
"to execute or view/download this notebook on\n",
"[GitHub](https://github.com/speechbrain/speechbrain/tree/develop/docs/tutorials/preprocessing/speech-features.ipynb)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7d8HzInf8-j2"
},
"source": [
"# Speech Features\n",
"\n",
"Speech is a very **high-dimensional** signal. For instance, when the sampling frequency is 16 kHz, we have 16000 samples for each second. Working with such very high dimensional data can be critical from a machine learning perspective. The goal of feature extraction is to find **more compact** ways to represent speech.\n",
"\n",
"The study of proper speech features was a very active field for research some years ago. With the advent of deep learning, however, the trend is to feed neural networks with **simple features**. We then leave the network itself discovering higher-level representations.\n",
"\n",
"In this tutorial, we will describe the two most popular speech features:\n",
"\n",
"* Filter Banks (FBANKs)\n",
"* Mel-Frequency Cepstral Coefficients (MFCCs)\n",
"\n",
"We will then mention common techniques to add context information.\n",
"\n",
"\n",
"## 1. Filter Banks (FBANKs)\n",
"FBANKs are time-frequency representations computed by applying a **set of filters** to the spectrogram of a speech signal. Please, [take a look at this tutorial](https://speechbrain.readthedocs.io/en/latest/tutorials/preprocessing/fourier-transform-and-spectrograms.html) for a detailed overview on Fourier transform and spectrograms.\n",
"\n",
"First of all, let's download some speech signals and install SpeechBrain:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PiZVuhfB7bN5"
},
"outputs": [],
"source": [
"%%capture\n",
"# Installing SpeechBrain via pip\n",
"BRANCH = 'develop'\n",
"!python -m pip install git+https://github.com/speechbrain/speechbrain.git@$BRANCH\n",
"\n",
"# Clone SpeechBrain repository\n",
"!git clone https://github.com/speechbrain/speechbrain/\n",
"%cd /content/speechbrain/"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "is5-1Vuq86b6"
},
"outputs": [],
"source": [
"%%capture\n",
"!wget https://www.dropbox.com/s/u8qyvuyie2op286/spk1_snt1.wav"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Cc_06vdTENJz"
},
"source": [
"Let's now compute the spectrogram of a speech signal:\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PopykIe5EUpf"
},
"outputs": [],
"source": [
"import torch\n",
"import matplotlib.pyplot as plt\n",
"from speechbrain.dataio.dataio import read_audio\n",
"from speechbrain.processing.features import STFT\n",
"\n",
"signal = read_audio('spk1_snt1.wav').unsqueeze(0) # [batch, time]\n",
"\n",
"compute_STFT = STFT(sample_rate=16000, win_length=25, hop_length=10, n_fft=400)\n",
"signal_STFT = compute_STFT(signal)\n",
"\n",
"spectrogram = signal_STFT.pow(2).sum(-1) # Power spectrogram\n",
"spectrogram = spectrogram.squeeze(0).transpose(0,1)\n",
"spectrogram = torch.log(spectrogram)\n",
"\n",
"plt.imshow(spectrogram.squeeze(0), cmap='hot', interpolation='nearest', origin='lower')\n",
"plt.xlabel('Time')\n",
"plt.ylabel('Frequency')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WSXyUVmdJNwg"
},
"source": [
"One way to compress the signal is to average the spectrogram over the frequency axis. This is done with a set of filters:"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "F5KHdgsEJqYL"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FzrBe84lJuv1"
},
"source": [
"From the spectrogram, we can notice that most of the energy is concentrated in the **lower part of the spectrum**. It is convenient to allocate more filters in the lower part of the spectrum and fewer filters for high-frequency components. This is what the mel filter banks are doing.\n",
"\n",
"Each filter is **triangular** and has a response of 1 at the center frequency. The response decreases linearly towards 0 till it reaches the center frequencies of the two adjacent filters (see figure). There is thus some **overlap** between adjacent filters.\n",
"\n",
"The filters are designed to be equally-spaced in the mel-frequency domain. It is possible to pass from the linear frequency domain to the mel one (and vice-versa) with the following non-linear transformations:\n",
"\n",
"$ m=2595log10(1+f/700)$\n",
"\n",
"$f=700(10m/2595−1)$,\n",
"\n",
"where $m$ is the mel frequency component and $f$ the standard frequency one (in Hz). The mel-frequency domain is compressed with a logarithm. The result is that filters equally spaced in the mel-domain will not be equally spaced in the target linear domain. We indeed have more filters in the lower part of the spectrum and less in the higher as desired.\n",
"\n",
"Let's now compute FBANKs using SpeechBrain:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "0xW3t0ywPsNa"
},
"outputs": [],
"source": [
"from speechbrain.processing.features import spectral_magnitude\n",
"from speechbrain.processing.features import Filterbank\n",
"\n",
"compute_fbanks = Filterbank(n_mels=40)\n",
"\n",
"STFT = compute_STFT(signal)\n",
"mag = spectral_magnitude(STFT)\n",
"fbanks = compute_fbanks(mag)\n",
"\n",
"print(STFT.shape)\n",
"print(mag.shape)\n",
"print(fbanks.shape)\n",
"\n",
"plt.imshow(fbanks.squeeze(0).t(), cmap='hot', interpolation='nearest', origin='lower')\n",
"plt.xlabel('Time')\n",
"plt.ylabel('Frequency')\n",
"plt.show()\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VRjJ9HaSRcuc"
},
"source": [
"Normally, 40 or 80 FBANKs are computed. As you can observe from the shapes, the dimensionality of the time axis is the same. The dimensionality of the frequency axis, instead, has been reduced. You can see FBANKs just as a simple way to **compress** the rich information embedded in the spectrogram.\n",
"\n",
"The SpeechBrain implementation of the filterbanks is designed to support different shapes of the filters (triangular, rectangular, gaussian). Moreover, when freeze=False, the filters are not frozen and can be tuned during training.\n",
"\n",
"To make it easier the computation of FBANKs, we created a lobe that performs all the needed steps in a single function:"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "S0cqrIjpSKOQ"
},
"source": [
"The SpeechBrain implementation of the filterbanks is designed to support different shapes for the filters (triangular, rectangular, gaussian). Morever, when `freeze=False`, the filters are not frozen and can be tuned during training."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Jb0yE3ZyUgNN"
},
"source": [
"To make it easier the computation of FBANKs, we created a lobe that performs all the needed steps in a single functions:\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "oSq0Sfc_UqDE"
},
"outputs": [],
"source": [
"from speechbrain.lobes.features import Fbank\n",
"fbank_maker = Fbank()\n",
"fbanks = fbank_maker(signal)\n",
"\n",
"plt.imshow(fbanks.squeeze(0).t(), cmap='hot', interpolation='nearest', origin='lower')\n",
"plt.xlabel('Time')\n",
"plt.ylabel('Frequency')\n",
"plt.show()\n",
"\n",
"# Zoom of first 80 steps\n",
"plt.imshow(fbanks.squeeze(0).t()[:,0:80], cmap='hot', interpolation='nearest', origin='lower')\n",
"plt.xlabel('Time')\n",
"plt.ylabel('Frequency')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "QIs_l3UfTES6"
},
"source": [
"## 2. Mel-Frequency Cepstral Coefficients (MFCCs)\n",
"MFCCs are computed by applying a Discrete Cosine Transform (DCT) on the top of the FBANKs. The DCT is a transformation that decorrelates the features and can be used to further compress them.\n",
"\n",
"To make the computation of MFCCs easier, we created a lobe for that:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2TgofojnW_mo"
},
"outputs": [],
"source": [
"from speechbrain.lobes.features import MFCC\n",
"mfcc_maker = MFCC(n_mfcc=13, deltas=False, context=False)\n",
"mfccs = mfcc_maker(signal)\n",
"\n",
"plt.imshow(mfccs.squeeze(0).t(), cmap='hot', interpolation='nearest', origin='lower')\n",
"plt.xlabel('Time')\n",
"plt.ylabel('Frequency')\n",
"plt.show()\n",
"\n",
"#Zoom of the first 25 steps\n",
"plt.imshow(mfccs.squeeze(0).t()[:,0:25], cmap='hot', interpolation='nearest', origin='lower')\n",
"plt.xlabel('Time')\n",
"plt.ylabel('Frequency')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1LAYg7YpXAzZ"
},
"source": [
"In the past, working on decorrelated features was essential. Past machine learning techniques such as Gaussian Mixture Model (GMMS) were not suitable to model correlated data. Deep Neural networks, instead, can be work very well also with **correlated data** and FBANKs are nowadays the preferred choice.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Eeqnb_c2fFLs"
},
"source": [
"## 3. Context Information\n",
"Proper management of local context is essential to most speech processing tasks. The dominant solution in the past was about setting a “hand-crafted” context with the following approaches:\n",
"\n",
"* Derivatives \n",
"* Context windows\n",
"\n",
"### 3.1 Derivatives\n",
"The idea behind derivatives is to introduce a local context by simply computing the **difference** with adjacent features. The derivatives are often computed with MFCCS coefficients:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "mqZKFjQ0hydd"
},
"outputs": [],
"source": [
"from speechbrain.lobes.features import MFCC\n",
"mfcc_maker = MFCC(n_mfcc=13, deltas=True, context=False)\n",
"mfccs_with_deltas = mfcc_maker(signal)\n",
"\n",
"print(mfccs.shape)\n",
"print(mfccs_with_deltas.shape)\n",
"\n",
"plt.imshow(mfccs_with_deltas.squeeze(0).t(), cmap='hot', interpolation='nearest', origin='lower')\n",
"plt.xlabel('Time')\n",
"plt.ylabel('Frequency')\n",
"plt.show()\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qs5h8Z4iiEeZ"
},
"source": [
"The first and second-order derivatives are called delta and delta-delta coefficients and are concatenated with the static coefficients. In the examples, the dimensionality is thus 39 (13 statics, 13 deltas, 13 delta-deltas)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "scbtvj-BjZUh"
},
"source": [
"### 3.2 Context Windows\n",
"Context windows adds a local context by simply **concatenating** multiple consecutive features. The result is a bigger feature vector that is better \"aware\" of the local information.\n",
"\n",
"Let's see an example:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "FIwV2I6ckQb1"
},
"outputs": [],
"source": [
"from speechbrain.lobes.features import MFCC\n",
"mfcc_maker = MFCC(n_mfcc=13,\n",
" deltas=True,\n",
" context=True,\n",
" left_frames=5,\n",
" right_frames=5)\n",
"mfccs_with_context = mfcc_maker(signal)\n",
"\n",
"print(mfccs.shape)\n",
"print(mfccs_with_deltas.shape)\n",
"print(mfccs_with_context.shape)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "iNfhmNtmk1Pr"
},
"source": [
"In this case, we concatenate the current frame with 5 past and 5 future frames. The total dimesionality is thus $39 * (5+5+1)= 429$"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mPoAwpV8lpmL"
},
"source": [
"Instead of using the aforementioned solutions, the current trend is to use static features and progressively add a **learnable context** through the **receptive field** on a **Convolutional Neural Network** (CNN). CNNs are often used in the early layers of the neural speech processing system to derive robust and context-aware representations."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TGKN8Vspm3Zn"
},
"source": [
"## 4. Other Features\n",
"A recent trend is to feed the neural network with the **raw data**. It is getting quite common to feed neural networks with the **spectrogram **directly or even with the **STFT**. It is also possible to feed the neural network with the **raw time-domain samples** directly. This is made easier with properly designed networks such as SincNEt. SincNEt uses a parametrized convolutional layers called SincConv that learns from the raw samples. SincNet is described in [this tutorial](add link)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "w2_iy_smtCsS"
},
"source": [
"## References\n",
"[1] P. Mermelstein (1976), \"Distance measures for speech recognition, psychological and instrumental,\" in Pattern Recognition and Artificial Intelligence. [pdf (Web Archive)](https://web.archive.org/web/20200714014004/http://www.haskins.yale.edu/sr/SR047/SR047_07.pdf)\n",
"\n",
"[2] X. Huang, A. Acero (Author), H.-W. Hon, \"Spoken Language Processing: A Guide to Theory, Algorithm and System Development Paperback – 2001\n",
"\n",
"[3] \n",
"\n",
"[4] M. Ravanelli, M. Omologo, \"Automatic context window composition for distant speech recognition\", Speech Communication, 2018 [ArXiv](https://arxiv.org/abs/1805.10498)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sb_auto_footer",
"tags": [
"sb_auto_footer"
]
},
"source": [
"## Citing SpeechBrain\n",
"\n",
"If you use SpeechBrain in your research or business, please cite it using the following BibTeX entry:\n",
"\n",
"```bibtex\n",
"@misc{speechbrainV1,\n",
" title={Open-Source Conversational AI with {SpeechBrain} 1.0},\n",
" author={Mirco Ravanelli and Titouan Parcollet and Adel Moumen and Sylvain de Langen and Cem Subakan and Peter Plantinga and Yingzhi Wang and Pooneh Mousavi and Luca Della Libera and Artem Ploujnikov and Francesco Paissan and Davide Borra and Salah Zaiem and Zeyu Zhao and Shucong Zhang and Georgios Karakasidis and Sung-Lin Yeh and Pierre Champion and Aku Rouhe and Rudolf Braun and Florian Mai and Juan Zuluaga-Gomez and Seyed Mahed Mousavi and Andreas Nautsch and Xuechen Liu and Sangeet Sagar and Jarod Duret and Salima Mdhaffar and Gaelle Laperriere and Mickael Rouvier and Renato De Mori and Yannick Esteve},\n",
" year={2024},\n",
" eprint={2407.00463},\n",
" archivePrefix={arXiv},\n",
" primaryClass={cs.LG},\n",
" url={https://arxiv.org/abs/2407.00463},\n",
"}\n",
"@misc{speechbrain,\n",
" title={{SpeechBrain}: A General-Purpose Speech Toolkit},\n",
" author={Mirco Ravanelli and Titouan Parcollet and Peter Plantinga and Aku Rouhe and Samuele Cornell and Loren Lugosch and Cem Subakan and Nauman Dawalatabad and Abdelwahab Heba and Jianyuan Zhong and Ju-Chieh Chou and Sung-Lin Yeh and Szu-Wei Fu and Chien-Feng Liao and Elena Rastorgueva and François Grondin and William Aris and Hwidong Na and Yan Gao and Renato De Mori and Yoshua Bengio},\n",
" year={2021},\n",
" eprint={2106.04624},\n",
" archivePrefix={arXiv},\n",
" primaryClass={eess.AS},\n",
" note={arXiv:2106.04624}\n",
"}\n",
"```"
]
}
],
"metadata": {
"colab": {
"provenance": [
{
"file_id": "1iH1Tvgrsi7nD_uxh5jEj7DAXaL4gdjze",
"timestamp": 1612452449642
}
]
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.6"
}
},
"nbformat": 4,
"nbformat_minor": 4
}