speechbrain.processing.features moduleο
Low-level feature pipeline components
This library gathers functions that compute popular speech features over batches of data. All the classes are of type nn.Module. This gives the possibility to have end-to-end differentiability and to backpropagate the gradient through them. Our functions are a modified version the ones in torch audio toolkit (https://github.com/pytorch/audio).
Example
>>> import torch
>>> from speechbrain.dataio.dataio import read_audio
>>> signal =read_audio('tests/samples/single-mic/example1.wav')
>>> signal = signal.unsqueeze(0)
>>> compute_STFT = STFT(
... sample_rate=16000, win_length=25, hop_length=10, n_fft=400
... )
>>> features = compute_STFT(signal)
>>> features = spectral_magnitude(features)
>>> compute_fbanks = Filterbank(n_mels=40)
>>> features = compute_fbanks(features)
>>> compute_mfccs = DCT(input_size=40, n_out=20)
>>> features = compute_mfccs(features)
>>> compute_deltas = Deltas(input_size=20)
>>> delta1 = compute_deltas(features)
>>> delta2 = compute_deltas(delta1)
>>> features = torch.cat([features, delta1, delta2], dim=2)
>>> compute_cw = ContextWindow(left_frames=5, right_frames=5)
>>> features = compute_cw(features)
>>> norm = InputNormalization()
>>> features = norm(features, torch.tensor([1]).float())
- Authors
Mirco Ravanelli 2020
Summaryο
Classes:
Computes the context window. |
|
Computes the discrete cosine transform. |
|
Computes delta coefficients (time derivatives). |
|
Dynamic range compression for audio signals - clipped log scale with an optional multiplier |
|
computes filter bank (FBANK) features given spectral magnitudes. |
|
A global normalization module - computes a single mean and standard deviation for the entire batch across unmasked positions and uses it to normalize the inputs to the desired mean and standard deviation. |
|
Computes the Inverse Short-Term Fourier Transform (ISTFT) |
|
Performs mean and variance normalization of the input tensor. |
|
A commonly used normalization for the decibel scale |
|
computes the Short-Term Fourier Transform (STFT). |
Functions:
Returns the magnitude of a complex spectrogram. |
Referenceο
- class speechbrain.processing.features.STFT(sample_rate, win_length=25, hop_length=10, n_fft=400, window_fn=<built-in method hamming_window of type object>, normalized_stft=False, center=True, pad_mode='constant', onesided=True)[source]ο
Bases:
Module
computes the Short-Term Fourier Transform (STFT).
This class computes the Short-Term Fourier Transform of an audio signal. It supports multi-channel audio inputs (batch, time, channels).
- Parameters:
sample_rate (int) β Sample rate of the input audio signal (e.g 16000).
win_length (float) β Length (in ms) of the sliding window used to compute the STFT.
hop_length (float) β Length (in ms) of the hope of the sliding window used to compute the STFT.
n_fft (int) β Number of fft point of the STFT. It defines the frequency resolution (n_fft should be <= than win_len).
window_fn (function) β A function that takes an integer (number of samples) and outputs a tensor to be multiplied with each window before fft.
normalized_stft (bool) β If True, the function returns the normalized STFT results, i.e., multiplied by win_length^-0.5 (default is False).
center (bool) β If True (default), the input will be padded on both sides so that the t-th frame is centered at time tΓhop_length. Otherwise, the t-th frame begins at time tΓhop_length.
pad_mode (str) β It can be βconstantβ,βreflectβ,βreplicateβ, βcircularβ, βreflectβ (default). βconstantβ pads the input tensor boundaries with a constant value. βreflectβ pads the input tensor using the reflection of the input boundary. βreplicateβ pads the input tensor using replication of the input boundary. βcircularβ pads using circular replication.
onesided (True) β If True (default) only returns nfft/2 values. Note that the other samples are redundant due to the Fourier transform conjugate symmetry.
Example
>>> import torch >>> compute_STFT = STFT( ... sample_rate=16000, win_length=25, hop_length=10, n_fft=400 ... ) >>> inputs = torch.randn([10, 16000]) >>> features = compute_STFT(inputs) >>> features.shape torch.Size([10, 101, 201, 2])
- forward(x)[source]ο
Returns the STFT generated from the input waveforms.
- Parameters:
x (torch.Tensor) β A batch of audio signals to transform.
- Returns:
stft
- Return type:
torch.Tensor
- get_filter_properties() FilterProperties [source]ο
- class speechbrain.processing.features.ISTFT(sample_rate, n_fft=None, win_length=25, hop_length=10, window_fn=<built-in method hamming_window of type object>, normalized_stft=False, center=True, onesided=True, epsilon=1e-12)[source]ο
Bases:
Module
Computes the Inverse Short-Term Fourier Transform (ISTFT)
This class computes the Inverse Short-Term Fourier Transform of an audio signal. It supports multi-channel audio inputs (batch, time_step, n_fft, 2, n_channels [optional]).
- Parameters:
sample_rate (int) β Sample rate of the input audio signal (e.g. 16000).
n_fft (int) β Number of points in FFT.
win_length (float) β Length (in ms) of the sliding window used when computing the STFT.
hop_length (float) β Length (in ms) of the hope of the sliding window used when computing the STFT.
window_fn (function) β A function that takes an integer (number of samples) and outputs a tensor to be used as a window for ifft.
normalized_stft (bool) β If True, the function assumes that itβs working with the normalized STFT results. (default is False)
center (bool) β If True (default), the function assumes that the STFT result was padded on both sides.
onesided (True) β If True (default), the function assumes that there are n_fft/2 values for each time frame of the STFT.
epsilon (float) β A small value to avoid division by 0 when normalizing by the sum of the squared window. Playing with it can fix some abnormalities at the beginning and at the end of the reconstructed signal. The default value of epsilon is 1e-12.
Example
>>> import torch >>> compute_STFT = STFT( ... sample_rate=16000, win_length=25, hop_length=10, n_fft=400 ... ) >>> compute_ISTFT = ISTFT( ... sample_rate=16000, win_length=25, hop_length=10 ... ) >>> inputs = torch.randn([10, 16000]) >>> outputs = compute_ISTFT(compute_STFT(inputs)) >>> outputs.shape torch.Size([10, 16000])
- forward(x, sig_length=None)[source]ο
Returns the ISTFT generated from the input signal.
- Parameters:
x (torch.Tensor) β A batch of audio signals in the frequency domain to transform.
sig_length (int) β The length of the output signal in number of samples. If not specified will be equal to: (time_step - 1) * hop_length + n_fft
- Returns:
istft
- Return type:
torch.Tensor
- speechbrain.processing.features.spectral_magnitude(stft, power: int = 1, log: bool = False, eps: float = 1e-14)[source]ο
Returns the magnitude of a complex spectrogram.
- Parameters:
stft (torch.Tensor) β A tensor, output from the stft function.
power (int) β What power to use in computing the magnitude. Use power=1 for the power spectrogram. Use power=0.5 for the magnitude spectrogram.
log (bool) β Whether to apply log to the spectral features.
eps (float) β A small value to prevent square root of zero.
- Returns:
spectr
- Return type:
torch.Tensor
Example
>>> a = torch.Tensor([[3, 4]]) >>> spectral_magnitude(a, power=0.5) tensor([5.])
- class speechbrain.processing.features.Filterbank(n_mels=40, log_mel=True, filter_shape='triangular', f_min=0, f_max=8000, n_fft=400, sample_rate=16000, power_spectrogram=2, amin=1e-10, ref_value=1.0, top_db=80.0, param_change_factor=1.0, param_rand_factor=0.0, freeze=True)[source]ο
Bases:
Module
computes filter bank (FBANK) features given spectral magnitudes.
- Parameters:
n_mels (float) β Number of Mel filters used to average the spectrogram.
log_mel (bool) β If True, it computes the log of the FBANKs.
filter_shape (str) β Shape of the filters (βtriangularβ, βrectangularβ, βgaussianβ).
f_min (int) β Lowest frequency for the Mel filters.
f_max (int) β Highest frequency for the Mel filters.
n_fft (int) β Number of fft points of the STFT. It defines the frequency resolution (n_fft should be<= than win_len).
sample_rate (int) β Sample rate of the input audio signal (e.g, 16000)
power_spectrogram (float) β Exponent used for spectrogram computation.
amin (float) β Minimum amplitude (used for numerical stability).
ref_value (float) β Reference value used for the dB scale.
top_db (float) β Minimum negative cut-off in decibels.
param_change_factor (bool) β If freeze=False, this parameter affects the speed at which the filter parameters (i.e., central_freqs and bands) can be changed. When high (e.g., param_change_factor=1) the filters change a lot during training. When low (e.g. param_change_factor=0.1) the filter parameters are more stable during training
param_rand_factor (float) β This parameter can be used to randomly change the filter parameters (i.e, central frequencies and bands) during training. It is thus a sort of regularization. param_rand_factor=0 does not affect, while param_rand_factor=0.15 allows random variations within +-15% of the standard values of the filter parameters (e.g., if the central freq is 100 Hz, we can randomly change it from 85 Hz to 115 Hz).
freeze (bool) β If False, it the central frequency and the band of each filter are added into nn.parameters. If True, the standard frozen features are computed.
Example
>>> import torch >>> compute_fbanks = Filterbank() >>> inputs = torch.randn([10, 101, 201]) >>> features = compute_fbanks(inputs) >>> features.shape torch.Size([10, 101, 40])
- class speechbrain.processing.features.DCT(input_size, n_out=20, ortho_norm=True)[source]ο
Bases:
Module
Computes the discrete cosine transform.
This class is primarily used to compute MFCC features of an audio signal given a set of FBANK features as input.
- Parameters:
Example
>>> import torch >>> inputs = torch.randn([10, 101, 40]) >>> compute_mfccs = DCT(input_size=inputs.size(-1)) >>> features = compute_mfccs(inputs) >>> features.shape torch.Size([10, 101, 20])
- class speechbrain.processing.features.Deltas(input_size, window_length=5)[source]ο
Bases:
Module
Computes delta coefficients (time derivatives).
- Parameters:
Example
>>> inputs = torch.randn([10, 101, 20]) >>> compute_deltas = Deltas(input_size=inputs.size(-1)) >>> features = compute_deltas(inputs) >>> features.shape torch.Size([10, 101, 20])
- class speechbrain.processing.features.ContextWindow(left_frames=0, right_frames=0)[source]ο
Bases:
Module
Computes the context window.
This class applies a context window by gathering multiple time steps in a single feature vector. The operation is performed with a convolutional layer based on a fixed kernel designed for that.
- Parameters:
Example
>>> import torch >>> compute_cw = ContextWindow(left_frames=5, right_frames=5) >>> inputs = torch.randn([10, 101, 20]) >>> features = compute_cw(inputs) >>> features.shape torch.Size([10, 101, 220])
- class speechbrain.processing.features.InputNormalization(mean_norm=True, std_norm=True, norm_type='global', avg_factor=None, requires_grad=False, update_until_epoch=3)[source]ο
Bases:
Module
Performs mean and variance normalization of the input tensor.
- Parameters:
mean_norm (True) β If True, the mean will be normalized.
std_norm (True) β If True, the standard deviation will be normalized.
norm_type (str) β It defines how the statistics are computed (βsentenceβ computes them at sentence level, βbatchβ at batch level, βspeakerβ at speaker level, while global computes a single normalization vector for all the sentences in the dataset). Speaker and global statistics are computed with a moving average approach.
avg_factor (float) β It can be used to manually set the weighting factor between current statistics and accumulated ones.
requires_grad (bool) β Whether this module should be updated using the gradient during training.
update_until_epoch (int) β The epoch after which updates to the norm stats should stop.
Example
>>> import torch >>> norm = InputNormalization() >>> inputs = torch.randn([10, 101, 20]) >>> inp_len = torch.ones([10]) >>> features = norm(inputs, inp_len)
- forward(x, lengths, spk_ids=tensor([]), epoch=0)[source]ο
Returns the tensor with the surrounding context.
- Parameters:
x (torch.Tensor) β A batch of tensors.
lengths (torch.Tensor) β A batch of tensors containing the relative length of each sentence (e.g, [0.7, 0.9, 1.0]). It is used to avoid computing stats on zero-padded steps.
spk_ids (torch.Tensor containing the ids of each speaker (e.g, [0 10 6]).) β It is used to perform per-speaker normalization when norm_type=βspeakerβ.
epoch (int) β The epoch count.
- Returns:
x β The normalized tensor.
- Return type:
torch.Tensor
- class speechbrain.processing.features.GlobalNorm(norm_mean=0.0, norm_std=1.0, update_steps=None, length_dim=2, mask_value=0.0)[source]ο
Bases:
Module
A global normalization module - computes a single mean and standard deviation for the entire batch across unmasked positions and uses it to normalize the inputs to the desired mean and standard deviation.
This normalization is reversible - it is possible to use the .denormalize() method to recover the original values.
- Parameters:
norm_mean (float) β the desired normalized mean
norm_std (float) β the desired normalized standard deviation
update_steps (float) β the number of steps over which statistics will be collected
length_dim (int) β the dimension used to represent the length
mask_value (float) β the value with which to fill masked positions without a mask_value, the masked positions would be normalized, which might not be desired
Example
>>> import torch >>> from speechbrain.processing.features import GlobalNorm >>> global_norm = GlobalNorm( ... norm_mean=0.5, ... norm_std=0.2, ... update_steps=3, ... length_dim=1 ... ) >>> x = torch.tensor([[1., 2., 3.]]) >>> x_norm = global_norm(x) >>> x_norm tensor([[0.3000, 0.5000, 0.7000]]) >>> x = torch.tensor([[5., 10., -4.]]) >>> x_norm = global_norm(x) >>> x_norm tensor([[0.6071, 0.8541, 0.1623]]) >>> x_denorm = global_norm.denormalize(x_norm) >>> x_denorm tensor([[ 5.0000, 10.0000, -4.0000]]) >>> x = torch.tensor([[100., -100., -50.]]) >>> global_norm.freeze() >>> global_norm(x) tensor([[ 5.3016, -4.5816, -2.1108]]) >>> global_norm.denormalize(x_norm) tensor([[ 5.0000, 10.0000, -4.0000]]) >>> global_norm.unfreeze() >>> global_norm(x) tensor([[ 5.3016, -4.5816, -2.1108]]) >>> global_norm.denormalize(x_norm) tensor([[ 5.0000, 10.0000, -4.0000]])
- forward(x, lengths=None, mask_value=None, skip_update=False)[source]ο
Normalizes the tensor provided
- Parameters:
x (torch.Tensor) β the tensor to normalize
lengths (torch.Tensor) β a tensor of relative lengths (padding will not count towards normalization)
mask_value (float) β the value to use for masked positions
skip_update (false) β whether to skip updates to the norm
- Returns:
result β the normalized tensor
- Return type:
torch.Tensor
- normalize(x)[source]ο
Performs the normalization operation against the running mean and standard deviation
- Parameters:
x (torch.Tensor) β the tensor to normalize
- Returns:
result β the normalized tensor
- Return type:
torch.Tensor
- get_mask(x, lengths)[source]ο
Returns the length mask for the specified tensor
- Parameters:
x (torch.Tensor) β the tensor for which the mask will be obtained
lengths (torch.Tensor) β the length tensor
- Returns:
mask β the mask tensor
- Return type:
torch.Tensor
- class speechbrain.processing.features.MinLevelNorm(min_level_db)[source]ο
Bases:
Module
A commonly used normalization for the decibel scale
The scheme is as follows
x_norm = (x - min_level_db)/-min_level_db * 2 - 1
The rationale behind the scheme is as follows:
The top of the scale is assumed to be 0db. x_rel = (x - min) / (max - min) gives the relative position on the scale between the minimum and the maximum where the minimum is 0. and the maximum is 1.
The subsequent rescaling (x_rel * 2 - 1) puts it on a scale from -1. to 1. with the middle of the range centered at zero.
- Parameters:
min_level_db (float) β the minimum level
Example
>>> norm = MinLevelNorm(min_level_db=-100.) >>> x = torch.tensor([-50., -20., -80.]) >>> x_norm = norm(x) >>> x_norm tensor([ 0.0000, 0.6000, -0.6000])
- class speechbrain.processing.features.DynamicRangeCompression(multiplier=1, clip_val=1e-05)[source]ο
Bases:
Module
Dynamic range compression for audio signals - clipped log scale with an optional multiplier
- Parameters:
Example
>>> drc = DynamicRangeCompression() >>> x = torch.tensor([10., 20., 0., 30.]) >>> drc(x) tensor([ 2.3026, 2.9957, -11.5129, 3.4012]) >>> drc = DynamicRangeCompression(2.) >>> x = torch.tensor([10., 20., 0., 30.]) >>> drc(x) tensor([ 2.9957, 3.6889, -10.8198, 4.0943])