speechbrain.inference.interfaces module

Defines interfaces for simple inference with pretrained models

Authors:

  • Aku Rouhe 2021

  • Peter Plantinga 2021

  • Loren Lugosch 2020

  • Mirco Ravanelli 2020

  • Titouan Parcollet 2021

  • Abdel Heba 2021

  • Andreas Nautsch 2022, 2023

  • Pooneh Mousavi 2023

  • Sylvain de Langen 2023

  • Adel Moumen 2023

  • Pradnya Kandarkar 2023

Summary

Classes:

EncodeDecodePipelineMixin

A mixin for pretrained models that makes it possible to specify an encoding pipeline and a decoding pipeline

Pretrained

Takes a trained model and makes predictions on new data.

Functions:

foreign_class

Thin wrapper for pretrained_from_hparams() that fetches and loads a custom class.

pretrained_from_hparams

Fetch and load an interface from an outside source

Reference

speechbrain.inference.interfaces.foreign_class(source, hparams_file='hyperparams.yaml', pymodule_file='custom.py', classname='CustomInterface', savedir=None, local_strategy: LocalStrategy = LocalStrategy.SYMLINK, fetch_config: FetchConfig = FetchConfig(overwrite=False, allow_updates=False, allow_network=True, token=False, revision=None, huggingface_cache_dir=None), **kwargs)[source]

Thin wrapper for pretrained_from_hparams() that fetches and loads a custom class.

The pymodule file should contain a class with the given classname. An instance of that class is returned. The idea is to have a custom Pretrained subclass in the file. The pymodule file is also added to the python path before the Hyperparams YAML file is loaded, so it can contain any custom implementations that are needed.

Warning

Caution should be used with this function as it can download and run arbitrary code onto the machine this function is used on. Only use this function when the target module is from a highly trusted source!

Parameters:

  • source (str or Path or FetchSource) – The location to use for finding the model. See speechbrain.utils.fetching.fetch for details.

  • hparams_file (str) – The name of the hyperparameters file to use for constructing the modules necessary for inference. Must contain two keys: “modules” and “pretrainer”, as described in pretrained_from_hparams.

  • pymodule_file (str) – The name of the Python file containing the model’s python class. The file will be fetched from source and will be used to load the class code.

  • classname (str) – The name of the model’s Python class, which should be present in the code of the pymodule_file.

  • savedir (Optional[Union[str, Path]]) – Where to put the pretraining material. If not given, just use cache.

  • local_strategy (LocalStrategy, default LocalStrategy.SYMLINK) – Type of caching to use for keeping a local copy.

  • fetch_config (FetchConfig) – Configuration options for caching and other fetch behavior.

  • **kwargs – Arguments to pass to pretrained_from_hparams

Returns:

An instance of a class with the given classname from the given pymodule file.

Return type:

object

speechbrain.inference.interfaces.pretrained_from_hparams(cls, source, hparams_file='hyperparams.yaml', overrides={}, overrides_must_match=True, savedir=None, download_only=False, local_strategy: LocalStrategy = LocalStrategy.SYMLINK, fetch_config: FetchConfig = FetchConfig(overwrite=False, allow_updates=False, allow_network=True, token=False, revision=None, huggingface_cache_dir=None), **kwargs)[source]

Fetch and load an interface from an outside source

The source can be a location on the filesystem or online/huggingface

The hyperparams file should contain a “modules” key, which is a dictionary of torch modules used for computation.

The hyperparams file should contain a “pretrainer” key, which is a speechbrain.utils.parameter_transfer.Pretrainer

Warning

Caution should be used with this function as it can download and run arbitrary code onto the machine this function is used on. Only use this function when the target hparams file is from a highly trusted source!

Parameters:

  • cls (Type[Pretrained]) – The class to construct an instance of, usually a sub-type of Pretrained

  • source (str or Path or FetchSource) – The location to use for finding the model. See speechbrain.utils.fetching.fetch for details.

  • hparams_file (str) – The name of the hyperparameters file to use for constructing the modules necessary for inference. Must contain two keys: “modules” and “pretrainer”, as described.

  • overrides (dict) – Any changes to make to the hparams file when it is loaded.

  • overrides_must_match (bool) – Whether an error will be thrown when an override does not match a corresponding key in the yaml_stream.

  • savedir (str or Path) – Where to put the pretraining material. If not given, just use cache.

  • download_only (bool (default: False)) – If true, class and instance creation is skipped.

  • local_strategy (LocalStrategy, default LocalStrategy.SYMLINK) – Type of caching to use for keeping a local copy.

  • fetch_config (FetchConfig) – Configuration options for caching and other fetch behavior.

  • **kwargs (dict) – Arguments to forward to class constructor.

Returns:

object – An instance of a Pretrained class, constructed from the hparams. None is returned if the argument download_only is True.

Return type:

Optional[Pretrained]

class speechbrain.inference.interfaces.Pretrained(modules=None, hparams=None, run_opts=None, freeze_params=True)[source]

Bases: Module

Takes a trained model and makes predictions on new data.

This is a base class which handles some common boilerplate. It intentionally has an interface similar to Brain - these base classes handle similar things.

Subclasses of Pretrained should implement the actual logic of how the pretrained system runs, and add methods with descriptive names (e.g. transcribe_file() for ASR).

Pretrained is a torch.nn.Module so that methods like .to() or .eval() can work. Subclasses should provide a suitable forward() implementation: by convention, it should be a method that takes a batch of audio signals and runs the full model (as applicable).

Parameters:

  • modules (dict of str:torch.nn.Module pairs) – The Torch modules that make up the learned system. These can be treated in special ways (put on the right device, frozen, etc.). These are available as attributes under self.mods, like self.mods.model(x)

  • hparams (dict) – Each key:value pair should consist of a string key and a hyperparameter that is used within the overridden methods. These will be accessible via an hparams attribute, using “dot” notation: e.g., self.hparams.model(x).

  • run_opts (Optional[Union[RunOptions, dict]]) – A set of options to change the runtime environment, see RunOptions for a complete list. Some options are meant for training, and will not apply for this instance intended for inference.

  • freeze_params (bool) – To freeze (requires_grad=False) parameters or not. Normally in inference you want to freeze the params. Also calls .eval() on all modules.

HPARAMS_NEEDED = []
MODULES_NEEDED = []
load_audio(path, savedir=None)[source]

Load an audio file with this model’s input spec

When using a speech model, it is important to use the same type of data, as was used to train the model. This means for example using the same sampling rate and number of channels. It is, however, possible to convert a file from a higher sampling rate to a lower one (downsampling). Similarly, it is simple to downmix a stereo file to mono. The path can be a local path, a web url, or a link to a huggingface repo.

classmethod from_hparams(source, hparams_file='hyperparams.yaml', **kwargs)[source]

Fetch and load based from outside source based on HyperPyYAML file

The source can be a location on the filesystem or online/huggingface

The hyperparams file should contain a “modules” key, which is a dictionary of torch modules used for computation.

The hyperparams file should contain a “pretrainer” key, which is a speechbrain.utils.parameter_transfer.Pretrainer

Warning

Caution should be used with this function as it can download and run arbitrary code onto the machine this function is used on. Only use this function when the target hparams file is from a highly trusted source!

Parameters:

  • source (str) – The location to use for finding the model. See speechbrain.utils.fetching.fetch for details.

  • hparams_file (str) – The name of the hyperparameters file to use for constructing the modules necessary for inference. Must contain two keys: “modules” and “pretrainer”, as described.

  • **kwargs (dict) – Arguments to forward to pretrained_from_hparams.

Return type:

Instance of cls

class speechbrain.inference.interfaces.EncodeDecodePipelineMixin[source]

Bases: object

A mixin for pretrained models that makes it possible to specify an encoding pipeline and a decoding pipeline

create_pipelines()[source]

Initializes the encode and decode pipeline

to_dict(data)[source]

Converts padded batches to dictionaries, leaves other data types as is

Parameters:

data (object) – a dictionary or a padded batch

Returns:

results – the dictionary

Return type:

dict

property batch_inputs

Determines whether the input pipeline operates on batches or individual examples (true means batched)

Returns:

batch_inputs

Return type:

bool

property input_use_padded_data

If turned on, raw PaddedData instances will be passed to the model. If turned off, only .data will be used

Returns:

result – whether padded data is used as is

Return type:

bool

property batch_outputs

Determines whether the output pipeline operates on batches or individual examples (true means batched)

Returns:

batch_outputs

Return type:

bool

encode_input(input)[source]

Encodes the inputs using the pipeline

Parameters:

input (dict) – the raw inputs

Returns:

results

Return type:

object

decode_output(output)[source]

Decodes the raw model outputs

Parameters:

output (tuple) – raw model outputs

Returns:

result – the output of the pipeline

Return type:

dict or list