speechbrain.utils.hpopt module

Utilities for hyperparameter optimization. This wrapper has an optional dependency on Oríon

https://orion.readthedocs.io/en/stable/ https://github.com/Epistimio/orion

Authors

Artem Ploujnikov 2021

Summary

Classes:

`GenericHyperparameterOptimizationReporter`	A generic hyperparameter fit reporter that outputs the result as JSON to an arbitrary data stream, which may be read as a third-party tool
`HyperparameterOptimizationContext`	A convenience context manager that makes it possible to conditionally enable hyperparameter optimization for a recipe.
`HyperparameterOptimizationReporter`	A base class for hyperparameter fit reporters
`OrionHyperparameterOptimizationReporter`	A result reporter implementation based on Orion

Functions:

`get_reporter`	Attempts to get the reporter specified by the mode and reverts to a generic one if it is not available
`get_trial_id`	Returns the ID of the current hyperparameter optimization trial, used primarily for the name of experiment folders.
`hpopt_mode`	A decorator to register a reporter implementation for a hyperparameter optimization mode
`hyperparameter_optimization`	Initializes the hyperparameter optimization context
`report_result`	Reports the result using the current reporter, if available.

Reference

speechbrain.utils.hpopt.hpopt_mode(mode)[source]

A decorator to register a reporter implementation for a hyperparameter optimization mode

Parameters:: mode (str) – the mode to register
Returns:: f – a callable function that registers and returns the reporter class
Return type:: callable

Example

>>> @hpopt_mode("raw")
... class RawHyperparameterOptimizationReporter(
...     HyperparameterOptimizationReporter
... ):
...     def __init__(self, *args, **kwargs):
...         super().__init__(*args, **kwargs)
...
...     def report_objective(self, result):
...         objective = result[self.objective_key]
...         print(f"Objective: {objective}")

>>> reporter = get_reporter("raw", objective_key="error")
>>> result = {"error": 1.2, "train_loss": 7.2}
>>> reporter.report_objective(result)
Objective: 1.2

class speechbrain.utils.hpopt.HyperparameterOptimizationReporter(objective_key)[source]

Bases: object

A base class for hyperparameter fit reporters

Parameters:: objective_key (str) – the key from the result dictionary to be used as the objective

report_objective(result)[source]

Reports the objective for hyperparameter optimization.

Parameters:: result (dict) – a dictionary with the run result.
Returns:: objective – A mapping from metric to score.
Return type:: dict

property is_available: Determines whether this reporter is available

property trial_id: The unique ID of this trial (used for folder naming)

class speechbrain.utils.hpopt.GenericHyperparameterOptimizationReporter(reference_date=None, output=None, *args, **kwargs)[source]

Bases: HyperparameterOptimizationReporter

A generic hyperparameter fit reporter that outputs the result as JSON to an arbitrary data stream, which may be read as a third-party tool

Parameters:

reference_date (datetime.datetime) – The date used to create trial id
output (stream) – The stream to report the results to
*args (tuple) – Arguments to be forwarded to parent class
**kwargs (dict) – Arguments to be forwarded to parent class

report_objective(result)[source]

Reports the objective for hyperparameter optimization.

Parameters:: result (dict) – a dictionary with the run result.

Example

>>> reporter = GenericHyperparameterOptimizationReporter(
...     objective_key="error"
... )
>>> result = {"error": 1.2, "train_loss": 7.2}
>>> reporter.report_objective(result)
{"error": 1.2, "train_loss": 7.2, "objective": 1.2}

property trial_id

The unique ID of this trial (used mainly for folder naming)

Example

>>> import datetime
>>> reporter = GenericHyperparameterOptimizationReporter(
...     objective_key="error",
...     reference_date=datetime.datetime(2021, 1, 3),
... )
>>> print(reporter.trial_id)
20210103000000000000

class speechbrain.utils.hpopt.OrionHyperparameterOptimizationReporter(objective_key)[source]

Bases: HyperparameterOptimizationReporter

A result reporter implementation based on Orion

Parameters:: objective_key (str) – the key from the result dictionary to be used as the objective

report_objective(result)[source]

Reports the objective for hyperparameter optimization.

Parameters:: result (dict) – a dictionary with the run result.

property trial_id: The unique ID of this trial (used mainly for folder naming)

property is_available: Determines if Orion is available. In order for it to be available, the library needs to be installed, and at least one of ORION_EXPERIMENT_NAME, ORION_EXPERIMENT_VERSION, ORION_TRIAL_ID needs to be set

speechbrain.utils.hpopt.get_reporter(mode, *args, **kwargs)[source]

Attempts to get the reporter specified by the mode and reverts to a generic one if it is not available

Parameters:

mode (str) – a string identifier for a registered hyperparameter optimization mode, corresponding to a specific reporter instance
*args (tuple) – Arguments to forward to the reporter class.
**kwargs (dict) – Arguments to forward to the reporter class.

Returns:

reporter – a reporter instance

Return type:

HyperparameterOptimizationReporter

Example

>>> reporter = get_reporter("generic", objective_key="error")
>>> result = {"error": 3.4, "train_loss": 1.2}
>>> reporter.report_objective(result)
{"error": 3.4, "train_loss": 1.2, "objective": 3.4}

class speechbrain.utils.hpopt.HyperparameterOptimizationContext(reporter_args=None, reporter_kwargs=None)[source]

Bases: object

A convenience context manager that makes it possible to conditionally enable hyperparameter optimization for a recipe.

Parameters:

reporter_args (list) – arguments to the reporter class
reporter_kwargs (dict) – keyword arguments to the reporter class

Example

>>> ctx = HyperparameterOptimizationContext(
...     reporter_args=[], reporter_kwargs={"objective_key": "error"}
... )

parse_arguments(arg_list, pass_hpopt_args=None, pass_trial_id=True)[source]

A version of speechbrain.parse_arguments enhanced for hyperparameter optimization.

If a parameter named ‘hpopt’ is provided, hyperparameter optimization and reporting will be enabled.

If the parameter value corresponds to a filename, it will be read as a hyperpyyaml file, and the contents will be added to “overrides”. This is useful for cases where the values of certain hyperparameters are different during hyperparameter optimization vs during full training (e.g. number of epochs, saving files, etc)

Parameters:

arg_list (list) – a list of arguments
pass_hpopt_args (enumerable) – forces arguments that are normally suppressed and only used for hyperparameter optimization to be passed into overrides
pass_trial_id (bool) – whether the “trial_id” argument is passed through (enabled by default)

Returns:

param_file (str) – The location of the parameters file.
run_opts (dict) – Run options, such as distributed, device, etc.
overrides (dict) – The overrides to pass to load_hyperpyyaml.

Example

>>> ctx = HyperparameterOptimizationContext()
>>> arg_list = ["hparams.yaml", "--x", "1", "--y", "2"]
>>> hparams_file, run_opts, overrides = ctx.parse_arguments(arg_list)
>>> print(f"File: {hparams_file}, Overrides: {overrides}")
File: hparams.yaml, Overrides: {'x': 1, 'y': 2}

speechbrain.utils.hpopt.hyperparameter_optimization(*args, **kwargs)[source]

Initializes the hyperparameter optimization context

Parameters:

*args (tuple) – Arguments to forward to HyperparameterOptimizationContext
**kwargs (dict) – Arguments to forward to HyperparameterOptimizationContext

Return type:

HyperparameterOptimizationContext

Example

>>> import sys
>>> with hyperparameter_optimization(
...     objective_key="error", output=sys.stdout
... ) as hp_ctx:
...     result = {"error": 3.5, "train_loss": 2.1}
...     report_result(result)
{"error": 3.5, "train_loss": 2.1, "objective": 3.5}

speechbrain.utils.hpopt.report_result(result)[source]

Reports the result using the current reporter, if available. When not in hyperparameter optimization mode, this function does nothing.

Parameters:: result (dict) – A dictionary of stats to be reported

Example

>>> result = {"error": 3.5, "train_loss": 2.1}
>>> report_result(result["error"])

speechbrain.utils.hpopt.get_trial_id()[source]

Returns the ID of the current hyperparameter optimization trial, used primarily for the name of experiment folders.

When using a context, the convention for identifying the trial ID will depend on the reporter being used. The default implementation returns a fixed value (“hpopt”)

Returns:: trial_id – the trial identifier
Return type:: str

Example

>>> trial_id = get_trial_id()
>>> trial_id
'hpopt'