speechbrain.utils.hpopt module
Utilities for hyperparameter optimization. This wrapper has an optional dependency on Oríon
https://orion.readthedocs.io/en/stable/ https://github.com/Epistimio/orion
- Authors
Artem Ploujnikov 2021
Summary
Classes:
A generic hyperparameter fit reporter that outputs the result as JSON to an arbitrary data stream, which may be read as a third-party tool |
|
A convenience context manager that makes it possible to conditionally enable hyperparameter optimization for a recipe. |
|
A base class for hyperparameter fit reporters |
|
A result reporter implementation based on Orion |
Functions:
Attempts to get the reporter specified by the mode and reverts to a generic one if it is not available |
|
Returns the ID of the current hyperparameter optimization trial, used primarily for the name of experiment folders. |
|
A decorator to register a reporter implementation for a hyperparameter optimization mode |
|
Initializes the hyperparameter optimization context |
|
Reports the result using the current reporter, if available. |
Reference
- speechbrain.utils.hpopt.hpopt_mode(mode)[source]
A decorator to register a reporter implementation for a hyperparameter optimization mode
- Parameters
mode (str) – the mode to register
- Returns
f – a callable function that registers and returns the reporter class
- Return type
callable
Example
>>> @hpopt_mode("raw") ... class RawHyperparameterOptimizationReporter(HyperparameterOptimizationReporter): ... def __init__(self, *args, **kwargs): ... super().__init__( *args, **kwargs) ... def report_objective(self, result): ... objective = result[self.objective_key] ... print(f"Objective: {objective}")
>>> reporter = get_reporter("raw", objective_key="error") >>> result = {"error": 1.2, "train_loss": 7.2} >>> reporter.report_objective(result) Objective: 1.2
- class speechbrain.utils.hpopt.HyperparameterOptimizationReporter(objective_key)[source]
Bases:
object
A base class for hyperparameter fit reporters
- Parameters
objective_key (str) – the key from the result dictionary to be used as the objective
- report_objective(result)[source]
Reports the objective for hyperparameter optimization.
- Parameters
result (dict) – a dictionary with the run result.
- property is_available
Determines whether this reporter is available
- property trial_id
The unique ID of this trial (used for folder naming)
- class speechbrain.utils.hpopt.GenericHyperparameterOptimizationReporter(reference_date=None, output=None, *args, **kwargs)[source]
Bases:
HyperparameterOptimizationReporter
A generic hyperparameter fit reporter that outputs the result as JSON to an arbitrary data stream, which may be read as a third-party tool
- Parameters
objective_key (str) – the key from the result dictionary to be used as the objective
- report_objective(result)[source]
Reports the objective for hyperparameter optimization.
- Parameters
result (dict) – a dictionary with the run result.
Example
>>> reporter = GenericHyperparameterOptimizationReporter( ... objective_key="error" ... ) >>> result = {"error": 1.2, "train_loss": 7.2} >>> reporter.report_objective(result) {"error": 1.2, "train_loss": 7.2, "objective": 1.2}
- property trial_id
The unique ID of this trial (used mainly for folder naming)
Example
>>> import datetime >>> reporter = GenericHyperparameterOptimizationReporter( ... objective_key="error", ... reference_date=datetime.datetime(2021, 1, 3) ... ) >>> print(reporter.trial_id) 20210103000000000000
- class speechbrain.utils.hpopt.OrionHyperparameterOptimizationReporter(objective_key)[source]
Bases:
HyperparameterOptimizationReporter
A result reporter implementation based on Orion
- Parameters
orion_client (module) – the Python module for Orion
- report_objective(result)[source]
Reports the objective for hyperparameter optimization.
- Parameters
result (dict) – a dictionary with the run result.
- property trial_id
The unique ID of this trial (used mainly for folder naming)
- property is_available
Determines if Orion is available. In order for it to be available, the library needs to be installed, and at least one of ORION_EXPERIMENT_NAME, ORION_EXPERIMENT_VERSION, ORION_TRIAL_ID needs to be set
- speechbrain.utils.hpopt.get_reporter(mode, *args, **kwargs)[source]
Attempts to get the reporter specified by the mode and reverts to a generic one if it is not available
- Parameters
mode (str) – a string identifier for a registered hyperparametr optimization mode, corresponding to a specific reporter instance
- Returns
reporter – a reporter instance
- Return type
Example
>>> reporter = get_reporter("generic", objective_key="error") >>> result = {"error": 3.4, "train_loss": 1.2} >>> reporter.report_objective(result) {"error": 3.4, "train_loss": 1.2, "objective": 3.4}
- class speechbrain.utils.hpopt.HyperparameterOptimizationContext(reporter_args=None, reporter_kwargs=None)[source]
Bases:
object
A convenience context manager that makes it possible to conditionally enable hyperparameter optimization for a recipe.
- Parameters
Example
>>> ctx = HyperparameterOptimizationContext( ... reporter_args=[], ... reporter_kwargs={"objective_key": "error"} ... )
- parse_arguments(arg_list)[source]
A version of speechbrain.parse_arguments enhanced for hyperparameter optimization.
If a parameter named ‘hpopt’ is provided, hyperparameter optimization and reporting will be enabled.
If the parameter value corresponds to a filename, it will be read as a hyperpyaml file, and the contents will be added to “overrides”. This is useful for cases where the values of certain hyperparameters are different during hyperparameter optimization vs during full training (e.g. number of epochs, saving files, etc)
- Parameters
arg_list (a list of arguments) –
- Returns
param_file (str) – The location of the parameters file.
run_opts (dict) – Run options, such as distributed, device, etc.
overrides (dict) – The overrides to pass to
load_hyperpyyaml
.
Example
>>> ctx = HyperparameterOptimizationContext() >>> arg_list = ["hparams.yaml", "--x", "1", "--y", "2"] >>> hparams_file, run_opts, overrides = ctx.parse_arguments(arg_list) >>> print(f"File: {hparams_file}, Overrides: {overrides}") File: hparams.yaml, Overrides: {'x': 1, 'y': 2}
- speechbrain.utils.hpopt.hyperparameter_optimization(*args, **kwargs)[source]
Initializes the hyperparameter optimization context
Example
>>> import sys >>> with hyperparameter_optimization(objective_key="error", output=sys.stdout) as hp_ctx: ... result = {"error": 3.5, "train_loss": 2.1} ... report_result(result) ... {"error": 3.5, "train_loss": 2.1, "objective": 3.5}
- speechbrain.utils.hpopt.report_result(result)[source]
Reports the result using the current reporter, if available. When not in hyperparameter optimization mode, this function does nothing.
- Parameters
result (dict) – A dictionary of stats to be reported
Example
>>> result = {"error": 3.5, "train_loss": 2.1} >>> report_result(result["error"])
- speechbrain.utils.hpopt.get_trial_id()[source]
Returns the ID of the current hyperparameter optimization trial, used primarily for the name of experiment folders.
When using a context, the convention for identifying the trial ID will depend on the reporter being used. The default implementation returns a fixed value (“hpopt”)
- Returns
trial_id – the trial identifier
- Return type
Example
>>> trial_id = get_trial_id() >>> trial_id 'hpopt'