speechbrain.utils.recipe_tests module

Library for running recipe tests.

Authors
  • Mirco Ravanelli 2022

  • Andreas Nautsch 2022

Summary

Functions:

check_files

Checks if the output folder created by the test has the expected files.

check_performance

Checks if the performance achieved by the recipe matches with the expectations.

check_row_for_test

Checks if the current row of the csv recipe file has a test to run.

check_threshold

Checks if the value satisfied the threshold constraints.

load_yaml_test

Tests if the yaml files can be loaded without errors.

prepare_test

Extracts all the needed information to run the recipe test.

run_recipe_tests

Runs the recipes tests.

run_test_cmd

Runs the command corresponding to a recipe test.

Reference

speechbrain.utils.recipe_tests.check_row_for_test(row, filter_fields, filters, test_field)[source]

Checks if the current row of the csv recipe file has a test to run.

Parameters
  • row (dict) – Line of the csv file (in dict from).

  • filter_fields (list) – This can be used with the “filter” variable to run only some tests. For instance, filter_fileds=[‘Task’] and filters=[‘ASR’]) will only run tests for ASR recipes.

  • filters (list) – See above.

  • test_field (string) – Key of the input dictionary that contains the test flags.

Returns

test – True if the line must be tested, False otherwise.

Return type

bool

speechbrain.utils.recipe_tests.prepare_test(recipe_csvfile='tests/recipes.csv', script_field='Script_file', hparam_field='Hparam_file', recipe_id_field='RecipeID', test_field='test_debug_flags', check_field='test_debug_checks', filters_fields=[], filters=[])[source]

Extracts all the needed information to run the recipe test.

Parameters
  • recipe_csvfile (path) – Path of the csv recipe file summarizing all the recipes in the repo.

  • script_field (str) – Field of the csv recipe file containing the path of the script to run.

  • hparam_field (str) – Field of the csv recipe file containing the path of the hparam file.

  • recipe_id_field (str) – Field of the csv recipe file containing the unique recipe ID.

  • test_field (string) – Field of the csv recipe file containing the test flags.

  • check_field (string) – Field of the csv recipe file containing the checks to perform.

  • filter_fields (list) – This can be used with the “filter” variable to run only some tests. For instance, filter_fileds=[‘Task’] and filters=[‘ASR’]) will only run tests for ASR recipes.

  • filters (list) – See above.

Returns

  • test_script (dict) – A Dictionary containing recipe IDs as keys and test_scripts as values.

  • test_hparam (dict) – A dictionary containing recipe IDs as keys and hparams as values.

  • test_flag (dict) – A dictionary containing recipe IDs as keys and the test flags as values.

  • test_check (dict) – A dictionary containing recipe IDs as keys and the checks as values.

speechbrain.utils.recipe_tests.check_files(check_str, output_folder, recipe_id, pattern='file_exists=\\[(.*?)\\]')[source]

Checks if the output folder created by the test has the expected files.

Parameters
  • check_str (str) – String summarizing the checks to perform.

  • output_folder (path) – The path where to check the files.

  • recipe_id (str) – Unique ID of the recipe.

  • pattern (str) – The pattern used to extract the list of files to check from check_str.

Returns

check – True if all the files are found, False otherwise.

Return type

bool

speechbrain.utils.recipe_tests.check_performance(check_str, output_folder, recipe_id, pattern='performance_check=\\[(.*?)\\]')[source]

Checks if the performance achieved by the recipe matches with the expectations. This is done by adding a performance_check entry in the recipe check field of the csv recipe file For instance: performance_check=[train_log.txt, train loss, <=15, epoch: 2]), will check the variable “train_loss” in the train_log.txt at epoch 2. It will raise an error if the train_loss is >15.

Parameters
  • check_str (str) – String summarizing the checks to perform.

  • output_folder (path) – The path where the recipe files are stored.

  • recipe_id (str) – Unique ID of the recipe.

  • pattern (str) – The pattern used to extract the list of files to check from check_str.

Returns

check – True if all the files are found, False otherwise.

Return type

bool

speechbrain.utils.recipe_tests.check_threshold(threshold, value)[source]

Checks if the value satisfied the threshold constraints.

Parameters
  • threshold (str) – String that contains the contains. E.g, “>=10” or “==15” or “<5”.

  • value (float) – Float corresponding to the value to test

Returns

True if the constraint is satisfied, False otherwise.

Return type

bool

speechbrain.utils.recipe_tests.run_test_cmd(cmd, stdout_file, stderr_file)[source]

Runs the command corresponding to a recipe test. The standard output and the standard error is saved in the specified paths.

Parameters
  • cmd (str) – String corresponding to the command to run.

  • stdout_file (path) – File where standard output is stored.

  • stderr_file (path) – File where standard error is stored.

Returns

rc – The return code obtained after running the command. If 0, the test is run without errors. If >0 the execution failed.

Return type

bool

speechbrain.utils.recipe_tests.run_recipe_tests(recipe_csvfile='tests/recipes.csv', script_field='Script_file', hparam_field='Hparam_file', recipe_id_field='RecipeID', test_field='test_debug_flags', check_field='test_debug_checks', run_opts='--device=cpu', output_folder='tests/tmp/recipes/', filters_fields=[], filters=[], do_checks=True)[source]

Runs the recipes tests.

Parameters
  • recipe_csvfile (path) – Path of the csv recipe file summarizing all the recipes in the repo.

  • script_field (str) – Field of the csv recipe file containing the path of the script to run.

  • hparam_field (str) – Field of the csv recipe file containing the path of the hparam file.

  • recipe_id_field (str) – Field of the csv recipe file containing the unique recipe ID.

  • test_field (string) – Field of the csv recipe file containing the test flags.

  • check_field (string) – Field of the csv recipe file containing the checks to perform.

  • run_opts (string) – Additional flags to add for the tests (see run_opts of speechbrain/core.py).

  • output_folder (string) – Folder where the output of the tests are saved.

  • filter_fields (list) – This can be used with the “filter” variable to run only some tests. For instance, filter_fileds=[‘Task’] and filters=[‘ASR’]) will only run tests for ASR recipes.

  • filters (list) – See above.

  • do_checks – If True performs the checks on the output folder (when the check_field is not empty).

Returns

check – True if all the recipe tests pass, False otherwise.

Return type

True

speechbrain.utils.recipe_tests.load_yaml_test(recipe_csvfile='tests/recipes.csv', script_field='Script_file', hparam_field='Hparam_file', test_field='Hparam_file', filters_fields=[], filters=[], avoid_list=['templates/hyperparameter_optimization_speaker_id/train.yaml', 'templates/speaker_id/train.yaml', 'recipes/timers-and-such/multistage/hparams/train_LS_LM.yaml', 'recipes/timers-and-such/multistage/hparams/train_TAS_LM.yaml', 'recipes/timers-and-such/direct/hparams/train.yaml', 'recipes/timers-and-such/decoupled/hparams/train_LS_LM.yaml', 'recipes/timers-and-such/decoupled/hparams/train_TAS_LM.yaml', 'recipes/fluent-speech-commands/direct/hparams/train.yaml', 'recipes/CommonLanguage/lang_id/hparams/train_ecapa_tdnn.yaml', 'recipes/SLURP/direct/hparams/train.yaml'], rir_folder='tests/tmp/rir', data_folder='tests/tmp/yaml', output_folder='tests/tmp/yaml')[source]

Tests if the yaml files can be loaded without errors.

Parameters
  • recipe_csvfile (path) – Path of the csv recipe file summarizing all the recipes in the repo.

  • script_field (str) – Field of the csv recipe file containing the path of the script to run.

  • hparam_field (str) – Field of the csv recipe file containing the path of the hparam file.

  • test_field (string) – Field of the csv recipe file containing the test flags.

  • filter_fields (list) – This can be used with the “filter” variable to run only some tests. For instance, filter_fileds=[‘Task’] and filters=[‘ASR’]) will only run tests for ASR recipes.

  • filters (list) – See above.

  • avoid_list (list) – List of hparam file not to check.

  • rir_folder – This overrides the rir_folder; rir_path, and openrir_folder usually specified in the hparam files.

  • data_folder – This overrides the data_folder usually specified in the hparam files.

  • output_folder – This overrides the output_folder usually specified in the hparam files.

Returns

check – True if all the hparam files are loaded correctly, False otherwise.

Return type

True