speechbrain.utils.recipe_tests module
Library for running recipe tests.
- Authors
Mirco Ravanelli 2022
Andreas Nautsch 2022
Summary
Functions:
Checks if the output folder created by the test has the expected files. |
|
Checks if the performance achieved by the recipe matches with the expectations. |
|
Checks if the current row of the csv recipe file has a test to run. |
|
Checks if the value satisfied the threshold constraints. |
|
Tests if the yaml files can be loaded without errors. |
|
Extracts all the needed information to run the recipe test. |
|
Runs the recipes tests. |
|
Runs the command corresponding to a recipe test. |
Reference
- speechbrain.utils.recipe_tests.check_row_for_test(row, filter_fields, filters, test_field)[source]
Checks if the current row of the csv recipe file has a test to run.
- Parameters
row (dict) – Line of the csv file (in dict from).
filter_fields (list) – This can be used with the “filter” variable to run only some tests. For instance, filter_fileds=[‘Task’] and filters=[‘ASR’]) will only run tests for ASR recipes.
filters (list) – See above.
test_field (string) – Key of the input dictionary that contains the test flags.
- Returns
test – True if the line must be tested, False otherwise.
- Return type
- speechbrain.utils.recipe_tests.prepare_test(recipe_csvfile='tests/recipes.csv', script_field='Script_file', hparam_field='Hparam_file', recipe_id_field='RecipeID', test_field='test_debug_flags', check_field='test_debug_checks', filters_fields=[], filters=[])[source]
Extracts all the needed information to run the recipe test.
- Parameters
recipe_csvfile (path) – Path of the csv recipe file summarizing all the recipes in the repo.
script_field (str) – Field of the csv recipe file containing the path of the script to run.
hparam_field (str) – Field of the csv recipe file containing the path of the hparam file.
recipe_id_field (str) – Field of the csv recipe file containing the unique recipe ID.
test_field (string) – Field of the csv recipe file containing the test flags.
check_field (string) – Field of the csv recipe file containing the checks to perform.
filter_fields (list) – This can be used with the “filter” variable to run only some tests. For instance, filter_fileds=[‘Task’] and filters=[‘ASR’]) will only run tests for ASR recipes.
filters (list) – See above.
- Returns
test_script (dict) – A Dictionary containing recipe IDs as keys and test_scripts as values.
test_hparam (dict) – A dictionary containing recipe IDs as keys and hparams as values.
test_flag (dict) – A dictionary containing recipe IDs as keys and the test flags as values.
test_check (dict) – A dictionary containing recipe IDs as keys and the checks as values.
- speechbrain.utils.recipe_tests.check_files(check_str, output_folder, recipe_id, pattern='file_exists=\\[(.*?)\\]')[source]
Checks if the output folder created by the test has the expected files.
- Parameters
- Returns
check – True if all the files are found, False otherwise.
- Return type
- speechbrain.utils.recipe_tests.check_performance(check_str, output_folder, recipe_id, pattern='performance_check=\\[(.*?)\\]')[source]
Checks if the performance achieved by the recipe matches with the expectations. This is done by adding a performance_check entry in the recipe check field of the csv recipe file For instance: performance_check=[train_log.txt, train loss, <=15, epoch: 2]), will check the variable “train_loss” in the train_log.txt at epoch 2. It will raise an error if the train_loss is >15.
- Parameters
- Returns
check – True if all the files are found, False otherwise.
- Return type
- speechbrain.utils.recipe_tests.check_threshold(threshold, value)[source]
Checks if the value satisfied the threshold constraints.
- speechbrain.utils.recipe_tests.run_test_cmd(cmd, stdout_file, stderr_file)[source]
Runs the command corresponding to a recipe test. The standard output and the standard error is saved in the specified paths.
- Parameters
cmd (str) – String corresponding to the command to run.
stdout_file (path) – File where standard output is stored.
stderr_file (path) – File where standard error is stored.
- Returns
rc – The return code obtained after running the command. If 0, the test is run without errors. If >0 the execution failed.
- Return type
- speechbrain.utils.recipe_tests.run_recipe_tests(recipe_csvfile='tests/recipes.csv', script_field='Script_file', hparam_field='Hparam_file', recipe_id_field='RecipeID', test_field='test_debug_flags', check_field='test_debug_checks', run_opts='--device=cpu', output_folder='tests/tmp/recipes/', filters_fields=[], filters=[], do_checks=True)[source]
Runs the recipes tests.
- Parameters
recipe_csvfile (path) – Path of the csv recipe file summarizing all the recipes in the repo.
script_field (str) – Field of the csv recipe file containing the path of the script to run.
hparam_field (str) – Field of the csv recipe file containing the path of the hparam file.
recipe_id_field (str) – Field of the csv recipe file containing the unique recipe ID.
test_field (string) – Field of the csv recipe file containing the test flags.
check_field (string) – Field of the csv recipe file containing the checks to perform.
run_opts (string) – Additional flags to add for the tests (see run_opts of speechbrain/core.py).
output_folder (string) – Folder where the output of the tests are saved.
filter_fields (list) – This can be used with the “filter” variable to run only some tests. For instance, filter_fileds=[‘Task’] and filters=[‘ASR’]) will only run tests for ASR recipes.
filters (list) – See above.
do_checks – If True performs the checks on the output folder (when the check_field is not empty).
- Returns
check – True if all the recipe tests pass, False otherwise.
- Return type
True
- speechbrain.utils.recipe_tests.load_yaml_test(recipe_csvfile='tests/recipes.csv', script_field='Script_file', hparam_field='Hparam_file', test_field='Hparam_file', filters_fields=[], filters=[], avoid_list=['templates/hyperparameter_optimization_speaker_id/train.yaml', 'templates/speaker_id/train.yaml', 'recipes/timers-and-such/multistage/hparams/train_LS_LM.yaml', 'recipes/timers-and-such/multistage/hparams/train_TAS_LM.yaml', 'recipes/timers-and-such/direct/hparams/train.yaml', 'recipes/timers-and-such/decoupled/hparams/train_LS_LM.yaml', 'recipes/timers-and-such/decoupled/hparams/train_TAS_LM.yaml', 'recipes/fluent-speech-commands/direct/hparams/train.yaml', 'recipes/CommonLanguage/lang_id/hparams/train_ecapa_tdnn.yaml', 'recipes/SLURP/direct/hparams/train.yaml'], rir_folder='tests/tmp/rir', data_folder='tests/tmp/yaml', output_folder='tests/tmp/yaml')[source]
Tests if the yaml files can be loaded without errors.
- Parameters
recipe_csvfile (path) – Path of the csv recipe file summarizing all the recipes in the repo.
script_field (str) – Field of the csv recipe file containing the path of the script to run.
hparam_field (str) – Field of the csv recipe file containing the path of the hparam file.
test_field (string) – Field of the csv recipe file containing the test flags.
filter_fields (list) – This can be used with the “filter” variable to run only some tests. For instance, filter_fileds=[‘Task’] and filters=[‘ASR’]) will only run tests for ASR recipes.
filters (list) – See above.
avoid_list (list) – List of hparam file not to check.
rir_folder – This overrides the rir_folder; rir_path, and openrir_folder usually specified in the hparam files.
data_folder – This overrides the data_folder usually specified in the hparam files.
output_folder – This overrides the output_folder usually specified in the hparam files.
- Returns
check – True if all the hparam files are loaded correctly, False otherwise.
- Return type
True