hyperpyyaml.core module
This library gathers utilities for hyperpyyaml loading
- Authors
Peter Plantinga 2020
Aku Rouhe 2020
Jianchen Li 2022
Summary
Classes:
Class for dumping !PLACEHOLDER tags to yaml |
|
Class for dumping !ref tags to yaml |
Functions:
Find the value referred to by a reference in dot-notation |
|
Dump yaml including placeholder and reference tags. |
|
This function implements the HyperPyYAML syntax |
|
Parses simple arithmetic operations in references |
|
Resolve a reference to a value, following chained references |
|
Similar function to dict.update, but for a nested dict. |
|
Resolves inter-document references, a component of HyperPyYAML. |
Reference
- hyperpyyaml.core.load_hyperpyyaml(yaml_stream, overrides=None, overrides_must_match=True)[source]
This function implements the HyperPyYAML syntax
The purpose for this syntax is a compact, structured hyperparameter and function definition. This function implements a few extensions to the yaml syntax, listed below.
PyYAML complex tag shortcuts
Part of our clean structured hyperparameter interface is being able to specify python objects easily and cleanly. This is possible with native YAML using the following syntax:
alignment_saver: !!python/object/new:speechbrain.data_io.TensorSaver kwargs: {save_dir: results/asr/ali}
However, due to the extensive use within speechbrain yaml files, we have added a shortcut for this that has the following syntax:
alignment_saver: !new:speechbrain.data_io.TensorSaver save_dir: results/asr/ali
In this example, the alignment_saver will be an instance of the
TensorSaver
class, with'exp/asr/ali'
passed to the__init__()
method as a keyword argument. This is equivalent to:import speechbrain.data_io.data_io alignment_saver = speechbrain.data_io.TensorSaver( save_dir='exp/asr/ali' )
We have also implemented a few more shortcuts::
!!python/name: => !name: !!python/module: => !module: !!python/object/apply: => !apply:
References and copies
Allows internal references to any node in the file. Any node with tag
!ref
will create an object reference to the yaml object at the<key.subkey>
location within the yaml itself, following reference chains.output_folder: results/asr alignment_saver: !new:speechbrain.data_io.TensorSaver save_dir: !ref <output_folder>
Strings values are handled specially: references are substituted but the rest of the string is left in place, allowing filepaths to be easily extended:
output_folder: results/asr alignment_saver: !new:speechbrain.data_io.TensorSaver save_dir: !ref <output_folder>/ali # results/asr/ali
A more complex example for demonstration purposes:
key1: {a: !new:object {arg1: 1}} key2: !ref <key1[a]>
Here,
key2
will contain a reference to thea
object, so changinga.arg1
will also changekey2.arg1
. If you need a deep copy of the object instead of a shallow reference, you can use a similar syntax with the tag!copy
. For example:key1: {a: !new:object {arg1: 1}} key2: !copy <key1[a]>
These will also implement very basic arithmetic, so:
key1: 1 key2: !ref <key1> + 3 # this is 4
Tuples
One last minor enhancement is an implicit tuple resolver. Passing a string value of
(3, 4)
will be given a tag of!tuple
which is then interpreted as a tuple.- Parameters
yaml_stream (stream) – A file-like object or string from which to read.
overrides (mapping or str) – A set of overrides for the values read from the stream. As yaml implements a nested structure, so can the overrides. See speechbrain.utils.data_utils.recursive_update
overrides_must_match (bool) – Whether an error will be thrown when an override does not match a corresponding key in the yaml_stream.
return_dict (bool) – Whether to return a dictionary rather than the default namespace.
- Returns
hparams – Reflects the structure of
yaml_stream
.- Return type
Example
>>> yaml_string = """ ... a: 3 ... thing: !new:collections.Counter ... b: !ref <a> ... """ >>> params = load_hyperpyyaml(yaml_string) >>> params["thing"] Counter({'b': 3})
- class hyperpyyaml.core.RefTag(ref_str)[source]
Bases:
object
Class for dumping !ref tags to yaml
- Parameters
ref_str (str) – String including yaml keys in <key> notation
Example
See
dump_hyperpyyaml
- yaml_tag = '!ref'
- class hyperpyyaml.core.Placeholder[source]
Bases:
object
Class for dumping !PLACEHOLDER tags to yaml
Example
See
dump_hyperpyyaml
- yaml_tag = '!PLACEHOLDER'
- hyperpyyaml.core.dump_hyperpyyaml(yaml_tree, output_stream, *args, **kwargs)[source]
Dump yaml including placeholder and reference tags.
- Parameters
yaml_tree (dict) – An object to dump
output_stream (stream) – A file stream for putting the yaml
*args – Arguments to forward to ruamel.yaml.YAML().dump()
**kwargs – Arguments to forward to ruamel.yaml.YAML().dump()
Example
>>> to_yaml = {'a': Placeholder(), 'b': RefTag('<a>')} >>> stringio = StringIO() >>> dump_hyperpyyaml(to_yaml, stringio) >>> stringio.getvalue() 'a: !PLACEHOLDER\nb: !ref <a>\n'
- hyperpyyaml.core.resolve_references(yaml_stream, overrides=None, overrides_must_match=False)[source]
Resolves inter-document references, a component of HyperPyYAML.
- Parameters
yaml_stream (stream) – A file-like object or string with the contents of a yaml file written with the HyperPyYAML syntax.
overrides (mapping or str) – Replacement values, either in a yaml-formatted string or a dict.
overrides_must_match (bool) – Whether an error will be thrown when an override does not match a corresponding key in the yaml_stream. This is the opposite default from
load_hyperpyyaml
becauseresolve_references
doesn’t need to be as strict by default.
- Returns
A yaml-formatted stream with all references and overrides resolved.
- Return type
stream
Example
>>> yaml_string = """ ... constants: ... a: 3 ... b: !ref <constants[a]> ... """ >>> overrides = {'constants': {'a': 4}} >>> resolve_references(yaml_string, overrides).getvalue() 'constants:\n a: 4\n b: 4\n'
- hyperpyyaml.core.deref(ref, full_tree, copy_mode=False)[source]
Find the value referred to by a reference in dot-notation
- Parameters
- Returns
The node in the full_tree dictionary referenced by
ref
.- Return type
node
Example
>>> deref('constants[a][b]', {'constants': {'a': {'b': 'c'}}}) 'c'
- hyperpyyaml.core.recursive_resolve(reference, reference_list, full_tree, copy_mode=False)[source]
Resolve a reference to a value, following chained references
- Parameters
reference (str) – a string containing ‘<x[y]>’ in it where x[y] refers to a scalar node in the file.
reference_list (list) – list of prior references in the chain, in order to catch circular references.
full_tree (dict) – the dictionary in which to find all references and their values.
copy_mode (bool) – Whether to perform a deep copy of the referenced node, rather than a shallow reference to the same object.
- Returns
The dereferenced value, with possible string interpolation and arithmetic parsing.
- Return type
scalar
Example
>>> tree = {'a': 3, 'b': 'x', 'c': '<a>', 'd': '<c>/<c>', 'e': '<b>/<b>'} >>> recursive_resolve('<d>', [], tree) 1.0 >>> recursive_resolve('<e>', [], tree) 'x/x'
- hyperpyyaml.core.parse_arithmetic(reference_string)[source]
Parses simple arithmetic operations in references
Adapted from https://stackoverflow.com/a/9558001/1761970
- Parameters
reference_string (str) – A string with references and possible arithmetic operations.
- Returns
Result of parsing and applying the arithmetic.
- Return type
Example
>>> parse_arithmetic('2 * 6') 12
- hyperpyyaml.core.recursive_update(d, u, must_match=False)[source]
Similar function to dict.update, but for a nested dict.
From: https://stackoverflow.com/a/3233356
If you have to a nested mapping structure, for example:
{“a”: 1, “b”: {“c”: 2}}
Say you want to update the above structure with:
{“b”: {“d”: 3}}
This function will produce:
{“a”: 1, “b”: {“c”: 2, “d”: 3}}
Instead of:
{“a”: 1, “b”: {“d”: 3}}
- Parameters
Example
>>> d = {'a': 1, 'b': {'c': 2}} >>> recursive_update(d, {'b': {'d': 3}}) >>> d {'a': 1, 'b': {'c': 2, 'd': 3}}