hyperpyyaml.core module

This library gathers utilities for hyperpyyaml loading

Authors

• Peter Plantinga 2020

• Aku Rouhe 2020

• Jianchen Li 2022

Summary

Classes:

Placeholder	Class for dumping !PLACEHOLDER tags to yaml
RefTag	Class for dumping !ref tags to yaml

Functions:

deref	Find the value referred to by a reference in dot-notation
dump_hyperpyyaml	Dump yaml including placeholder and reference tags.
load_hyperpyyaml	This function implements the HyperPyYAML syntax
parse_arithmetic	Parses simple arithmetic operations in references
recursive_resolve	Resolve a reference to a value, following chained references
recursive_update	Similar function to dict.update, but for a nested dict.
resolve_references	Resolves inter-document references, a component of HyperPyYAML.

Reference

hyperpyyaml.core.load_hyperpyyaml(yaml_stream, overrides=None, overrides_must_match=True, loader=<class 'ruamel.yaml.loader.Loader'>)

This function implements the HyperPyYAML syntax

The purpose for this syntax is a compact, structured hyperparameter and function definition. This function implements a few extensions to the yaml syntax, listed below.

PyYAML complex tag shortcuts

Part of our clean structured hyperparameter interface is being able to specify python objects easily and cleanly. This is possible with native YAML using the following syntax:

alignment_saver: !!python/object/new:speechbrain.data_io.TensorSaver
    kwargs: {save_dir: results/asr/ali}

However, due to the extensive use within speechbrain yaml files, we have added a shortcut for this that has the following syntax:

alignment_saver: !new:speechbrain.data_io.TensorSaver
    save_dir: results/asr/ali

In this example, the alignment_saver will be an instance of the TensorSaver class, with 'exp/asr/ali' passed to the __init__() method as a keyword argument. This is equivalent to:

import speechbrain.data_io.data_io
alignment_saver = speechbrain.data_io.TensorSaver(
    save_dir='exp/asr/ali'
)

We have also implemented a few more shortcuts::

!!python/name: => !name:
!!python/module: => !module:
!!python/object/apply: => !apply:

References and copies

Allows internal references to any node in the file. Any node with tag !ref will create an object reference to the yaml object at the <key.subkey> location within the yaml itself, following reference chains.

output_folder: results/asr
alignment_saver: !new:speechbrain.data_io.TensorSaver
    save_dir: !ref <output_folder>

Strings values are handled specially: references are substituted but the rest of the string is left in place, allowing filepaths to be easily extended:

output_folder: results/asr
alignment_saver: !new:speechbrain.data_io.TensorSaver
    save_dir: !ref <output_folder>/ali  # results/asr/ali

A more complex example for demonstration purposes:

key1: {a: !new:object {arg1: 1}}
key2: !ref <key1[a]>

Here, key2 will contain a reference to the a object, so changing a.arg1 will also change key2.arg1. If you need a deep copy of the object instead of a shallow reference, you can use a similar syntax with the tag !copy. For example:

key1: {a: !new:object {arg1: 1}}
key2: !copy <key1[a]>

These will also implement very basic arithmetic, so:

key1: 1
key2: !ref <key1> + 3  # this is 4

Tuples

One last minor enhancement is an implicit tuple resolver. Passing a string value of (3, 4) will be given a tag of !tuple which is then interpreted as a tuple.

Parameters:

• yaml_stream (stream) – A file-like object or string from which to read.

• overrides (mapping or str) – A set of overrides for the values read from the stream. As yaml implements a nested structure, so can the overrides. See speechbrain.utils.data_utils.recursive_update

• overrides_must_match (bool) – Whether an error will be thrown when an override does not match a corresponding key in the yaml_stream.

• return_dict (bool) – Whether to return a dictionary rather than the default namespace.

• loader (Loader) – Use to parse the yaml_stream, could be ruamel.yaml.Loader, yaml.Loader, etc.

Returns:

hparams – Reflects the structure of yaml_stream.

Return type:

dict

Example

>>> yaml_string = """
... a: 3
... thing: !new:collections.Counter
...     b: !ref <a>
... """
>>> params = load_hyperpyyaml(yaml_string)
>>> params["thing"]
Counter({'b': 3})

class hyperpyyaml.core.RefTag(ref_str)

Bases: object

Class for dumping !ref tags to yaml

Parameters:

ref_str (str) – String including yaml keys in <key> notation

Example

See dump_hyperpyyaml

yaml_tag = '!ref'

classmethod to_yaml(representer, node)

class hyperpyyaml.core.Placeholder

Bases: object

Class for dumping !PLACEHOLDER tags to yaml

Example

See dump_hyperpyyaml

yaml_tag = '!PLACEHOLDER'

classmethod to_yaml(representer, node)

hyperpyyaml.core.dump_hyperpyyaml(yaml_tree, output_stream, *args, **kwargs)

Dump yaml including placeholder and reference tags.

Parameters:

• yaml_tree (dict) – An object to dump

• output_stream (stream) – A file stream for putting the yaml

• *args – Arguments to forward to ruamel.yaml.YAML().dump()

• **kwargs – Arguments to forward to ruamel.yaml.YAML().dump()

Example

>>> to_yaml = {'a': Placeholder(), 'b': RefTag('<a>')}
>>> stringio = StringIO()
>>> dump_hyperpyyaml(to_yaml, stringio)
>>> stringio.getvalue()
'a: !PLACEHOLDER\nb: !ref <a>\n'

hyperpyyaml.core.resolve_references(yaml_stream, overrides=None, overrides_must_match=False)

Resolves inter-document references, a component of HyperPyYAML.

Parameters:

• yaml_stream (stream) – A file-like object or string with the contents of a yaml file written with the HyperPyYAML syntax.

• overrides (mapping or str) – Replacement values, either in a yaml-formatted string or a dict.

• overrides_must_match (bool) – Whether an error will be thrown when an override does not match a corresponding key in the yaml_stream. This is the opposite default from load_hyperpyyaml because resolve_references doesn’t need to be as strict by default.

Returns:

A yaml-formatted stream with all references and overrides resolved.

Return type:

stream

Example

>>> yaml_string = """
... constants:
...     a: 3
...     b: !ref <constants[a]>
... """
>>> overrides = {'constants': {'a': 4}}
>>> resolve_references(yaml_string, overrides).getvalue()
'constants:\n  a: 4\n  b: 4\n'

hyperpyyaml.core.deref(ref, full_tree, copy_mode=False)

Find the value referred to by a reference in dot-notation

Parameters:

• ref (str) – The location of the requested value, e.g. ‘constants.param’

• full_tree (dict) – The dictionary to use for finding values

• copy_mode (bool) – Whether to copy the node before dereferencing.

Returns:

The node in the full_tree dictionary referenced by ref.

Return type:

node

Example

>>> deref('constants[a][b]', {'constants': {'a': {'b': 'c'}}})
'c'

hyperpyyaml.core.recursive_resolve(reference, reference_list, full_tree, copy_mode=False)

Resolve a reference to a value, following chained references

Parameters:

• reference (str) – a string containing ‘<x[y]>’ in it where x[y] refers to a scalar node in the file.

• reference_list (list) – list of prior references in the chain, in order to catch circular references.

• full_tree (dict) – the dictionary in which to find all references and their values.

• copy_mode (bool) – Whether to perform a deep copy of the referenced node, rather than a shallow reference to the same object.

Returns:

The dereferenced value, with possible string interpolation and arithmetic parsing.

Return type:

scalar

Example

>>> tree = {'a': 3, 'b': 'x', 'c': '<a>', 'd': '<c>/<c>', 'e': '<b>/<b>'}
>>> recursive_resolve('<d>', [], tree)
1.0
>>> recursive_resolve('<e>', [], tree)
'x/x'

hyperpyyaml.core.parse_arithmetic(reference_string)

Parses simple arithmetic operations in references

Adapted from https://stackoverflow.com/a/9558001/1761970

Parameters:

reference_string (str) – A string with references and possible arithmetic operations.

Returns:

Result of parsing and applying the arithmetic.

Return type:

str

Example

>>> parse_arithmetic('2 * 6')
12

hyperpyyaml.core.recursive_update(d, u, must_match=False)

Similar function to dict.update, but for a nested dict.

From: https://stackoverflow.com/a/3233356

If you have to a nested mapping structure, for example:

{“a”: 1, “b”: {“c”: 2}}

Say you want to update the above structure with:

{“b”: {“d”: 3}}

This function will produce:

{“a”: 1, “b”: {“c”: 2, “d”: 3}}

Instead of:

{“a”: 1, “b”: {“d”: 3}}

Parameters:

• d (dict) – mapping to be updated

• u (dict) – mapping to update with

• must_match (bool) – Whether to throw an error if the key in u does not exist in d.

Example

>>> d = {'a': 1, 'b': {'c': 2}}
>>> recursive_update(d, {'b': {'d': 3}})
>>> d
{'a': 1, 'b': {'c': 2, 'd': 3}}