Flexible Python configuration system. The last one you will ever need.

Overview

OmegaConf

Description
Project PyPI versionDownloadsPyPI - Python Version
Code quality CircleCICoverage StatusTotal alertsLanguage grade: Python
Docs and support Documentation StatusBinder

OmegaConf is a hierarchical configuration system, with support for merging configurations from multiple sources (YAML config files, dataclasses/objects and CLI arguments) providing a consistent API regardless of how the configuration was created.

Releases

Stable (2.0)

OmegaConf 2.0 stable version.

Install with pip install --upgrade omegaconf

Live tutorial

Run the live tutorial : Binder

Issues
  • Improve handling of interpolations pointing to missing nodes

    Improve handling of interpolations pointing to missing nodes

    • Interpolations are never considered to be missing anymore, even if they point to a missing node

    • When resolving an expression containing an interpolation pointing to a missing node, an InterpolationToMissingValueError exception is raised

    • When resolving an expression containing an interpolation pointing to a node that does not exist, an InterpolationKeyError exception is raised

    • key in cfg returns True whenever key is an interpolation, even if it cannot be resolved (including interpolations to missing nodes)

    • get() and pop() no longer return the default value in case of interpolation resolution failure (same thing for OmegaConf.select())

    • If throw_on_resolution_failure is False, then resolving an interpolation resulting in a resolution failure always leads to the result being None (instead of potentially being an expression computed from None)

    Fixes #543 Fixes #561 Fixes #562 Fixes #565

    opened by odelalleau 56
  • Introduce new `oc.env` and `oc.decode` resolvers

    Introduce new `oc.env` and `oc.decode` resolvers

    • Restore and deprecate the old env resolver for backward compatibility with OmegaConf 2.0

    • The new oc.env resolver keeps the string representation of environment variables, and does not use the cache

    • The new oc.decode resolver can be used to parse and evaluate strings according to the OmegaConf grammar

    Fixes #383 Fixes #573 Fixes #574

    Notes:

    • the diff for the notebook can be seen at https://odelalleau.github.io/diff/diff_notebook_decode.html
    • the diff of test_interpolation.py is hard to read because there was an indentation change => I created a dummy diff at https://github.com/odelalleau/omegaconf/pull/2/files that doesn't have the indentation change for better readability
    opened by odelalleau 27
  • Major changes to interpolations and resolvers

    Major changes to interpolations and resolvers

    This is a squashed version of #321.

    • Add a grammar to parse interpolations
    • Add support for nested interpolations
    • Deprecate register_resolver() and introduce register_new_resolver() (that allows resolvers to use non-string arguments, and to decide whether or not to use the cache)
    • The env resolver now parses environment variables in a way that is consistent with the new interpolation grammar

    Fixes #100 #230 #266 #318

    opened by odelalleau 25
  • [Feature Request] resolve individual resolver arguments

    [Feature Request] resolve individual resolver arguments

    It would be nice to enable recursive interpolation.

    Example :

    import omegaconf
    from omegaconf import OmegaConf
    
    OmegaConf.register_resolver("plus", lambda x,y: float(x)+float(y))
    
    c = OmegaConf.create({
            "i": 1,
            'key1': '${plus:1,2}',
            'key2': '${plus:${i},2}'
     })
    print(c.key1)
    # 3.0
    print(c.key2)
    # '${plus:1,2}'
    

    But one would hope to get 3.0 out

    Note that 2 recursion would be enough to enable the user to create deeper interpolation if they need it (by calling an other key which had already been interpolated).

    enhancement 
    opened by YannDubs 22
  • Implement nested interpolations

    Implement nested interpolations

    This is to fix #100

    Doc and tests are missing, I'm aware of it, I wanted to first get feedback on the implementation itself, in case there'd be any major concern.

    The main parsing logic is in _evaluate_complex(), it's actually not too complex since we can sequentially iterate on all characters in the string and evaluate interpolations in the order we encounter their end }. So the core logic is simple but there are still "details" to be addressed (see concerns below).

    This is not a super efficient implementation: (i) there are many string copies in the process, and (ii) the worst case complexity is quadratic in case of deep nesting (Edit: not anymore, see comment below). That being said, I don't expect it to be a real issue in the large majority of practical use cases.

    A few potential concerns:

    1. Although all tests currently pass, this can actually break backward compatibility since for instance if you had a: 1 as config, then ${${a}} used to evaluate to the string"${1}", when now it would crash when trying to resolve this new interpolation. Hopefully people were not relying on this!
    2. My current implementation returns a None value instead of crashing in case of syntax error. This isn't good, but right now the only two flags to trigger a crash are throw_on_missing and throw_on_resolution_failure, because there could be no syntax error in the previous code (for instance ${a}_${a would be resolved into "1_${a"). Should I add a throw_on_parsing_failure flag?
    3. With my current implementation ${a resolves to "${a" while ${a}_${a triggers a parsing failure. This inconsistency isn't good => I was thinking of using my new interpolation logic whenever it's not a simple interpolation but there is still a ${ in the string to be parsed, which should address this issue. Thoughts?
    4. I'm not planning to allow escaping of the special ${ and } tokens, although it probably wouldn't be that hard. Let me know if you think this is important.
    opened by odelalleau 20
  • OmegaConf.select is relative to the node it's called on

    OmegaConf.select is relative to the node it's called on

    The support added for relative keys in select (#656) was a breaking change. in 2.0, when select is called on a nested node, it's already relative. #656 made that select absolute, which is a breaking change.

    A non breaking fix is to keep nested selects relative, but add support for relative syntax (specifically for going up the hierarchy). The resulting behavior is shown below and can be a bit surprising:

    from omegaconf import OmegaConf
    
    cfg = OmegaConf.create(
        {
            "a": {
                "b": {
                    "c": 10,
                }
            },
            "z": 10,
        },
    )
    # works on 2.0
    assert OmegaConf.select(cfg.a, "") == {"b": {"c": 10}}
    assert OmegaConf.select(cfg.a, "b") == {"c": 10}
    # new behavior, going up one level (as an example):
    assert OmegaConf.select(cfg.a, ".") == {"a": {"b": {"c": 10}}, "z": 10}
    assert OmegaConf.select(cfg.a, ".a") == {"b": {"c": 10}}
    assert OmegaConf.select(cfg.a, ".z") == 10
    

    In most scenarios, going up a level requires two dots. but since OmegaConf.select on a nested node is already relative to that node, only one dot is needed.

    The API is establishing the non-breaking implementation. An alternative is to make this breaking (the behavior from #656).

    The non-breaking behavior will look weird when using select inside custom resolvers using select as an alternative way to access nodes (this is using a planned oc.select resolver that will just call OmegaConf.select):

    a: 10
    foo:
      a: 20
      b: ${oc.select: .a) # relative to foo, one level up: 10
      b: ${oc.select: a) # relative to foo: 20
      # as opposed to:
      c: ${a}  # absolute: 10
      c: ${.a}  # relative: 20
    

    I am not particularly happy with this. On the one hand, select being relative to the node it's called on is intuitive and is the current behavior. On the other hand, it's inconsistent with interpolations. Thoughts?

    I am planning to merge this because this is fixing an unintentional breaking change, but let's decide if we actually want to make this an intentional breaking change (and at the point, users will need to use relative select syntax to get relative behavior, like in interpolations.

    EDIT: Final solution is different than anything above. read the code and look at the tests to understand it.

    opened by omry 18
  • Allow resolvers access to the config

    Allow resolvers access to the config

    Currently resolvers are not getting access to the config structure. It will be helpful to allow it.

    Example use case:

    OmegaConf.register_resolver("len", lambda root, value: len(OmegaConf.select(root, value)))
    c = OmegaConf.create({
      'list': [1,2,3],
      'list_len': '${len:list}'
    })
    assert cfg.list_len == 3
    
    enhancement 
    opened by omry 18
  • Structured configs do not respect `dataclasses.field(init=False)`

    Structured configs do not respect `dataclasses.field(init=False)`

    The documentation of dataclasses has this example for init=False:

    @dataclass
    class C:
        a: float
        b: float
        c: float = field(init=False)
    
        def __post_init__(self):
            self.c = self.a + self.b
    

    It just means that c is not an argument to the __init__. I find this useful sometimes when I have a configuration value that can be computed from other configuration values.

    Describe the solution you'd like

    OmegaConf should just ignore fields that have init=False set.

    Describe alternatives you've considered

    One alternative is to not give a type annotation to the field and just set it to MISSING:

    >>> @dataclass
    ... class C:
    ...     a: float
    ...     b: float
    ...     c = MISSING
    ...     def __post_init__(self):
    ...         self.c = self.a + self.b
    ...
    >>> d = OmegaConf.structured(C)
    >>> d.a = 3
    >>> d.b = 4
    >>> o = OmegaConf.to_object(d)
    >>> o.c
    7.0
    

    This works, but it is a bit sad that we have to accept the loss of the type annotation.

    Additional context

    This is what currently happens when you use init=False.
    >>> from dataclasses import dataclass, field
    >>> @dataclass
    ... class C:
    ...     a: float
    ...     b: float
    ...     c: float = field(init=False)
    ...     def __post_init__(self):
    ...         self.c = self.a + self.b
    ...
    >>> c = C(a=3, b=4)
    >>> c.c
    7
    >>> from omegaconf import OmegaConf
    >>> d = OmegaConf.structured(C)
    >>> d.a = 3
    >>> d.b = 4
    >>> OmegaConf.to_object(d)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/home/tmk/.conda/envs/palbolts/lib/python3.9/site-packages/omegaconf/omegaconf.py", line 574, in to_object
        return OmegaConf.to_container(
      File "/home/tmk/.conda/envs/palbolts/lib/python3.9/site-packages/omegaconf/omegaconf.py", line 553, in to_container
        return BaseContainer._to_content(
      File "/home/tmk/.conda/envs/palbolts/lib/python3.9/site-packages/omegaconf/basecontainer.py", line 249, in _to_content
        return conf._to_object()
      File "/home/tmk/.conda/envs/palbolts/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 735, in _to_object
        self._format_and_raise(
      File "/home/tmk/.conda/envs/palbolts/lib/python3.9/site-packages/omegaconf/base.py", line 190, in _format_and_raise
        format_and_raise(
      File "/home/tmk/.conda/envs/palbolts/lib/python3.9/site-packages/omegaconf/_utils.py", line 821, in format_and_raise
        _raise(ex, cause)
      File "/home/tmk/.conda/envs/palbolts/lib/python3.9/site-packages/omegaconf/_utils.py", line 719, in _raise
        raise ex.with_traceback(sys.exc_info()[2])  # set end OC_CAUSE=1 for full backtrace
    omegaconf.errors.MissingMandatoryValue: Structured config of type `C` has missing mandatory value: c
        full_key: c
        object_type=C
    >>> d.c = 0
    >>> OmegaConf.to_object(d)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/home/tmk/.conda/envs/palbolts/lib/python3.9/site-packages/omegaconf/omegaconf.py", line 574, in to_object
        return OmegaConf.to_container(
      File "/home/tmk/.conda/envs/palbolts/lib/python3.9/site-packages/omegaconf/omegaconf.py", line 553, in to_container
        return BaseContainer._to_content(
      File "/home/tmk/.conda/envs/palbolts/lib/python3.9/site-packages/omegaconf/basecontainer.py", line 249, in _to_content
        return conf._to_object()
      File "/home/tmk/.conda/envs/palbolts/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 752, in _to_object
        result = object_type(**field_items)
    TypeError: __init__() got an unexpected keyword argument 'c'
    
    enhancement structured config 
    opened by thomkeh 17
  • Grammar support for interpolations like ${.}. ${..} etc.

    Grammar support for interpolations like ${.}. ${..} etc.

    This enables implementing a form of select with a default value: Side note, we should rethink support for passing the parent node automatically on demand (${.}), it will simplify this use case.

    from typing import Any
    
    from omegaconf import OmegaConf
    from omegaconf.omegaconf import _EMPTY_MARKER_
    
    yaml = """\
    a: 10
    no_default: ${oc.select:${.}, a}
    not_using_default: ${oc.select:${.}, a, 20}
    using_default: ${oc.select:${.}, not_there, 20}
    """
    
    
    def oc_select(node: Any, key: str, default: Any = _EMPTY_MARKER_):
        if default is _EMPTY_MARKER_:
            return OmegaConf.select(node, key)
        else:
            return OmegaConf.select(node, key, default=default)
    
    
    OmegaConf.register_new_resolver("oc.select", oc_select)
    
    cfg = OmegaConf.create(yaml)
    assert cfg.a == 10
    assert cfg.no_default == 10
    assert cfg.not_using_default == 10
    assert cfg.using_default == 20
    

    Notes:

    1. I did not like the complexity of the testing in test_grammar.py. seems like a new testing framework instead of a test designed to test some logic. I created a simpler more localized test for interpolations. happy to discuss.

    2. while it's not very big, it can be easier to review one commit at a time.

    opened by omry 17
  • Type validation of interpolations

    Type validation of interpolations

    Validate and convert interpolation results to their intended type

    This PR fixes several issues:

    • For nested resolvers (ex: ${f:${g:x}}), intermediate resolver outputs (of g in this example) were wrapped in a ValueNode just to be unwrapped immediately, which was wasteful. This commit pushes the node wrapping to the very last step of the interpolation resolution.

    • There was no type checking to make sure that the result of an interpolation had a type consistent with the node's type (when specified). Now a check is made and the interpolation result may be converted into the desired type (see #488).

    • If a resolver interpolation returns a dict / list, it is now wrapped into a DictConfig / ListConfig, instead of a ValueNode. This makes it possible to generate configs from resolvers (see #540). These configs are read-only since they are being re-generated on each access.

    Fixes #488 Fixes #540

    opened by odelalleau 17
  • support merge keys (YAML >> tag), also with interpolation

    support merge keys (YAML >> tag), also with interpolation

    Is your feature request related to a problem? Please describe.

    I would like to be able to merge one config section into another, and augment the result with new keys. In vanilla YAML, this is straightforwardly accomplished via anchors and the merge key tag. The following piece of code illustrates this:

    import yaml
    print(yaml.safe_load("""
            a: &A
                x: 1
                y: 2
    
            b:
                <<: *A
                x: 3
                z: 1
    """))
    

    results in

    {'a': {'x': 1, 'y': 2}, 'b': {'x': 1, 'y': 2, 'z': 1}}
    

    Loading the same YAML into omegaconf results in an exception:

    from omegaconf import *
    conf = OmegaConf.create("""
            a: &A
                x: 1
                y: 2
    
            b:
                <<: *A
                x: 3
                z: 1
    """)
    
    yaml.constructor.ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:merge'
    

    Describe the solution you'd like

    Well it would be nice if the vanilla YAML merge key worked. However, this whole anchor/alias business is rather clumsy. Top prize would be if this could work with the interpolation feature, like so:

            a:
                x: 1
                y: 2
    
            b:
                <<: ${a}
                x: 3
                z: 1
    

    Even better would be if this worked across a merge of configs (similar to what's discussed in https://github.com/omry/omegaconf/issues/442). I.e.

    from omegaconf import *
    conf1 = OmegaConf.create("""
            a: 
                x: 1
                y: 2
    """)
    conf2 = OmegaConf.create("""
            b:
                <<: ${a}
                x: 3
                z: 1
    """)
    conf3 = OmegaConf.merge(conf1, conf2)
    
    

    Describe alternatives you've considered I could use lists with nested mappings in them, and merge the keys in the code. Seems ugly though. I don't see any other way around it.

    Additional context Current onegaconf master, python 3.6.9.

    opened by o-smirnov 16
  • Generated files use obsolete typing.io import

    Generated files use obsolete typing.io import

    Describe the bug Two generated files in omegaconf 2.2.1 use the typing.io namespace, which has been deprecated since Python 3.8 and will be removed in Python 3.12:

    • omegaconf/grammar/gen/OmegaConfGrammarLexer.py
    • omegaconf/grammar/gen/OmegaConfGrammarParser.py

    This import can simply be replaced with an import from the main typing namespace as TextIO is available there.

    Additional context

    • [ ] OmegaConf version: 2.2.1
    • [ ] Python version: any
    • [ ] Operating system: any
    • [ ] Please provide a minimal repro
    bug 
    opened by srittau 0
  • Implicit Path() to str conversion fails in 2.2.1

    Implicit Path() to str conversion fails in 2.2.1

    Describe the bug

    When assigning a value to a structured dataclass, attempting to convert a Path() object implicitly to str fails -

    Traceback (most recent call last):
      File "tests/core_ptl/check_for_ranks.py", line 154, in <module>
        run_checks()
      File "tests/core_ptl/check_for_ranks.py", line 144, in run_checks
        trainer = instantiate_multinode_ddp_if_possible()
      File "tests/core_ptl/check_for_ranks.py", line 83, in instantiate_multinode_ddp_if_possible
        exp_manager(trainer, cfg=OmegaConf.structured(exp_manager_cfg))
      File "/opt/conda/lib/python3.8/site-packages/nemo/utils/exp_manager.py", line 332, in exp_manager
        configure_checkpointing(
      File "/opt/conda/lib/python3.8/site-packages/nemo/utils/exp_manager.py", line 871, in configure_checkpointing
        params.dirpath = Path(log_dir / 'checkpoints')
      File "/opt/conda/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 340, in __setattr__
        self._format_and_raise(key=key, value=value, cause=e)
      File "/opt/conda/lib/python3.8/site-packages/omegaconf/base.py", line 231, in _format_and_raise
        format_and_raise(
      File "/opt/conda/lib/python3.8/site-packages/omegaconf/_utils.py", line 873, in format_and_raise
        _raise(ex, cause)
      File "/opt/conda/lib/python3.8/site-packages/omegaconf/_utils.py", line 771, in _raise
        raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
      File "/opt/conda/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 336, in __setattr__
        self.__set_impl(key, value)
      File "/opt/conda/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 320, in __set_impl
        self._set_item_impl(key, value)
      File "/opt/conda/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 602, in _set_item_impl
        self.__dict__["_content"][key]._set_value(value)
      File "/opt/conda/lib/python3.8/site-packages/omegaconf/nodes.py", line 46, in _set_value
        self._val = self.validate_and_convert(value)
      File "/opt/conda/lib/python3.8/site-packages/omegaconf/nodes.py", line 76, in validate_and_convert
        return self._validate_and_convert_impl(value)
      File "/opt/conda/lib/python3.8/site-packages/omegaconf/nodes.py", line 194, in _validate_and_convert_impl
        raise ValidationError("Cannot convert '$VALUE_TYPE' to string: '$VALUE'")
    omegaconf.errors.ValidationError: Cannot convert 'pathlib.PosixPath' to string: 'ddp_check/default/version_0/checkpoints'
        full_key: checkpoint_callback_params.dirpath
        reference_type=Optional[CallbackParams]
        object_type=CallbackParams 
    

    To Reproduce

    Python file :

    from dataclasses import dataclass
    from typing import Any, Dict, List, Optional, Union
    from omegaconf import OmegaConf
    from pathlib import Path
    import omegaconf
    
    @dataclass
    class CallbackParams:
        dirpath: Optional[str] = None  # If None, exp_manager will attempt to handle the filepath
    
    if __name__ == '__main__':
        print("OmegaConf version :", omegaconf.version.__version__)
    
        cfg = OmegaConf.structured(CallbackParams)  # type: CallbackParams
        logdir = Path.cwd()
        cfg.dirpath = logdir / Path('logs')
    
    

    Output :

    /home/smajumdar/miniconda3/envs/NeMo/bin/python /home/smajumdar/PycharmProjects/nemo-eval/tmp/temp.py
    OmegaConf version : 2.2.1
    
    Traceback (most recent call last):
      File "/home/smajumdar/PycharmProjects/nemo-eval/tmp/temp.py", line 18, in <module>
        cfg.dirpath = logdir / Path('logs')
      File "/home/smajumdar/miniconda3/envs/NeMo/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 340, in __setattr__
        self._format_and_raise(key=key, value=value, cause=e)
      File "/home/smajumdar/miniconda3/envs/NeMo/lib/python3.8/site-packages/omegaconf/base.py", line 231, in _format_and_raise
        format_and_raise(
      File "/home/smajumdar/miniconda3/envs/NeMo/lib/python3.8/site-packages/omegaconf/_utils.py", line 873, in format_and_raise
        _raise(ex, cause)
      File "/home/smajumdar/miniconda3/envs/NeMo/lib/python3.8/site-packages/omegaconf/_utils.py", line 771, in _raise
        raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
      File "/home/smajumdar/miniconda3/envs/NeMo/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 336, in __setattr__
        self.__set_impl(key, value)
      File "/home/smajumdar/miniconda3/envs/NeMo/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 320, in __set_impl
        self._set_item_impl(key, value)
      File "/home/smajumdar/miniconda3/envs/NeMo/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 602, in _set_item_impl
        self.__dict__["_content"][key]._set_value(value)
      File "/home/smajumdar/miniconda3/envs/NeMo/lib/python3.8/site-packages/omegaconf/nodes.py", line 46, in _set_value
        self._val = self.validate_and_convert(value)
      File "/home/smajumdar/miniconda3/envs/NeMo/lib/python3.8/site-packages/omegaconf/nodes.py", line 76, in validate_and_convert
        return self._validate_and_convert_impl(value)
      File "/home/smajumdar/miniconda3/envs/NeMo/lib/python3.8/site-packages/omegaconf/nodes.py", line 194, in _validate_and_convert_impl
        raise ValidationError("Cannot convert '$VALUE_TYPE' to string: '$VALUE'")
    omegaconf.errors.ValidationError: Cannot convert 'pathlib.PosixPath' to string: '/home/smajumdar/PycharmProjects/nemo-eval/tmp/logs'
        full_key: dirpath
        object_type=CallbackParams
    
    Process finished with exit code 1
    

    Expected behavior

    /home/smajumdar/miniconda3/envs/NeMo/bin/python /home/smajumdar/PycharmProjects/nemo-eval/tmp/temp.py
    OmegaConf version : 2.1.2
    
    Process finished with exit code 0
    

    Additional context

    • [x] OmegaConf version: 2.2.1
    • [x] Python version: 3.8
    • [x] Operating system: Linux, Ubuntu 20.04
    • [x] Please provide a minimal repro
    bug 
    opened by titu1994 4
  • `OmegaConf.get_type` inconsistent on `NoneType`

    `OmegaConf.get_type` inconsistent on `NoneType`

    Describe the bug The OmegaConf.get_type method is supposed to return the object type of a given OmegaConf node. I've discovered some inconsistent behavior in handling of nodes with a None value. If the node in question is an AnyNode, get_type returns type(None). If the node in question is a DictConfig, get_type returns None.

    To Reproduce

    from dataclasses import dataclass
    from typing import Any, Dict, Optional
    
    from omegaconf import OmegaConf
    
    
    @dataclass
    class HasOptDict:
        foo: Optional[Dict[Any, Any]] = None
    
    
    cfg1 = OmegaConf.create({"foo": None})
    cfg2 = OmegaConf.structured(HasOptDict())
    
    assert cfg1 == {"foo": None}
    assert cfg2 == {"foo": None}
    
    assert OmegaConf.get_type(cfg1, "foo") == type(None)
    assert OmegaConf.get_type(cfg2, "foo") == type(None)  # Raises AssertionError
    

    The assertion on the last line above fails, as OmegaConf.get_type returns None instead of type(None).

    Expected behavior All assertions above should succeed.

    Additional context

    • [X] OmegaConf version: latest
    • [X] Python version: 3.10
    bug 
    opened by Jasha10 0
  • Add support for 4 merging modes of DictConfig

    Add support for 4 merging modes of DictConfig

    See the description of omegaconf/omegaconf.py. Essentially, the idea is to introduce 4 different modes of merging DictConfigs on their keys, where the keys are to be understood as nested keys. E.g. the dictionary

    • a1: 1
    • a2:
      • b1: 2
      • b2:
        • c1: 3
        • c2: 4

    or {a1: 1, a2: {b1: 2, b2: {c1: 3, c2: 4}}} can be understood as a dictionary mapping nested keys to values as

    • a1: 1
    • a2.b1: 2
    • a2.b2.c1: 3
    • a2.b2.c2: 4

    The way of merging can be provided through the how keyword in OmegaConf.merge(), for example as OmegaConf.merge(config1, config2, how='left').

    The four different modes are as follows:

    1. left:

    Only merge on keys from the left DictConfig. E.g. {a1: 1, a2: {b1: 2, b2: 3}}, {a2: {b2: 11, b2: 12}, a3: 13} -> {a1: 1, a2: {b1: 2, b2: 11}}

    2. inner:

    Only merge on keys from both DictConfigs. E.g. {a1: 1, a2: {b1: 2, b2: 3}}, {a2: {b2: 11, b2: 12}, a3: 13} -> {a2: {b2: 11}}

    3. outer-left:

    Merge on keys from both DictConfigs. E.g. {a1: 1, a2: {b1: 2, b2: 3}}, {a2: {b2: 11, b2: 12}, a3: 13} -> {a1: 1, a2: {b1: 2, b2: 11, b3: }} If nesting levels conflict, use keys on the left. E.g. {a1: {b1: 9}}, {a1: {b1: {c1: 21}}, a2: 22} -> {a1: {b1: 9}, a2: 22}

    4. outer-right:

    Merge on keys from both DictConfigs (see outer-left). If nesting levels conflict, use keys on the right. E.g. {a1: {b1: 9}}, {a1: {b1: {c1: 21}}, a2: 22} -> {a1: {b1: {c1: 21}}, a2: 22}

    The current default is "outer-right", which corresponds to the current Omegaconf merging behavior.

    opened by luisherrmann 2
  • Allow assigning variable interpolation to structured config during merge

    Allow assigning variable interpolation to structured config during merge

    Is your feature request related to a problem? Please describe. When merging a DictConfig containing variable interpolation with a compatible structured config, the following error is thrown:

    Merge error: str is not a subclass of FooCfg. value: ${elsewhere}
        full_key:
        object_type=FooCfg
    

    Example code:

    @dataclass
    class FooCfg:
        a: int = 1
        b: int = 2
    
    @dataclass
    class AppCfg:
        foo: FooCfg = field(default_factory=FooCfg)
        bar: bool = False
    
    # override.yaml
    # this config file will be merged with the default
    foo: ${elsewhere}
    elsewhere:
        a: 3
        b: 4
    
    schema = OmegaConf.create(AppCfg)
    overrides = OmegaConf.load("override.yaml")
    # error is thrown below
    cfg = OmegaConf.unsafe_merge(schema, overrides)
    

    Describe alternatives you've considered Currently, my workaround is to convert the Structured Config back to normal to disable runtime validation:

    schema = OmegaConf.to_container(schema, structured_config_mode=SCMode.DICT)
    

    On a side note, is there a function or flag to explicitly disable runtime validation?

    Describe the solution you'd like OmegaConf should ignore the incompatibility between str and FooCfg when merging if the value appears to be a variable interpolation. Optionally, it can attempt to check if the node referred to by the interpolation is compatible with FooCfg.

    bug interpolation structured config 
    opened by Interpause 2
  • `OmegaConf.structured(...)` may mutate its input

    `OmegaConf.structured(...)` may mutate its input

    Describe the bug OmegaConf.structured(...) mutates its input when called on a dataclass or dataclass instance that has a structured DictConfig instance as one of its fields.

    To Reproduce

    from dataclasses import dataclass
    from omegaconf import OmegaConf
    
    @dataclass
    class User:
        name: str = "Bond"
        age: int = 7
    
    @dataclass
    class HasUser:
        user: User
    
    has_user1 = OmegaConf.structured(HasUser(User()))
    assert has_user1.user._parent is has_user1
    has_user2 = OmegaConf.structured(HasUser(has_user1.user))
    assert has_user2.user._parent is has_user2
    assert has_user1.user._parent is None  # has_user1.user has been mutated
    assert has_user1.user is not has_user2.user
    

    Expected behavior The creation of has_user2 should not mutate has_user1.user. Most OmegaConf creation and assignment methods do not mutate their inputs.

    Additional context

    • OmegaConf version: latest dev version (main branch)
    bug 
    opened by Jasha10 0
Releases(v2.2.1)
  • v2.2.1(May 17, 2022)

    2.2.1 (2022-05-17)

    OmegaConf 2.2 is a major release. The most significant area of improvement in 2.2 is support for more flexible type hints in structured configs. In addition, OmegaConf now natively supports two new primitive types, bytes and pathlib.Path.

    Features

    • Support unions of primitive types in structured config type hints (typing.Union) (#144)
    • Support nested container type hints in structured configs, e.g. dict-of-dict and list-of-list (#427)
    • Improve support for optional element types in structured config container type hints (typing.Optional) (#460)
    • Add support for bytes-typed values (#844)
    • Add support for pathlib.Path-typed values (#97)
    • ListConfig now implements slice assignment (#736)
    • Enable adding a ListConfig to a list via the ListConfig.__radd__ dunder method (#849)
    • Add OmegaConf.missing_keys(), a method that returns the missing keys in a config object (#720)
    • Add OmegaConf.clear_resolver(), a method to remove interpolation resolvers by name (#769)
    • Enable the use of a pipe symbol | in unquoted strings in OmegaConf interpolations (#799)

    Bug Fixes

    • OmegaConf.to_object now works properly with structured configs that have init=False fields (#789)
    • Fix bugs related to creation of structured configs from dataclasses having fields with a default_factory (#831)

    API changes and deprecations

    • Removed support for OmegaConf.is_none(cfg, "key"). Please use cfg.key is None instead. (#547)
    • Removed support for ${env} interpolations. ${oc.env} should be used instead. (#573)
    • Removed OmegaConf.get_resolver(). Please use OmegaConf.has_resolver() instead. (#608)
    • Removed support for OmegaConf.is_optional(). (#698)
    • Improved error message when assigning an invalid value to int or float config nodes (#743)
    • To conform with the MutableMapping API, the DictConfig.items method now returns an object of type ItemsView, and DictConfig.keys will now always return a KeysView (#848)
    Source code(tar.gz)
    Source code(zip)
  • v2.1.1(Aug 19, 2021)

    2.1.1 (2021-08-19)

    Features

    • Add a throw_on_missing keyword argument to the signature of OmegaConf.to_container, which controls whether MissingMandatoryValue exceptions are raised. (#501)

    Miscellaneous changes

    • Update pyyaml dependency specification for compatibility with PEP440 (#758)
    • Fix a packaging issue (missing sdist dependency) (#772
    Source code(tar.gz)
    Source code(zip)
  • v2.1.0(Jun 7, 2021)

    2.1.0 (2021-06-07)

    OmegaConf 2.1 is a major release introducing substantial new features, and introducing some incompatible changes. The biggest area of improvement in 2.1 is interpolations and resolvers. In addition - OmegaConf containers are now much more compatible with their plain Python container counterparts (dict and list).

    Features

    API Enhancements

    • OmegaConf.select() now takes an optional default value to return if a key is not found (#228)
    • flag_override can now override multiple flags at the same time (#400)
    • Add the OmegaConf.to_object method, which converts Structured Configs to native instances of the underlying @dataclass or @attr.s class. (#472)
    • Add OmegaConf.unsafe_merge(), a fast merge variant that destroys the input configs (#482)
    • New function OmegaConf.has_resolver() allows checking whether a resolver has already been registered. (#608)
    • Adds OmegaConf.resolve(cfg) for in-place interpolation resolution on cfg (#640)
    • force_add flag added to OmegaConf.update(), ensuring that the path is created even if it will result in insertion of new values into struct nodes. (#664)
    • Add DictConfig support for keys of type int, float and bool (#149), (#483)
    • Structured Configs fields without a value are now automatically treated as OmegaConf.MISSING (#390)
    • Add minimal support for typing.TypedDict (#473)
    • OmegaConf.to_container now takes a structured_config_mode keyword argument. Setting structured_config_mode=SCMode.DICT_CONFIG causes to_container to not convert Structured Config objects to python dicts (it leaves them as DictConfig objects). (#548)

    Interpolation and resolvers

    • Support for relative interpolation (#48)
    • Add ability to nest interpolations, e.g. ${foo.${bar}}}, ${oc.env:{$var1},${var2}}, or ${${func}:x1,x2} (#445)
    • Custom resolvers can now access the parent and the root config nodes (#266)
    • For OmegaConf.{update, select} and in interpolations, bracketed keys may be used as an alternative form to dot notation, e.g. foo.1 is equivalent to foo[1], [foo].1 and [foo][1]. (#179)
    • Custom resolvers may take non string arguments as input, and control whether to use the cache. (#445)
    • Dots may now be used in resolver names to denote namespaces (e.g: ${namespace.my_func:123}) (#539)
    • New resolver oc.select, enabling node selection with a default value to use if the node cannot be selected (#541)
    • New resolver oc.decode that can be used to automatically convert a string to bool, int, float, dict, list, etc. (#574)
    • New resolvers oc.dict.keys and oc.dict.values provide a list view of the keys or values of a DictConfig node. (#643)
    • New resolver oc.create can be used to dynamically generate config nodes (#645)
    • New resolver oc.deprecated, that enables deprecating config nodes (#681)
    • The dollar character $ is now allowed in interpolated key names, e.g. ${$var} (#600)

    Misc

    • New PyDev.Debugger resolver plugin for easier debugging in PyCharm and VSCode (#214)
    • OmegaConf now supports Python 3.9 (#447)
    • Support for Python 3.10 postponed annotation evaluation (#303)
    • Experimental support for enabling objects in config via "allow_objects" flag (#382)

    Bug Fixes

    • ListConfig.append() now copies input config nodes (#601)
    • Fix loading of OmegaConf 2.0 pickled configs (#718)
    • Fix support for forward declarations in Dict and Lists (#378)
    • Fix bug that allowed instances of Structured Configs to be assigned to DictConfig with different element type. (#386)
    • Fix exception raised when checking for the existence of a key with an incompatible type in DictConfig (#394)
    • Fix loading of an empty file via a file-pointer to return an empty dictionary (#403)
    • Fix pickling of Structured Configs with fields annotated as Dict[KT, VT] or List[T] on Python 3.6. (#407)
    • Assigning a primitive type to a Subscripted Dict now raises a descriptive message. (#409)
    • Fix assignment of an invalid value to a DictConfig to raise an exception without modifying the config object (#409)
    • Assigning a Structured Config to a Dict annotation now raises a descriptive error message. (#410)
    • OmegaConf.to_container() raises a ValueError on invalid input (#418)
    • Fix ConfigKeyError in some cases when merging lists containing interpolation values (#422)
    • DictConfig.get() in struct mode return None like standard Dict for non-existing keys (#425)
    • Fix bug where interpolations were unnecessarily resolved during merge (#431)
    • Fix bug where assignment of an invalid value to a ListConfig raised an exception but left the object modified. (#433)
    • When initializing a Structured Config with an incorrectly-typed value, the resulting ValidationError now properly reports the offending value in its error message. (#435)
    • Fix assignment of a Container to itself causing it to clear its content (#449)
    • Fix bug where DictConfig's shallow copy didn't work properly in some cases. (#450)
    • Fix support for merge tags in YAML files (#470)
    • Fix merge into a custom resolver node that raises an exception (#486)
    • Fix merge when element type is a Structured Config (#496)
    • Fix ValidationError when setting to None an optional field currently interpolated to a non-optional one (#524)
    • Fix OmegaConf.to_yaml(cfg) when keys are of Enum type (#531)
    • When a DictConfig has enum-typed keys, __delitem__ can now be called with a string naming the enum member to be deleted. (#554)
    • OmegaConf.select() of a missing (???) node from a ListConfig with throw_on_missing set to True now raises the intended exception. (#563)
    • DictConfig.{get(),pop()} now return None when the accessed key evaluates to None, instead of the specified default value (for consistency with regular Python dictionaries). (#583)
    • ListConfig.get() now return None when the accessed key evaluates to None, instead of the specified default value (for consistency with DictConfig). (#583)
    • Fix creation of structured config from a dict subclass: data from the dict is no longer thrown away. (#584)
    • Assignment of a dict/list to an existing node in a parent in struct mode no longer raises ValidationError (#586)
    • Nested flag_override now properly restore the original state (#589)
    • Fix OmegaConf.create() to set the provided parent when creating a config from a YAML string. (#648)
    • OmegaConf.select now returns None when attempting to select a child of a value or None node (#678)
    • Improve error message when creating a config from a Structured Config that fails type validation (#697)

    API changes and deprecations

    • DictConfig __getattr__ access, e.g. cfg.foo, is now raising a AttributeError if the key "foo" does not exist (#515)
    • DictConfig __getitem__ access, e.g. cfg["foo"], is now raising a KeyError if the key "foo" does not exist (#515)
    • DictConfig get access, e.g. cfg.get("foo"), now returns None if the key "foo" does not exist (#527)
    • Omegaconf.select(cfg, key, default, throw_on_missing) now requires keyword arguments for everything after key (#228)
    • Structured Configs with nested Structured config field that does not specify a default value are now interpreted as MISSING (???) instead of auto-expanding (#411)
    • OmegaConf.update() is now merging dict/list values into the destination node by default. Call with merge=False to replace instead. (#667)
    • register_resolver() is deprecated in favor of register_new_resolver(), allowing resolvers to (i) take non-string arguments like int, float, dict, interpolations, etc. and (ii) control the cache behavior (now disabled by default) (#426)
    • Merging a MISSING value onto an existing value no longer changes the target value to MISSING. (#462)
    • When resolving an interpolation of a config value with a primitive type, the interpolated value is validated and possibly converted based on the node's type. (#488)
    • DictConfig and ListConfig shallow copy is now performing a deepcopy (#492)
    • OmegaConf.select(), DictConfig.{get(),pop()}, ListConfig.{get(),pop()} no longer return the specified default value when the accessed key is an interpolation that cannot be resolved: instead, an exception is raised. (#543)
    • OmegaConf.{merge, unsafe_merge, to_yaml} now raises a ValueError when called on a str input. Previously an AssertionError was raised. (#560)
    • All exceptions raised during the resolution of an interpolation are either InterpolationResolutionError or a subclass of it. (#561)
    • key in cfg now returns True when key is an interpolation even if the interpolation target is a missing ("???") value. (#562)
    • OmegaConf.select() as well as container methods get() and pop() do not return their default value anymore when the accessed key is an interpolation that cannot be resolved: instead, an exception is raised. (#565)
    • Implicitly empty resolver arguments (e.g., ${foo:a,}) are deprecated in favor of explicit quoted strings (e.g., ${foo:a,""}) (#572)
    • The env resolver is deprecated in favor of oc.env, which keeps the string representation of environment variables, does not cache the resulting value, and handles "null" as default value. (#573)
    • OmegaConf.get_resolver() is deprecated: use the new OmegaConf.has_resolver() to check for the existence of a resolver. (#608)
    • Interpolation cycles are now forbidden and will trigger an InterpolationResolutionError on access. (#662)
    • Support for Structured Configs that subclass typing.Dict is now deprecated. (#663)
    • Remove BaseContainer.{pretty,select,update_node} that have been deprecated since OmegaConf 2.0. (#671)

    Miscellaneous changes

    • Optimized config creation time. Faster by 1.25x to 4x in benchmarks (#477)
    • ListConfig.contains optimized, about 15x faster in a benchmark (#529)
    • Optimized ListConfig iteration by 12x in a benchmark (#532)
    Source code(tar.gz)
    Source code(zip)
  • v2.1.0.rc1(May 13, 2021)

    2.1.0.rc1 (2021-05-12)

    OmegaConf 2.1 is a major release introducing substantial new features, and introducing some incompatible changes. The biggest area of improvement in 2.1 is interpolations and resolvers. In addition - OmegaConf containers are now much more compatible with their plain Python container counterparts (dict and list).

    Features

    API Enhancements

    • OmegaConf.select() now takes an optional default value to return if a key is not found (#228)
    • flag_override can now override multiple flags at the same time (#400)
    • Add the OmegaConf.to_object method, which converts Structured Configs to native instances of the underlying @dataclass or @attr.s class. (#472)
    • Add OmegaConf.unsafe_merge(), a fast merge variant that destroys the input configs (#482)
    • New function OmegaConf.has_resolver() allows checking whether a resolver has already been registered. (#608)
    • Adds OmegaConf.resolve(cfg) for in-place interpolation resolution on cfg (#640)
    • force_add flag added to OmegaConf.update(), ensuring that the path is created even if it will result in insertion of new values into struct nodes. (#664)
    • Add DictConfig support for keys of type int, float and bool (#149), (#483)
    • Structured Configs fields without a value are now automatically treated as OmegaConf.MISSING (#390)
    • Add minimal support for typing.TypedDict (#473)
    • OmegaConf.to_container now takes a structured_config_mode keyword argument. Setting structured_config_mode=SCMode.DICT_CONFIG causes to_container to not convert Structured Config objects to python dicts (it leaves them as DictConfig objects). (#548)

    Interpolation and resolvers

    • Support for relative interpolation (#48)
    • Add ability to nest interpolations, e.g. ${foo.${bar}}}, ${oc.env:{$var1},${var2}}, or ${${func}:x1,x2} (#445)
    • Custom resolvers can now access the parent and the root config nodes (#266)
    • For OmegaConf.{update, select} and in interpolations, bracketed keys may be used as an alternative form to dot notation, e.g. foo.1 is equivalent to foo[1], [foo].1 and [foo][1]. (#179)
    • Custom resolvers may take non string arguments as input, and control whether to use the cache. (#445)
    • Dots may now be used in resolver names to denote namespaces (e.g: ${namespace.my_func:123}) (#539)
    • New resolver oc.select, enabling node selection with a default value to use if the node cannot be selected (#541)
    • New resolver oc.decode that can be used to automatically convert a string to bool, int, float, dict, list, etc. (#574)
    • New resolvers oc.dict.keys and oc.dict.values provide a list view of the keys or values of a DictConfig node. (#643)
    • New resolver oc.create can be used to dynamically generate config nodes (#645)
    • New resolver oc.deprecated, that enables deprecating config nodes (#681)
    • The dollar character $ is now allowed in interpolated key names, e.g. ${$var} (#600)

    Misc

    • New PyDev.Debugger resolver plugin for easier debugging in PyCharm and VSCode (#214)
    • OmegaConf now supports Python 3.9 (#447)
    • Support for Python 3.10 postponed annotation evaluation (#303)
    • Experimental support for enabling objects in config via "allow_objects" flag (#382)

    Bug Fixes

    • Fix support for forward declarations in Dict and Lists (#378)
    • Fix bug that allowed instances of Structured Configs to be assigned to DictConfig with different element type. (#386)
    • Fix exception raised when checking for the existence of a key with an incompatible type in DictConfig (#394)
    • Fix loading of an empty file via a file-pointer to return an empty dictionary (#403)
    • Fix pickling of Structured Configs with fields annotated as Dict[KT, VT] or List[T] on Python 3.6. (#407)
    • Assigning a primitive type to a Subscripted Dict now raises a descriptive message. (#409)
    • Fix assignment of an invalid value to a DictConfig to raise an exception without modifying the config object (#409)
    • Assigning a Structured Config to a Dict annotation now raises a descriptive error message. (#410)
    • OmegaConf.to_container() raises a ValueError on invalid input (#418)
    • Fix ConfigKeyError in some cases when merging lists containing interpolation values (#422)
    • DictConfig.get() in struct mode return None like standard Dict for non-existing keys (#425)
    • Fix bug where interpolations were unnecessarily resolved during merge (#431)
    • Fix bug where assignment of an invalid value to a ListConfig raised an exception but left the object modified. (#433)
    • When initializing a Structured Config with an incorrectly-typed value, the resulting ValidationError now properly reports the offending value in its error message. (#435)
    • Fix assignment of a Container to itself causing it to clear its content (#449)
    • Fix bug where DictConfig's shallow copy didn't work properly in some cases. (#450)
    • Fix support for merge tags in YAML files (#470)
    • Fix merge into a custom resolver node that raises an exception (#486)
    • Fix merge when element type is a Structured Config (#496)
    • Fix ValidationError when setting to None an optional field currently interpolated to a non-optional one (#524)
    • Fix OmegaConf.to_yaml(cfg) when keys are of Enum type (#531)
    • When a DictConfig has enum-typed keys, __delitem__ can now be called with a string naming the enum member to be deleted. (#554)
    • OmegaConf.select() of a missing (???) node from a ListConfig with throw_on_missing set to True now raises the intended exception. (#563)
    • DictConfig.{get(),pop()} now return None when the accessed key evaluates to None, instead of the specified default value (for consistency with regular Python dictionaries). (#583)
    • ListConfig.get() now return None when the accessed key evaluates to None, instead of the specified default value (for consistency with DictConfig). (#583)
    • Fix creation of structured config from a dict subclass: data from the dict is no longer thrown away. (#584)
    • Assignment of a dict/list to an existing node in a parent in struct mode no longer raises ValidationError (#586)
    • Nested flag_override now properly restore the original state (#589)
    • Fix OmegaConf.create() to set the provided parent when creating a config from a YAML string. (#648)
    • OmegaConf.select now returns None when attempting to select a child of a value or None node (#678)
    • Improve error message when creating a config from a Structured Config that fails type validation (#697)

    API changes and deprecations

    • DictConfig __getattr__ access, e.g. cfg.foo, is now raising a AttributeError if the key "foo" does not exist (#515)
    • DictConfig __getitem__ access, e.g. cfg["foo"], is now raising a KeyError if the key "foo" does not exist (#515)
    • DictConfig get access, e.g. cfg.get("foo"), now returns None if the key "foo" does not exist (#527)
    • Omegaconf.select(cfg, key, default, throw_on_missing) now requires keyword arguments for everything after key (#228)
    • Structured Configs with nested Structured config field that does not specify a default value are now interpreted as MISSING (???) instead of auto-expanding (#411)
    • OmegaConf.update() is now merging dict/list values into the destination node by default. Call with merge=False to replace instead. (#667)
    • register_resolver() is deprecated in favor of register_new_resolver(), allowing resolvers to (i) take non-string arguments like int, float, dict, interpolations, etc. and (ii) control the cache behavior (now disabled by default) (#426)
    • Merging a MISSING value onto an existing value no longer changes the target value to MISSING. (#462)
    • When resolving an interpolation of a config value with a primitive type, the interpolated value is validated and possibly converted based on the node's type. (#488)
    • DictConfig and ListConfig shallow copy is now performing a deepcopy (#492)
    • OmegaConf.select(), DictConfig.{get(),pop()}, ListConfig.{get(),pop()} no longer return the specified default value when the accessed key is an interpolation that cannot be resolved: instead, an exception is raised. (#543)
    • OmegaConf.{merge, unsafe_merge, to_yaml} now raises a ValueError when called on a str input. Previously an AssertionError was raised. (#560)
    • All exceptions raised during the resolution of an interpolation are either InterpolationResolutionError or a subclass of it. (#561)
    • key in cfg now returns True when key is an interpolation even if the interpolation target is a missing ("???") value. (#562)
    • OmegaConf.select() as well as container methods get() and pop() do not return their default value anymore when the accessed key is an interpolation that cannot be resolved: instead, an exception is raised. (#565)
    • Implicitly empty resolver arguments (e.g., ${foo:a,}) are deprecated in favor of explicit quoted strings (e.g., ${foo:a,""}) (#572)
    • The env resolver is deprecated in favor of oc.env, which keeps the string representation of environment variables, does not cache the resulting value, and handles "null" as default value. (#573)
    • OmegaConf.get_resolver() is deprecated: use the new OmegaConf.has_resolver() to check for the existence of a resolver. (#608)
    • Interpolation cycles are now forbidden and will trigger an InterpolationResolutionError on access. (#662)
    • Support for Structured Configs that subclass typing.Dict is now deprecated. (#663)
    • Remove BaseContainer.{pretty,select,update_node} that have been deprecated since OmegaConf 2.0. (#671)

    Miscellaneous changes

    • Optimized config creation time. Up to 4x faster in benchmarks (#477)
    • ListConfig.__contains__ optimized, about 15x faster in a benchmark (#529)
    • Optimized ListConfig iteration by 12x in a benchmark (#532)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.6(Jan 19, 2021)

  • v2.0.5(Nov 11, 2020)

  • v2.0.4(Nov 3, 2020)

  • v2.0.3(Oct 19, 2020)

    2.0.3 (2020-10-19)

    Minor release deprecating an uncommonly used feature.

    Deprecations and Removals

    • Automatic expansion of nested dataclasses without a default value is deprecated (#412)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.2(Sep 10, 2020)

    2.0.2 (2020-09-10)

    Features

    • OmegaConf.update() now takes a merge flag to indicate merge or set for config values (#363)

    Bug Fixes

    • Fix cfg.pretty() deprecation warning (#358)
    • Properly crash when accessing ${foo.bar} if foo is a value node (instead of silently returning ${foo}) (#364)

    Deprecations and Removals

    • OmegaConf.update() now warns if the merge flag is not specified (#367)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.1(Sep 1, 2020)

    2.0.1 (2020-09-01)

    This is mostly a bugfix release. The notable change is the config.pretty() is now deprecated in favor of OmegaConf.to_yaml().

    Bug Fixes

    • Fixes merging of dict into a Dict[str, str] (#246)
    • Fix DictConfig created from another DictConfig drops node types (#252)
    • Relax save and load APIs to accept IO[Any] (#253)
    • Report errors when loading YAML files with duplicate keys (#257)
    • Fix a bug initializing config with field typed as Any with Structured Config object (#260)
    • Merging into a MISSING Structured config node expands the node first to ensure the result is legal (#269)
    • Fix merging into a config with a read only node if merge is not mutating that node (#271)
    • Fix OmegaConf.to_container() failing in some cases when the config is read-only (#275)
    • Optional[Tuple] types are now supported as type annotation in Structured Configs. (#279)
    • Support indirect interpolation (#283)
    • OmegaConf.save() can now save dataclass and attr classes and instances (#287)
    • OmegaConf.create() doesn't modify yaml.loader.SafeLoader (#289)
    • Fix merging a sublcass Structured Config that adds a field (#291)
    • strings containing valid ints and floats represented are converted to quoted strings instead of the primitives in pretty() (#296)
    • Loading an empty YAML file now returns an empty DictConfig (#297)
    • Fix bug that allowed an annotated List and Dict field in a Structured Config to be assigned a value of a different type. (#300)
    • merge_with() now copied flags (readonly, struct) into target config (#301)
    • Fix DictConfig setdefault method to behave as it should (#304)
    • Merging a missing list onto an existing one makes the target missing (#306)
    • Fix error when merging a structured config into a field with None value (#310)
    • Fix a bug that allowed the assignment of containers onto fields annotated as primitive types (#324)
    • Merging a List of a structured with a different type now raises an error. (#327)
    • Remove dot-keys usage warning (#332)
    • Fix merging into an Optional[List[Any]] = None (#336)
    • Fix to properly merge list of dicts into a list of dataclasses (#348)
    • OmegaConf.to_yaml() now properly support Structured Configs (#350)

    Deprecations and Removals

    • cfg.pretty() is deprecated in favor of OmegaConf.to_yaml(config). (#263)

    Improved Documentation

    • Document serialization APIs (#278)
    • Document OmegaConf.is_interpolation and OmegaConf.is_none (#286)
    • Document OmegaConf.get_type() (#343)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.0(May 4, 2020)

    2.0.0 (2020-05-04)

    OmegaConf 2.0 is a major release introducing substantial new features, and introducing some incompatible changes. The biggest new feature is Structured Configs, which extends OmegaConf into an schema validation system as well as a configuration system. With Structured Configs you can create OmegaConf objects from standard dataclasses or attr classes (or objects). OmegaConf will retain the type information from the source object/class and validate that config mutations are legal.

    This is the biggest OmegaConf release ever, the number of unit tests more than trippled (485 to 1571).

    Features

    • Add support for initializing OmegaConf from typed objects and classes (#87)
    • DictConfig and ListConfig now implements typing.MutableMapping and typing.MutableSequence. (#114)
    • Enums can now be used as values and keys (#87),(#137)
    • Standardize exception messages (#186)
    • In struct mode, exceptions raised on invalid access are now consistent with Python (#138),(#94)
      • KeyError is raised when using dictionary access style for a missing key: cfg["foo"]
      • AttributeError is raised when using attribute access style for a missing attribute: cfg.foo
    • Structured configs can now inherit from Dict, making them open to arbitrary fields (#134)
    • Container.pretty() now preserves insertion order by default. override with sort_keys=True (#161)
    • Merge into node interpolation is now by value (copying target node) (#184)
    • Add OmegaConf.{is_config, is_list, is_dict} to test if an Object is an OmegaConf object, and if it's a list or a dict (#101)
    • Add OmegaConf.is_missing(cfg, key) to test if a key is missing ('???') in a config (#102)
    • OmegaConf.is_interpolation queries if a node is an interpolation (#239)
    • OmegaConf.is_missing queries if a node is missing (has the value '???') (#239)
    • OmegaConf.is_optional queries if a node in the config is optional (can take None) (#239)
    • OmegaConf.is_none queries if a node represents None (#239)
    • OmegaConf now passes strict mypy tests (#105)
    • Add isort to ensure imports are kept sorted (#107)

    Bug Fixes

    • Disable automatic conversion of date strings in yaml decoding (#95)
    • Fixed pretty to handle strings with unicode characters correctly (#111)
    • Fix eq fails if object contains unresolveable values (#124)
    • Correctly throw MissingMandatoryValue on indirect access of missing value (#99)
    • DictConfig pop now returns the underlying value and not ValueNode (#127)
    • OmegaConf.select(key) now returns the root node when key is "" (#135)
    • Add support for loading/saving config files by using pathlib.Path objects (#159)
    • Fix AttributeError when accessing config in struct-mode with get() while providing None as default (#174)

    Deprecations and Removals

    • Renamed omegaconf.Config to omegaconf.Container (#103)
    • Dropped support Python 2.7 and 3.5 (#88)
    • cfg.select(key) deprecated in favor of OmegaConf.select(cfg, key) (#116)
    • cfg.update(key, value) deprecated in favor of OmegaConf.update(cfg, key, value) (#116)
    • Container.pretty() behavior change: sorted keys -> unsorted keys by default. override with sort_keys=True. (#161)
    • cfg.to_container() is removed, deprecated since 1.4.0. Use OmegaConf.to_container() (#188)
    • cfg.save() is removed, deprecated since 1.4.0, use OmegaConf.save() (#188)
    • DictConfig item deletion now throws ConfigTypeError if the config is in struct mode (#225)
    • DictConfig.pop() now throws ConfigTypeError if the config is in struct mode (#225)
    Source code(tar.gz)
    Source code(zip)
  • v1.4.1(Dec 25, 2019)

  • v1.4.0(Nov 20, 2019)

    1.4.0 (2019-11-19)

    Features

    • ListConfig now implements + operator (Allowing concatenation with other ListConfigs) (#36)
    • OmegaConf.save() now takes a resolve flag (defaults False) (#37)
    • Add OmegaConf.masked_copy(keys) function that returns a copy of a config with a subset of the keys (#42)
    • Improve built-in env resolver to return properly typed values ("1" -> int, "1.0" -> float etc) (#44)
    • Resolvers can now accept a list of zero or more arguments, for example: "${foo:a,b,..,n}" (#46)
    • Change semantics of contains check ('x' in conf): Missing mandatory values ('???') are now considered not included and contains test returns false for them (#49)
    • Allow assignment of a tuple value into a Config (#74)

    Bug Fixes

    • Read-only list can no longer be replaced with command line override (#39)
    • Fix an error when expanding an empty dictionary in PyCharm debugger (#40)
    • Fix a bug in open_dict causing improper restoration of struct flag in some cases (#47)
    • Fix a bug preventing dotlist values from containing '=' (foo=bar=10 -> key: foo, value: bar=10) (#56)
    • Config.merge_with_dotlist() now throws if input is not a list or tuple of strings (#72)
    • Add copy method for DictConfig and improve shallow copy support (#82)

    Deprecations and Removals

    • Deprecated Config.to_container() in favor of OmegaConf.to_container() (#41)
    • Deprecated config.save(file) in favor of OmegaConf.save(config, file) (#66)
    • Remove OmegaConf.{empty(), from_string(), from_dict(), from_list()}. Use OmegaConf.create() (deprecated since 1.1.5) (#67)
    • Remove Config.merge_from(). Use Config.merge_with() (deprecated since 1.1.0) (#67)
    • Remove OmegaConf.from_filename() and OmegaConf.from_file(). Use OmegaConf.load() (deprecated since 1.1.5) (#67)

    Miscellaneous changes

    • Switch from tox to nox for test automation (#54)
    • Formatting code with Black (#54)
    • Switch from Travis to CircleCI for CI (#54)
    Source code(tar.gz)
    Source code(zip)
Owner
Omry Yadan
Author of OmegaConf and Hydra
Omry Yadan
KConfig Browser is a graphical application which allows you to modify KDE configuration files found in ~/.config

kconfig_browser KConfig Browser is a graphical application which allows you to modify KDE configuration files found in ~/.config Screenshot Why I crea

null 10 Dec 29, 2021
Configuration Management for Python ⚙

dynaconf - Configuration Management for Python. Features Inspired by the 12-factor application guide Settings management (default values, validation,

Bruno Rocha 2.4k May 26, 2022
Python Marlin Configurator to make valid configuration files to be used to compile Marlin with.

marlin-configurator Concept originally imagined by The-EG using PowerShell Build Script for Marlin Configurations The purpose of this project is to pa

DevPeeps 2 Oct 9, 2021
Simple dataclasses configuration management for Python with hocon/json/yaml/properties/env-vars/dict support.

Simple dataclasses configuration management for Python with hocon/json/yaml/properties/env-vars/dict support, based on awesome and lightweight pyhocon parsing library.

Teo Stocco 43 May 11, 2022
A Python library to parse PARI/GP configuration and header files

pari-utils A Python library to parse PARI/GP configuration and header files. This is mainly used in the code generation of https://github.com/sagemath

Sage Mathematical Software System 2 Dec 30, 2021
Pydantic-ish YAML configuration management.

Pydantic-ish YAML configuration management.

Dribia Data Research 16 Apr 13, 2022
Configuration Extractor for EXE4J PE files

EXE4J Configuration Extractor This script helps reverse engineering Portable Executable files created with EXE4J by extracting their configuration dat

Karsten Hahn 5 Feb 11, 2022
Sync any your configuration file to remote. Currently only support gist.

Sync your configuration to remote, such as vimrc. You can use EscSync to manage your configure of editor, shell, etc.

Me1onRind 1 Nov 28, 2021
Tools to assist with the configuration and maintenance of fapolicyd.

Tools to assist with the configuration and maintenance of fapolicyd.

Concurrent Technologies Corporation (CTC) 5 Jan 17, 2022
A tool to manage configuration files, build scripts etc. across multiple projects.

A tool to manage configuration files, build scripts etc. across multiple projects.

null 6 Feb 28, 2022
filetailor is a peer-based configuration management utility for plain-text files such as dotfiles.

filetailor filetailor is a peer-based configuration management utility for plain-text files (and directories) such as dotfiles. Files are backed up to

null 4 Jan 15, 2022
An application pulls configuration information from JSON files generated

AP Provisioning Automation An application pulls configuration information from JSON files generated by Ekahau and then uses Netmiko to configure the l

Cisco GVE DevNet Team 1 Dec 17, 2021
Secsie is a configuration language made for speed, beauty, and ease of use.

secsie-conf pip3 install secsie-conf Secsie is a configuration language parser for Python, made for speed and beauty. Instead of writing config files

Noah Broyles 3 Feb 19, 2022
A slightly opinionated template for iPython configuration for interactive development

A slightly opinionated template for iPython configuration for interactive development. Auto-reload and no imports for packages and modules in the project.

Seva Zhidkov 24 Feb 16, 2022
Load Django Settings from Environmental Variables with One Magical Line of Code

DjEnv: Django + Environment Load Django Settings Directly from Environmental Variables features modify django configuration without modifying source c

Daniel J. Dufour 27 Apr 20, 2022
Django-environ allows you to utilize 12factor inspired environment variables to configure your Django application.

Django-environ django-environ allows you to use Twelve-factor methodology to configure your Django application with environment variables. import envi

Daniele Faraglia 2.5k May 25, 2022
Python 3+ compatible port of the configobj library

configobj Python 3+ compatible port of the configobj library. Documentation You can find a full manual on how to use ConfigObj at readthedocs. If you

Differently Sized Kittens 273 May 29, 2022
🤫 Easily manage configs and secrets in your Python projects (with CLI support)

Installation pip install confidential How does it work? Confidential manages secrets for your project, using AWS Secrets Manager. First, store a secr

Candid™️ 61 May 22, 2022
A modern simfile parsing & editing library for Python 3

A modern simfile parsing & editing library for Python 3

ash garcia 30 May 11, 2022