Simple dataclasses configuration management for Python with hocon/json/yaml/properties/env-vars/dict support.

Teo Stocco

Last update: Dec 23, 2022

Related tags

Configuration python yaml serialization json parsing deserialization properties hocon dataclasses env-vars

Overview

Dataconf

Simple dataclasses configuration management for Python with hocon/json/yaml/properties/env-vars/dict support, based on awesome and lightweight pyhocon parsing library.

Getting started

Requires at least Python 3.8.

# pypi
pip install dataconf
poetry add dataconf

# remote master
pip install --upgrade git+https://github.com/zifeo/dataconf.git
poetry add git+https://github.com/zifeo/dataconf.git

# local repo/dev
poetry install
pre-commit install

Usage

import os
from dataclasses import dataclass, field
from typing import List, Dict, Text, Union
from dateutil.relativedelta import relativedelta
from datetime import datetime
import dataconf

conf = """
str_name = test
str_name = ${?HOME}
dash-to-underscore = true
float_num = 2.2
iso_datetime = "2000-01-01T20:00:00"
# this is a comment
list_data = [
    a
    b
]
nested {
    a = test
    b : 1
}
nested_list = [
    {
        a = test1
        b : 2.5
    }
]
duration = 2s
union = 1
people {
    name = Thailand
}
zone {
    area_code = 42
}
"""

class AbstractBaseClass:
    pass
    
@dataclass
class Person(AbstractBaseClass):
    name: Text
        
@dataclass
class Zone(AbstractBaseClass):
    area_code: int

@dataclass
class Nested:
    a: Text
    b: float

@dataclass
class Config:
    str_name: Text
    dash_to_underscore: bool
    float_num: float
    iso_datetime: datetime
    list_data: List[Text]
    nested: Nested
    nested_list: List[Nested]
    duration: relativedelta
    union: Union[Text, int]
    people: AbstractBaseClass
    zone: AbstractBaseClass
    default: Text = 'hello'
    default_factory: Dict[Text, Text] = field(default_factory=dict)

print(dataconf.string(conf, Config))
# Config(
#   str_name='/users/root',
#   dash_to_underscore=True,
#   float_num=2.2,
#   list_data=['a', 'b'],
#   nested=Nested(a='test'),
#   nested_list=[Nested(a='test1', b=2.5)],
#   duration=relativedelta(seconds=+2), 
#   union=1, 
#   people=Person(name='Thailand'), 
#   zone=Zone(area_code=42),
#   default='hello', 
#   default_factory={}
# )

@dataclass
class Example:
    hello: string
    world: string

os.environ['DC_WORLD'] = 'monde'

print(
    dataconf
    .multi
    .url('https://raw.githubusercontent.com/zifeo/dataconf/master/confs/simple.hocon')
    .env('DC')
    .on(Example)
)
# Example(hello='bonjour',world='monde')

API

import dataconf

conf = dataconf.string('{ name: Test }', Config)
conf = dataconf.env('PREFIX_', Config)
conf = dataconf.dict({'name': 'Test'}, Config)
conf = dataconf.url('https://raw.githubusercontent.com/zifeo/dataconf/master/confs/test.hocon', Config)
conf = dataconf.file('confs/test.{hocon,json,yaml,properties}', Config)

# Aggregation
conf = dataconf.multi.string(...).env(...).url(...).file(...).dict(...).on(Config)

# Same api as Python json/yaml packages (e.g. `load`, `loads`, `dump`, `dumps`)
conf = dataconf.load('confs/test.{hocon,json,yaml,properties}', Config)
dataconf.dump('confs/test.hocon', conf, out='hocon')
dataconf.dump('confs/test.json', conf, out='json')
dataconf.dump('confs/test.yaml', conf, out='yaml')
dataconf.dump('confs/test.properties', conf, out='properties')

For full HOCON capabilities see here.

Env dict/list parsing

PREFIX_VAR=a
PREFIX_VAR_NAME=b
PREFIX_TEST__NAME=c
PREFIX_LS_0=d
PREFIX_LS_1=e
PREFIX_LSLS_0_0=f
PREFIX_LSOB_0__NAME=g
PREFIX_NESTED_="{ name: Test }"
PREFIX_SUB_="{ value: ${PREFIX_VAR} }"

is equivalent to

{
    var = a
    var_name = b
    test {
        name = c
    }
    ls = [
        d
        e
    ]
    lsls = [
        [
            f
        ]
    ]
    lsob = [
        {
            name = g
        }
    ]
    nested {
        # parse nested config by suffixing env var with `_`
        name: Test
    }
    sub {
        # will have value "a" at parsing, useful for aliases
        value = ${PREFIX_VAR}
    }
}

CLI usage

Can be used for validation or converting between supported file formats (-o).

dataconf -c confs/test.hocon -m tests.configs -d TestConf -o hocon
# dataconf.exceptions.TypeConfigException: expected type 
   
     at .duration, got

Comments

Scala sealed trait ability (mostly) in dataconf using dataclass
@zifeo added the ability for nested dataclass methods (sealed trait pureconfig functionality in Scala), updated README, incremented to 0.1.6, and created version.py which will provide version and get the latest from the pyproject.toml

Also, on my local, the following test always fails. How do I get it to pass?

tests/test_parser.py:186 (TestParser.test_missing_type) self = <tests.test_parser.TestParser object at 0x7fe398c417c0> def test_missing_type(self) -> None: with pytest.raises(MissingTypeException): > loads("", Dict) E Failed: DID NOT RAISE <class 'dataconf.exceptions.MissingTypeException'>
opened by dwsmith1983 15

pyhocon can't parse nested YAML maps

I discovered this while trying to use dataconf to parse a YAML config file with one level of nesting.

For example, adding the following to test_parse.py:

    def test_yaml_nested(self) -> None:

        @dataclass
        class B:
            c: Text

        @dataclass
        class A:
            b: B

        conf = """
        b:
          c: test
        """
        assert loads(conf, A) == A(b=B(c="test"))

This test should pass, but results in the following test failure:

=================================================== FAILURES ====================================================
__________________________________________ TestParser.test_yaml_nested __________________________________________

self = <tests.test_parse.TestParser object at 0x7f440c099f90>

    def test_yaml_nested(self) -> None:
    
        @dataclass
        class B:
            c: Text
    
        @dataclass
        class A:
            b: B
    
        conf = """
        b:
          c: test
        """
>       assert loads(conf, A) == A(b=B(c="test"))

tests/test_parse.py:298: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
dataconf/main.py:102: in loads
    return string(s, clazz, **kwargs)
dataconf/main.py:82: in string
    return multi.string(s, **kwargs).on(clazz)
dataconf/main.py:67: in on
    return parse(conf, clazz, self.strict, **self.kwargs)
dataconf/main.py:17: in parse
    return utils.__parse(conf, clazz, "", strict, ignore_unexpected)
dataconf/utils.py:76: in __parse
    fs[f.name] = __parse(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

value = 'c: test', clazz = <class 'tests.test_parse.TestParser.test_yaml_nested.<locals>.B'>, path = '.b'
strict = True, ignore_unexpected = False

    def __parse(value: any, clazz: Type, path: str, strict: bool, ignore_unexpected: bool):
    
        if is_dataclass(clazz):
    
            if not isinstance(value, ConfigTree):
>               raise TypeConfigException(
                    f"expected type {clazz} at {path}, got {type(value)}"
                )
E               dataconf.exceptions.TypeConfigException: expected type <class 'tests.test_parse.TestParser.test_yaml_nested.<locals>.B'> at .b, got <class 'str'>

dataconf/utils.py:55: TypeConfigException
============================================ short test summary info ============================================
FAILED tests/test_parse.py::TestParser::test_yaml_nested - dataconf.exceptions.TypeConfigException: expected t...
========================================= 1 failed, 30 passed in 0.40s ==========================================

Changing the input to:

        conf = """
        b:
            {c: test}
        """

causes the test to pass, but I suspect this is because it is coincedentally also valid HOCON.

opened by deleted 10

python 3.10 support through github actions

@zifeo looks like the python on github ci isn't updated for python 3.10 release. I just pushed a dev branch to test if everything would work but issues with python 3.10 being looked up as 3.1
enhancement

opened by dwsmith1983 9
feat: add support for disambiguating subtypes
In the rare case that there is more than one subtype that matches the given fields:

class AmbigImplBase: pass @dataclass(init=True, repr=True) class AmbigImplOne(AmbigImplBase): bar: str @dataclass(init=True, repr=True) class AmbigImplTwo(AmbigImplBase): bar: str

With config:

{ a: Europe foo { bar: Baz } }

Add support to select which subclass to use based on just the class name or the module path using the _type field.

{ a: Europe foo { _type: AmbigImplTwo bar: Baz } }

The module path is tail-matched, meaning a class with the fully qualified name of a.b.c.d.AmbigImplTwo can be matched by:

a.b.c.d.AmbigImplTwo

c.d.AmbigImplTwo

d.AmbigImplTwo

AmbigImplTwo

If the _type field matches don't narrow it down enough it will still throw an error.
opened by slyons 6

Python 3.10 Import error with collections

With Python 3.10, we encounter an import error related to collections.

Similar issues: https://github.com/rmartin16/qbittorrent-api/issues/45 https://github.com/carbonblack/cbapi-python/issues/298

[ImportError]
cannot import name 'Mapping' from 'collections' (/opt/hostedtoolcache/Python/3.10.0/x64/lib/python3.10/collections/__init__.py)

From this issue, they found an issue with

*rmartin Feb 2021
AttrDict finally broke with Python 3.10 since abstract base classes can no
longer be imported from collections but should use collections.abc instead.
Since AttrDict is abandoned, I've consolidated the code here for future use.
AttrMap and AttrDefault are left for posterity but commented out.
"""

from abc import ABCMeta
from abc import abstractmethod
from re import match as re_match

try:  # python 3
    from collections.abc import Mapping
    from collections.abc import MutableMapping
    from collections.abc import Sequence
except ImportError:  # python 2
    from collections import Mapping
    from collections import MutableMapping
    from collections import Sequence

bug

opened by dwsmith1983 5

Can not pass sample codes from README.md.
I can not pass below sample codes from README.md.

@dataclass class Example: hello: string world: string os.environ['DC_WORLD'] = 'monde' print( dataconf .multi .url('https://raw.githubusercontent.com/zifeo/dataconf/master/confs/simple.hocon') .env('DC') .on(Example) ) # Example(hello='bonjour',world='monde')

First error is about below line. hello: string =>no string

After replace string to str, I got below error msgs.

" dataconf ../../../opt/anaconda3/envs/test/lib/python3.8/site-packages/dataconf/main.py:89: in on conf = ConfigTree.merge_configs(conf, nxt) ../../../opt/anaconda3/envs/test/lib/python3.8/site-packages/pyhocon/config_tree.py:61: in merge_configs a[key] = value E TypeError: list indices must be integers or slices, not str "
opened by ChiahungTai 4
Support mypy imports
Description

mypy report error when importing dataconf

Expected Behavior

mypy checks pass without any error without the need to use ignore_missing_imports = True option

Actual Behavior

error: Skipping analyzing "dataconf": module is installed, but missing library stubs or py.typed marker [import]
opened by kavinvin 3

load when calling hocon file

In your parsing operation, you dont allow for optional parameters that are list or dicts.

    if origin is list:
        if len(args) != 1:
            raise MissingTypeException("excepted list with type information: List[?]")
        return [__parse(v, args[0], f"{path}[]") for v in value]

    if origin is dict:
        if len(args) != 2:
            raise MissingTypeException(
                "excepted dict with type information: Dict[?, ?]"
            )
        return {k: __parse(v, args[1], f"{path}.{k}") for k, v in value.items()}

Consider the following dataclass:

        @dataclass
        class Base:
            data_root: Text
            pipeline_name: Text
            data_type: Text
            production: bool
            conn: Optional[Conn] = None
            data_split: Optional[Dict[Text, int]] = None
            tfx_root: Optional[Text] = None
            metadata_root: Optional[Text] = None
            beam_args: Optional[List[Text]] = field(
                default_factory=lambda: ["--direct_running_mode=multi_processing", "--direct_num_workers=0"]
            )

An optional parameter with None shouldn't return a failure. We use HOCON configs all the time with case classes in JVM languages and optional parameter parsing should be able to be by passed if None is specified.

opened by dwsmith1983 3

Strict mode

This allows to disable the strict mode (off by default when using .env as a source). Not sure this is the best solution, I will let open for a while to see if a better solution can be designed for #35.

Fix https://github.com/zifeo/dataconf/issues/35.

opened by zifeo 2
Replicated the behavior with Scala sealed traits and pureconfig on HOCON

I replicated the behavior of using seal trait case classes with Scala and pureconfig. It may not be the cleanest but it is working. I have some Todo notes in case you have any ideas.

opened by dwsmith1983 2

Error with `from future import annotations`

from __future__ import annotations

import dataconf

from dataclasses import dataclass


@dataclass
class Model():
    token: str
    
dataconf.env("TEST_", Model)

Error:

Traceback (most recent call last):
  File "main.py", line 13, in <module>
    dataconf.env("ITBUTKA_", Model)
  File "/.venv/lib/python3.9/site-packages/dataconf/main.py", line 64, in env
    return multi.env(prefix, **kwargs).on(clazz)
  File "/.venv/lib/python3.9/site-packages/dataconf/main.py", line 57, in on
    return parse(conf, clazz, self.strict, **self.kwargs)
  File "/.venv/lib/python3.9/site-packages/dataconf/main.py", line 16, in parse
    return utils.__parse(conf, clazz, "", strict, ignore_unexpected)
  File "/.venv/lib/python3.9/site-packages/dataconf/utils.py", line 68, in __parse
    fs[f.name] = __parse(
  File "/.venv/lib/python3.9/site-packages/dataconf/utils.py", line 186, in __parse
    for child_clazz in sorted(clazz.__subclasses__(), key=lambda c: c.__name__):
AttributeError: 'str' object has no attribute '__subclasses__'

Process finished with exit code 1

opened by LEv145 2