Simple dataclasses configuration management for Python with hocon/json/yaml/properties/env-vars/dict support.

Overview

Dataconf

Actions Status PyPI version

Simple dataclasses configuration management for Python with hocon/json/yaml/properties/env-vars/dict support, based on awesome and lightweight pyhocon parsing library.

Getting started

Requires at least Python 3.8.

# pypi
pip install dataconf
poetry add dataconf

# remote master
pip install --upgrade git+https://github.com/zifeo/dataconf.git
poetry add git+https://github.com/zifeo/dataconf.git

# local repo/dev
poetry install
pre-commit install

Usage

import os
from dataclasses import dataclass, field
from typing import List, Dict, Text, Union
from dateutil.relativedelta import relativedelta
from datetime import datetime
import dataconf

conf = """
str_name = test
str_name = ${?HOME}
dash-to-underscore = true
float_num = 2.2
iso_datetime = "2000-01-01T20:00:00"
# this is a comment
list_data = [
    a
    b
]
nested {
    a = test
    b : 1
}
nested_list = [
    {
        a = test1
        b : 2.5
    }
]
duration = 2s
union = 1
people {
    name = Thailand
}
zone {
    area_code = 42
}
"""

class AbstractBaseClass:
    pass
    
@dataclass
class Person(AbstractBaseClass):
    name: Text
        
@dataclass
class Zone(AbstractBaseClass):
    area_code: int

@dataclass
class Nested:
    a: Text
    b: float

@dataclass
class Config:
    str_name: Text
    dash_to_underscore: bool
    float_num: float
    iso_datetime: datetime
    list_data: List[Text]
    nested: Nested
    nested_list: List[Nested]
    duration: relativedelta
    union: Union[Text, int]
    people: AbstractBaseClass
    zone: AbstractBaseClass
    default: Text = 'hello'
    default_factory: Dict[Text, Text] = field(default_factory=dict)

print(dataconf.string(conf, Config))
# Config(
#   str_name='/users/root',
#   dash_to_underscore=True,
#   float_num=2.2,
#   list_data=['a', 'b'],
#   nested=Nested(a='test'),
#   nested_list=[Nested(a='test1', b=2.5)],
#   duration=relativedelta(seconds=+2), 
#   union=1, 
#   people=Person(name='Thailand'), 
#   zone=Zone(area_code=42),
#   default='hello', 
#   default_factory={}
# )

@dataclass
class Example:
    hello: string
    world: string

os.environ['DC_WORLD'] = 'monde'

print(
    dataconf
    .multi
    .url('https://raw.githubusercontent.com/zifeo/dataconf/master/confs/simple.hocon')
    .env('DC')
    .on(Example)
)
# Example(hello='bonjour',world='monde')

API

import dataconf

conf = dataconf.string('{ name: Test }', Config)
conf = dataconf.env('PREFIX_', Config)
conf = dataconf.dict({'name': 'Test'}, Config)
conf = dataconf.url('https://raw.githubusercontent.com/zifeo/dataconf/master/confs/test.hocon', Config)
conf = dataconf.file('confs/test.{hocon,json,yaml,properties}', Config)

# Aggregation
conf = dataconf.multi.string(...).env(...).url(...).file(...).dict(...).on(Config)

# Same api as Python json/yaml packages (e.g. `load`, `loads`, `dump`, `dumps`)
conf = dataconf.load('confs/test.{hocon,json,yaml,properties}', Config)
dataconf.dump('confs/test.hocon', conf, out='hocon')
dataconf.dump('confs/test.json', conf, out='json')
dataconf.dump('confs/test.yaml', conf, out='yaml')
dataconf.dump('confs/test.properties', conf, out='properties')

For full HOCON capabilities see here.

Env dict/list parsing

PREFIX_VAR=a
PREFIX_VAR_NAME=b
PREFIX_TEST__NAME=c
PREFIX_LS_0=d
PREFIX_LS_1=e
PREFIX_LSLS_0_0=f
PREFIX_LSOB_0__NAME=g
PREFIX_NESTED_="{ name: Test }"
PREFIX_SUB_="{ value: ${PREFIX_VAR} }"

is equivalent to

{
    var = a
    var_name = b
    test {
        name = c
    }
    ls = [
        d
        e
    ]
    lsls = [
        [
            f
        ]
    ]
    lsob = [
        {
            name = g
        }
    ]
    nested {
        # parse nested config by suffixing env var with `_`
        name: Test
    }
    sub {
        # will have value "a" at parsing, useful for aliases
        value = ${PREFIX_VAR}
    }
}

CLI usage

Can be used for validation or converting between supported file formats (-o).

dataconf -c confs/test.hocon -m tests.configs -d TestConf -o hocon
# dataconf.exceptions.TypeConfigException: expected type 
   
     at .duration, got 
    
   
Comments
  • Scala sealed trait ability (mostly) in dataconf using dataclass

    Scala sealed trait ability (mostly) in dataconf using dataclass

    @zifeo added the ability for nested dataclass methods (sealed trait pureconfig functionality in Scala), updated README, incremented to 0.1.6, and created version.py which will provide version and get the latest from the pyproject.toml

    Also, on my local, the following test always fails. How do I get it to pass?

    tests/test_parser.py:186 (TestParser.test_missing_type)
    self = <tests.test_parser.TestParser object at 0x7fe398c417c0>
    
        def test_missing_type(self) -> None:
        
            with pytest.raises(MissingTypeException):
    >           loads("", Dict)
    E           Failed: DID NOT RAISE <class 'dataconf.exceptions.MissingTypeException'>
    
    opened by dwsmith1983 15
  • pyhocon can't parse nested YAML maps

    pyhocon can't parse nested YAML maps

    I discovered this while trying to use dataconf to parse a YAML config file with one level of nesting.

    For example, adding the following to test_parse.py:

        def test_yaml_nested(self) -> None:
    
            @dataclass
            class B:
                c: Text
    
            @dataclass
            class A:
                b: B
    
            conf = """
            b:
              c: test
            """
            assert loads(conf, A) == A(b=B(c="test"))
    

    This test should pass, but results in the following test failure:

    =================================================== FAILURES ====================================================
    __________________________________________ TestParser.test_yaml_nested __________________________________________
    
    self = <tests.test_parse.TestParser object at 0x7f440c099f90>
    
        def test_yaml_nested(self) -> None:
        
            @dataclass
            class B:
                c: Text
        
            @dataclass
            class A:
                b: B
        
            conf = """
            b:
              c: test
            """
    >       assert loads(conf, A) == A(b=B(c="test"))
    
    tests/test_parse.py:298: 
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    dataconf/main.py:102: in loads
        return string(s, clazz, **kwargs)
    dataconf/main.py:82: in string
        return multi.string(s, **kwargs).on(clazz)
    dataconf/main.py:67: in on
        return parse(conf, clazz, self.strict, **self.kwargs)
    dataconf/main.py:17: in parse
        return utils.__parse(conf, clazz, "", strict, ignore_unexpected)
    dataconf/utils.py:76: in __parse
        fs[f.name] = __parse(
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    
    value = 'c: test', clazz = <class 'tests.test_parse.TestParser.test_yaml_nested.<locals>.B'>, path = '.b'
    strict = True, ignore_unexpected = False
    
        def __parse(value: any, clazz: Type, path: str, strict: bool, ignore_unexpected: bool):
        
            if is_dataclass(clazz):
        
                if not isinstance(value, ConfigTree):
    >               raise TypeConfigException(
                        f"expected type {clazz} at {path}, got {type(value)}"
                    )
    E               dataconf.exceptions.TypeConfigException: expected type <class 'tests.test_parse.TestParser.test_yaml_nested.<locals>.B'> at .b, got <class 'str'>
    
    dataconf/utils.py:55: TypeConfigException
    ============================================ short test summary info ============================================
    FAILED tests/test_parse.py::TestParser::test_yaml_nested - dataconf.exceptions.TypeConfigException: expected t...
    ========================================= 1 failed, 30 passed in 0.40s ==========================================
    

    Changing the input to:

            conf = """
            b:
                {c: test}
            """
    

    causes the test to pass, but I suspect this is because it is coincedentally also valid HOCON.

    opened by deleted 10
  • python 3.10 support through github actions

    python 3.10 support through github actions

    @zifeo looks like the python on github ci isn't updated for python 3.10 release. I just pushed a dev branch to test if everything would work but issues with python 3.10 being looked up as 3.1

    enhancement 
    opened by dwsmith1983 9
  • feat: add support for disambiguating subtypes

    feat: add support for disambiguating subtypes

    In the rare case that there is more than one subtype that matches the given fields:

    class AmbigImplBase:
        pass
    
    
    @dataclass(init=True, repr=True)
    class AmbigImplOne(AmbigImplBase):
        bar: str
    
    
    @dataclass(init=True, repr=True)
    class AmbigImplTwo(AmbigImplBase):
        bar: str
    

    With config:

    {
        a: Europe
        foo {
            bar: Baz
        }
    }
    

    Add support to select which subclass to use based on just the class name or the module path using the _type field.

    {
        a: Europe
        foo {
            _type: AmbigImplTwo
            bar: Baz
        }
    }
    

    The module path is tail-matched, meaning a class with the fully qualified name of a.b.c.d.AmbigImplTwo can be matched by:

    • a.b.c.d.AmbigImplTwo
    • c.d.AmbigImplTwo
    • d.AmbigImplTwo
    • AmbigImplTwo

    If the _type field matches don't narrow it down enough it will still throw an error.

    opened by slyons 6
  • Parsing dictionaries with Any

    Parsing dictionaries with Any

    It appears that Any can cause some problems.

    from dataclasses import dataclass, field
    import dataconf
    from typing import Any, Dict, Text
    
    
    @dataclass
    class Test:
        name: Text
        items: Dict[Text, Any] = field(default_factory=dict)
    
    
    config = """
    name: letters
    items: {
        a: d, 
        b: e, 
        c: f
    }
    """
    
    conf = dataconf.string(config, Test)
    

    Traceback:

    AttributeError                            Traceback (most recent call last)
    /var/folders/kh/n0p_nl6d7sg0hfqljfwmlhcr0000gq/T/ipykernel_6851/3484491678.py in <module>
    ----> 1 conf = dataconf.string(config, Test)
    
    ~/opt/anaconda3/lib/python3.8/site-packages/dataconf/main.py in string(s, clazz)
         65 
         66 def string(s: str, clazz):
    ---> 67     return multi.string(s).on(clazz)
         68 
         69 
    
    ~/opt/anaconda3/lib/python3.8/site-packages/dataconf/main.py in on(self, clazz)
         50         for nxt in nxts:
         51             conf = ConfigTree.merge_configs(conf, nxt)
    ---> 52         return parse(conf, clazz)
         53 
         54 
    
    ~/opt/anaconda3/lib/python3.8/site-packages/dataconf/main.py in parse(conf, clazz)
         12 def parse(conf: ConfigTree, clazz):
         13     try:
    ---> 14         return utils.__parse(conf, clazz, "")
         15     except pyparsing.ParseSyntaxException as e:
         16         raise MalformedConfigException(
    
    ~/opt/anaconda3/lib/python3.8/site-packages/dataconf/utils.py in __parse(value, clazz, path)
         66 
         67             if not isinstance(val, _MISSING_TYPE):
    ---> 68                 fs[f.name] = __parse(val, f.type, f"{path}.{f.name}")
         69 
         70             elif is_optional(f.type):
    
    ~/opt/anaconda3/lib/python3.8/site-packages/dataconf/utils.py in __parse(value, clazz, path)
        107         left, right = args
        108         try:
    --> 109             return __parse(value, left if right is NoneType else right, path)
        110         except TypeConfigException:
        111             # cannot parse Optional
    
    ~/opt/anaconda3/lib/python3.8/site-packages/dataconf/utils.py in __parse(value, clazz, path)
        101             )
        102         if value is not None:
    --> 103             return {k: __parse(v, args[1], f"{path}.{k}") for k, v in value.items()}
        104         return None
        105 
    
    ~/opt/anaconda3/lib/python3.8/site-packages/dataconf/utils.py in <dictcomp>(.0)
        101             )
        102         if value is not None:
    --> 103             return {k: __parse(v, args[1], f"{path}.{k}") for k, v in value.items()}
        104         return None
        105 
    
    ~/opt/anaconda3/lib/python3.8/site-packages/dataconf/utils.py in __parse(value, clazz, path)
        155 
        156     child_failures = []
    --> 157     for child_clazz in sorted(clazz.__subclasses__(), key=lambda c: c.__name__):
        158         if is_dataclass(child_clazz):
        159             try:
    
    AttributeError: '_SpecialForm' object has no attribute '__subclasses__'
    
    bug 
    opened by dwsmith1983 6
  • Python 3.10 Import error with collections

    Python 3.10 Import error with collections

    With Python 3.10, we encounter an import error related to collections.

    Similar issues: https://github.com/rmartin16/qbittorrent-api/issues/45 https://github.com/carbonblack/cbapi-python/issues/298

    [ImportError]
    cannot import name 'Mapping' from 'collections' (/opt/hostedtoolcache/Python/3.10.0/x64/lib/python3.10/collections/__init__.py)
    

    From this issue, they found an issue with

    *rmartin Feb 2021
    AttrDict finally broke with Python 3.10 since abstract base classes can no
    longer be imported from collections but should use collections.abc instead.
    Since AttrDict is abandoned, I've consolidated the code here for future use.
    AttrMap and AttrDefault are left for posterity but commented out.
    """
    
    from abc import ABCMeta
    from abc import abstractmethod
    from re import match as re_match
    
    try:  # python 3
        from collections.abc import Mapping
        from collections.abc import MutableMapping
        from collections.abc import Sequence
    except ImportError:  # python 2
        from collections import Mapping
        from collections import MutableMapping
        from collections import Sequence
    
    bug 
    opened by dwsmith1983 5
  • Can not pass sample codes from README.md.

    Can not pass sample codes from README.md.

    I can not pass below sample codes from README.md.

    @dataclass
    class Example:
        hello: string
        world: string
    
    os.environ['DC_WORLD'] = 'monde'
    
    print(
        dataconf
        .multi
        .url('https://raw.githubusercontent.com/zifeo/dataconf/master/confs/simple.hocon')
        .env('DC')
        .on(Example)
    )
    # Example(hello='bonjour',world='monde')
    

    First error is about below line. hello: string =>no string

    After replace string to str, I got below error msgs.

    " dataconf ../../../opt/anaconda3/envs/test/lib/python3.8/site-packages/dataconf/main.py:89: in on conf = ConfigTree.merge_configs(conf, nxt) ../../../opt/anaconda3/envs/test/lib/python3.8/site-packages/pyhocon/config_tree.py:61: in merge_configs a[key] = value E TypeError: list indices must be integers or slices, not str "

    opened by ChiahungTai 4
  • Support mypy imports

    Support mypy imports

    Description

    mypy report error when importing dataconf

    Expected Behavior

    mypy checks pass without any error without the need to use ignore_missing_imports = True option

    Actual Behavior

    error: Skipping analyzing "dataconf": module is installed, but missing library stubs or py.typed marker  [import]
    
    opened by kavinvin 3
  • load when calling hocon file

    load when calling hocon file

    In your parsing operation, you dont allow for optional parameters that are list or dicts.

        if origin is list:
            if len(args) != 1:
                raise MissingTypeException("excepted list with type information: List[?]")
            return [__parse(v, args[0], f"{path}[]") for v in value]
    
        if origin is dict:
            if len(args) != 2:
                raise MissingTypeException(
                    "excepted dict with type information: Dict[?, ?]"
                )
            return {k: __parse(v, args[1], f"{path}.{k}") for k, v in value.items()}
    

    Consider the following dataclass:

            @dataclass
            class Base:
                data_root: Text
                pipeline_name: Text
                data_type: Text
                production: bool
                conn: Optional[Conn] = None
                data_split: Optional[Dict[Text, int]] = None
                tfx_root: Optional[Text] = None
                metadata_root: Optional[Text] = None
                beam_args: Optional[List[Text]] = field(
                    default_factory=lambda: ["--direct_running_mode=multi_processing", "--direct_num_workers=0"]
                )
    

    An optional parameter with None shouldn't return a failure. We use HOCON configs all the time with case classes in JVM languages and optional parameter parsing should be able to be by passed if None is specified.

    opened by dwsmith1983 3
  • Strict mode

    Strict mode

    This allows to disable the strict mode (off by default when using .env as a source). Not sure this is the best solution, I will let open for a while to see if a better solution can be designed for #35.

    Fix https://github.com/zifeo/dataconf/issues/35.

    opened by zifeo 2
  • Replicated the behavior with Scala sealed traits and pureconfig on HOCON

    Replicated the behavior with Scala sealed traits and pureconfig on HOCON

    I replicated the behavior of using seal trait case classes with Scala and pureconfig. It may not be the cleanest but it is working. I have some Todo notes in case you have any ideas.

    opened by dwsmith1983 2
  • Error with `from __future__ import annotations`

    Error with `from __future__ import annotations`

    from __future__ import annotations
    
    import dataconf
    
    from dataclasses import dataclass
    
    
    @dataclass
    class Model():
        token: str
        
    dataconf.env("TEST_", Model)
    

    Error:

    Traceback (most recent call last):
      File "main.py", line 13, in <module>
        dataconf.env("ITBUTKA_", Model)
      File "/.venv/lib/python3.9/site-packages/dataconf/main.py", line 64, in env
        return multi.env(prefix, **kwargs).on(clazz)
      File "/.venv/lib/python3.9/site-packages/dataconf/main.py", line 57, in on
        return parse(conf, clazz, self.strict, **self.kwargs)
      File "/.venv/lib/python3.9/site-packages/dataconf/main.py", line 16, in parse
        return utils.__parse(conf, clazz, "", strict, ignore_unexpected)
      File "/.venv/lib/python3.9/site-packages/dataconf/utils.py", line 68, in __parse
        fs[f.name] = __parse(
      File "/.venv/lib/python3.9/site-packages/dataconf/utils.py", line 186, in __parse
        for child_clazz in sorted(clazz.__subclasses__(), key=lambda c: c.__name__):
    AttributeError: 'str' object has no attribute '__subclasses__'
    
    Process finished with exit code 1
    
    opened by LEv145 2
Releases(v2.1.3)
Owner
Teo Stocco
Chief of Technology, Data & Innovation at Smood. MSc in data science and computational neuroscience (EPFL).
Teo Stocco
Read configuration settings from python configuration files.

Maison Read configuration settings from python configuration files. Motivation When developing a python application, e.g a command-line tool, it can b

null 9 Jan 4, 2023
Configuration Management for Python ⚙

dynaconf - Configuration Management for Python. Features Inspired by the 12-factor application guide Settings management (default values, validation,

Bruno Rocha 2.8k Jan 6, 2023
An application pulls configuration information from JSON files generated

AP Provisioning Automation An application pulls configuration information from JSON files generated by Ekahau and then uses Netmiko to configure the l

Cisco GVE DevNet Team 1 Dec 17, 2021
filetailor is a peer-based configuration management utility for plain-text files such as dotfiles.

filetailor filetailor is a peer-based configuration management utility for plain-text files (and directories) such as dotfiles. Files are backed up to

null 5 Dec 23, 2022
Sync any your configuration file to remote. Currently only support gist.

Sync your configuration to remote, such as vimrc. You can use EscSync to manage your configure of editor, shell, etc.

Me1onRind 0 Nov 21, 2022
Apt2sbom python package generates SPDX or YAML files

Welcome to apt2sbom This package contains a library and a CLI tool to convert a Ubuntu software package inventory to a software bill of materials. You

Eliot Lear 15 Nov 13, 2022
Python YAML Environment (ymlenv) by Problem Fighter Library

In the name of God, the Most Gracious, the Most Merciful. PF-PY-YMLEnv Documentation Install and update using pip: pip install -U PF-PY-YMLEnv Please

Problem Fighter 2 Jan 20, 2022
A YAML validator for Programming Historian lessons.

phyaml A simple YAML validator for Programming Historian lessons. USAGE: python3 ph-lesson-yaml-validator.py lesson.md The script automatically detect

Riva Quiroga 1 Nov 7, 2021
Dag-bakery - Dag Bakery enables the capability to define Airflow DAGs via YAML.

DAG Bakery - WIP ?? dag-bakery aims to simplify our DAG development by removing all the boilerplate and duplicated code when defining multiple DAG cro

Typeform 2 Jan 8, 2022
Yamale (ya·ma·lē) - A schema and validator for YAML.

Yamale (ya·ma·lē) ⚠️ Ensure that your schema definitions come from internal or trusted sources. Yamale does not protect against intentionally maliciou

23andMe 534 Dec 21, 2022
Flexible Python configuration system. The last one you will ever need.

OmegaConf Description Project Code quality Docs and support OmegaConf is a hierarchical configuration system, with support for merging configurations

Omry Yadan 1.4k Jan 2, 2023
Python Marlin Configurator to make valid configuration files to be used to compile Marlin with.

marlin-configurator Concept originally imagined by The-EG using PowerShell Build Script for Marlin Configurations The purpose of this project is to pa

DevPeeps 2 Oct 9, 2021
A Python library to parse PARI/GP configuration and header files

pari-utils A Python library to parse PARI/GP configuration and header files. This is mainly used in the code generation of https://github.com/sagemath

Sage Mathematical Software System 3 Sep 18, 2022
Configuration Extractor for EXE4J PE files

EXE4J Configuration Extractor This script helps reverse engineering Portable Executable files created with EXE4J by extracting their configuration dat

Karsten Hahn 6 Jun 29, 2022
Tools to assist with the configuration and maintenance of fapolicyd.

Tools to assist with the configuration and maintenance of fapolicyd.

Concurrent Technologies Corporation (CTC) 7 Dec 27, 2022
A tool to manage configuration files, build scripts etc. across multiple projects.

A tool to manage configuration files, build scripts etc. across multiple projects.

null 8 Dec 14, 2022
KConfig Browser is a graphical application which allows you to modify KDE configuration files found in ~/.config

kconfig_browser KConfig Browser is a graphical application which allows you to modify KDE configuration files found in ~/.config Screenshot Why I crea

null 11 Sep 15, 2022
Secsie is a configuration language made for speed, beauty, and ease of use.

secsie-conf pip3 install secsie-conf Secsie is a configuration language parser for Python, made for speed and beauty. Instead of writing config files

Noah Broyles 3 Feb 19, 2022
A slightly opinionated template for iPython configuration for interactive development

A slightly opinionated template for iPython configuration for interactive development. Auto-reload and no imports for packages and modules in the project.

Seva Zhidkov 24 Feb 16, 2022