Transparently load variables from environment or JSON/YAML file.

Overview

Goodconf

A thin wrapper over Pydantic's settings management. Allows you to define configuration variables and load them from environment or JSON/YAML file. Also generates initial configuration files and documentation for your defined configuration.

Installation

pip install goodconf or pip install goodconf[yaml] if parsing/generating YAML files is required.

Quick Start

Let's use configurable Django settings as an example.

First, create a conf.py file in your project's directory, next to settings.py:

import base64
import os

from goodconf import GoodConf, Field
from pydantic import PostgresDsn

class AppConfig(GoodConf):
    "Configuration for My App"
    DEBUG: bool
    DATABASE_URL: PostgresDsn = "postgres://localhost:5432/mydb"
    SECRET_KEY: str = Field(
        initial=lambda: base64.b64encode(os.urandom(60)).decode(),
        description="Used for cryptographic signing. "
        "https://docs.djangoproject.com/en/2.0/ref/settings/#secret-key")

    class Config:
        default_files = ["/etc/myproject/myproject.yaml", "myproject.yaml"]

config = AppConfig()

Next, use the config in your settings.py file:

import dj_database_url
from .conf import config

config.load()

DEBUG = config.DEBUG
SECRET_KEY = config.SECRET_KEY
DATABASES = {"default": dj_database_url.parse(config.DATABASE_URL)}

In your initial developer installation instructions, give some advice such as:

python -c "import myproject; print(myproject.conf.config.generate_yaml(DEBUG=True))" > myproject.yaml

Better yet, make it a function and entry point so you can install your project and run something like generate-config > myproject.yaml.

Usage

GoodConf

Your subclassed GoodConf object can include a Config class with the following attributes:

file_env_var
The name of an environment variable which can be used for the name of the configuration file to load.
default_files
If no file is passed to the load method, try to load a configuration from these files in order.

It also has one method:

load
Trigger the load method during instantiation. Defaults to False.

Use plain-text docstring for use as a header when generating a configuration file.

Environment variables always take precedence over variables in the configuration files.

See Pydantic's docs for examples of loading:

Fields

Declare configuration values by subclassing GoodConf and defining class attributes which are standard Python type definitions or Pydantic FieldInfo instances generated by the Field function.

Goodconf can use one extra argument provided to the Field to define an function which can generate an initial value for the field:

initial
Callable to use for initial value when generating a config

Django Usage

A helper is provided which monkey-patches Django's management commands to accept a --config argument. Replace your manage.py with the following:

# Define your GoodConf in `myproject/conf.py`
from myproject.conf import config

if __name__ == '__main__':
    config.django_manage()

Why?

I took inspiration from logan (used by Sentry) and derpconf (used by Thumbor). Both, however used Python files for configuration. I wanted a safer format and one that was easier to serialize data into from a configuration management system.

Environment Variables

I don't like working with environment variables. First, there are potential security issues:

  1. Accidental leaks via logging or error reporting services.
  2. Child process inheritance (see ImageTragick for an idea why this could be bad).

Second, in practice on deployment environments, environment variables end up getting written to a number of files (cron, bash profile, service definitions, web server config, etc.). Not only is it cumbersome, but also increases the possibility of leaks via incorrect file permissions.

I prefer a single structured file which is explicitly read by the application. I also want it to be easy to run my applications on services like Heroku where environment variables are the preferred configuration method.

This module let's me do things the way I prefer in environments I control, but still run them with environment variables on environments I don't control with minimal fuss.

Contribute

Create virtual environment and install package and dependencies.

pip install -e ".[tests]"

Run tests

pytest
Comments
  • Deprecation Warning depending on ruamel version

    Deprecation Warning depending on ruamel version

    Thanks for this module , super useful quickly making configs.

    So nothing is broken just getting deprecation warnings when running code under pytest. I get the following:

    PendingDeprecationWarning: safe_load will be removed, use

    yaml=YAML(typ='safe', pure=True) yaml.load(...)

    Maybe can do setup based based on ruamel version ? Details in project description: https://pypi.org/project/ruamel.yaml/

    Also unrelated, if none of the "default_files" are found there is no error and FileNotFoundError is not thrown, is that intended ?

    opened by jonans 3
  • Switch to pydantic

    Switch to pydantic

    Pydantic is widely used and does much of what this lib was doing. The maintainer has explicitly stated they are not interested in file loading. The other parts that still remain in Goodconf are generating configuration files and the markdown documentation.

    opened by ipmb 1
  • Doc incorrect

    Doc incorrect

    Hi, thanks for goodconf!

    In the example in the docs a Config object is initialised and assigned to a variable config. However the load method returns nothing. Therefore the config var is None.

    Also in the modified manage.py I had to include this line for it to work normally:

    if __name__ == '__main__':
        os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'main.settings')
        settings.config.django_manage()
    
    opened by mjmare 1
  • Export typing information when packaging

    Export typing information when packaging

    Whenever I import Goodconf I get the next mypy error:

    Skipping analyzing "goodconf": found module but no type hints or library stubs
    

    It would be nice if goodconf exposed the typing information at package level, probably adding the py.typed file would be enough

    opened by lyz-code 0
  • Switch to pydantic

    Switch to pydantic

    Pydantic is widely used and does much of what this lib was doing. The maintainer has explicitly stated they are not interested in file loading. The other parts that still remain in Goodconf are generating configuration files and the markdown documentation.

    opened by ipmb 0
  • Automatically generate and expose settings - even less setup code!

    Automatically generate and expose settings - even less setup code!

    Thanks for this solid tool! I wonder if we can take it further to meet my madcap schemes!

    Or maybe this is already possible but I just haven't understood yet.

    Here's my idealised solution:

    Firstly, no need for two files: conf.py goes away and settings.py remains. I preferably only want to look at a single file for all the configuration declarations (goodconf actually makes it feasible to have this file declarative, short and easy to grok, woot woot).

    Then, in my project-name/settings.py, I define (names not fixed, bikeshed as necessary):

    from goodconf import GoodConf, generate_settings
    
    class Base(GoodConf): # django core settings ...
    class Dev(Base): # my dev specific settings ...
    class Stage(Base): # my staging specific settings ...
    class Prod(Base): # my production specific settings
    
    generate_settings(  # `GoodConf` API more or less
        load=True,
        file_env_var='DJANGO_GOODCONF',
        default_class=Dev,
        settings_directory='./settings/',
    )
    

    So,

    • If no DJANGO_GOODCONF is supplied, we generate a settings/dev.yml
    • If DJANGO_GOODCONF=Stage, we generate a settings/stage.yml
    • We load the settings/<ENV>.yml and programmatically make all settings available for Django (no need to X = config.X boilerplate) through some globals trick or something

    The reason we need generate_settings (or something that isn't an environment specific subclass of GoodConf, like, Dev) is so that we can put the logic for the selection of the environment and generation of settings into goodconf core and not leave it for the user to configure. The DJANGO_GOODCONF env var is the only thing the user needs to consider when deploying to anything that isn't the localhost (Dev used by default otherwise).

    Thoughts? Happy to get stuck in on implementation if we're in agreement!

    PS. A lot of this comes from my past experiences with https://github.com/jazzband/django-configurations. I think if we join these two approaches, we have a really flexible method for Django configuration.

    opened by decentral1se 0
  • Use pydantic settings source for loading files

    Use pydantic settings source for loading files

    This is more of the "pydantic way" to do things and restores some functionality that was broken from the BaseSettings class (see #10 and #9)

    It is a backwards incompatible change. The class is now validated on instantiation. The load method has been removed and a class method from_file has been added.

    • [ ] update docs
    • [ ] update changelog
    • [ ] make new major release
    opened by ipmb 0
  • Allow config saving

    Allow config saving

    Right now we can only load the settings, but it would be nice to have a save method to save changed configuration values. It should keep the user comments on the file

    opened by lyz-code 3
  • Support `env_prefix`

    Support `env_prefix`

    Hi, I'm trying to use the BaseSettings env_prefix Config property to preprend a string on the environment variables without success. A simple snippet that shows the desired behaviour is:

    import os
    
    from goodconf import GoodConf
    
    
    class AppConfig(GoodConf):
        "Configuration for My App"
        DEBUG: bool = False
    
        class Config:
            env_previx = "GOOD_"
    
    
    os.environ["GOOD_DEBUG"] = "True"
    
    config = AppConfig(load=True)
    
    assert config.DEBUG
    

    Using os.environ['DEBUG'] = "True" works though

    opened by lyz-code 0
  • Allow the initialization of values

    Allow the initialization of values

    Hi, first thanks for your awesome project :)

    To build the test cases, I want to create a GoodConf instance with some values that are different from the default, for example the DATABASE_URL attribute. It will be nice that we could do something like:

    AppConfig(DATABASE_URL='fake://address')
    

    I know BaseSettings from pydantic supports it, so maybe we can call to super().__init__ in the GoodConf``init`. What do you think?

    opened by lyz-code 0
Owner
Lincoln Loop
Makers of high performance web applications.
Lincoln Loop
Process RunGap output file of a workout and load data into Apple Numbers Spreadsheet and my website with API calls

BSD 3-Clause License Copyright (c) 2020, Mike Bromberek All rights reserved. ProcessWorkout Exercise data is exported in JSON format to iCloud using

Mike Bromberek 1 Jan 3, 2022
C++ Environment InitiatorVisual Studio Code C / C++ Environment Initiator

Visual Studio Code C / C++ Environment Initiator Latest Version : v 1.0.1(2021/11/08) .exe link here About : Visual Studio Code에서 C/C++환경을 MinGW GCC/G

Junho Yoon 2 Dec 19, 2021
skimpy is a light weight tool that provides summary statistics about variables in data frames within the console.

skimpy Welcome Welcome to skimpy! skimpy is a light weight tool that provides summary statistics about variables in data frames within the console. Th

null 267 Dec 29, 2022
A package with multiple bias correction methods for climatic variables, including the QM, DQM, QDM, UQM, and SDM methods

A package with multiple bias correction methods for climatic variables, including the QM, DQM, QDM, UQM, and SDM methods

Sebastián A. Aedo Quililongo 9 Nov 18, 2022
Python plugin/extra to load data files from an external source (such as AWS S3) to a local directory

Data Loader Plugin - Python Table of Content (ToC) Data Loader Plugin - Python Table of Content (ToC) Overview References Python module Python virtual

Cloud Helpers 2 Jan 10, 2022
Load dependent libraries dynamically.

dypend dypend Load dependent libraries dynamically. A few days ago, I encountered many users feedback in an open source project. The Problem is they c

Louis 5 Mar 2, 2022
Automatically load and dump your dataclasses 📂🙋

file dataclasses Installation By default, filedataclasses comes with support for JSON files only. To support other formats like YAML and TOML, filedat

Alon 1 Dec 30, 2021
ioztat is a storage load analysis tool for OpenZFS

ioztat is a storage load analysis tool for OpenZFS. It provides iostat-like statistics at an individual dataset/zvol level.

Jim Salter 116 Nov 25, 2022
A Python3 script to decode an encoded VBScript file, often seen with a .vbe file extension

vbe-decoder.py Decode one or multiple encoded VBScript files, often seen with a .vbe file extension. Usage usage: vbe-decoder.py [-h] [-o output] file

John Hammond 147 Nov 15, 2022
This app converts an pdf file into the audio file.

PDF-to-Audio This app takes an pdf as an input and convert it into audio, and the library text-to-speech starts speaking the preffered page given in t

Ojas Barawal 3 Aug 4, 2021
Fetch data from an excel file and create HTML file

excel-to-html Problem Statement! - Fetch data from excel file and create html file Excel.xlsx file contain the information.in multiple rows that is ne

Vivek Kashyap 1 Oct 25, 2021
JD-backup is an advanced Python script, that will extract all links from a jDownloader 2 file list and export them to a text file.

JD-backup is an advanced Python script, that will extract all links from a jDownloader 2 file list and export them to a text file.

Kraken.snv 3 Jun 7, 2022
RELATE is an Environment for Learning And TEaching

RELATE Relate is an Environment for Learning And TEaching RELATE is a web-based courseware package. It is set apart by the following features: Focus o

Andreas Klöckner 311 Dec 25, 2022
AndroidEnv is a Python library that exposes an Android device as a Reinforcement Learning (RL) environment.

AndroidEnv is a Python library that exposes an Android device as a Reinforcement Learning (RL) environment.

DeepMind 814 Dec 26, 2022
My solution for a MARL problem on a Grid Environment with Q-tables.

To run the project, run: conda create --name env python=3.7 pip install -r requirements.txt python run.py To-do: Add direction to the state space Take

Merve Noyan 12 Dec 25, 2021
A step-by-step tutorial for how to work with some of the most basic features of Nav2 using a Jupyter Notebook in a warehouse environment to create a basic application.

This project has a step-by-step tutorial for how to work with some of the most basic features of Nav2 using a Jupyter Notebook in a warehouse environment to create a basic application.

Steve Macenski 49 Dec 22, 2022
BridgeWalk is a partially-observed reinforcement learning environment with dynamics of varying stochasticity.

BridgeWalk is a partially-observed reinforcement learning environment with dynamics of varying stochasticity. The player needs to walk along a bridge to reach a goal location. When the player walks off the bridge into the water, the current will move it randomly until it gets washed back on the shore. A good agent in this environment avoids this stochastic trap

Danijar Hafner 6 Jun 13, 2022
pydock - Docker-based environment manager for Python

pydock - Docker-based environment manager for Python ⚠️ pydock is still in beta mode, and very unstable. It is not recommended for anything serious. p

Alejandro Piad 16 Sep 18, 2021
Minimalistic Gridworld Environment (MiniGrid)

Minimalistic Gridworld Environment (MiniGrid) There are other gridworld Gym environments out there, but this one is designed to be particularly simple

Maxime Chevalier-Boisvert 1.7k Jan 3, 2023