🤫 Easily manage configs and secrets in your Python projects (with CLI support)

Candid™️

Last update: Oct 30, 2022

Related tags

Overview

Installation

pip install confidential

How does it work?

Confidential manages secrets for your project, using AWS Secrets Manager.

First, store a secret in AWS Secrets Manager. Then, create a secrets file, say my_secrets.json. A value will be decrypted if the word secret precedes it, like the database value below:

{
  "database": "secret:database_details",
  "environment": "production",
  "debug_mode": false
}

You can decrypt this file either in Python, or directly using the CLI. Ensure AWS CLI is set up, then run:

confidential my_secrets.json

which outputs the file with decrypted values

{
  "database": {
    "url": "https://example.com",
    "username": "admin",
    "password": "p@55w0rd",
    "port": 5678
  },
  "environment": "production",
  "debug_mode": false
}

Can I use it in my Python projects?

Yes, simply import and instantiate SecretsManager, like so:

settings.py

from confidential import SecretsManager


secrets = SecretManager(
    secrets_file=".secrets/production.json",
    secrets_file_default=".secrets/defaults.json",  # Overridable defaults you can use in common environments
    region_name="us-east-1",
)

DATABASES = {
    'default': secrets["database"]
}

Testing

First, install all dependencies:

poetry install

Then run the tests

poetry run pytest

Comments

Handle user permissions error
Old behavior: Throw TypeError, unhelpful error message, hard to debug underlying issue.

New behavior: Throw IOError with helpful error message indicating cause of failure.

Also decided to remove the .idea folder. Let me know if that should be kept for some reason.

Checklist:

[x] Bump minor version to 2.3.0

[x] Add appropriate pytests
opened by candid-elliott 2

Make confidential say when it doesn’t have access to decrypt

If the SECRETS_FILE is not available it'll drop an error and trace:

root@asdadsad:/opt/app# export SECRETS_FILE=".secrets/stage.json"
root@asdadsad:/opt/app# python manage.py shell
Traceback (most recent call last):
  File "manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 325, in execute
    settings.INSTALLED_APPS
  File "/usr/local/lib/python3.6/site-packages/django/conf/__init__.py", line 57, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.6/site-packages/django/conf/__init__.py", line 44, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.6/site-packages/django/conf/__init__.py", line 107, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/app/project/__init__.py", line 8, in <module>
    if settings.DOGSTATSD_ENABLED:
  File "/usr/local/lib/python3.6/site-packages/django/conf/__init__.py", line 57, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.6/site-packages/django/conf/__init__.py", line 44, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.6/site-packages/django/conf/__init__.py", line 107, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/app/project/settings.py", line 27, in <module>
    region_name=AWS_REGION,
  File "/usr/local/lib/python3.6/site-packages/confidential/secrets_manager.py", line 23, in __init__
    secrets = self.parse_secrets_file(secrets_file) if secrets_file else {}
  File "/usr/local/lib/python3.6/site-packages/confidential/secrets_manager.py", line 85, in parse_secrets_file
    config[key] = json.loads(decrypted_string)
  File "/usr/local/lib/python3.6/json/__init__.py", line 348, in loads
    'not {!r}'.format(s.__class__.__name__))
TypeError: the JSON object must be str, bytes or bytearray, not 'NoneType'

opened by lassiter 2

Handle nested keys
what

Addresses #12, an issue with accessing secrets in nested keys.

Example:

{ "secrets": { "key": "secret:value" } }
opened by vagelim 1
Support objects in json structure.
Currently, if you have a key inside an object in <env>.json that is

{ 'service': { 'api_key': 'secret:env/app/my_api_key', 'base_url': 'url' } }

It will not see the secret inside the service object.

However, if it's top level, it'll properly decrypt:

{ 'another_key': 'secret:env/app/my_api_key' }

Steps to Reproduce

Take a working key from a top level of the json and place it inside the object like the first example. You should only see the the string stored in .json from where you store your secret paths.
opened by lassiter 1
add private pypi push

what

Adds push to private pypi.

why

Confidential was apparently pushed manually to pypi using dvf's account. Until we get that access we should push to our internal pypi so that we can still iterate.

opened by vagelim 0

Raise permission error on none-type SecretString response

Helpful error messaging for AWS permissions error. Currently unhelpful response is:

manager.py", line 100, in decrypt_string
    result = json.loads(decrypted_string)
  File "/Users/dvf/.pyenv/versions/3.7.4/lib/python3.7/json/__init__.py", line 341, in loads
    raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not NoneType

⭐️ Added test for {"SecretString": None} secrets manager response.

opened by candid-elliott 0

add optional profile parameter
What does this do?

Adds an optional parameter (-p,--profile) to allow the user to pass in an alternative profile.

Why did you do this?

The current configuration limits confidential to the default profile. For users with multiple profiles, they may wish to use confidential with something beyond the default profile.

How did you test this change?

Ran with a known working profile, with a known non-working profile, with a profile that doesn't exist, and lastly, with no profile specified at all.

etc

I had thought of catching the exception raised when a profile is specified but could not be found (botocore.exceptions.ProfileNotFound) but found the error message to be descriptive enough on its own.

ex:

botocore.exceptions.ProfileNotFound: The config profile (candidco) could not be found
opened by vagelim 0
Fix regression

We introduced a regression in v.2.1.0 where we don't check for an integer. This should fix that. Also added some more tests.

Thanks to @jsundy for finding it.

opened by dvf 0
fix builds

what

fixes builds

why

they dont work, specifically because of deprecated options passed to black. also because of an unavailable flag being passed to poetry ([ValueError] Setting settings.virtualenvs.create does not exist ##[error]Process completed with exit code 1.)

https://github.com/candidco/confidential/runs/428287858

opened by vagelim 0

Added deep overrides of secrets

Secrets files can now be deeply-merged:

# defaults.json
{
  "django": {
    "debug": true,
    "database": {
      "hostname": "123",
      "port": 8000,
    }
  }
}

# overrides.json
{
  "django": {
    "database": {
      "hostname": "456",
    }
  }
}

Result:

{
  "django": {
    "debug": true,
    "database": {
      "hostname": "456",
      "port": 8000,
    }
  }
}

Also fixed a minor typo.

opened by dvf 0

Bump certifi from 2021.10.8 to 2022.12.7
Bumps certifi from 2021.10.8 to 2022.12.7.

Commits

9e9e840 2022.12.07

b81bdb2 2022.09.24

939a28f 2022.09.14

aca828a 2022.06.15.2

de0eae1 Only use importlib.resources's new files() / Traversable API on Python ≥3.11 ...

b8eb5e9 2022.06.15.1

47fb7ab Fix deprecation warning on Python 3.11 (#199)

b0b48e0 fixes #198 -- update link in license

9d514b4 2022.06.15

4151e88 Add py.typed to MANIFEST.in to package in sdist (#196)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 0

Releases(v2.3.2)

v2.3.2(Apr 26, 2022)
Upgrading version of Click to 8.0.3

Source code(tar.gz)
Source code(zip)

Owner

Candid™️

GitHub

Sync any your configuration file to remote. Currently only support gist.

Sync your configuration to remote, such as vimrc. You can use EscSync to manage your configure of editor, shell, etc.

0 Nov 21, 2022

Organize Django settings into multiple files and directories. Easily override and modify settings. Use wildcards and optional settings files.

Organize Django settings into multiple files and directories. Easily override and modify settings. Use wildcards in settings file paths and mark setti