Python dictionaries with advanced dot notation access

Potentially dangerous regression in config box, due to the new ruamel.yaml dependency.

>>> from box import ConfigBox
>>> b = ConfigBox.from_yaml("kill_all_humans_mode: no")
>>> if b.kill_all_humans_mode: 
...     print("ok, killing all humans...") 
...
ok, killing all humans...

Same problem with box.Box. Our test suite caught this, but others might not be so lucky. ruamel isn't exactly a drop-in replacement for yaml, despite claims to the contrary (ruamel defaults to spec 1.2 and yaml is spec 1.1).

I would advise to make the toml and ruamel.yaml dependencies optional (or "soft" dependencies, like it was in 3.x) since users might prefer other providers such as pytoml and PyYAML and you don't want to tie their hands here.

Personally, I can not use box>=4 at all because the ruamel.yaml has an insane installer which, for various reasons, does not work in the prod environment at $EMPLOYER.

from box import Box

class test(object):
	def __init__(self, **kwargs):
		d = Box({
		'item':'value', 
		'item2': self.ok
		})
		self.d = d 

	def ok(self, **kwargs):
		print "test"
		print kwargs.keys()
		for key in kwargs.keys():
			print kwargs[key]
		return 

data={
	'test':'value',
	'test1':'value1'
}

t = test()
print t.d.item2(**data)

From t.d.item2 how do I get that path while in the 'ok' function.

Thanks!

Is Box.update supposed to be much slower than dict.update? I expected the performance to be comparable, and ran into trouble. The code below shows the timing for dict.update and Box. .update

import random
import box
import string
import time

def random_string(N):
    return ''.join(random.choices(string.ascii_uppercase + string.digits, k=N))


a = dict((random_string(64), 0) for i in range(10**3))
for i in a:
    a[i.upper()] = 0

b = box.Box(dict((random_string(64), 0) for i in range(10)))
c = dict((random_string(64), 0) for i in range(10))

st = time.time()
c.update(a)
et = time.time()
print(et - st)

st = time.time()
b.update(a)
et = time.time()
print(et - st)

Output:

2.09808349609375e-05
4.840667724609375

With

Python 3.7.5 (default, Nov  1 2019, 02:16:32) 
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin

Given this sample code:

import box

if __name__ == '__main__':
    b = box.Box({
        'one': {
            'sub1': 'monkey',
            'sub2': 'goat',
        }
    }, box_it_up = True)

    print(f'box: {b}')
    print(f'one is a {type(b.one)}')
    print(f'expect monkey: {b.one.sub1}')
    print(f'expect goat: {b.one.sub2}')
            
    b.update({'one': {'sub2': 'alpaca'}})
    print(f'one is a {type(b.one)}')
    print(f'expect monkey: {b.one.sub1}')
    print(f'expect alpaca: {b.one.sub2}')

It works with < 3.4.3:

box: {'one': {'sub1': 'monkey', 'sub2': 'goat'}}
one is a <class 'box.Box'>
expect monkey: monkey
expect goat: goat
one is a <class 'box.Box'>
expect monkey: monkey
expect alpaca: alpaca

But using >= 3.4.3, this is the result:

box: {'one': {'sub1': 'monkey', 'sub2': 'goat'}}
one is a <class 'box.Box'>
expect monkey: monkey
expect goat: goat
one is a <class 'dict'>
Traceback (most recent call last):
  File "./test.py", line 16, in <module>
    print(f'expect monkey: {b.one.sub1}')
AttributeError: 'dict' object has no attribute 'sub1'

I notice there were changes to the update() method.

(edited: added some code to print out the types, which show the difference)

Partially solves #46 . I have limited time today so I only fixed the default_box + camel_killer issue.

Now, "camel-killed" attributes will be looked up before the default_box logic is used.

I've added regression test for this case.

Would you please explain the rationale behind allowing only string.ascii_letters + string.digits + "_" in keys? I would like to use greek letters and end up with silent errors like:

>>> a = Box()
>>> a.σeq = 1
>>> a.µeq = 2
>>> a
<Box: {'σeq': 2}>

Is there anything wrong with adding more utf-8 characters to allowed?

I have implemented support for dot keys by sub-classing Box and using it for two years by now. Among other things it provides an alternative data model which essentially flattens the hierarchy - a useful approach in certain cases. To fully support this model I needed to allow also flat iteration over the data elements. It was implemented by adding optional parameter in keys and items and values members.

For example, for keys:

def keys(self, depth=False):
    ...

So that

box.keys(depth=True)
# ['a.b.c', 'a.b.d', 'a.x.y']

That makes the 'flat' model complete and can be implemented with minimal code and performance overhead.

Do you think it could be generic enough to include into the package?

#78

from box import Box

class MyList(list): 
    def sq(self):
        return [x*x for x in self]
	
b = Box(a = MyList([1,2,4]), box_intact_types = (MyList, ))
b.a.sq()
# [1,4,16]

Wow, sorry this took so long to get around to (new job, life, etc...).

@matan129 and @polishmatt, Thank you for finding those bugs and code contributions to fix them. I needed to modify how they were handled and this PR happens to (hopefully) handle both those issues.

If you could both please look at the last file, the tests, and just make sure that your respective bug would be caught with it, would be much appreciated.

I get the following error when I have this line in my code: from box import Box

ModuleNotFoundError: No module named 'box'

I've installed on RasPi Zero using pip3 install box. Also tried the regular pip install box.

Am running Python 3 3.7.3

Am I missing something obvious?

Thanks, Ken

Hi,

thanks a lot for your project, it's currently helping me a lot :) one feature i missed is the ability to treat None values as if there was no key defined.

like in this example.

b=Box({'brol': None}, default_box=True)
b.brol.truc

gives

Traceback (most recent call last): File "", line 1, in AttributeError: 'NoneType' object has no attribute 'truc'

but this works as exepected

b=Box({}, default_box=True)
b.brol.truc

many many thanks 👍

Frozen boxes are great for enforcing a functional style of programming at runtime. But the implementation of Box.__add__ assumes that the left-hand side is non-frozen.

The following piece of code fails in box/box.py:274: box.box.Box.__add__ with box.exceptions.BoxError: Box is frozen

foo = box.Box(frozen_box=True)
foobar = foo + {"bar": 1}

As far as I can judge, there is no reason to restrict __add__ (and __or__) operations to non-frozen boxes. The implementation of Box.__add__ creates a copy of the input box before merging/updating, thus the operations leave the input boxes unmodified.

This implementation of Box.__or__ would solve the issue:

...
new_box = other.copy()
new_box._box_config["frozen_box"] = False
new_box.update(other)
new_box._box_config["frozen_box"] = other._box_config["frozen_box"]
...

I've made a quick check that this works for flat and nested structures. But there might be some unintended side effects for nested structures. But this would be an issue even for non-frozen boxes. I guess most programmers will assume that writing foobar = foo + {"bar": 1} won't mutate foo.

What are the objections against frozen boxes in Box.__add__ (Box.__or__)? And, how should this be implemented?

Bug of keys in Box:

When you use keys() method in a "dotted" situation, meaning we have a tree of keys in a Box, and you want to get the keys of a given node, one can notice that we get back a list of keys that are multiplicated if the value of a key is a list. (and the multiplication is by the lengh of the list)

For example:

b = Box({
"Animals" : {"Land_animals" : ["Lion", "Elephant"], "Sea_animals" : ["Dolphin", "Shark"]},
"Trees" : ["Palm_tree", "Coconut_tree"]
}, box_dots = True)

 

When you are using keys = b.keys(dotted = True) you get -

['Animals.Land_animals[0]',
'Animals.Land_animals[1]',
'Animals.Sea_animals[0]',
'Animals.Sea_animals[1]',
'Trees[0]',
'Trees[1]']

and we can see the multiple occurances of the keys (with indexes of their number of values), and the method len(keys) would have wrong answer - 6 instead of 3.

The output which I thought would be right is: ['Animals.Land_animals', 'Animals.Sea_animals', 'Trees']

A workaround is to add to the box this:

box_intact_types = (tuple,list)

so that lists wouldn't be converted, and than it works fine.

Box is really useful when accessing non-existing keys/attributes and assigning new keys on them e.g. box.foo.bar.baz = 1.

Noting down here an issue I have found. When IPython's help using ? is called it adds a key getdoc in Box object.

Version

python-box v6.1.0

Reproducer

Cause

This seems expected and is a side-effect of this line in IPython which tries a find a custom method getdoc() to get docstring for an object.

Is there a workaround here?

Hi,

An __or__ method is not meant to be cumulative, as we know. In box-box operations new_box = box_1 | box_2 , everything is working properly That said, new_boxwould be different if we would write new_box = box_2 | box_1

The same is with dictionaries ( as introduced in PEP 584), and dict-dict operations.

The problem starts when you combine Box with a dict in an __or__/__ror__ method, for example new_box = box_1 | dict_1.

You can check that:

new_box1 = box_1 | dict_1 , new_box2 = dict_1 | box_1

And you get that - new_box1 **==** new_box2

As opposed to the noncumulative attribute of OR operation in Dicts and Boxes.

If you would make the change: new_box1 = box_1 | Box(dict_1) , new_box2 = Box(dict_1) | box_1

you would get - new_box1 **!=** new_box2.

But that means that you can't do box operations with normal dicts (at least with OR dunder method)

Hello,

I noticed two small typos on the Wiki page Types of Boxes:

Types of Boxes / Box Dots

Support for traversing box lists as well!

my_box = Box({'data': [ {'rabbit': 'hole'} ] }, box_dots=True)
print(data.data[0].rabbit)
# hole

To change

my_box = Box({'data': [ {'rabbit': 'hole'} ] }, box_dots=True)
- print(data.data[0].rabbit)
+ print(my_box.data[0].rabbit)
# hole

Types of Boxes / Box Recast Values

If it cannot be converted, it will raise a BoxValueError (catachable with either BoxError or ValueError as well)

To change

- If it cannot be converted, it will raise a `BoxValueError` (catachable with either `BoxError` or `ValueError` as well)
+ If it cannot be converted, it will raise a `BoxValueError` (catchable with either `BoxError` or `ValueError` as well)

Thank you for your work on the Box library and kind regards Martin

I have some nested Boxes using an overridden __convert_and_store in a subclass to convert values to an expected type (eg: I want all values to be an int or some other custom class). For that conversion, I would like to know the "namespace" of the Box/value in order to reference it in the conversion and show nicer error messages.

For example, when handling the conversion of 5 in the example below:

box = MyBox({"a": {"b": {"c": 5}}})

I would like to know that I'm assigning to ("a", "b", "c").

This is quite hard to setup only in a subclass because of the recursive value conversion (and copying) during __init__ and conversion doesn't give a good hook to intercept after the box is created, but before sub-boxes are created.

I think the easiest way to support this is by adding a box_path or box_namespace param to __init__ (defaulting to ()) and then into _box_config, which can be passed through (and extended) in __get_default and __convert_and_store when creating sub-boxes.

Is this something you'd be open to? I'd be happy to make a PR for this if so!