Deep Difference and search of any Python object/data.

Overview

DeepDiff v 5.6.0

Downloads Python Versions License Build Status codecov

DeepDiff Overview

  • DeepDiff: Deep Difference of dictionaries, iterables, strings and other objects. It will recursively look for all the changes.
  • DeepSearch: Search for objects within other objects.
  • DeepHash: Hash any object based on their content.

Tested on Python 3.6+ and PyPy3.

NOTE: Python 2 is not supported any more. DeepDiff v3.3.0 was the last version to support Python 2

NOTE: The last version of DeepDiff to work on Python 3.5 was DeepDiff 5-0-2

What is new?

DeepDiff 5-6-0 allows you to pass custom operators.

>> ">
>>> from deepdiff import DeepDiff
>>> from deepdiff.operator import BaseOperator
>>> class CustomClass:
...     def __init__(self, d: dict, l: list):
...         self.dict = d
...         self.dict['list'] = l
...
>>>
>>> custom1 = CustomClass(d=dict(a=1, b=2), l=[1, 2, 3])
>>> custom2 = CustomClass(d=dict(c=3, d=4), l=[1, 2, 3, 2])
>>> custom3 = CustomClass(d=dict(a=1, b=2), l=[1, 2, 3, 4])
>>>
>>>
>>> class ListMatchOperator(BaseOperator):
...     def give_up_diffing(self, level, diff_instance):
...         if set(level.t1.dict['list']) == set(level.t2.dict['list']):
...             return True
...
>>>
>>> DeepDiff(custom1, custom2, custom_operators=[
...     ListMatchOperator(types=[CustomClass])
... ])
{}
>>>
>>>
>>> DeepDiff(custom2, custom3, custom_operators=[
...     ListMatchOperator(types=[CustomClass])
... ])
{'dictionary_item_added': [root.dict['a'], root.dict['b']], 'dictionary_item_removed': [root.dict['c'], root.dict['d']], 'values_changed': {"root.dict['list'][3]": {'new_value': 4, 'old_value': 2}}}
>>>

New in 5-6-0: Dynamic ignore order function

Ignoring order when certain word in the path

>>> from deepdiff import DeepDiff
>>> t1 = {'a': [1, 2], 'b': [3, 4]}
>>> t2 = {'a': [2, 1], 'b': [4, 3]}
>>> DeepDiff(t1, t2, ignore_order=True)
{}
>>> def ignore_order_func(level):
...     return 'a' in level.path()
...
>>> DeepDiff(t1, t2, ignore_order=True, ignore_order_func=ignore_order_func)
{'values_changed': {"root['b'][0]": {'new_value': 4, 'old_value': 3}, "root['b'][1]": {'new_value': 3, 'old_value': 4}}}

Installation

Install from PyPi:

pip install deepdiff

If you want to use DeepDiff from commandline:

pip install "deepdiff[cli]"

Importing

>>> from deepdiff import DeepDiff  # For Deep Difference of 2 objects
>>> from deepdiff import grep, DeepSearch  # For finding if item exists in an object
>>> from deepdiff import DeepHash  # For hashing objects based on their contents

Note: if you want to use DeepDiff via commandline, make sure to run pip install "deepdiff[cli]". Then you can access the commands via:

  • DeepDiff
    • $ deep diff --help
  • Delta
    • $ deep patch --help
  • grep
    • $ deep grep --help
  • extract
    • $ deep extract --help

Deep Diff

DeepDiff gets the difference of 2 objects.

A few Examples

Note: This is just a brief overview of what DeepDiff can do. Please visit https://zepworks.com/deepdiff/5.6.0/ for full documentation.

List difference ignoring order or duplicates

>> t2 = {1:1, 2:2, 3:3, 4:{"a":"hello", "b":[1, 3, 2, 3]}} >>> ddiff = DeepDiff(t1, t2, ignore_order=True) >>> print (ddiff) {} ">
>>> t1 = {1:1, 2:2, 3:3, 4:{"a":"hello", "b":[1, 2, 3]}}
>>> t2 = {1:1, 2:2, 3:3, 4:{"a":"hello", "b":[1, 3, 2, 3]}}
>>> ddiff = DeepDiff(t1, t2, ignore_order=True)
>>> print (ddiff)
{}

Report repetitions

This flag ONLY works when ignoring order is enabled.

t1 = [1, 3, 1, 4]
t2 = [4, 4, 1]
ddiff = DeepDiff(t1, t2, ignore_order=True, report_repetition=True)
print(ddiff)

which will print you:

{'iterable_item_removed': {'root[1]': 3},
  'repetition_change': {'root[0]': {'old_repeat': 2,
                                    'old_indexes': [0, 2],
                                    'new_indexes': [2],
                                    'value': 1,
                                    'new_repeat': 1},
                        'root[3]': {'old_repeat': 1,
                                    'old_indexes': [3],
                                    'new_indexes': [0, 1],
                                    'value': 4,
                                    'new_repeat': 2}}}

Exclude certain types from comparison:

>> l2 = logging.getLogger("test2") >>> t1 = {"log": l1, 2: 1337} >>> t2 = {"log": l2, 2: 1337} >>> print(DeepDiff(t1, t2, exclude_types={logging.Logger})) {} ">
>>> l1 = logging.getLogger("test")
>>> l2 = logging.getLogger("test2")
>>> t1 = {"log": l1, 2: 1337}
>>> t2 = {"log": l2, 2: 1337}
>>> print(DeepDiff(t1, t2, exclude_types={logging.Logger}))
{}

Exclude part of your object tree from comparison

>> t2 = {"for life": "vegan", "ingredients": ["veggies", "tofu", "soy sauce"]} >>> print (DeepDiff(t1, t2, exclude_paths={"root['ingredients']"})) {} ">
>>> t1 = {"for life": "vegan", "ingredients": ["no meat", "no eggs", "no dairy"]}
>>> t2 = {"for life": "vegan", "ingredients": ["veggies", "tofu", "soy sauce"]}
>>> print (DeepDiff(t1, t2, exclude_paths={"root['ingredients']"}))
{}

Exclude Regex Paths

You can also exclude using regular expressions by using exclude_regex_paths and pass a set or list of path regexes to exclude. The items in the list could be raw regex strings or compiled regex objects.

>> exclude_path = re.compile(r"root\[\d+\]\['b'\]") >>> print(DeepDiff(t1, t2, exclude_regex_paths=[exclude_path])) {} ">
>>> t1 = [{'a': 1, 'b': 2}, {'c': 4, 'b': 5}]
>>> t2 = [{'a': 1, 'b': 3}, {'c': 4, 'b': 5}]
>>> print(DeepDiff(t1, t2, exclude_regex_paths={r"root\[\d+\]\['b'\]"}))
{}
>>> exclude_path = re.compile(r"root\[\d+\]\['b'\]")
>>> print(DeepDiff(t1, t2, exclude_regex_paths=[exclude_path]))
{}

Significant Digits

Digits after the decimal point. Internally it uses "{:.Xf}".format(Your Number) to compare numbers where X=significant_digits

>>> t1 = Decimal('1.52')
>>> t2 = Decimal('1.57')
>>> DeepDiff(t1, t2, significant_digits=0)
{}
>>> DeepDiff(t1, t2, significant_digits=1)
{'values_changed': {'root': {'old_value': Decimal('1.52'), 'new_value': Decimal('1.57')}}}

Ignore Type Number - List that contains float and integer:

>>> from deepdiff import DeepDiff
>>> from pprint import pprint
>>> t1 = [1, 2, 3]
>>> t2 = [1.0, 2.0, 3.0]
>>> ddiff = DeepDiff(t1, t2)
>>> pprint(ddiff, indent=2)
{ 'type_changes': { 'root[0]': { 'new_type': <class 'float'>,
                         'new_value': 1.0,
                         'old_type': <class 'int'>,
                         'old_value': 1},
            'root[1]': { 'new_type': <class 'float'>,
                         'new_value': 2.0,
                         'old_type': <class 'int'>,
                         'old_value': 2},
            'root[2]': { 'new_type': <class 'float'>,
                         'new_value': 3.0,
                         'old_type': <class 'int'>,
                         'old_value': 3}}}
>>> ddiff = DeepDiff(t1, t2, ignore_type_in_groups=[(int, float)])
{}

Views

Starting with DeepDiff v 3, there are two different views into your diffed data: text view (original) and tree view (new).

Text View

Text view is the original and currently the default view of DeepDiff.

It is called text view because the results contain texts that represent the path to the data:

Example of using the text view.

>>> from deepdiff import DeepDiff
>>> t1 = {1:1, 3:3, 4:4}
>>> t2 = {1:1, 3:3, 5:5, 6:6}
>>> ddiff = DeepDiff(t1, t2)
>>> print(ddiff)
{'dictionary_item_added': {'root[5]', 'root[6]'}, 'dictionary_item_removed': {'root[4]'}}

So for example ddiff['dictionary_item_removed'] is a set if strings thus this is called the text view.

The following examples are using the *default text view.*
The Tree View is introduced in DeepDiff v3
and provides traversing capabilities through your diffed data and more!
Read more about the Tree View at the [tree view section](#tree-view) of this page.

Tree View

Starting the version v3 You can choose the view into the deepdiff results. The tree view provides you with tree objects that you can traverse through to find the parents of the objects that are diffed and the actual objects that are being diffed.

Value of an item has changed (Tree View)

>>> from deepdiff import DeepDiff
>>> from pprint import pprint
>>> t1 = {1:1, 2:2, 3:3}
>>> t2 = {1:1, 2:4, 3:3}
>>> ddiff_verbose0 = DeepDiff(t1, t2, verbose_level=0, view='tree')
>>> ddiff_verbose0
{'values_changed': {<root[2]>}}
>>>
>>> ddiff_verbose1 = DeepDiff(t1, t2, verbose_level=1, view='tree')
>>> ddiff_verbose1
{'values_changed': {<root[2] t1:2, t2:4>}}
>>> set_of_values_changed = ddiff_verbose1['values_changed']
>>> # since set_of_values_changed includes only one item in a set
>>> # in order to get that one item we can:
>>> (changed,) = set_of_values_changed
>>> changed  # Another way to get this is to do: changed=list(set_of_values_changed)[0]
<root[2] t1:2, t2:4>
>>> changed.t1
2
>>> changed.t2
4
>>> # You can traverse through the tree, get to the parents!
>>> changed.up
<root t1:{1: 1, 2: 2,...}, t2:{1: 1, 2: 4,...}>

Serialization

In order to convert the DeepDiff object into a normal Python dictionary, use the to_dict() method. Note that to_dict will use the text view even if you did the diff in tree view.

Example:

>> t2 = {1: 1, 2: 2, 3: 3, 4: {"a": "hello", "b": "world\n\n\nEnd"}} >>> ddiff = DeepDiff(t1, t2, view='tree') >>> ddiff.to_dict() {'type_changes': {"root[4]['b']": {'old_type': , 'new_type': , 'old_value': [1, 2, 3], 'new_value': 'world\n\n\nEnd'}}} ">
>>> t1 = {1: 1, 2: 2, 3: 3, 4: {"a": "hello", "b": [1, 2, 3]}}
>>> t2 = {1: 1, 2: 2, 3: 3, 4: {"a": "hello", "b": "world\n\n\nEnd"}}
>>> ddiff = DeepDiff(t1, t2, view='tree')
>>> ddiff.to_dict()
{'type_changes': {"root[4]['b']": {'old_type': <class 'list'>, 'new_type': <class 'str'>, 'old_value': [1, 2, 3], 'new_value': 'world\n\n\nEnd'}}}

In order to do safe json serialization, use the to_json() method.

Example:

>> t2 = {1: 1, 2: 2, 3: 3, 4: {"a": "hello", "b": "world\n\n\nEnd"}} >>> ddiff = DeepDiff(t1, t2, view='tree') >>> ddiff.to_json() '{"type_changes": {"root[4][\'b\']": {"old_type": "list", "new_type": "str", "old_value": [1, 2, 3], "new_value": "world\\n\\n\\nEnd"}}}' ">
>>> t1 = {1: 1, 2: 2, 3: 3, 4: {"a": "hello", "b": [1, 2, 3]}}
>>> t2 = {1: 1, 2: 2, 3: 3, 4: {"a": "hello", "b": "world\n\n\nEnd"}}
>>> ddiff = DeepDiff(t1, t2, view='tree')
>>> ddiff.to_json()
'{"type_changes": {"root[4][\'b\']": {"old_type": "list", "new_type": "str", "old_value": [1, 2, 3], "new_value": "world\\n\\n\\nEnd"}}}'

Deep Search

DeepDiff comes with a utility to find the path to the item you are looking for. It is called DeepSearch and it has a similar interface to DeepDiff.

Let's say you have a huge nested object and want to see if any item with the word somewhere exists in it. Just grep through your objects as you would in shell!

from deepdiff import grep
obj = {"long": "somewhere", "string": 2, 0: 0, "somewhere": "around"}
ds = obj | grep("somewhere")
print(ds)

Which will print:

{'matched_paths': {"root['somewhere']"},
 'matched_values': {"root['long']"}}

And you can pass all the same kwargs as DeepSearch to grep too:

>>> obj | grep(item, verbose_level=2)
{'matched_paths': {"root['somewhere']": 'around'}, 'matched_values': {"root['long']": 'somewhere'}}

Deep Hash

(New in v4-0-0)

DeepHash is designed to give you hash of ANY python object based on its contents even if the object is not considered hashable! DeepHash is supposed to be deterministic in order to make sure 2 objects that contain the same data, produce the same hash.

Let's say you have a dictionary object.

>>> from deepdiff import DeepHash
>>>
>>> obj = {1: 2, 'a': 'b'}

If you try to hash it:

", line 1, in TypeError: unhashable type: 'dict' ">
>>> hash(obj)
Traceback (most recent call last):
  File "
     
      "
     , line 1, in <module>
TypeError: unhashable type: 'dict'

But with DeepHash:

>>> from deepdiff import DeepHash
>>> obj = {1: 2, 'a': 'b'}
>>> DeepHash(obj)
{4355639248: 2468916477072481777512283587789292749, 4355639280: -35787773492556653776377555218122431491, 4358636128: -88390647972316138151822486391929534118, 4358009664: 8833996863197925870419376694314494743, 4357467952: 34150898645750099477987229399128149852}

So what is exactly the hash of obj in this case? DeepHash is calculating the hash of the obj and any other object that obj contains. The output of DeepHash is a dictionary of object IDs to their hashes. In order to get the hash of obj itself, you need to use the object (or the id of object) to get its hash:

>>> hashes = DeepHash(obj)
>>> hashes[obj]
34150898645750099477987229399128149852

Which you can write as:

>>> hashes = DeepHash(obj)[obj]

At first it might seem weird why DeepHash(obj)[obj] but remember that DeepHash(obj) is a dictionary of hashes of all other objects that obj contains too.

Using DeepDiff in unit tests

result is the output of the function that is being tests. expected is the expected output of the function.

self.assertEqual(DeepDiff(expected, result), {})

or if you are using Pytest:

assert not DeepDiff(expected, result)

In other words, assert that there is no diff between the expected and the result.

Difference with Json Patch

Unlike Json Patch which is designed only for Json objects, DeepDiff is designed specifically for almost all Python types. In addition to that, DeepDiff checks for type changes and attribute value changes that Json Patch does not cover since there are no such things in Json. Last but not least, DeepDiff gives you the exact path of the item(s) that were changed in Python syntax.

Example in Json Patch for replacing:

{ "op": "replace", "path": "/a/b/c", "value": 42 }

Example in DeepDiff for the same operation:

, 'new_value': 42, 'old_value': 'foo', 'new_type': }}} ">
>>> item1 = {'a':{'b':{'c':'foo'}}}
>>> item2 = {'a':{'b':{'c':42}}}
>>> DeepDiff(item1, item2)
{'type_changes': {"root['a']['b']['c']": {'old_type': <type 'str'>, 'new_value': 42, 'old_value': 'foo', 'new_type': <type 'int'>}}}

Documentation

https://zepworks.com/deepdiff/current/

Pycon 2016

I was honored to give a talk about the basics of how DeepDiff does what it does at Pycon 2016. Please check out the video and let me know what you think:

Diff It To Dig It Video And here is more info: http://zepworks.com/blog/diff-it-to-digg-it/

ChangeLog

Please take a look at the CHANGELOG file.

Releases

We use bump2version to bump and tag releases.

git checkout master && git pull
bumpversion {patch|minor|major}
git push && git push --tags

Contribute

  1. Please make your PR against the dev branch
  2. Please make sure that your PR has tests. Since DeepDiff is used in many sensitive data driven projects, we strive to maintain around 100% test coverage on the code.

Please run pytest --cov=deepdiff --runslow to see the coverage report. Note that the --runslow flag will run some slow tests too. In most cases you only want to run the fast tests which so you won't add the --runslow flag.

Or to see a more user friendly version, please run: pytest --cov=deepdiff --cov-report term-missing --runslow.

Thank you!

Authors

Please take a look at the AUTHORS file.

Comments
  • V3

    V3

    • A new diff model to replace parents. This model allows an easy way to traverse through parent, child in a diff graph. There is one graph per diff just like how there was one parents per diff.
    opened by seperman 55
  • Return object refs

    Return object refs

    Hi Sep,

    I finally got around to implement the feature I talked about earlier. I changed DeepDiff to provide and use internally a new, less processed view of a DeepDiff result. This "reference-style" view provides a change as a linked list of the object path leading to the actual change, allowing access to all objects in this path on both sides of the comparison.

    An individual result is now of the new type "DiffLevel", which is the leaf of this linked list. I tried to depict the broad idea in refdesign.odt (which is kinda ugly and should probably be removed before merging this).

    The basic idea of this was to allow a program employing DeepDiff to further process it's results. For example, I'm currently using DeepDiff in a script that fetches and displays an overview of permission data in some web application. I use DeepDiff to figure out where this data has changed. Afterwards, I mark the data DeepDiff has found as added or deleted in an HTML view.

    Used in this way DeepDiff can free the developer from writing custom comparison code.

    The new reference-style view for DeepDiff allows me to do something like this:

    permdata = self.model.permissions
    olddata = self.cmp.permissions
    diff = DeepDiff(olddata, permdata, default_view='ref')  # enable new view
    if 'set_item_removed' in diff:
     for change in diff['set_item_removed']:
       parentset = change.up.t2
       parentset.add("<del>" + change.t1 + "</del>")
    

    With my changes DeepDiff now uses this new structure to recreate the traditional result format. All existing tests pass and I'm positive this is fully compatible to the way DeepDiff behaved previously.

    I know this is a huge change to DeepDiff's internals but I hope you can see the benefit of it :) There are still a bunch of open TODOs in the code but I'd love to work with you to get this merged.

    Best, Victor

    opened by victorhahncastell 18
  • Bugfix for significant_digits (signed zero); Numpy-friendlyness; Issue #49; Failing tests for #50;

    Bugfix for significant_digits (signed zero); Numpy-friendlyness; Issue #49; Failing tests for #50;

    This pullrequests makes several small changes. Fell free to merge only selected commits if desired. Most commit contain changes to the main source code and tests.

    • I add 2 failing tests to document issue #50
    • I compare self.down.t1 to None instead of using if self.down.ta in auto_generate_child_rel, because numpy arrays rise an error when used as booleans. I hope this changed behaviour does not introduce any bugs with other edge-cases.
    • I added a bugfix to __diff_numbers for the significant_digits option. Now -0 and +0 compare equal.
    • Finally I use __diff_dict instead of __diff_iterable for the comparison of collections.abc.Mapping instances (before __diff_dict required the container to be a MutableMapping) - see issue #49
    bug 
    opened by Bernhard10 15
  • How to align objects in lists properly?

    How to align objects in lists properly?

    Hi. Sorry, I don't know whether it's a bug/feature or just a question. I've asked it some time ago on StackOverflow, but didn't receive any answer, so I decided to try here.

    I'm trying to compare two lists of objects (dicts in this case) with deepdiff:

    old = [
           {'name': 'war', 'status': 'active'},
           {'name': 'drought', 'status': 'pending'}
    ]
    
    new = [
           {'name': 'war', 'status': 'pending'},
           {'name': 'fire', 'status': 'pending'}]
    
    DeepDiff(old, new)
    
    # Result:
    {'values_changed': 
      {"root[0]['status']": {'new_value': 'pending', 'old_value': 'active'},
       "root[1]['name']": {'new_value': 'fire', 'old_value': 'drought'}}}
    

    The problem is that I need a different way of aligning objects. In my project a particular state (for example war) have a strict life cycle: appears as pending, transforms to active and disappears. I want to use deepdiff to track these changes. Objects with different names are different objects and I don't want them to align with each other.

    So the result I expect is:

    'values_changed':
        {"root[0]['status']": {'new_value': 'pending', 'old_value': 'active'},
     'iterable_item_removed':
        {'root[1]': {'name': 'drought', 'status': 'pending'}}}
     'iterable_item_added':
        {'root[1]': {'name': 'fire', 'status': 'pending'}}}
    

    Is there a way I can modify my object to align it properly? I've tried replacing the dict with a class with custom __eq__ method, but it didn't work. Do you have any other suggestion how I can make objects with the same name align only with each other?

    Here is the original stack overflow question.

    opened by MKaras93 14
  • DeepDiff to run in multiple passes to diff combinations of results when ignore_order=True

    DeepDiff to run in multiple passes to diff combinations of results when ignore_order=True

    DeepDiff to run in 2 passes. And diff combinations of results when ignore_order=True.

    Example:

    Currently:

    In [2]: from deepdiff import DeepDiff
    
    In [3]: DeepDiff({'a': [1,2,3]}, {'a': [3,2,1, 0]}, ignore_order=True)
    Out[3]: {'iterable_item_added': {"root['a'][3]": 0}}
    
    In [4]: DeepDiff({'a': [{'b': [1,2,3]}]}, {'a': [{'b': [3,2,1, 0]}]}, ignore_order=True)
    Out[4]:
    {'iterable_item_added': {"root['a'][0]": {'b': [3, 2, 1, 0]}},
     'iterable_item_removed': {"root['a'][0]": {'b': [1, 2, 3]}}}
    

    But if deepdiff compares the items between the iterable item added and removed, it should be spitting out the following results instead:

    In [4]: DeepDiff({'a': [{'b': [1,2,3]}]}, {'a': [{'b': [3,2,1, 0]}]}, ignore_order=True)
    Out[4]:  {'iterable_item_added': {"root['a'][0][3]": 0}}
    
    enhancement 
    opened by seperman 13
  • Support searching with regular expressions

    Support searching with regular expressions

    I'd like to use regular expressions to find elements of a dictionary with grep.

    (Pdb) entity_attributes
    {'alone': {'id_': 'alone', 'name': 'Timothy', 'last_name': 'Jackson', 'country': 'Korea', 'rating': 5491}, 'film': {'id_': 'film', 'name': 'David', 'last_name': 'Howell', 'country': 'Moldova', 'rating': 9731}, 'thought': {'id_': 'thought'
    , 'name': 'Jason', 'last_name': 'Roberts', 'country': 'Western Sahara', 'rating': 5484}}
    (Pdb) entity_attributes | grep('Da')
    {'matched_values': ["root['film']['name']"]}
    (Pdb) entity_attributes | grep('Da.*')
    {}
    

    If you give me some guidance on how to do it, I can make a PR

    opened by lyz-code 11
  • Delta + Numpy structure throws ambiguous true value

    Delta + Numpy structure throws ambiguous true value

    Describe the bug I am using the delta of the diff to generate the other input again. When using numpy arrays as input (or dictionaries of lists of arrays, etc.) it throws an exception, ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all().

    To Reproduce Steps to reproduce the behavior:

    1. Input Code
    a1 = np.array([1,2,3,4])
    a2 =np.array([5,6,7,8,9,10])
    mydiff = DeepDiff(a1, a2)
    delta = Delta(mydiff)
    delta + a1
    
    1. Error Trace
    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    <ipython-input-62-bbde3ce70e97> in <module>
          3 mydiff = DeepDiff(a1, a2)
          4 delta = Delta(mydiff)
    ----> 5 delta + a1
    
    ~\AppData\Local\Continuum\anaconda3\lib\site-packages\deepdiff\delta.py in __add__(self, other)
        146         else:
        147             self.root = deepcopy(other)
    --> 148         self._do_values_changed()
        149         self._do_set_item_added()
        150         self._do_set_item_removed()
    
    ~\AppData\Local\Continuum\anaconda3\lib\site-packages\deepdiff\delta.py in _do_values_changed(self)
        300         values_changed = self.diff.get('values_changed')
        301         if values_changed:
    --> 302             self._do_values_or_type_changed(values_changed)
        303 
        304     def _do_type_changes(self):
    
    ~\AppData\Local\Continuum\anaconda3\lib\site-packages\deepdiff\delta.py in _do_values_or_type_changed(self, changes, is_type_change)
        358 
        359             self._set_new_value(parent, parent_to_obj_elem, parent_to_obj_action,
    --> 360                                 obj, elements, path, elem, action, new_value)
        361 
        362             self._do_verify_changes(path, expected_old_value, current_old_value)
    
    ~\AppData\Local\Continuum\anaconda3\lib\site-packages\deepdiff\delta.py in _set_new_value(self, parent, parent_to_obj_elem, parent_to_obj_action, obj, elements, path, elem, action, new_value)
        234         self._simple_set_elem_value(obj=obj, path_for_err_reporting=path, elem=elem,
        235                                     value=new_value, action=action)
    --> 236         if obj_is_new and parent:
        237             # Making sure that the object is re-instated inside the parent especially if it was immutable
        238             # and we had to turn it into a mutable one. In such cases the object has a new id.
    
    ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
    
    

    Expected behavior return of data structure matching numpy array a2

    OS, DeepDiff version and Python version (please complete the following information):

    • OS: Win7
    • Python: 3.7.4 (default, Aug 9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)]
    • DeepDiff Version: '5.0.0' (From Dev today)
    bug 
    opened by David-Herman 11
  • option to ignore type changes (unicode vs string), or (datetime vs string).

    option to ignore type changes (unicode vs string), or (datetime vs string).

    working with comparing previous versions of objects to the api, the revision storage creates some natural deltas from strings to unicode, and from datetime objects to iso strings.

    enhancement 
    opened by kapilt 11
  • added an option for a custom type with a custom comparison function.

    added an option for a custom type with a custom comparison function.

    input is a python dict() in which the keys are custom types and the values are custom function handles which compare (x, y). tested locally on the data-structures for which I added this features; without this addition the code got choked. with it the comparison works like a charm.

    opened by roi-f 10
  • Pretty form

    Pretty form

    Feature requested in issue #85

    Added method pretty_form() to DeepDiff class. Method returns a string with verbose description of changes, for example:

    Item root[5] added to dictionary.
    Item root[3] removed from dictionary.
    Type of root[2] changed from int to str and value changed from 2 to "b".
    Value of root[4] changed from 4 to 5.
    

    There is one text pattern for each available report_type.

    opened by MKaras93 10
  • Pretty Difference output mode?

    Pretty Difference output mode?

    I was wondering if a pretty output wouldn't be a nice addon to the library. Something more human readable ?

    For instance the below : {'type_changes': {"root['a']['b']['c']": {'old_type': <type 'str'>, 'new_value': 42, 'old_value': 'foo', 'new_type': <type 'int'>}}}

    could be replaced with something like : type_changes for a.b.c, "foo" (string type) replaced by 42 (integer type)

    Creating a wrapper for the library doesn't seem like the best way to overcome this, as a new update of the core library could break the wrapper.

    enhancement help wanted 
    opened by dbrrt 10
  • tests: Use tmp_path fixture

    tests: Use tmp_path fixture

    Fixes permission errors when running tests inside a sandbox, using the tmp_path pytest fixture https://docs.pytest.org/en/6.2.x/tmpdir.html

    FAILED test_command.py::TestCommands::test_deeppatch_command[t1.json-t2.json-args0-0] - PermissionError: [Errno 13] Permission denied: '/tmp/t1.json'
    FAILED test_command.py::TestCommands::test_deeppatch_command[t1_corrupt.json-t2.json-args1-1] - PermissionError: [Errno 13] Permission denied: '/tmp/t1_corrupt.json'
    FAILED test_command.py::TestCommands::test_deeppatch_command[t1.json-t2_json.csv-args2-0] - PermissionError: [Errno 13] Permission denied: '/tmp/t1.json'
    FAILED test_command.py::TestCommands::test_deeppatch_command[t2_json.csv-t1.json-args3-0] - PermissionError: [Errno 13] Permission denied: '/tmp/delta.pickle'
    FAILED test_command.py::TestCommands::test_deeppatch_command[t1.csv-t2.csv-args4-0] - PermissionError: [Errno 13] Permission denied: '/tmp/delta.pickle'
    FAILED test_command.py::TestCommands::test_deeppatch_command[t1.toml-t2.toml-args5-0] - PermissionError: [Errno 13] Permission denied: '/tmp/delta.pickle'
    FAILED test_command.py::TestCommands::test_deeppatch_command[t1.pickle-t2.pickle-args6-0] - PermissionError: [Errno 13] Permission denied: '/tmp/delta.pickle'
    FAILED test_command.py::TestCommands::test_deeppatch_command[t1.yaml-t2.yaml-args7-0] - PermissionError: [Errno 13] Permission denied: '/tmp/delta.pickle'
    
    opened by martin-kokos 0
  • Option for acknowledging item ordering

    Option for acknowledging item ordering

    I've noticed that the hash of an object is the same if a list contained in the object has items moved around. E.g.

    obj1 = {"key": [{1: "one"},{2: "two"}]}
    obj2 = {"key": [{2: "two"},{1: "one"}]}
    

    Both of the above result in the same hash.

    Strictly speaking this should be considered a change and should result in a different hash.

    Describe the solution you'd like I'd ideally like the default handling to acknowledge item ordering and result in a different hash value.

    Describe alternatives you've considered Failing it being the default handling, it would be good if there was an option to set item ordering acknowledgements, resulting in a different hash value.

    opened by ianyoung 2
  • when accessing property raises an exception

    when accessing property raises an exception

    Describe the bug Deepdiff 5.8.1 added support for comparing @property values, but when calling a @property getter raises an exception (for some reason), deepdiff doesn't perform a comparison.

    To Reproduce

    class A:
        def __init__(self, fname=None):
            self.fname = fname
        @property
        def content(self):
            return open(self.fname).read()
         
    print(DeepDiff(A("/not/existing/file"),A("/not/existing/file")))
    

    Expected behavior deepdiff compares exceptions from two objects and checks if they are the same exception. Or have an option to change the behavior to do so.

    opened by keighrim 0
  • Type hints/stubs

    Type hints/stubs

    It would be really helpful to have type hints or stubs for this package to help the developer in his experience using the library*.

    I have found no alternative solution, for now.

    opened by ethiy 3
  • Delta not working when dictionary keys contain quotes

    Delta not working when dictionary keys contain quotes

    Describe the bug

    If a dictionary key contains a quote character (' or ") then deltas cannot be applied to it.

    To Reproduce

    test = {"test'":  3}
    new_test = {"test'": 4}
    diff = DeepDiff(test, new_test)
    delta = Delta(diff)
    print(test + delta)
    

    Gives the following:

    Unable to get the item at root['test'']
    {"test'": 3}
    

    Expected behavior

    {"test'": 4}
    

    OS, DeepDiff version and Python version (please complete the following information):

    • OS: [e.g. Ubuntu] MacOS
    • Version [e.g. 20LTS]
    • Python Version [e.g. 3.9.12] 3.9.13
    • DeepDiff Version [e.g. 5.8.0] deepdiff6==6.2.0

    Additional context

    Happy to work on a fix if it would help :smiley:

    opened by ethanwharris 1
Releases(6.2.1)
  • 6.2.1(Oct 30, 2022)

  • 6.1.0(Aug 28, 2022)

    • [x] DeepDiff.affected_paths can be used to get the list of all paths where a change, addition, or deletion was reported for.
    • [x] DeepDiff.affected_root_keys can be used to get the list of all paths where a change, addition, or deletion was reported for.
    • [x] Bugfix: ValueError when using Decimal 0.x #339 by Enric Pou
    • [x] Serialization of UUID
    Source code(tar.gz)
    Source code(zip)
  • 6.0.0(Aug 26, 2022)

    Note: This is a version change for DeepDiff since we are still waiting for Pypi to let us publish a new version of DeepDiff. In the meantime we are publishing under DeepDiff6 package name on pypi.

    So you will need to do pip install deepdiff6

    Source code(tar.gz)
    Source code(zip)
  • v5.8.2(May 17, 2022)

  • v5.8.1(May 13, 2022)

    New In DeepDiff 5-8-1

    DeepDiff 5-8-1 includes bug fixes: - Fixed test suite for 32bit systems (https://github.com/seperman/deepdiff/issues/302) by Louis-Philippe Véronneau_ - Fixed the issue when using ignore_order=True and group_by simultaneously - Added the support for diffing object properties (@property) (https://github.com/seperman/deepdiff/issues/312) - Better support of diffing private variables

    .. _Louis-Philippe Véronneau: https://github.com/baldurmen

    New In DeepDiff 5-8-0

    DeepDiff 5-8-0 includes bug fixes and improvements:

    • Fixed the bug with delta randomly not producing the same results when ignore_order=True
    • Display detailed pretty when verbose
    • Allow ordered-set version 4.1.x
    • Removing extra logging when key is not found in DeepHash
    • Fixed error when comparing non-utf8 byte strings with ignore_order=True
    • Fixed Tests fail after 2022-05-14
    • Fixed TypeError is thrown when comparing bool and str
    Source code(tar.gz)
    Source code(zip)
  • 5.6.0(Oct 13, 2021)

    • v5-6-0: Adding custom operators, and ignore_order_func. Bugfix: verbose_level==0 should disable values_changes. Bugfix: unprocessed key error.
    Source code(tar.gz)
    Source code(zip)
  • 5.5.0(Apr 29, 2021)

    • v5-5-0: adding iterable_compare_func for DeepDiff, adding output_format of list for path() in tree view.
    • v5-4-0: adding strict_checking for numbers in DeepSearch.
    Source code(tar.gz)
    Source code(zip)
  • 5.3.0(Apr 16, 2021)

  • 5.2.3(Feb 16, 2021)

    • v5-2-3: Retaining the order of multiple dictionary items added via Delta. Fixed the typo with yml files in deep cli. Fixing Grep RecursionError where using non UTF-8 character. Allowing kwargs to be passed to to_json method.
    Source code(tar.gz)
    Source code(zip)
  • 5.2.2(Jan 15, 2021)

  • 5.2.1(Jan 2, 2021)

    • v5-2-0: Removed Murmur3 as the preferred hashing method. Using SHA256 by default now. Added commandline for deepdiff. Added group_by. Added math_epsilon. Improved ignoring of NoneType.
    Source code(tar.gz)
    Source code(zip)
  • 5.0.2(Jul 23, 2020)

    • v5-0-2: Bug Fix NoneType in ignore type groups https://github.com/seperman/deepdiff/issues/207
    • v5-0-1: Bug fix to not apply format to non numbers.
    Source code(tar.gz)
    Source code(zip)
  • 5.0.0(Jun 23, 2020)

    • v5-0-0: Introducing the Delta object, Improving Numpy support, Fixing tuples comparison when ignore_order=True, Dramatically improving the results when ignore_order=True by running in passes, Introducing pretty print view, deep_distance, purge, progress logging, cache and truncate_datetime.
    Source code(tar.gz)
    Source code(zip)
  • 4.3.2(Mar 19, 2020)

  • 4.3.1(Mar 11, 2020)

    • v4-3-1: Fixing the issue with exclude_path and hash calculations when dictionaries were inside iterables. https://github.com/seperman/deepdiff/issues/174
    • v4-3-0: adding exclude_obj_callback
    Source code(tar.gz)
    Source code(zip)
  • 4.2.0(Jan 30, 2020)

    4.2.0 Release

    • .json property is finally removed.

    • Fix for Py3.10.

    • Dropping support for EOL Python 3.4.

    • Ignoring private keys when calculating hashes. For example __init__ is not a part of hash calculation anymore.

    • Fix for #166 Problem with comparing lists, with boolean as element.

    • v4-0-9: Fixing the bug for hashing custom unhashable objects

    • v4-0-8: Adding ignore_nan_inequality for float('nan')

    Source code(tar.gz)
    Source code(zip)
  • 4.0.7(Jul 12, 2019)

  • 4.0.6(Apr 13, 2019)

  • 4.0.5(Apr 7, 2019)

  • 4.0.4(Apr 5, 2019)

    • v4-0-4: Adding ignore_string_case and ignore_type_subclasses
    • v4-0-3: Adding versionbump tool for release
    • v4-0-2: Fixing installation issue where rst files are missing.
    • v4-0-1: Fixing installation Tarball missing requirements.txt . DeepDiff v4+ should not show up as pip installable for Py2. Making Murmur3 installation optional.
    Source code(tar.gz)
    Source code(zip)
  • 4.0.0(Mar 20, 2019)

    Ending Python 2 support, Adding more functionalities and documentation for DeepHash. Switching to Pytest for testing. Switching to Murmur3 128bit for hashing. Fixing classes which inherit from classes with slots didn't have all of their slots compared. Renaming ContentHash to DeepHash. Adding exclude by path and regex path to DeepHash. Adding ignore_type_in_groups. Adding match_string to DeepSearch. Adding Timedelta object diffing.

    Source code(tar.gz)
    Source code(zip)
Owner
Sep Dehpour
Just released DeepDiff 5.6.0 !
Sep Dehpour
Easy compression and extraction for any compression or archival format.

Tzar: Tar, Zip, Anything Really Easy compression and extraction for any compression or archival format. Usage/Examples tzar compress large-dir compres

DanielVZ 37 Nov 2, 2022
Convert any-bit number to decimal number and vise versa.

2deci Convert any-bit number to decimal number and vise versa. --bit n to set bit to n --exp xxx to set expression to xxx --r to run reversely (from d

null 3 Sep 15, 2021
🌲 A simple BST (Binary Search Tree) generator written in python

Tree-Traversals (BST) ?? A simple BST (Binary Search Tree) generator written in python Installation Use the package manager pip to install BST. Usage

Jan Kupczyk 1 Dec 12, 2021
Check the basic quality of any dataset

Data Quality Checker in Python Check the basic quality of any dataset. Sneak Peek Read full tutorial at Medium. Explore the app Requirements python 3.

MalaDeep 8 Feb 23, 2022
Search, generate & deliver Msfvenom payloads in an quick and easy way

Goal Search, generate & deliver payloads in an quick and easy way Be as simple as possible BUT with all msfvenom payloads. Ever lost time searching th

null 2 Mar 3, 2022
Tool for generating Memory.scan() compatible instruction search patterns

scanpat Tool for generating Frida Memory.scan() compatible instruction search patterns. Powered by r2. Examples $ ./scanpat.py arm.ks:64 'sub sp, sp,

Ole André Vadla Ravnås 13 Sep 19, 2022
Import the module and create an object of the class LocalVariable.

LocalVariable Import the module and create an object of the class LocalVariable. Call the save method with the name and the value of a variable as arg

Sajedur Rahman Fiad 2 Dec 14, 2022
🍰 ConnectMP - An easy and efficient way to share data between Processes in Python.

ConnectMP - Taking Multi-Process Data Sharing to the moon ?? Contribute · Community · Documentation ?? Introduction : ?? ConnectMP is the easiest and

Aiden Ellis 1 Dec 24, 2021
A library for interacting with Path of Exile game and economy data, and a unique loot filter generation framework.

wraeblast A library for interfacing with Path of Exile game and economy data, and a set of item filters geared towards trade league players. Filter Ge

David Gidwani 29 Aug 28, 2022
Python based tool to extract forensic info from EventTranscript.db (Windows Diagnostic Data)

EventTranscriptParser EventTranscriptParser is python based tool to extract forensically useful details from EventTranscript.db (Windows Diagnostic Da

P. Abhiram Kumar 24 Nov 18, 2022
This is a python table of data implementation with styles, colors

Table This is a python table of data implementation with styles, colors Example Table adapts to the lack of data Lambda color features Full power of l

Урядов Алексей 5 Nov 9, 2021
Aggregating gridded data (xarray) to polygons

A package to aggregate gridded data in xarray to polygons in geopandas using area-weighting from the relative area overlaps between pixels and polygons.

Kevin Schwarzwald 42 Nov 9, 2022
Raganarok X: Next Generation Data Dump

Raganarok X Data Dump Raganarok X: Next Generation Data Dump More interesting Files File Name Contains en_langs All the variables you need in English

null 14 Jul 15, 2022
Creates a C array from a hex-string or a stream of binary data.

hex2array-c Creates a C array from a hex-string. Usage Usage: python3 hex2array_c.py HEX_STRING [-h|--help] Use '-' to read the hex string from STDIN.

John Doe 3 Nov 24, 2022
jsoooooooon derulo - Make sure your 'jason derulo' is featured as the first part of your json data

jsonderulo Make sure your 'jason derulo' is featured as the first part of your json data Install: # python pip install jsonderulo poetry add jsonderul

jesse 3 Sep 13, 2021
Tools for binary data on cassette

Micro Manchester Tape Storage Tools for storing binary data on cassette Includes: Python script for encoding Arduino sketch for decoding Eagle CAD fil

Zack Nelson 28 Dec 25, 2022
Early version for manipulate Geo localization data trough API REST.

Backend para obtener los datos (beta) Descripción El servidor está diseñado para recibir y almacenar datos enviados en forma de JSON por una aplicació

Víctor Omar Vento Hernández 1 Nov 14, 2021
Data Utilities e.g. for importing files to onetask

Use this repository to easily convert your source files (csv, txt, excel, json, html) into record-oriented JSON files that can be uploaded into onetask.

onetask.ai 1 Jul 18, 2022
Script to generate a massive volume of data in sql, csv, json or xml format

DataGenerator Made with Python Open for pull requests 1. Dependencies To install required dependencies run pip install -r requirements.txt 2. Executi

icrescenti 3 Sep 20, 2022