Deep Difference and search of any Python object/data.

Sep Dehpour

Last update: Jan 8, 2023

Related tags

General Utilities python diff tree distance comparison delta recursive nested difference deep-search repetition report-repetition distance-calculation deepdiff deephash

Overview

DeepDiff v 5.6.0

DeepDiff Overview

DeepDiff: Deep Difference of dictionaries, iterables, strings and other objects. It will recursively look for all the changes.
DeepSearch: Search for objects within other objects.
DeepHash: Hash any object based on their content.

Tested on Python 3.6+ and PyPy3.

NOTE: Python 2 is not supported any more. DeepDiff v3.3.0 was the last version to support Python 2

NOTE: The last version of DeepDiff to work on Python 3.5 was DeepDiff 5-0-2

Documentation

What is new?

DeepDiff 5-6-0 allows you to pass custom operators.

>> ">

>>> from deepdiff import DeepDiff
>>> from deepdiff.operator import BaseOperator
>>> class CustomClass:
...     def __init__(self, d: dict, l: list):
...         self.dict = d
...         self.dict['list'] = l
...
>>>
>>> custom1 = CustomClass(d=dict(a=1, b=2), l=[1, 2, 3])
>>> custom2 = CustomClass(d=dict(c=3, d=4), l=[1, 2, 3, 2])
>>> custom3 = CustomClass(d=dict(a=1, b=2), l=[1, 2, 3, 4])
>>>
>>>
>>> class ListMatchOperator(BaseOperator):
...     def give_up_diffing(self, level, diff_instance):
...         if set(level.t1.dict['list']) == set(level.t2.dict['list']):
...             return True
...
>>>
>>> DeepDiff(custom1, custom2, custom_operators=[
...     ListMatchOperator(types=[CustomClass])
... ])
{}
>>>
>>>
>>> DeepDiff(custom2, custom3, custom_operators=[
...     ListMatchOperator(types=[CustomClass])
... ])
{'dictionary_item_added': [root.dict['a'], root.dict['b']], 'dictionary_item_removed': [root.dict['c'], root.dict['d']], 'values_changed': {"root.dict['list'][3]": {'new_value': 4, 'old_value': 2}}}
>>>

New in 5-6-0: Dynamic ignore order function

Ignoring order when certain word in the path

>>> from deepdiff import DeepDiff
>>> t1 = {'a': [1, 2], 'b': [3, 4]}
>>> t2 = {'a': [2, 1], 'b': [4, 3]}
>>> DeepDiff(t1, t2, ignore_order=True)
{}
>>> def ignore_order_func(level):
...     return 'a' in level.path()
...
>>> DeepDiff(t1, t2, ignore_order=True, ignore_order_func=ignore_order_func)
{'values_changed': {"root['b'][0]": {'new_value': 4, 'old_value': 3}, "root['b'][1]": {'new_value': 3, 'old_value': 4}}}

Installation

Install from PyPi:

pip install deepdiff

If you want to use DeepDiff from commandline:

pip install "deepdiff[cli]"

Importing

>>> from deepdiff import DeepDiff  # For Deep Difference of 2 objects
>>> from deepdiff import grep, DeepSearch  # For finding if item exists in an object
>>> from deepdiff import DeepHash  # For hashing objects based on their contents

Note: if you want to use DeepDiff via commandline, make sure to run pip install "deepdiff[cli]". Then you can access the commands via:

DeepDiff
- $ deep diff --help
Delta
- $ deep patch --help
grep
- $ deep grep --help
extract
- $ deep extract --help

Deep Diff

DeepDiff gets the difference of 2 objects.

Please take a look at the DeepDiff docs

The full documentation of all modules can be found on https://zepworks.com/deepdiff/5.6.0/

Tutorials and posts about DeepDiff can be found on https://zepworks.com/tags/deepdiff/

A few Examples

Note: This is just a brief overview of what DeepDiff can do. Please visit https://zepworks.com/deepdiff/5.6.0/ for full documentation.

List difference ignoring order or duplicates

>> t2 = {1:1, 2:2, 3:3, 4:{"a":"hello", "b":[1, 3, 2, 3]}} >>> ddiff = DeepDiff(t1, t2, ignore_order=True) >>> print (ddiff) {} ">

>>> t1 = {1:1, 2:2, 3:3, 4:{"a":"hello", "b":[1, 2, 3]}}
>>> t2 = {1:1, 2:2, 3:3, 4:{"a":"hello", "b":[1, 3, 2, 3]}}
>>> ddiff = DeepDiff(t1, t2, ignore_order=True)
>>> print (ddiff)
{}

Report repetitions

This flag ONLY works when ignoring order is enabled.

t1 = [1, 3, 1, 4]
t2 = [4, 4, 1]
ddiff = DeepDiff(t1, t2, ignore_order=True, report_repetition=True)
print(ddiff)

which will print you:

{'iterable_item_removed': {'root[1]': 3},
  'repetition_change': {'root[0]': {'old_repeat': 2,
                                    'old_indexes': [0, 2],
                                    'new_indexes': [2],
                                    'value': 1,
                                    'new_repeat': 1},
                        'root[3]': {'old_repeat': 1,
                                    'old_indexes': [3],
                                    'new_indexes': [0, 1],
                                    'value': 4,
                                    'new_repeat': 2}}}

Exclude certain types from comparison:

>> l2 = logging.getLogger("test2") >>> t1 = {"log": l1, 2: 1337} >>> t2 = {"log": l2, 2: 1337} >>> print(DeepDiff(t1, t2, exclude_types={logging.Logger})) {} ">

>>> l1 = logging.getLogger("test")
>>> l2 = logging.getLogger("test2")
>>> t1 = {"log": l1, 2: 1337}
>>> t2 = {"log": l2, 2: 1337}
>>> print(DeepDiff(t1, t2, exclude_types={logging.Logger}))
{}

Exclude part of your object tree from comparison

>> t2 = {"for life": "vegan", "ingredients": ["veggies", "tofu", "soy sauce"]} >>> print (DeepDiff(t1, t2, exclude_paths={"root['ingredients']"})) {} ">

>>> t1 = {"for life": "vegan", "ingredients": ["no meat", "no eggs", "no dairy"]}
>>> t2 = {"for life": "vegan", "ingredients": ["veggies", "tofu", "soy sauce"]}
>>> print (DeepDiff(t1, t2, exclude_paths={"root['ingredients']"}))
{}

Exclude Regex Paths

You can also exclude using regular expressions by using exclude_regex_paths and pass a set or list of path regexes to exclude. The items in the list could be raw regex strings or compiled regex objects.

>> exclude_path = re.compile(r"root\[\d+\]\['b'\]") >>> print(DeepDiff(t1, t2, exclude_regex_paths=[exclude_path])) {} ">

>>> t1 = [{'a': 1, 'b': 2}, {'c': 4, 'b': 5}]
>>> t2 = [{'a': 1, 'b': 3}, {'c': 4, 'b': 5}]
>>> print(DeepDiff(t1, t2, exclude_regex_paths={r"root\[\d+\]\['b'\]"}))
{}
>>> exclude_path = re.compile(r"root\[\d+\]\['b'\]")
>>> print(DeepDiff(t1, t2, exclude_regex_paths=[exclude_path]))
{}

Significant Digits

Digits after the decimal point. Internally it uses "{:.Xf}".format(Your Number) to compare numbers where X=significant_digits

>>> t1 = Decimal('1.52')
>>> t2 = Decimal('1.57')
>>> DeepDiff(t1, t2, significant_digits=0)
{}
>>> DeepDiff(t1, t2, significant_digits=1)
{'values_changed': {'root': {'old_value': Decimal('1.52'), 'new_value': Decimal('1.57')}}}

Ignore Type Number - List that contains float and integer:

>>> from deepdiff import DeepDiff
>>> from pprint import pprint
>>> t1 = [1, 2, 3]
>>> t2 = [1.0, 2.0, 3.0]
>>> ddiff = DeepDiff(t1, t2)
>>> pprint(ddiff, indent=2)
{ 'type_changes': { 'root[0]': { 'new_type': <class 'float'>,
                         'new_value': 1.0,
                         'old_type': <class 'int'>,
                         'old_value': 1},
            'root[1]': { 'new_type': <class 'float'>,
                         'new_value': 2.0,
                         'old_type': <class 'int'>,
                         'old_value': 2},
            'root[2]': { 'new_type': <class 'float'>,
                         'new_value': 3.0,
                         'old_type': <class 'int'>,
                         'old_value': 3}}}
>>> ddiff = DeepDiff(t1, t2, ignore_type_in_groups=[(int, float)])
{}

Views

Starting with DeepDiff v 3, there are two different views into your diffed data: text view (original) and tree view (new).

Text View

Text view is the original and currently the default view of DeepDiff.

It is called text view because the results contain texts that represent the path to the data:

Example of using the text view.

>>> from deepdiff import DeepDiff
>>> t1 = {1:1, 3:3, 4:4}
>>> t2 = {1:1, 3:3, 5:5, 6:6}
>>> ddiff = DeepDiff(t1, t2)
>>> print(ddiff)
{'dictionary_item_added': {'root[5]', 'root[6]'}, 'dictionary_item_removed': {'root[4]'}}

So for example ddiff['dictionary_item_removed'] is a set if strings thus this is called the text view.

The following examples are using the *default text view.*
The Tree View is introduced in DeepDiff v3
and provides traversing capabilities through your diffed data and more!
Read more about the Tree View at the [tree view section](#tree-view) of this page.

Tree View

Starting the version v3 You can choose the view into the deepdiff results. The tree view provides you with tree objects that you can traverse through to find the parents of the objects that are diffed and the actual objects that are being diffed.

Value of an item has changed (Tree View)

>>> from deepdiff import DeepDiff
>>> from pprint import pprint
>>> t1 = {1:1, 2:2, 3:3}
>>> t2 = {1:1, 2:4, 3:3}
>>> ddiff_verbose0 = DeepDiff(t1, t2, verbose_level=0, view='tree')
>>> ddiff_verbose0
{'values_changed': {<root[2]>}}
>>>
>>> ddiff_verbose1 = DeepDiff(t1, t2, verbose_level=1, view='tree')
>>> ddiff_verbose1
{'values_changed': {<root[2] t1:2, t2:4>}}
>>> set_of_values_changed = ddiff_verbose1['values_changed']
>>> # since set_of_values_changed includes only one item in a set
>>> # in order to get that one item we can:
>>> (changed,) = set_of_values_changed
>>> changed  # Another way to get this is to do: changed=list(set_of_values_changed)[0]
<root[2] t1:2, t2:4>
>>> changed.t1
2
>>> changed.t2
4
>>> # You can traverse through the tree, get to the parents!
>>> changed.up
<root t1:{1: 1, 2: 2,...}, t2:{1: 1, 2: 4,...}>

Serialization

In order to convert the DeepDiff object into a normal Python dictionary, use the to_dict() method. Note that to_dict will use the text view even if you did the diff in tree view.

Example:

 >> t2 = {1: 1, 2: 2, 3: 3, 4: {"a": "hello", "b": "world\n\n\nEnd"}} >>> ddiff = DeepDiff(t1, t2, view='tree') >>> ddiff.to_dict() {'type_changes': {"root[4]['b']": {'old_type': 
  , 'new_type': 
  
   , 'old_value': [1, 2, 3], 'new_value': 'world\n\n\nEnd'}}} "> 
   >>> t1 = {1: 1, 2: 2, 3: 3, 4: {"a": "hello", "b": [1, 2, 3]}}
>>> t2 = {1: 1, 2: 2, 3: 3, 4: {"a": "hello", "b": "world\n\n\nEnd"}}
>>> ddiff = DeepDiff(t1, t2, view='tree')
>>> ddiff.to_dict()
{'type_changes': {"root[4]['b']": {'old_type': <class 'list'>, 'new_type': <class 'str'>, 'old_value': [1, 2, 3], 'new_value': 'world\n\n\nEnd'}}} 
  
 

In order to do safe json serialization, use the to_json() method.

Example:

>> t2 = {1: 1, 2: 2, 3: 3, 4: {"a": "hello", "b": "world\n\n\nEnd"}} >>> ddiff = DeepDiff(t1, t2, view='tree') >>> ddiff.to_json() '{"type_changes": {"root[4][\'b\']": {"old_type": "list", "new_type": "str", "old_value": [1, 2, 3], "new_value": "world\\n\\n\\nEnd"}}}' ">

>>> t1 = {1: 1, 2: 2, 3: 3, 4: {"a": "hello", "b": [1, 2, 3]}}
>>> t2 = {1: 1, 2: 2, 3: 3, 4: {"a": "hello", "b": "world\n\n\nEnd"}}
>>> ddiff = DeepDiff(t1, t2, view='tree')
>>> ddiff.to_json()
'{"type_changes": {"root[4][\'b\']": {"old_type": "list", "new_type": "str", "old_value": [1, 2, 3], "new_value": "world\\n\\n\\nEnd"}}}'

Please take a look at the DeepDiff docs

The full documentation can be found on https://zepworks.com/deepdiff/5.6.0/

Deep Search

DeepDiff comes with a utility to find the path to the item you are looking for. It is called DeepSearch and it has a similar interface to DeepDiff.

Let's say you have a huge nested object and want to see if any item with the word somewhere exists in it. Just grep through your objects as you would in shell!

from deepdiff import grep
obj = {"long": "somewhere", "string": 2, 0: 0, "somewhere": "around"}
ds = obj | grep("somewhere")
print(ds)

Which will print:

{'matched_paths': {"root['somewhere']"},
 'matched_values': {"root['long']"}}

And you can pass all the same kwargs as DeepSearch to grep too:

>>> obj | grep(item, verbose_level=2)
{'matched_paths': {"root['somewhere']": 'around'}, 'matched_values': {"root['long']": 'somewhere'}}

Please take a look at the DeepSearch docs

The full documentation can be found on https://zepworks.com/deepdiff/5.6.0/

Deep Hash

(New in v4-0-0)

DeepHash is designed to give you hash of ANY python object based on its contents even if the object is not considered hashable! DeepHash is supposed to be deterministic in order to make sure 2 objects that contain the same data, produce the same hash.

Please take a look at the DeepHash docs

The full documentation can be found on https://zepworks.com/deepdiff/5.6.0/

Let's say you have a dictionary object.

>>> from deepdiff import DeepHash
>>>
>>> obj = {1: 2, 'a': 'b'}

If you try to hash it:

 
  ", line 1, in 
  
    TypeError: unhashable type: 'dict' "> 
   >>> hash(obj)
Traceback (most recent call last):
  File "
     
      "
     , line 1, in <module>
TypeError: unhashable type: 'dict' 
  
 

But with DeepHash:

>>> from deepdiff import DeepHash
>>> obj = {1: 2, 'a': 'b'}
>>> DeepHash(obj)
{4355639248: 2468916477072481777512283587789292749, 4355639280: -35787773492556653776377555218122431491, 4358636128: -88390647972316138151822486391929534118, 4358009664: 8833996863197925870419376694314494743, 4357467952: 34150898645750099477987229399128149852}

So what is exactly the hash of obj in this case? DeepHash is calculating the hash of the obj and any other object that obj contains. The output of DeepHash is a dictionary of object IDs to their hashes. In order to get the hash of obj itself, you need to use the object (or the id of object) to get its hash:

>>> hashes = DeepHash(obj)
>>> hashes[obj]
34150898645750099477987229399128149852

Which you can write as:

>>> hashes = DeepHash(obj)[obj]

At first it might seem weird why DeepHash(obj)[obj] but remember that DeepHash(obj) is a dictionary of hashes of all other objects that obj contains too.

Please take a look at the DeepHash docs

The full documentation can be found on https://zepworks.com/deepdiff/5.6.0/

Using DeepDiff in unit tests

result is the output of the function that is being tests. expected is the expected output of the function.

self.assertEqual(DeepDiff(expected, result), {})

or if you are using Pytest:

assert not DeepDiff(expected, result)

In other words, assert that there is no diff between the expected and the result.

Difference with Json Patch

Unlike Json Patch which is designed only for Json objects, DeepDiff is designed specifically for almost all Python types. In addition to that, DeepDiff checks for type changes and attribute value changes that Json Patch does not cover since there are no such things in Json. Last but not least, DeepDiff gives you the exact path of the item(s) that were changed in Python syntax.

Example in Json Patch for replacing:

{ "op": "replace", "path": "/a/b/c", "value": 42 }

Example in DeepDiff for the same operation:

 , 'new_value': 42, 'old_value': 'foo', 'new_type': 
  }}} "> 
  >>> item1 = {'a':{'b':{'c':'foo'}}}
>>> item2 = {'a':{'b':{'c':42}}}
>>> DeepDiff(item1, item2)
{'type_changes': {"root['a']['b']['c']": {'old_type': <type 'str'>, 'new_value': 42, 'old_value': 'foo', 'new_type': <type 'int'>}}} 
 

Documentation

https://zepworks.com/deepdiff/current/

Pycon 2016

I was honored to give a talk about the basics of how DeepDiff does what it does at Pycon 2016. Please check out the video and let me know what you think:

Diff It To Dig It Video And here is more info: http://zepworks.com/blog/diff-it-to-digg-it/

ChangeLog

Please take a look at the CHANGELOG file.

Releases

We use bump2version to bump and tag releases.

git checkout master && git pull
bumpversion {patch|minor|major}
git push && git push --tags

Contribute

Please make your PR against the dev branch
Please make sure that your PR has tests. Since DeepDiff is used in many sensitive data driven projects, we strive to maintain around 100% test coverage on the code.

Please run pytest --cov=deepdiff --runslow to see the coverage report. Note that the --runslow flag will run some slow tests too. In most cases you only want to run the fast tests which so you won't add the --runslow flag.

Or to see a more user friendly version, please run: pytest --cov=deepdiff --cov-report term-missing --runslow.

Thank you!

Authors

Please take a look at the AUTHORS file.

Comments

V3
A new diff model to replace parents. This model allows an easy way to traverse through parent, child in a diff graph. There is one graph per diff just like how there was one parents per diff.
opened by seperman 55
Return object refs
Hi Sep,

I finally got around to implement the feature I talked about earlier. I changed DeepDiff to provide and use internally a new, less processed view of a DeepDiff result. This "reference-style" view provides a change as a linked list of the object path leading to the actual change, allowing access to all objects in this path on both sides of the comparison.

An individual result is now of the new type "DiffLevel", which is the leaf of this linked list. I tried to depict the broad idea in refdesign.odt (which is kinda ugly and should probably be removed before merging this).

The basic idea of this was to allow a program employing DeepDiff to further process it's results. For example, I'm currently using DeepDiff in a script that fetches and displays an overview of permission data in some web application. I use DeepDiff to figure out where this data has changed. Afterwards, I mark the data DeepDiff has found as added or deleted in an HTML view.

Used in this way DeepDiff can free the developer from writing custom comparison code.

The new reference-style view for DeepDiff allows me to do something like this:

permdata = self.model.permissions olddata = self.cmp.permissions diff = DeepDiff(olddata, permdata, default_view='ref') # enable new view if 'set_item_removed' in diff: for change in diff['set_item_removed']: parentset = change.up.t2 parentset.add("<del>" + change.t1 + "</del>")

With my changes DeepDiff now uses this new structure to recreate the traditional result format. All existing tests pass and I'm positive this is fully compatible to the way DeepDiff behaved previously.

I know this is a huge change to DeepDiff's internals but I hope you can see the benefit of it :) There are still a bunch of open TODOs in the code but I'd love to work with you to get this merged.

Best, Victor
opened by victorhahncastell 18
Bugfix for significant_digits (signed zero); Numpy-friendlyness; Issue #49; Failing tests for #50;
This pullrequests makes several small changes. Fell free to merge only selected commits if desired. Most commit contain changes to the main source code and tests.

I add 2 failing tests to document issue #50

I compare self.down.t1 to None instead of using if self.down.ta in auto_generate_child_rel, because numpy arrays rise an error when used as booleans. I hope this changed behaviour does not introduce any bugs with other edge-cases.

I added a bugfix to __diff_numbers for the significant_digits option. Now -0 and +0 compare equal.

Finally I use __diff_dict instead of __diff_iterable for the comparison of collections.abc.Mapping instances (before __diff_dict required the container to be a MutableMapping) - see issue #49

bug
opened by Bernhard10 15
How to align objects in lists properly?
Hi. Sorry, I don't know whether it's a bug/feature or just a question. I've asked it some time ago on StackOverflow, but didn't receive any answer, so I decided to try here.

I'm trying to compare two lists of objects (dicts in this case) with deepdiff:

old = [ {'name': 'war', 'status': 'active'}, {'name': 'drought', 'status': 'pending'} ] new = [ {'name': 'war', 'status': 'pending'}, {'name': 'fire', 'status': 'pending'}] DeepDiff(old, new) # Result: {'values_changed': {"root[0]['status']": {'new_value': 'pending', 'old_value': 'active'}, "root[1]['name']": {'new_value': 'fire', 'old_value': 'drought'}}}

The problem is that I need a different way of aligning objects. In my project a particular state (for example war) have a strict life cycle: appears as pending, transforms to active and disappears. I want to use deepdiff to track these changes. Objects with different names are different objects and I don't want them to align with each other.

So the result I expect is:

'values_changed': {"root[0]['status']": {'new_value': 'pending', 'old_value': 'active'}, 'iterable_item_removed': {'root[1]': {'name': 'drought', 'status': 'pending'}}} 'iterable_item_added': {'root[1]': {'name': 'fire', 'status': 'pending'}}}

Is there a way I can modify my object to align it properly? I've tried replacing the dict with a class with custom __eq__ method, but it didn't work. Do you have any other suggestion how I can make objects with the same name align only with each other?

Here is the original stack overflow question.
opened by MKaras93 14

DeepDiff to run in multiple passes to diff combinations of results when ignore_order=True

DeepDiff to run in 2 passes. And diff combinations of results when ignore_order=True.

Example:

Currently:

In [2]: from deepdiff import DeepDiff

In [3]: DeepDiff({'a': [1,2,3]}, {'a': [3,2,1, 0]}, ignore_order=True)
Out[3]: {'iterable_item_added': {"root['a'][3]": 0}}

In [4]: DeepDiff({'a': [{'b': [1,2,3]}]}, {'a': [{'b': [3,2,1, 0]}]}, ignore_order=True)
Out[4]:
{'iterable_item_added': {"root['a'][0]": {'b': [3, 2, 1, 0]}},
 'iterable_item_removed': {"root['a'][0]": {'b': [1, 2, 3]}}}

But if deepdiff compares the items between the iterable item added and removed, it should be spitting out the following results instead:

In [4]: DeepDiff({'a': [{'b': [1,2,3]}]}, {'a': [{'b': [3,2,1, 0]}]}, ignore_order=True)
Out[4]:  {'iterable_item_added': {"root['a'][0][3]": 0}}

enhancement

opened by seperman 13

Support searching with regular expressions

I'd like to use regular expressions to find elements of a dictionary with grep.

(Pdb) entity_attributes
{'alone': {'id_': 'alone', 'name': 'Timothy', 'last_name': 'Jackson', 'country': 'Korea', 'rating': 5491}, 'film': {'id_': 'film', 'name': 'David', 'last_name': 'Howell', 'country': 'Moldova', 'rating': 9731}, 'thought': {'id_': 'thought'
, 'name': 'Jason', 'last_name': 'Roberts', 'country': 'Western Sahara', 'rating': 5484}}
(Pdb) entity_attributes | grep('Da')
{'matched_values': ["root['film']['name']"]}
(Pdb) entity_attributes | grep('Da.*')
{}

If you give me some guidance on how to do it, I can make a PR

opened by lyz-code 11

Delta + Numpy structure throws ambiguous true value

Describe the bug I am using the delta of the diff to generate the other input again. When using numpy arrays as input (or dictionaries of lists of arrays, etc.) it throws an exception, ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all().

To Reproduce Steps to reproduce the behavior:

Input Code

a1 = np.array([1,2,3,4])
a2 =np.array([5,6,7,8,9,10])
mydiff = DeepDiff(a1, a2)
delta = Delta(mydiff)
delta + a1

Error Trace

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-62-bbde3ce70e97> in <module>
      3 mydiff = DeepDiff(a1, a2)
      4 delta = Delta(mydiff)
----> 5 delta + a1

~\AppData\Local\Continuum\anaconda3\lib\site-packages\deepdiff\delta.py in __add__(self, other)
    146         else:
    147             self.root = deepcopy(other)
--> 148         self._do_values_changed()
    149         self._do_set_item_added()
    150         self._do_set_item_removed()

~\AppData\Local\Continuum\anaconda3\lib\site-packages\deepdiff\delta.py in _do_values_changed(self)
    300         values_changed = self.diff.get('values_changed')
    301         if values_changed:
--> 302             self._do_values_or_type_changed(values_changed)
    303 
    304     def _do_type_changes(self):

~\AppData\Local\Continuum\anaconda3\lib\site-packages\deepdiff\delta.py in _do_values_or_type_changed(self, changes, is_type_change)
    358 
    359             self._set_new_value(parent, parent_to_obj_elem, parent_to_obj_action,
--> 360                                 obj, elements, path, elem, action, new_value)
    361 
    362             self._do_verify_changes(path, expected_old_value, current_old_value)

~\AppData\Local\Continuum\anaconda3\lib\site-packages\deepdiff\delta.py in _set_new_value(self, parent, parent_to_obj_elem, parent_to_obj_action, obj, elements, path, elem, action, new_value)
    234         self._simple_set_elem_value(obj=obj, path_for_err_reporting=path, elem=elem,
    235                                     value=new_value, action=action)
--> 236         if obj_is_new and parent:
    237             # Making sure that the object is re-instated inside the parent especially if it was immutable
    238             # and we had to turn it into a mutable one. In such cases the object has a new id.

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Expected behavior return of data structure matching numpy array a2

OS, DeepDiff version and Python version (please complete the following information):

OS: Win7
Python: 3.7.4 (default, Aug 9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)]
DeepDiff Version: '5.0.0' (From Dev today)

bug

opened by David-Herman 11

option to ignore type changes (unicode vs string), or (datetime vs string).

working with comparing previous versions of objects to the api, the revision storage creates some natural deltas from strings to unicode, and from datetime objects to iso strings.
enhancement

opened by kapilt 11
added an option for a custom type with a custom comparison function.

input is a python dict() in which the keys are custom types and the values are custom function handles which compare (x, y). tested locally on the data-structures for which I added this features; without this addition the code got choked. with it the comparison works like a charm.

opened by roi-f 10
Pretty form
Feature requested in issue #85

Added method pretty_form() to DeepDiff class. Method returns a string with verbose description of changes, for example:

Item root[5] added to dictionary. Item root[3] removed from dictionary. Type of root[2] changed from int to str and value changed from 2 to "b". Value of root[4] changed from 4 to 5.

There is one text pattern for each available report_type.
opened by MKaras93 10
Pretty Difference output mode?

I was wondering if a pretty output wouldn't be a nice addon to the library. Something more human readable ?

For instance the below : {'type_changes': {"root['a']['b']['c']": {'old_type': <type 'str'>, 'new_value': 42, 'old_value': 'foo', 'new_type': <type 'int'>}}}

could be replaced with something like : type_changes for a.b.c, "foo" (string type) replaced by 42 (integer type)

Creating a wrapper for the library doesn't seem like the best way to overcome this, as a new update of the core library could break the wrapper.
enhancement help wanted

opened by dbrrt 10

tests: Use tmp_path fixture

Fixes permission errors when running tests inside a sandbox, using the tmp_path pytest fixture https://docs.pytest.org/en/6.2.x/tmpdir.html

FAILED test_command.py::TestCommands::test_deeppatch_command[t1.json-t2.json-args0-0] - PermissionError: [Errno 13] Permission denied: '/tmp/t1.json'
FAILED test_command.py::TestCommands::test_deeppatch_command[t1_corrupt.json-t2.json-args1-1] - PermissionError: [Errno 13] Permission denied: '/tmp/t1_corrupt.json'
FAILED test_command.py::TestCommands::test_deeppatch_command[t1.json-t2_json.csv-args2-0] - PermissionError: [Errno 13] Permission denied: '/tmp/t1.json'
FAILED test_command.py::TestCommands::test_deeppatch_command[t2_json.csv-t1.json-args3-0] - PermissionError: [Errno 13] Permission denied: '/tmp/delta.pickle'
FAILED test_command.py::TestCommands::test_deeppatch_command[t1.csv-t2.csv-args4-0] - PermissionError: [Errno 13] Permission denied: '/tmp/delta.pickle'
FAILED test_command.py::TestCommands::test_deeppatch_command[t1.toml-t2.toml-args5-0] - PermissionError: [Errno 13] Permission denied: '/tmp/delta.pickle'
FAILED test_command.py::TestCommands::test_deeppatch_command[t1.pickle-t2.pickle-args6-0] - PermissionError: [Errno 13] Permission denied: '/tmp/delta.pickle'
FAILED test_command.py::TestCommands::test_deeppatch_command[t1.yaml-t2.yaml-args7-0] - PermissionError: [Errno 13] Permission denied: '/tmp/delta.pickle'

opened by martin-kokos 0

Option for acknowledging item ordering
I've noticed that the hash of an object is the same if a list contained in the object has items moved around. E.g.

obj1 = {"key": [{1: "one"},{2: "two"}]} obj2 = {"key": [{2: "two"},{1: "one"}]}

Both of the above result in the same hash.

Strictly speaking this should be considered a change and should result in a different hash.

Describe the solution you'd like I'd ideally like the default handling to acknowledge item ordering and result in a different hash value.

Describe alternatives you've considered Failing it being the default handling, it would be good if there was an option to set item ordering acknowledgements, resulting in a different hash value.
opened by ianyoung 2
when accessing property raises an exception
Describe the bug Deepdiff 5.8.1 added support for comparing @property values, but when calling a @property getter raises an exception (for some reason), deepdiff doesn't perform a comparison.

To Reproduce

class A: def __init__(self, fname=None): self.fname = fname @property def content(self): return open(self.fname).read() print(DeepDiff(A("/not/existing/file"),A("/not/existing/file")))

Expected behavior deepdiff compares exceptions from two objects and checks if they are the same exception. Or have an option to change the behavior to do so.
opened by keighrim 0
Type hints/stubs

It would be really helpful to have type hints or stubs for this package to help the developer in his experience using the library*.

I have found no alternative solution, for now.

opened by ethiy 3
Delta not working when dictionary keys contain quotes
Describe the bug

If a dictionary key contains a quote character (' or ") then deltas cannot be applied to it.

To Reproduce

test = {"test'": 3} new_test = {"test'": 4} diff = DeepDiff(test, new_test) delta = Delta(diff) print(test + delta)

Gives the following:

Unable to get the item at root['test''] {"test'": 3}

Expected behavior

{"test'": 4}

OS, DeepDiff version and Python version (please complete the following information):

OS: [e.g. Ubuntu] MacOS

Version [e.g. 20LTS]

Python Version [e.g. 3.9.12] 3.9.13

DeepDiff Version [e.g. 5.8.0] deepdiff6==6.2.0

Additional context

Happy to work on a fix if it would help :smiley:
opened by ethanwharris 1

Releases(6.2.1)

6.2.1(Oct 30, 2022)

Major improvement in the diff report for lists when items are all hashable and the order of items is important.

Source code(tar.gz)
Source code(zip)

6.1.0(Aug 28, 2022)
[x] DeepDiff.affected_paths can be used to get the list of all paths where a change, addition, or deletion was reported for.

[x] DeepDiff.affected_root_keys can be used to get the list of all paths where a change, addition, or deletion was reported for.

[x] Bugfix: ValueError when using Decimal 0.x #339 by Enric Pou

[x] Serialization of UUID

Source code(tar.gz)
Source code(zip)
6.0.0(Aug 26, 2022)

Note: This is a version change for DeepDiff since we are still waiting for Pypi to let us publish a new version of DeepDiff. In the meantime we are publishing under DeepDiff6 package name on pypi.

So you will need to do pip install deepdiff6
Source code(tar.gz)
Source code(zip)
v5.8.2(May 17, 2022)

Fixing dependency for Py3.6
Source code(tar.gz)
Source code(zip)
v5.8.1(May 13, 2022)
New In DeepDiff 5-8-1

DeepDiff 5-8-1 includes bug fixes: - Fixed test suite for 32bit systems (https://github.com/seperman/deepdiff/issues/302) by Louis-Philippe Véronneau_ - Fixed the issue when using ignore_order=True and group_by simultaneously - Added the support for diffing object properties (@property) (https://github.com/seperman/deepdiff/issues/312) - Better support of diffing private variables

.. _Louis-Philippe Véronneau: https://github.com/baldurmen

New In DeepDiff 5-8-0

DeepDiff 5-8-0 includes bug fixes and improvements:

Fixed the bug with delta randomly not producing the same results when ignore_order=True

Display detailed pretty when verbose

Allow ordered-set version 4.1.x

Removing extra logging when key is not found in DeepHash

Fixed error when comparing non-utf8 byte strings with ignore_order=True

Fixed Tests fail after 2022-05-14

Fixed TypeError is thrown when comparing bool and str

Source code(tar.gz)
Source code(zip)
5.6.0(Oct 13, 2021)
v5-6-0: Adding custom operators, and ignore_order_func. Bugfix: verbose_level==0 should disable values_changes. Bugfix: unprocessed key error.

Source code(tar.gz)
Source code(zip)
5.5.0(Apr 29, 2021)
v5-5-0: adding iterable_compare_func for DeepDiff, adding output_format of list for path() in tree view.

v5-4-0: adding strict_checking for numbers in DeepSearch.

Source code(tar.gz)
Source code(zip)
5.3.0(Apr 16, 2021)
v5-3-0: add support for regular expressions in DeepSearch

Source code(tar.gz)
Source code(zip)
5.2.3(Feb 16, 2021)
v5-2-3: Retaining the order of multiple dictionary items added via Delta. Fixed the typo with yml files in deep cli. Fixing Grep RecursionError where using non UTF-8 character. Allowing kwargs to be passed to to_json method.

Source code(tar.gz)
Source code(zip)
5.2.2(Jan 15, 2021)

Fixed Delta serialization when None type is present.
Source code(tar.gz)
Source code(zip)
5.2.1(Jan 2, 2021)
v5-2-0: Removed Murmur3 as the preferred hashing method. Using SHA256 by default now. Added commandline for deepdiff. Added group_by. Added math_epsilon. Improved ignoring of NoneType.

Source code(tar.gz)
Source code(zip)
5.0.2(Jul 23, 2020)
v5-0-2: Bug Fix NoneType in ignore type groups https://github.com/seperman/deepdiff/issues/207

v5-0-1: Bug fix to not apply format to non numbers.

Source code(tar.gz)
Source code(zip)
5.0.0(Jun 23, 2020)
v5-0-0: Introducing the Delta object, Improving Numpy support, Fixing tuples comparison when ignore_order=True, Dramatically improving the results when ignore_order=True by running in passes, Introducing pretty print view, deep_distance, purge, progress logging, cache and truncate_datetime.

Source code(tar.gz)
Source code(zip)
4.3.2(Mar 19, 2020)

v4-3-2: Deprecation Warning Enhancement
Source code(tar.gz)
Source code(zip)
4.3.1(Mar 11, 2020)
v4-3-1: Fixing the issue with exclude_path and hash calculations when dictionaries were inside iterables. https://github.com/seperman/deepdiff/issues/174

v4-3-0: adding exclude_obj_callback

Source code(tar.gz)
Source code(zip)
4.2.0(Jan 30, 2020)
4.2.0 Release

.json property is finally removed.

Fix for Py3.10.

Dropping support for EOL Python 3.4.

Ignoring private keys when calculating hashes. For example __init__ is not a part of hash calculation anymore.

Fix for #166 Problem with comparing lists, with boolean as element.

v4-0-9: Fixing the bug for hashing custom unhashable objects

v4-0-8: Adding ignore_nan_inequality for float('nan')

Source code(tar.gz)
Source code(zip)
4.0.7(Jul 12, 2019)

v4-0-7: Hashing of the number 1 vs. True
Source code(tar.gz)
Source code(zip)
4.0.6(Apr 13, 2019)

Found a tiny bug in Python formatting of numbers in scientific notation. Added a workaround.
Source code(tar.gz)
Source code(zip)
4.0.5(Apr 7, 2019)
v4-0-5: Fixing number diffing. Adding number_format_notation and number_to_string_func.

Source code(tar.gz)
Source code(zip)
4.0.4(Apr 5, 2019)
v4-0-4: Adding ignore_string_case and ignore_type_subclasses

v4-0-3: Adding versionbump tool for release

v4-0-2: Fixing installation issue where rst files are missing.

v4-0-1: Fixing installation Tarball missing requirements.txt . DeepDiff v4+ should not show up as pip installable for Py2. Making Murmur3 installation optional.

Source code(tar.gz)
Source code(zip)
4.0.0(Mar 20, 2019)

Ending Python 2 support, Adding more functionalities and documentation for DeepHash. Switching to Pytest for testing. Switching to Murmur3 128bit for hashing. Fixing classes which inherit from classes with slots didn't have all of their slots compared. Renaming ContentHash to DeepHash. Adding exclude by path and regex path to DeepHash. Adding ignore_type_in_groups. Adding match_string to DeepSearch. Adding Timedelta object diffing.
Source code(tar.gz)
Source code(zip)