Hello there, and thanks for Pympler!

While doing a few experiments on a class that uses __slots__ to reduce memory footprint I ran into a few unexpected results.

Testcase:

import sys
print(sys.version)
from pympler import asizeof

SLOTS = ("foo", "bar")

class Foo(object):
  __slots__ = SLOTS

  def __init__(self):
    self.foo = SLOTS
    self.bar = SLOTS

print(asizeof.asized(Foo(), detail=1).format())
print("Same object? " + str(Foo().__slots__ is SLOTS))

Results:

3.6.5 (default, Jun 20 2018, 09:33:17) 
[GCC 5.4.0 20160609]
<__main__.Foo object at 0x7f143841ac88> size=304 flat=56
    foo size=176 flat=64
    __slots__ size=72 flat=72
    __class__ size=0 flat=0
    bar size=0 flat=0
Same object? True

Three things seem to be wrong in this case:

• the flat size of __slots__ is 8 bytes more than the object itself
• the total size of the Foo instance does not account for the fact that foo and __slots__ are pointing at the same object
• the flat size of __slots__ is shown to be the same as the size

Thus, whereas I would expect in this case the total size to be 56 + 176, pympler reports it as 56 + 176 + 72.

I was able to validate similar results with Python 2.7, so it doesn't seem to be something that may be off in any specific version.

Is this a pympler bug, or am I missing some important detail on how __slots__ are actually implemented that causes this?

It seems that pympler doesn't work with tensorflow, a popular deep learning toolkit. I suspect this has something to do with tensorflow being a python binding to a C++ implementation.

>>> import tensorflow
>>> from pympler.tracker import SummaryTracker
>>> tracker = SummaryTracker()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vlad/.local/lib/python3.5/site-packages/pympler/tracker.py", line 45, in __init__
    self.s0 = summary.summarize(muppy.get_objects())
  File "/home/vlad/.local/lib/python3.5/site-packages/pympler/muppy.py", line 37, in get_objects
    tmp = [o for o in tmp if not isframe(o)]
  File "/home/vlad/.local/lib/python3.5/site-packages/pympler/muppy.py", line 37, in <listcomp>
    tmp = [o for o in tmp if not isframe(o)]
  File "/usr/lib/python3.5/inspect.py", line 239, in isframe
    return isinstance(object, types.FrameType)
ReferenceError: weakly-referenced object no longer exists

Windows 11 Python 3.10.4 64 bits

Trying to measure the size of a SQLAlchemy create_engine instance

engine1 = create_engine(sqlite_url, echo=True) s = asizeof.asizeof(engine1)

Exception has occurred: TypeError cannot create weak reference to '_EmptyListener' object

I test the following code, and the summary is incorrect. a use about 0.8G memory but the summary doesn't contain this information.

from pympler import muppy, summary, asizeof
import numpy as np


class A(object):
    def __init__(self):
        self.a = np.empty((10000, 44, 44, 9, 5), dtype=np.uint8)


_sum = summary.summarize(muppy.get_objects(include_frames=True))
summary.print_(_sum)
a = A()
print(asizeof.asizeof(a) / 1024 / 1024 / 1024)
_sum = summary.summarize(muppy.get_objects(include_frames=True))
summary.print_(_sum)

Output:

                               types |   # objects |   total size
==================================== | =========== | ============
                         <class 'str |       17530 |      3.12 MB
                        <class 'dict |        2625 |      1.49 MB
                        <class 'code |        6004 |    847.42 KB
                        <class 'type |         712 |    726.35 KB
                         <class 'set |         579 |    196.66 KB
          <class 'wrapper_descriptor |        2140 |    167.19 KB
                       <class 'tuple |        2308 |    153.36 KB
                     <class 'weakref |        1519 |    118.67 KB
                        <class 'list |         695 |    109.71 KB
           <class 'method_descriptor |        1291 |     90.77 KB
  <class 'builtin_function_or_method |        1258 |     88.45 KB
                 <class 'abc.ABCMeta |          81 |     78.57 KB
           <class 'getset_descriptor |         976 |     68.62 KB
                         <class 'int |        2032 |     57.10 KB
                 function (__init__) |         338 |     44.89 KB
0.8113685846328735
                               types |   # objects |   total size
==================================== | =========== | ============
                         <class 'str |       21855 |      3.42 MB
                        <class 'dict |        2626 |      1.49 MB
                        <class 'code |        5999 |    846.72 KB
                        <class 'type |         712 |    726.35 KB
                        <class 'list |        4814 |    499.12 KB
                         <class 'set |         579 |    196.66 KB
          <class 'wrapper_descriptor |        2140 |    167.19 KB
                       <class 'tuple |        2287 |    151.31 KB
                     <class 'weakref |        1519 |    118.67 KB
           <class 'method_descriptor |        1291 |     90.77 KB
  <class 'builtin_function_or_method |        1258 |     88.45 KB
                 <class 'abc.ABCMeta |          81 |     78.57 KB
                         <class 'int |        2779 |     77.52 KB
           <class 'getset_descriptor |         976 |     68.62 KB
                 function (__init__) |         338 |     44.89 KB

Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/wsgiref/handlers.py", line 85, in run
    self.result = application(self.environ, self.start_response)
  File "/Users/markddavidoff/Dev/Envs/webenv/lib/python2.7/site-packages/django/contrib/staticfiles/handlers.py", line 63, in __call__
    return self.application(environ, start_response)
  File "/Users/markddavidoff/Dev/Envs/webenv/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 157, in __call__
    response = self.get_response(request)
  File "/Users/markddavidoff/Dev/Envs/webenv/lib/python2.7/site-packages/django/core/handlers/base.py", line 124, in get_response
    response = self._middleware_chain(request)
  File "/Users/markddavidoff/Dev/Envs/webenv/lib/python2.7/site-packages/django/core/handlers/exception.py", line 43, in inner
    response = response_for_exception(request, exc)
  File "/Users/markddavidoff/Dev/Envs/webenv/lib/python2.7/site-packages/django/core/handlers/exception.py", line 93, in response_for_exception
    response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
  File "/Users/markddavidoff/Dev/Envs/webenv/lib/python2.7/site-packages/django/core/handlers/exception.py", line 139, in handle_uncaught_exception
    return debug.technical_500_response(request, *exc_info)
  File "/Users/markddavidoff/Dev/Envs/webenv/lib/python2.7/site-packages/django/views/debug.py", line 84, in technical_500_response
    html = reporter.get_traceback_html()
  File "/Users/markddavidoff/Dev/Envs/webenv/lib/python2.7/site-packages/django/views/debug.py", line 326, in get_traceback_html
    t = DEBUG_ENGINE.from_string(TECHNICAL_500_TEMPLATE)
  File "/Users/markddavidoff/Dev/Envs/webenv/lib/python2.7/site-packages/django/template/engine.py", line 155, in from_string
    return Template(template_code, engine=self)
  File "/Users/markddavidoff/Dev/Envs/webenv/lib/python2.7/site-packages/pympler/classtracker.py", line 335, in <lambda>
    lambda *args, **kwds: func(self._observers[cls], *args, **kwds),
  File "/Users/markddavidoff/Dev/Envs/webenv/lib/python2.7/site-packages/pympler/classtracker.py", line 310, in _tracker
    _observer_.init(_self_, *args, **kwds)
  File "/Users/markddavidoff/Dev/Envs/webenv/lib/python2.7/site-packages/pympler/classtracker.py", line 335, in <lambda>
    lambda *args, **kwds: func(self._observers[cls], *args, **kwds),
  File "/Users/markddavidoff/Dev/Envs/webenv/lib/python2.7/site-packages/pympler/classtracker.py", line 310, in _tracker
    _observer_.init(_self_, *args, **kwds)
  File "/Users/markddavidoff/Dev/Envs/webenv/lib/python2.7/site-packages/pympler/classtracker.py", line 335, in <lambda>
    lambda *args, **kwds: func(self._observers[cls], *args, **kwds),
  File "/Users/markddavidoff/Dev/Envs/webenv/lib/python2.7/site-packages/pympler/classtracker.py", line 310, in _tracker
    _observer_.init(_self_, *args, **kwds)
  File "/Users/markddavidoff/Dev/Envs/webenv/lib/python2.7/site-packages/pympler/classtracker.py", line 335, in <lambda>
    lambda *args, **kwds: func(self._observers[cls], *args, **kwds),
KeyError: <class 'django.template.base.Template'>

Its similar with #36

Environment: ubuntu 1804, 64 bit python 3.7.5 numpy 1.17.3 Pympler 0.8

>>> import numpy as np
>>> from pympler import asizeof
>>> arr = np.arange(1000)
>>> asizeof.asizeof(arr)
8096
>>> arr = arr[:]
>>> asizeof.asizeof(arr)
96

The size is expected to the same.

dict variables and the values inside of them are not output when calling print_diff() with a SummaryTracker. Here's an example that demonstrates the behavior:

from pympler import tracker
from datetime import datetime

tr = tracker.SummaryTracker()
print('start')
tr.print_diff()

print('empty list')
list_test = []
tr.print_diff()

print('add to list')
list_test.append(datetime.now())
tr.print_diff()

print('empty dict')
dict_test = {}
tr.print_diff()

print('add to dict')
dict_test['test'] = datetime.now()
tr.print_diff()

Tested with 2.7.15 and 3.7.2 and neither output the any information about dict

Just a heads up, Django debug toolbar is changing how panels are processed to be more like new-style middleware, presumably for an upcoming 2.0 release (and also dropping Python 2 support).

As a result, Pympler panel is broken vs current Django debug toolbar master, it hits this error "'NoneType' object has no attribute 'get'" because debug_toolbar/middleware.py expects process_request to return a response object.

Using:

django debug toolbar master (https://github.com/jazzband/django-debug-toolbar/commit/b650a33bb02651903540d726704a9143c73eda0a) Pympler==0.7 Django==2.2.1

See simple example project that hits the bug on all requests: https://github.com/therefromhere/ddt_pympler_bug_example

Note that this doesn't affect the released version of Django debug toolbar yet.

Previously:

• Filtering by size could not be done without also filtering by type.
• Size filtering could not be done open ended (> min), both min and max had to be provided.
• Direction for min and max were backwards.

Pympler includes bottle (in pympler/util/bottle.py). I patched it to use system-wide bottle module and it seems to work fine (at least the tests pass). Please consider adding support for using the system version of bottle.

The behavior is similar to the bug from #151.

(test_asizeof) rxue@xenon:~$ ipython
Python 3.8.15 | packaged by conda-forge | (default, Nov 22 2022, 08:49:35) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.7.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import numpy as np
   ...: from pympler.asizeof import asizeof
   ...: x=np.str_(' '*100)
   ...: asizeof(x)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[1], line 4
      2 from pympler.asizeof import asizeof
      3 x=np.str_(' '*100)
----> 4 asizeof(x)

File /opt/mambaforge/envs/test_asizeof/lib/python3.8/site-packages/pympler/asizeof.py:2603, in asizeof(*objs, **opts)
   2601 if x:  # don't size, profile or rank _getobjects tuple
   2602     _asizer.exclude_objs(t)
-> 2603 s = _asizer.asizeof(*t)
   2604 _asizer.print_stats(objs=t, opts=opts)  # show opts as _kwdstr
   2605 _asizer._clear()

File /opt/mambaforge/envs/test_asizeof/lib/python3.8/site-packages/pympler/asizeof.py:2032, in Asizer.asizeof(self, *objs, **opts)
   2030     self.set(**opts)
   2031 self.exclude_refs(*objs)  # skip refs to objs
-> 2032 return sum(self._sizer(o, 0, 0, None) for o in objs)

File /opt/mambaforge/envs/test_asizeof/lib/python3.8/site-packages/pympler/asizeof.py:2032, in <genexpr>(.0)
   2030     self.set(**opts)
   2031 self.exclude_refs(*objs)  # skip refs to objs
-> 2032 return sum(self._sizer(o, 0, 0, None) for o in objs)

File /opt/mambaforge/envs/test_asizeof/lib/python3.8/site-packages/pympler/asizeof.py:1937, in Asizer._sizer(self, obj, pid, deep, sized)
   1935 v = _typedefs.get(k, None)
   1936 if not v:  # new typedef
-> 1937     _typedefs[k] = v = _typedef(obj, derive=self._derive_,
   1938                                      frames=self._frames_,
   1939                                       infer=self._infer_)
   1940 if (v.both or self._code_) and v.kind is not self._ign_d:
   1941     s = f = v.flat(obj, self._mask)  # flat size

File /opt/mambaforge/envs/test_asizeof/lib/python3.8/site-packages/pympler/asizeof.py:1597, in _typedef(obj, derive, frames, infer)
   1595     v.set(refs=_namedtuple_refs)
   1596 elif numpy and _isnumpy(obj):  # NumPy data
-> 1597     v.set(**_numpy_kwds(obj))
   1598     if _numpy_excl:
   1599         _getsizeof_excls_add(t)

File /opt/mambaforge/envs/test_asizeof/lib/python3.8/site-packages/pympler/asizeof.py:1183, in _Typedef.set(self, safe_len, **kwds)
   1181     d = self.kwds()
   1182     d.update(kwds)
-> 1183     self.reset(**d)
   1184 if safe_len and self.item:
   1185     self.leng = _len

File /opt/mambaforge/envs/test_asizeof/lib/python3.8/site-packages/pympler/asizeof.py:1192, in _Typedef.reset(self, base, item, leng, refs, both, kind, type, vari)
   1189 '''Reset all specified attributes.
   1190 '''
   1191 if base < 0:
-> 1192     raise ValueError('invalid option: %s=%r' % ('base', base))
   1193 else:
   1194     self.base = base

ValueError: invalid option: base=-219


In [2]: 

My understanding is that asizeof tries to register the base size of numpy.str_ when it came across this NumPy data type for the first time. However, numpy.str_ has a dynamical size, so the base from the diff between sys.getsizeof(obj) and obj.nbytes becomes a bit unpredictable?

On the other hand, starting fresh, this still works:

(test_asizeof) rxue@xenon:~$ ipython
Python 3.8.15 | packaged by conda-forge | (default, Nov 22 2022, 08:49:35) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.7.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import numpy as np
   ...: from pympler.asizeof import asizeof
   ...: 
   ...: x=np.str_()
   ...: asizeof(x)
   ...: 
   ...: x=np.str_(' '*100)
   ...: asizeof(x)
Out[1]: 200

tested with Pympler ver 1.0.1 and numpy ver 1.23.5

Hi!

After updating from Python 3.9.13 to Python 3.10.6, python setup.py test started failing like this:

======================================================================
FAIL: test_stracker_create_summary (muppy.test_tracker.TrackerTest)
Test that a summary is created correctly.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/marius/src/pympler/test/muppy/test_tracker.py", line 112, in test_stracker_create_summary
    self.assertEqual(len(tmp), 0, tmp)
AssertionError: 1 != 0 : [["<class 'code'>", 0, 37]]

Oddly, and frustratingly, that test does not fail when invoked directly through runtest.py:

$ python test/runtest.py test/muppy/test_tracker.py  -verbose 3
Python 3.10.6 (main, Jan  1 1970, 00:00:01) 64-bit [GCC 11.3.0]

test_otracker_diff (muppy.test_tracker.TrackerTest)
Test object tracker diff. ... ok
test_otracker_get_objects (muppy.test_tracker.TrackerTest)
Test object tracker. ... ok
test_stracker_create_summary (muppy.test_tracker.TrackerTest)
Test that a summary is created correctly. ... ok
[...]

Nor does python setup.py test fail if I change _glob_test_py in runtest.py to only include that one file! Making it difficult to troubleshoot.

The test succeeds in the CI with Python 3.10.6, so maybe there is something wrong with my Python? I haven't noticed problems in other packages however.

Any ideas what may be going on here?

Hi all, Thanks for this great tool!

I've used it successfully to track down a memory leak in a C extension that forgot to decref a list. But yesterday I discovered I was also leaking PyFloats (these were created by not decreff'd), which pympler did not pick up. I've been reading a bit, and it seems that it's related to (1) things tracked by GC and (2) things referenced from other objects. Is that correct?

To understand it a bit better, I wrote a small pure Python script:

import gc
import sys

import pympler.tracker

# Create tracker
tracker = pympler.tracker.SummaryTracker()

# Re-route stdout and perform first two calls: this gets rid of memory used by imports etc.
stdout = sys.stdout
sys.stdout = None
tracker.print_diff()
n_gc = len(gc.get_objects())
tracker.print_diff()
sys.stdout = stdout

print(f'1. Objects tracked by GC, since now: {len(gc.get_objects()) - n_gc})
a = 5
b = 1.2
c = []
d = {}
print()
tracker.print_diff()            # Show difference
print()
print(f'2. Objects tracked by GC, since 1: {len(gc.get_objects()) - n_gc}')

# Add object to list
c.append(4)

print()
tracker.print_diff()            # Show difference
print()
print(f'3. Objects tracked by GC, since 1: {len(gc.get_objects()) - n_gc}')

its output is

1. Objects tracked by GC, since first count: 0

  types |   # objects |   total size
======= | =========== | ============
   dict |           1 |     64     B
   list |           1 |     56     B

2. Objects tracked by GC, since first count: 1

  types |   # objects |   total size
======= | =========== | ============
   list |           0 |     32     B

3. Objects tracked by GC, since first count: 1

Can you help me understand this output? What I think is happening is:

1. The first bit gets rid of a lot of information about setting up the test case.
2. The first print_diff actually displayed shows the list (tracked by gc), the dict (not tracked by gc, not sure how pympler finds it anyway? does it scan the local namespace? if so, why aren't the int and float included?), but not the int or float (neither are tracked by gc)
3. Adding another int shows up as an increase in list size (32B) but not in the list object count (0 new lists), but also not as a new Int object (conscious choice to omit as its already shown in the list?)

My real use case is slightly more complex still (python objects created in C, so not in local or global python namespace), so I'd like to make sure I understand the above really well before thinking about that.

So my questions are:

1. Am I at all correct in my understanding of the above?
2. Is there some way to make pympler track python primitives as well, or does that require e.g. tracemalloc?

I am trying to estimate the size of a python object that is a subclass of an astropy.Table

However, I receive a ValueError: invalid option: base=-391848 error.

Does anyone have suggestions on why this is occurring/how I can fix this?

MWE:

%pip install -q lightkurve
%pip install -q pympler

import lightkurve as lk
from pympler import asizeof
import numpy as np
import os

lc = lk.search_lightcurve("TOI 588", author="SPOC").download_all().stitch().remove_nans()
asizeof.asizeof(lc)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-13-88b6425eddfd>](https://localhost:8080/#) in <module>()
      8 
      9 lc = lk.search_lightcurve("TOI 588", author="SPOC").download_all().stitch().remove_nans()
---> 10 asizeof.asizeof(lc)

30 frames
[/usr/local/lib/python3.7/dist-packages/pympler/asizeof.py](https://localhost:8080/#) in reset(self, base, item, leng, refs, both, kind, type, vari)
   1190         '''
   1191         if base < 0:
-> 1192             raise ValueError('invalid option: %s=%r' % ('base', base))
   1193         else:
   1194             self.base = base

ValueError: invalid option: base=-391848

The CI builds on Windows have been failing for quite a while now. It is unclear what the root cause of those failures is. It seems to be an CI issue and not a Pympler one, unconfirmed.

When running the Pympler-1.0.1 testsuite under python 3.11 on Fedora rawhide (development branch), the following failures can be observed:

$ PYTHONPATH=/builddir/build/BUILDROOT/python-Pympler-1.0.1-3.fc37.noarch/usr/lib/python3.11/site-packages /usr/bin/python3 setup.py test
running test
test_otracker_diff (muppy.test_tracker.TrackerTest.test_otracker_diff)
Test object tracker diff. ... ok
...
Test process sizes match: ps util vs getrusage ... ok
test_ps_vs_proc_sizes (test_process.ProcessMemoryTests.test_ps_vs_proc_sizes)
Test process sizes match: ps util vs /proc/self/stat ... ok
======================================================================
ERROR: test_all (asizeof.test_asizeof.AsizeofDemos.test_all)
Test all=True example
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/Pympler-1.0.1/test/asizeof/test_asizeof.py", line 522, in test_all
    asizeof.asizeof(limit=self.MAX, code=True, stats=self.MAX, all=True)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILDROOT/python-Pympler-1.0.1-3.fc37.noarch/usr/lib/python3.11/site-packages/pympler/asizeof.py", line 2603, in asizeof
    _asizer.print_stats(objs=t, opts=opts)  # show opts as _kwdstr
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILDROOT/python-Pympler-1.0.1-3.fc37.noarch/usr/lib/python3.11/site-packages/pympler/asizeof.py", line 2251, in print_stats
    self.print_profiles(w=w, cutoff=c, **print3options)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILDROOT/python-Pympler-1.0.1-3.fc37.noarch/usr/lib/python3.11/site-packages/pympler/asizeof.py", line 2191, in print_profiles
    for v, k in sorted(t, reverse=True):
                ^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '<' not supported between instances of 'type' and 'type'
======================================================================
FAIL: test_findgarbage (gui.test_garbage.GarbageTestCase.test_findgarbage)
Test garbage annotation.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/Pympler-1.0.1/test/gui/test_garbage.py", line 87, in test_findgarbage
    self.assertTrue(gbar.size > 0, gbar.size)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: False is not true : 0
======================================================================
FAIL: test_prune (gui.test_garbage.GarbageTestCase.test_prune)
Test pruning of reference graph.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/Pympler-1.0.1/test/gui/test_garbage.py", line 153, in test_prune
    self.assertEqual(gb2.num_in_cycles, 2)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 0 != 2
======================================================================
FAIL: test_get_tree (muppy.test_refbrowser.TreeTest.test_get_tree)
Test reference browser tree representation.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/Pympler-1.0.1/test/muppy/test_refbrowser.py", line 155, in test_get_tree
    self.assertTrue(r in children)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: False is not true
----------------------------------------------------------------------
Ran 126 tests in 49.562s
FAILED (failures=3, errors=1)
gc:0: ResourceWarning: gc: 139 uncollectable objects at shutdown; use gc.set_debug(gc.DEBUG_UNCOLLECTABLE) to list them