A columnar data container that can be compressed.

Overview

Unmaintained Package Notice

Unfortunately, and due to lack of resources, the Blosc Development Team is unable to maintain this package anymore. During the last 10 years we managed to find resources (even if in a quite irregular way) to develop what we think is a nice package for handling compressed data containers, especially tabular data. Regrettably, for the last years we did not found sponsorship enough to continue the maintenance of this package.

For those that depend on bcolz, a fork is welcome and we will try our best to provide advice for possible new maintainers. Indeed, if we manage to get some decent grants via Blosc (https://blosc.org/pages/donate/), our umbrella project, we would be glad to reconsider the maintenance of bcolz. But again, we would be very open and supportive for this project to get a new maintenance team.

Finally, thanks to all the people that used and contributed in one way or another to bcolz; it has been a nice ride! Let's hope it still would have a bright future ahead.

The Blosc Development Team

bcolz: columnar and compressed data containers

Join the chat at https://gitter.im/Blosc/bcolz
Version: version
Travis CI: travis
Appveyor: appveyor
Coveralls: coveralls
And...: powered

docs/bcolz.png

bcolz provides columnar, chunked data containers that can be compressed either in-memory and on-disk. Column storage allows for efficiently querying tables, as well as for cheap column addition and removal. It is based on NumPy, and uses it as the standard data container to communicate with bcolz objects, but it also comes with support for import/export facilities to/from HDF5/PyTables tables and pandas dataframes.

bcolz objects are compressed by default not only for reducing memory/disk storage, but also to improve I/O speed. The compression process is carried out internally by Blosc, a high-performance, multithreaded meta-compressor that is optimized for binary data (although it works with text data just fine too).

bcolz can also use numexpr internally (it does that by default if it detects numexpr installed) or dask so as to accelerate many vector and query operations (although it can use pure NumPy for doing so too). numexpr/dask can optimize the memory usage and use multithreading for doing the computations, so it is blazing fast. This, in combination with carray/ctable disk-based, compressed containers, can be used for performing out-of-core computations efficiently, but most importantly transparently.

Just to whet your appetite, here it is an example with real data, where bcolz is already fulfilling the promise of accelerating memory I/O by using compression:

http://nbviewer.ipython.org/github/Blosc/movielens-bench/blob/master/querying-ep14.ipynb

Rationale

By using compression, you can deal with more data using the same amount of memory, which is very good on itself. But in case you are wondering about the price to pay in terms of performance, you should know that nowadays memory access is the most common bottleneck in many computational scenarios, and that CPUs spend most of its time waiting for data. Hence, having data compressed in memory can reduce the stress of the memory subsystem as well.

Furthermore, columnar means that the tabular datasets are stored column-wise order, and this turns out to offer better opportunities to improve compression ratio. This is because data tends to expose more similarity in elements that sit in the same column rather than those in the same row, so compressors generally do a much better job when data is aligned in such column-wise order. In addition, when you have to deal with tables with a large number of columns and your operations only involve some of them, a columnar-wise storage tends to be much more effective because minimizes the amount of data that travels to CPU caches.

So, the ultimate goal for bcolz is not only reducing the memory needs of large arrays/tables, but also making bcolz operations to go faster than using a traditional data container like those in NumPy or Pandas. That is actually already the case in some real-life scenarios (see the notebook above) but that will become pretty more noticeable in combination with forthcoming, faster CPUs integrating more cores and wider vector units.

Requisites

  • Python >= 2.7 and >= 3.5
  • NumPy >= 1.8
  • Cython >= 0.22 (just for compiling the beast)
  • C-Blosc >= 1.8.0 (optional, as the internal Blosc will be used by default)

Optional:

  • numexpr >= 2.5.2
  • dask >= 0.9.0
  • pandas
  • tables (pytables)

Building

There are different ways to compile bcolz, depending if you want to link with an already installed Blosc library or not.

Compiling with an installed Blosc library (recommended)

Python and Blosc-powered extensions have a difficult relationship when compiled using GCC, so this is why using an external C-Blosc library is recommended for maximum performance (for details, see https://github.com/Blosc/python-blosc/issues/110).

Go to https://github.com/Blosc/c-blosc/releases and download and install the C-Blosc library. Then, you can tell bcolz where is the C-Blosc library in a couple of ways:

Using an environment variable:

$ BLOSC_DIR=/usr/local     (or "set BLOSC_DIR=\blosc" on Win)
$ export BLOSC_DIR         (not needed on Win)
$ python setup.py build_ext --inplace

Using a flag:

$ python setup.py build_ext --inplace --blosc=/usr/local

Compiling without an installed Blosc library

bcolz also comes with the Blosc sources with it so, assuming that you have a C++ compiler installed, do:

$ python setup.py build_ext --inplace

That's all. You can proceed with testing section now.

Note: The requirement for the C++ compiler is just for the Snappy dependency. The rest of the other components of Blosc are pure C (including the LZ4 and Zlib libraries).

Testing

After compiling, you can quickly check that the package is sane by running:

$ PYTHONPATH=.   (or "set PYTHONPATH=." on Windows)
$ export PYTHONPATH    (not needed on Windows)
$ python -c"import bcolz; bcolz.test()"  # add `heavy=True` if desired

Installing

Install it as a typical Python package:

$ pip install -U .

Optionally Install the additional dependencies:

$ pip install .[optional]

Documentation

You can find the online manual at:

http://bcolz.blosc.org

but of course, you can always access docstrings from the console (i.e. help(bcolz.ctable)).

Also, you may want to look at the bench/ directory for some examples of use.

Resources

Visit the main bcolz site repository at: http://github.com/Blosc/bcolz

Home of Blosc compressor: http://blosc.org

User's mail list: http://groups.google.com/group/bcolz ([email protected])

An introductory talk (20 min) about bcolz at EuroPython 2014. Slides here.

License

Please see BCOLZ.txt in LICENSES/ directory.

Share your experience

Let us know of any bugs, suggestions, gripes, kudos, etc. you may have.

Enjoy Data!

Comments
  • Minimally upgrade blosc 1.7.0

    Minimally upgrade blosc 1.7.0

    This PR contains the bare minimum required to upgrade c-blosc to 1.7.0.

    Test suite passes with 5 tests skipped on my Linux laptop so I don't believe any code changes are required within bcolz itself to accommodate this upgrade.

    opened by alimanfoo 33
  • Remove with nogil statement to avoid segfault

    Remove with nogil statement to avoid segfault

    Concurrent reads sometimes trigger a segfault somewhere within blosc_decompress.

    In principle the old bcolz code seems correct; presumably there is some issue lower down? For now a cheap solution seems to be to hold on to the GIL.

    Full trace is here:

    Program received signal SIGSEGV, Segmentation fault.
    [Switching to Thread 0x7fffd9c7d700 (LWP 28826)]
    _int_malloc (av=0x7fffa4000020, bytes=568) at malloc.c:3489
    3489    malloc.c: No such file or directory.
    (gdb) bt
    #0  _int_malloc (av=0x7fffa4000020, bytes=568) at malloc.c:3489
    #1  0x00007ffff70907b0 in __GI___libc_malloc (bytes=568) at malloc.c:2891
    #2  0x00007ffff707c44d in __fopen_internal (
        filename=0x7fffa4031e90 "/home/mrocklin/data/trip3.bcolz/rate_code/data/__445.blp", 
        mode=0x7fff3fc8bf80 "rb", is32=1) at iofopen.c:73
    #3  0x00007ffff7a66717 in open_the_file (f=0x7fffdc338e40, 
        name=0x7fffa4031e90 "/home/mrocklin/data/trip3.bcolz/rate_code/data/__445.blp", 
        mode=0x7ffff7ecddc4 "rb") at Objects/fileobject.c:374
    #4  0x00007ffff7a68b52 in file_init (self=0x7fffdc338e40, args=0x7fffa0776098, kwds=0x0)
        at Objects/fileobject.c:2430
    #5  0x00007ffff7aa3208 in type_call (type=<optimized out>, args=0x7fffa0776098, kwds=0x0)
        at Objects/typeobject.c:745
    #6  0x00007ffff7a40323 in PyObject_Call (func=0x7ffff7d9c820 <PyFile_Type>, arg=<optimized out>, 
        kw=<optimized out>) at Objects/abstract.c:2529
    #7  0x00007fffe1d7dac1 in __Pyx_PyObject_Call (func=0x7ffff7fcda70, arg=arg@entry=0x7fffa0776098, 
        kw=0x0) at bcolz/carray_ext.c:37718
    #8  0x00007fffe1dc70b6 in __pyx_f_5bcolz_10carray_ext_6chunks_read_chunk (
        __pyx_v_self=<optimized out>, __pyx_v_nchunk=0x11fce10) at bcolz/carray_ext.c:9434
    #9  0x00007fffe1d8532d in __pyx_pf_5bcolz_10carray_ext_6chunks_2__getitem__ (
        __pyx_v_nchunk=0x11fce10, __pyx_v_self=0x7fffe1b40ad0) at bcolz/carray_ext.c:9867
    #10 __pyx_pw_5bcolz_10carray_ext_6chunks_3__getitem__ (__pyx_v_self=0x7fffe1b40ad0, 
        __pyx_v_nchunk=0x11fce10) at bcolz/carray_ext.c:9809
    #11 0x00007fffe1d7a07a in __pyx_sq_item_5bcolz_10carray_ext_chunks (o=0x7fffe1b40ad0, 
        i=<optimized out>) at bcolz/carray_ext.c:34563
    #12 0x00007fffe1dba1d0 in __Pyx_GetItemInt_Fast (is_list=0, wraparound=1, boundscheck=1, i=445, 
        o=<optimized out>) at bcolz/carray_ext.c:38415
    #13 __pyx_pf_5bcolz_10carray_ext_6carray_38__getitem__ (__pyx_v_key=<optimized out>, 
        __pyx_v_self=<optimized out>) at bcolz/carray_ext.c:24885
    #14 __pyx_pw_5bcolz_10carray_ext_6carray_39__getitem__ (__pyx_v_self=<optimized out>, 
        __pyx_v_key=<optimized out>) at bcolz/carray_ext.c:23217
    #15 0x00007fffe1db8ab3 in __pyx_pf_5bcolz_10carray_ext_6carray_38__getitem__ (
        __pyx_v_key=0x7fffdb406d10, __pyx_v_self=0x7fffe2969bd0) at bcolz/carray_ext.c:23668
    #16 __pyx_pw_5bcolz_10carray_ext_6carray_39__getitem__ (__pyx_v_self=0x7fffe2969bd0, 
        __pyx_v_key=0x7fffdb406d10) at bcolz/carray_ext.c:23217
    #17 0x00007ffff54a3b3d in op_getitem (s=<optimized out>, a=<optimized out>)
        at -------src-dir-------/Python-2.7.9/Modules/operator.c:130
    #18 0x00007ffff7af022f in ext_do_call (nk=-1742972520, na=<optimized out>, flags=<optimized out>, 
        pp_stack=0x7fffd9c7b408, func=0x7ffff5f09f38) at Python/ceval.c:4343
    #19 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2718
    #20 0x00007ffff7af1c6e in PyEval_EvalCodeEx (co=0x7fffdc3b27b0, globals=<optimized out>, 
        locals=<optimized out>, args=<optimized out>, argcount=2, kws=0x7fffc4004910, kwcount=0, 
        defs=0x7fffdc3b3a28, defcount=1, closure=0x0) at Python/ceval.c:3265
    #21 0x00007ffff7af02aa in fast_function (nk=<optimized out>, na=2, n=<optimized out>, 
        pp_stack=0x7fffd9c7b608, func=0x7fffdc32b050) at Python/ceval.c:4129
    #22 call_function (oparg=<optimized out>, pp_stack=0x7fffd9c7b608) at Python/ceval.c:4054
    #23 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
    #24 0x00007ffff7a6421c in gen_send_ex (gen=0x7fffdb3f8820, arg=0x0, exc=<optimized out>)
        at Objects/genobject.c:85
    #25 0x00007ffff7a3f62b in PyIter_Next (iter=<optimized out>) at Objects/abstract.c:3103
    #26 0x00007ffff7ae44da in builtin_zip (self=<optimized out>, args=<optimized out>)
        at Python/bltinmodule.c:2555
    #27 0x00007ffff7af022f in ext_do_call (nk=-1603221672, na=<optimized out>, flags=<optimized out>, 
        pp_stack=0x7fffd9c7b7f8, func=0x7ffff7fcdea8) at Python/ceval.c:4343
    ---Type <return> to continue, or q <return> to quit---
    #28 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2718
    #29 0x00007ffff7af1c6e in PyEval_EvalCodeEx (co=0x7fffdc3b27b0, globals=<optimized out>, 
        locals=<optimized out>, args=<optimized out>, argcount=2, kws=0x7fffa40398f0, kwcount=0, 
        defs=0x7fffdc3b3a28, defcount=1, closure=0x0) at Python/ceval.c:3265
    #30 0x00007ffff7af02aa in fast_function (nk=<optimized out>, na=2, n=<optimized out>, 
        pp_stack=0x7fffd9c7b9f8, func=0x7fffdc32b050) at Python/ceval.c:4129
    #31 call_function (oparg=<optimized out>, pp_stack=0x7fffd9c7b9f8) at Python/ceval.c:4054
    #32 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
    #33 0x00007ffff7af1c6e in PyEval_EvalCodeEx (co=0x7fffdc3b27b0, globals=<optimized out>, 
        locals=<optimized out>, args=<optimized out>, argcount=2, kws=0x7fffc0002c90, kwcount=0, 
        defs=0x7fffdc3b3a28, defcount=1, closure=0x0) at Python/ceval.c:3265
    #34 0x00007ffff7af02aa in fast_function (nk=<optimized out>, na=2, n=<optimized out>, 
        pp_stack=0x7fffd9c7bbf8, func=0x7fffdc32b050) at Python/ceval.c:4129
    #35 call_function (oparg=<optimized out>, pp_stack=0x7fffd9c7bbf8) at Python/ceval.c:4054
    #36 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
    #37 0x00007ffff7af1c6e in PyEval_EvalCodeEx (co=0x7fffdc3b27b0, globals=<optimized out>, 
        locals=<optimized out>, args=<optimized out>, argcount=2, kws=0x7fffa4003750, kwcount=0, 
        defs=0x7fffdc3b3a28, defcount=1, closure=0x0) at Python/ceval.c:3265
    #38 0x00007ffff7af02aa in fast_function (nk=<optimized out>, na=2, n=<optimized out>, 
        pp_stack=0x7fffd9c7bdf8, func=0x7fffdc32b050) at Python/ceval.c:4129
    #39 call_function (oparg=<optimized out>, pp_stack=0x7fffd9c7bdf8) at Python/ceval.c:4054
    #40 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
    #41 0x00007ffff7af1c6e in PyEval_EvalCodeEx (co=0x7fffdc3b27b0, globals=<optimized out>, 
        locals=<optimized out>, args=<optimized out>, argcount=2, kws=0x7fffb4064210, kwcount=0, 
        defs=0x7fffdc3b3a28, defcount=1, closure=0x0) at Python/ceval.c:3265
    #42 0x00007ffff7af02aa in fast_function (nk=<optimized out>, na=2, n=<optimized out>, 
        pp_stack=0x7fffd9c7bff8, func=0x7fffdc32b050) at Python/ceval.c:4129
    #43 call_function (oparg=<optimized out>, pp_stack=0x7fffd9c7bff8) at Python/ceval.c:4054
    #44 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
    #45 0x00007ffff7af1c6e in PyEval_EvalCodeEx (co=0x7fffdc3b27b0, globals=<optimized out>, 
        locals=<optimized out>, args=<optimized out>, argcount=2, kws=0x7fffa4003968, kwcount=0, 
        defs=0x7fffdc3b3a28, defcount=1, closure=0x0) at Python/ceval.c:3265
    #46 0x00007ffff7af02aa in fast_function (nk=<optimized out>, na=2, n=<optimized out>, 
        pp_stack=0x7fffd9c7c1f8, func=0x7fffdc32b050) at Python/ceval.c:4129
    #47 call_function (oparg=<optimized out>, pp_stack=0x7fffd9c7c1f8) at Python/ceval.c:4054
    #48 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
    #49 0x00007ffff7af1c6e in PyEval_EvalCodeEx (co=0x7fffdc3b28b0, globals=<optimized out>, 
        locals=<optimized out>, args=<optimized out>, argcount=5, kws=0x7ffff7f92068, kwcount=0, 
        defs=0x7fffdc3b3b68, defcount=1, closure=0x0) at Python/ceval.c:3265
    #50 0x00007ffff7a6f958 in function_call (func=0x7fffdc32b0c8, arg=0x7fffdbcae4d0, kw=0x7fffdbed2910)
        at Objects/funcobject.c:526
    #51 0x00007ffff7a40323 in PyObject_Call (func=0x7fffdc32b0c8, arg=<optimized out>, 
        kw=<optimized out>) at Objects/abstract.c:2529
    #52 0x00007ffff7aee865 in ext_do_call (nk=-607460144, na=<optimized out>, flags=<optimized out>, 
        pp_stack=0x7fffd9c7c4c8, func=0x7fffdc32b0c8) at Python/ceval.c:4346
    #53 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2718
    #54 0x00007ffff7af1c6e in PyEval_EvalCodeEx (co=0x7fffdbece830, globals=<optimized out>, 
        locals=<optimized out>, args=<optimized out>, argcount=5, kws=0x7ffff7f92068, kwcount=0, 
        defs=0x7fffdc349f68, defcount=3, closure=0x0) at Python/ceval.c:3265
    #55 0x00007ffff7a6f958 in function_call (func=0x7fffdbed3b90, arg=0x7fffdbcae050, kw=0x7fffdbee15c8)
        at Objects/funcobject.c:526
    #56 0x00007ffff7a40323 in PyObject_Call (func=0x7fffdbed3b90, arg=<optimized out>, 
        kw=<optimized out>) at Objects/abstract.c:2529
    

    Relevant line numbers often point to this carray_ext.c file

    enhancement 
    opened by mrocklin 32
  • Will bcolz support numpy 1.9?

    Will bcolz support numpy 1.9?

    We are interested in including bcolz in the upcoming Anaconda release, which will use numpy 1.9. Do you have plans to support this? I hope so, since it's a very useful package. Thanks!

    opened by talumbau 28
  • conda recipe and binstar build yml file

    conda recipe and binstar build yml file

    This aids building bcolz with conda and hosting conda builds on binstar-build.

    Note that the Blaze channel is hosting bcolz builds for all platforms and common Python versions. These are accessible via the following:

    conda install -c blaze bcolz
    

    This doesn't require a compiler and tends to point to something closer to git master than the main conda channel.

    Also, just for acknowledgements, I stole a fair amount of this work from others, notably @cpcloud .

    opened by mrocklin 22
  • Chunks class iterator

    Chunks class iterator

    Hi,

    this PR is a proposal for a chunks class iterator, with two tests one for in-core and the other one for out-of-core carray. In case this changes are on the right path, would this fill the requirements to enable this kind of syntax for chunk_ in ca.chunks?

    Many thanks in advance for your input,

    Regards, F. Elies

    enhancement 
    opened by FrancescElies 20
  • tutorial notebook

    tutorial notebook

    Migrating tutorial to python notebook.

    To see the state of the notebook please visit: https://github.com/FrancescElies/bcolz/blob/fe/doc/ipython_notebook_tutorial/doc/tutorials.ipynb

    opened by FrancescElies 19
  • Make bcolz threadsafe

    Make bcolz threadsafe

    This PR builds on #285 to investigate the possibility to make bcolz safe to use within multi-threaded environments.

    The primary goal of this PR is to make bcolz threadsafe.

    The secondary goal of this PR is to investigate ways to optimise performance of multithreaded applications using bcolz. This includes looking for opportunities to safely release the GIL, and investigating the possibility for concurrent read operations on the same carray.

    opened by alimanfoo 18
  • Latest c-blosc causes test failures in bcolz master

    Latest c-blosc causes test failures in bcolz master

    @FrancescAlted It seems we've hit another hiccup with the Debian/Ubuntu packaging of c-blosc 1.14.2. The package has been held back because it causes test failures in bcolz.

    Here's the Ubuntu test log and here's the Debian test log.

    I was able to reproduce the failures using bcolz Git master branch in an Ubuntu 18.04 Docker container with the 1.14.2 package installed (see below). Any idea what could be the problem?

    root@f0b7772b23f5:~/bcolz# python3 -c"import bcolz; bcolz.test()"
    -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    bcolz version:     1.1.3.dev15
    bcolz git info:    b'1.1.2-15-g20cbcf3'
    NumPy version:     1.13.3
    Blosc version:     1.14.2 ($Date:: 2018-03-16 #$)
    Blosc compressors: ['blosclz', 'lz4', 'lz4hc', 'snappy', 'zlib', 'zstd']
    Numexpr version:   not available (version >= 2.5.2 not detected)
    Dask version:   not available (version >= 0.9.0 not detected)
    Python version:    3.6.5rc1 (default, Mar 14 2018, 06:54:23) 
    [GCC 7.3.0]
    Platform:          linux-x86_64
    Byte-ordering:     little
    Detected cores:    4
    -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    Performing only a light (yet comprehensive) subset of the test suite.
    If you want a more complete test, try passing the '-heavy' flag to this
    script (or set the 'heavy' parameter in case you are using bcolz.test()
    call).  The whole suite will take more than 30 seconds to complete on a
    relatively modern CPU and around 300 MB of RAM and 500 MB of disk
    [32-bit platforms will always run significantly more lightly].
    
    -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    ......s......s......s......s............................................................FF.............................................................................................ssssssssssssssssssssssssssss..............ssssssssssssssssssssssssssss..............ssssssssssssssssssssssssssss..............ssssssssssssssssssssssssssssssssssssssssssssssssssssssss...............FF.FFF................................................ss..........................................ssssss...............F..F..........FFF.............................................................................................................................................................................................FF.FFF................................................................................................................ssss...................../root/bcolz/bcolz/ctable.py:48: FutureWarning: split() requires a non-empty pattern match.
      return re_str_split.split(str(x))
    .............................sss.........s.......................F....................................................................................................................................................................................................................................................................................................................................................................
    ======================================================================
    FAIL: test01b (test_carray.bloscFiltersTest)
    Testing all available filters in big arrays (bcolz.defaults)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/bcolz/bcolz/tests/test_carray.py", line 2322, in test01b
        "carray does not seem to compress at all")
    AssertionError: False is not true : carray does not seem to compress at all
    
    ======================================================================
    FAIL: test01c (test_carray.bloscFiltersTest)
    Testing all available filters in big arrays (context)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/bcolz/bcolz/tests/test_carray.py", line 2344, in test01c
        self.assertTrue(bcolz.defaults.cparams['shuffle'] == bcolz.SHUFFLE)
    AssertionError: False is not true
    
    ======================================================================
    FAIL: test01b (test_carray.filtersDiskTest)
    Testing all available filters in big arrays (bcolz.defaults)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/bcolz/bcolz/tests/test_carray.py", line 2322, in test01b
        "carray does not seem to compress at all")
    AssertionError: False is not true : carray does not seem to compress at all
    
    ======================================================================
    FAIL: test01c (test_carray.filtersDiskTest)
    Testing all available filters in big arrays (context)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/bcolz/bcolz/tests/test_carray.py", line 2344, in test01c
        self.assertTrue(bcolz.defaults.cparams['shuffle'] == bcolz.SHUFFLE)
    AssertionError: False is not true
    
    ======================================================================
    FAIL: test01a (test_carray.filtersMemoryTest)
    Testing all available filters in big arrays (setdefaults)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/bcolz/bcolz/tests/test_carray.py", line 2302, in test01a
        "carray does not seem to compress at all")
    AssertionError: False is not true : carray does not seem to compress at all
    
    ======================================================================
    FAIL: test01b (test_carray.filtersMemoryTest)
    Testing all available filters in big arrays (bcolz.defaults)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/bcolz/bcolz/tests/test_carray.py", line 2322, in test01b
        "carray does not seem to compress at all")
    AssertionError: False is not true : carray does not seem to compress at all
    
    ======================================================================
    FAIL: test01c (test_carray.filtersMemoryTest)
    Testing all available filters in big arrays (context)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/bcolz/bcolz/tests/test_carray.py", line 2344, in test01c
        self.assertTrue(bcolz.defaults.cparams['shuffle'] == bcolz.SHUFFLE)
    AssertionError: False is not true
    
    ======================================================================
    FAIL: test01 (test_carray.miscDiskTest)
    Testing __sizeof__() (big carrays)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/bcolz/bcolz/tests/test_carray.py", line 639, in test01
        "carray does not seem to compress at all")
    AssertionError: False is not true : carray does not seem to compress at all
    
    ======================================================================
    FAIL: test01 (test_carray.miscMemoryTest)
    Testing __sizeof__() (big carrays)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/bcolz/bcolz/tests/test_carray.py", line 639, in test01
        "carray does not seem to compress at all")
    AssertionError: False is not true : carray does not seem to compress at all
    
    ======================================================================
    FAIL: test_repr_disk_array_append (test_carray.reprDiskTest)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/bcolz/bcolz/tests/test_carray.py", line 2430, in test_repr_disk_array_append
        self.assertEqual(expected, repr(y))
    AssertionError: "carr[90 chars]evel=5, shuffle=1, cname='blosclz', quantize=0[119 chars]\n[]" != "carr[90 chars]evel=9, shuffle=0, cname='blosclz', quantize=0[119 chars]\n[]"
      carray((0,), float64)
        nbytes := 0; cbytes := 16.00 KB; ratio: 0.00
    -   cparams := cparams(clevel=5, shuffle=1, cname='blosclz', quantize=0)
    ?                             ^          ^
    +   cparams := cparams(clevel=9, shuffle=0, cname='blosclz', quantize=0)
    ?                             ^          ^
        chunklen := 2048; chunksize: 16384; blocksize: 0
        rootdir := '/tmp/bcolz-reprDiskTest5yj4sw34'
        mode    := 'a'
      []
    
    ======================================================================
    FAIL: test_repr_disk_array_read (test_carray.reprDiskTest)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/bcolz/bcolz/tests/test_carray.py", line 2424, in test_repr_disk_array_read
        self.assertEqual(expected, repr(x))
    AssertionError: "carr[90 chars]evel=5, shuffle=1, cname='blosclz', quantize=0[119 chars]\n[]" != "carr[90 chars]evel=9, shuffle=0, cname='blosclz', quantize=0[119 chars]\n[]"
      carray((0,), float64)
        nbytes := 0; cbytes := 16.00 KB; ratio: 0.00
    -   cparams := cparams(clevel=5, shuffle=1, cname='blosclz', quantize=0)
    ?                             ^          ^
    +   cparams := cparams(clevel=9, shuffle=0, cname='blosclz', quantize=0)
    ?                             ^          ^
        chunklen := 2048; chunksize: 16384; blocksize: 0
        rootdir := '/tmp/bcolz-reprDiskTestiflbef87'
        mode    := 'r'
      []
    
    ======================================================================
    FAIL: test_repr_disk_array_write (test_carray.reprDiskTest)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/bcolz/bcolz/tests/test_carray.py", line 2419, in test_repr_disk_array_write
        self.assertEqual(expected, repr(x))
    AssertionError: "carr[90 chars]evel=5, shuffle=1, cname='blosclz', quantize=0[119 chars]\n[]" != "carr[90 chars]evel=9, shuffle=0, cname='blosclz', quantize=0[119 chars]\n[]"
      carray((0,), float64)
        nbytes := 0; cbytes := 16.00 KB; ratio: 0.00
    -   cparams := cparams(clevel=5, shuffle=1, cname='blosclz', quantize=0)
    ?                             ^          ^
    +   cparams := cparams(clevel=9, shuffle=0, cname='blosclz', quantize=0)
    ?                             ^          ^
        chunklen := 2048; chunksize: 16384; blocksize: 0
        rootdir := '/tmp/bcolz-reprDiskTestz450jgp7'
        mode    := 'w'
      []
    
    ======================================================================
    FAIL: test02 (test_ctable.copyDiskTest)
    Testing copy() with lower clevel
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/bcolz/bcolz/tests/test_ctable.py", line 1025, in test02
        self.assertTrue(t['f1'].cbytes < t2['f1'].cbytes, "clevel not changed")
    AssertionError: False is not true : clevel not changed
    
    ======================================================================
    FAIL: test03 (test_ctable.copyDiskTest)
    Testing copy() with no shuffle
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/bcolz/bcolz/tests/test_ctable.py", line 1036, in test03
        self.assertTrue(t['f1'].cbytes < t2['f1'].cbytes, "clevel not changed")
    AssertionError: False is not true : clevel not changed
    
    ======================================================================
    FAIL: test01 (test_ctable.copyMemoryTest)
    Testing copy() with higher clevel
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/bcolz/bcolz/tests/test_ctable.py", line 1014, in test01
        self.assertTrue(t['f1'].cbytes > t2['f1'].cbytes, "clevel not changed")
    AssertionError: False is not true : clevel not changed
    
    ======================================================================
    FAIL: test02 (test_ctable.copyMemoryTest)
    Testing copy() with lower clevel
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/bcolz/bcolz/tests/test_ctable.py", line 1025, in test02
        self.assertTrue(t['f1'].cbytes < t2['f1'].cbytes, "clevel not changed")
    AssertionError: False is not true : clevel not changed
    
    ======================================================================
    FAIL: test03 (test_ctable.copyMemoryTest)
    Testing copy() with no shuffle
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/bcolz/bcolz/tests/test_ctable.py", line 1036, in test03
        self.assertTrue(t['f1'].cbytes < t2['f1'].cbytes, "clevel not changed")
    AssertionError: False is not true : clevel not changed
    
    ======================================================================
    FAIL: test01 (test_ctable.specialTest)
    Testing __sizeof__() (big ctables)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/bcolz/bcolz/tests/test_ctable.py", line 1067, in test01
        "ctable does not seem to compress at all")
    AssertionError: False is not true : ctable does not seem to compress at all
    
    ----------------------------------------------------------------------
    Ran 1270 tests in 4.745s
    
    FAILED (failures=18, skipped=160)
    root@f0b7772b23f5:~/bcolz#
    

    As I'm not sure if it's a problem in Blosc or in bcolz, so I'm tentatively filing it here.

    @ginggs is the one who alerted me to the failure.

    opened by estan 17
  • Iterchunks

    Iterchunks

    Hi,

    I've created the iterchunks() function and added a test. Some comments:

    • Atm it fails, because _getitem expects a string and not a ndarray(!?!) If anyone could tell me what I'm doing wrong? For details see: https://groups.google.com/forum/#!topic/bcolz/Fk6YpxD5toc
    • It still calculates the leftover elements itself, because nleftover has not been merged into master yet

    Some tips to help me fix this, would be very much appreciated! BR

    Carst

    opened by CarstVaartjes 17
  • Disk handling

    Disk handling

    Hi,

    I've combined @cpcloud and @esc their improvements and added some extra checks and functionality (it should replace the older pulls + we should be able to close some issues after this) + did my long over-due auto-flush thing:

    • addcol/delcol #137 <- I also added something for a use-case where you add a diskbased carray to a diskbased ctable
    • purge #130
    • rootdir existence check keyerror #123 <- added a check for this
    • autoflush #127 (for disk-based ctable creation and append)

    So I hope this completely fixes all existing issues with disk-based ctables (we actually made some workarounds ourselves which we can also remove after this pull).

    BR

    Carst

    opened by CarstVaartjes 15
  • strange failure on linux and mac

    strange failure on linux and mac

    I'm unable to use bcolz with multiprocessing when my data is a particular size:

    here's a minimal failing example

    import shutil
    import numpy as np
    from math import ceil
    import bcolz
    from multiprocessing import Pool
    
    n = 2 ** 14
    rootdir = 'foo'
    
    try:
        shutil.rmtree(rootdir)
    except OSError:
        pass
    
    ct = bcolz.carray(np.random.rand(n) * 1000, rootdir=rootdir)
    cs = 1000
    
    slices = [slice(cs * i, cs * (i + 1)) for i in range(int(ceil(n / cs)))]
    
    
    class Sum(object):
        def __init__(self, data):
            self.data = data
    
        def __call__(self, index):
            return self.data[index].sum()
    
    
    if __name__ == '__main__':
        p = Pool()
        caller = Sum(ct)
        result = p.map(caller, slices)
        print(result)
    
    opened by cpcloud 15
  • macos Ventura 13.1  Python 3.7 pip install bcolz error

    macos Ventura 13.1 Python 3.7 pip install bcolz error

    Pip installing gives:

    (invest_research) natsu@Natsu-MacBook-Pro invest_research % pip install bcolz
    Looking in indexes: http://pypi.douban.com/simple, http://mirrors.aliyun.com/pypi/simple/, https://pypi.tuna.tsinghua.edu.cn/simple/, http://pypi.mirrors.ustc.edu.cn/simple/
    Collecting bcolz
      Using cached https://mirrors.bfsu.edu.cn/pypi/web/packages/5c/4e/23942de9d5c0fb16f10335fa83e52b431bcb8c0d4a8419c9ac206268c279/bcolz-1.2.1.tar.gz (1.5 MB)
      Preparing metadata (setup.py) ... done
    Requirement already satisfied: numpy>=1.7 in /opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages (from bcolz) (1.21.6)
    Building wheels for collected packages: bcolz
      Building wheel for bcolz (setup.py) ... error
      error: subprocess-exited-with-error
      
      × python setup.py bdist_wheel did not run successfully.
      │ exit code: 1
      ╰─> [156 lines of output]
          SSE2 detected
          /opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/setuptools/installer.py:30: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
            SetuptoolsDeprecationWarning,
          running bdist_wheel
          running build
          running build_py
          creating build
          creating build/lib.macosx-10.9-x86_64-cpython-37
          creating build/lib.macosx-10.9-x86_64-cpython-37/bcolz
          copying bcolz/py2help_tests.py -> build/lib.macosx-10.9-x86_64-cpython-37/bcolz
          copying bcolz/version.py -> build/lib.macosx-10.9-x86_64-cpython-37/bcolz
          copying bcolz/toplevel.py -> build/lib.macosx-10.9-x86_64-cpython-37/bcolz
          ...
          creating build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/zstd-1.3.4/dictBuilder
          creating build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/zstd-1.3.4/legacy
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c bcolz/carray_ext.c -o build/temp.macosx-10.9-x86_64-cpython-37/bcolz/carray_ext.o -DSHUFFLE_SSE2_ENABLED -msse2
          In file included from bcolz/carray_ext.c:525:
          In file included from /opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4:
          In file included from /opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12:
          In file included from /opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1969:
          /opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: "Using deprecated NumPy API, disable it with "          "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
          #warning "Using deprecated NumPy API, disable it with " \
           ^
          1 warning generated.
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/blosc/bitshuffle-generic.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/blosc/bitshuffle-generic.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/blosc/bitshuffle-sse2.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/blosc/bitshuffle-sse2.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/blosc/blosc.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/blosc/blosc.o -DSHUFFLE_SSE2_ENABLED -msse2
          c-blosc/blosc/blosc.c:684:11: warning: variable 'ctbytes' set but not used [-Wunused-but-set-variable]
            int32_t ctbytes = 0;           /* number of compressed bytes in block */
                    ^
          c-blosc/blosc/blosc.c:874:7: warning: unused variable 'rc' [-Wunused-variable]
            int rc;
                ^
          c-blosc/blosc/blosc.c:1656:7: warning: unused variable 'rc' [-Wunused-variable]
            int rc;
                ^
          c-blosc/blosc/blosc.c:2136:7: warning: unused variable 'rc' [-Wunused-variable]
            int rc;
                ^
          4 warnings generated.
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/blosc/blosclz.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/blosc/blosclz.o -DSHUFFLE_SSE2_ENABLED -msse2
          c-blosc/blosc/blosclz.c:82:24: warning: unused function 'get_run' [-Wunused-function]
          static inline uint8_t *get_run(uint8_t *ip, const uint8_t *ip_bound, const uint8_t *ref) {
                                 ^
          1 warning generated.
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/blosc/fastcopy.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/blosc/fastcopy.o -DSHUFFLE_SSE2_ENABLED -msse2
          c-blosc/blosc/fastcopy.c:246:30: warning: unused function 'chunk_memcpy_16' [-Wunused-function]
          static inline unsigned char *chunk_memcpy_16(unsigned char *out, const unsigned char *from, unsigned len) {
                                       ^
          c-blosc/blosc/fastcopy.c:270:30: warning: unused function 'chunk_memcpy_32' [-Wunused-function]
          static inline unsigned char *chunk_memcpy_32(unsigned char *out, const unsigned char *from, unsigned len) {
                                       ^
          c-blosc/blosc/fastcopy.c:294:30: warning: unused function 'chunk_memcpy_32_unrolled' [-Wunused-function]
          static inline unsigned char *chunk_memcpy_32_unrolled(unsigned char *out, const unsigned char *from, unsigned len) {
                                       ^
          c-blosc/blosc/fastcopy.c:402:30: warning: unused function 'chunk_memcpy_aligned' [-Wunused-function]
          static inline unsigned char *chunk_memcpy_aligned(unsigned char *out, const unsigned char *from, unsigned len) {
                                       ^
          4 warnings generated.
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/blosc/shuffle-generic.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/blosc/shuffle-generic.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/blosc/shuffle-sse2.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/blosc/shuffle-sse2.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/blosc/shuffle.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/blosc/shuffle.o -DSHUFFLE_SSE2_ENABLED -msse2
          In file included from c-blosc/blosc/shuffle.c:11:
          c-blosc/blosc/shuffle-generic.h:32:13: warning: unused function 'shuffle_generic_inline' [-Wunused-function]
          static void shuffle_generic_inline(const size_t type_size,
                      ^
          c-blosc/blosc/shuffle-generic.h:61:13: warning: unused function 'unshuffle_generic_inline' [-Wunused-function]
          static void unshuffle_generic_inline(const size_t type_size,
                      ^
          2 warnings generated.
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/internal-complibs/lz4-1.8.1.2/lz4.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/lz4-1.8.1.2/lz4.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/internal-complibs/lz4-1.8.1.2/lz4hc.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/lz4-1.8.1.2/lz4hc.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/internal-complibs/snappy-1.1.1/snappy-c.cc -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/snappy-1.1.1/snappy-c.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/internal-complibs/snappy-1.1.1/snappy-sinksource.cc -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/snappy-1.1.1/snappy-sinksource.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/internal-complibs/snappy-1.1.1/snappy-stubs-internal.cc -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/snappy-1.1.1/snappy-stubs-internal.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/internal-complibs/snappy-1.1.1/snappy.cc -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/snappy-1.1.1/snappy.o -DSHUFFLE_SSE2_ENABLED -msse2
          c-blosc/internal-complibs/snappy-1.1.1/snappy.cc:1013:33: warning: comparison of integers of different signs: 'int' and 'const size_t' (aka 'const unsigned long') [-Wsign-compare]
                  if (curr_iov_index_ + 1 >= output_iov_count_) {
                      ~~~~~~~~~~~~~~~~~~~ ^  ~~~~~~~~~~~~~~~~~
          c-blosc/internal-complibs/snappy-1.1.1/snappy.cc:1095:35: warning: comparison of integers of different signs: 'int' and 'const size_t' (aka 'const unsigned long') [-Wsign-compare]
                    if (curr_iov_index_ + 1 >= output_iov_count_) {
                        ~~~~~~~~~~~~~~~~~~~ ^  ~~~~~~~~~~~~~~~~~
          2 warnings generated.
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/internal-complibs/zlib-1.2.8/adler32.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/zlib-1.2.8/adler32.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/internal-complibs/zlib-1.2.8/compress.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/zlib-1.2.8/compress.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/internal-complibs/zlib-1.2.8/crc32.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/zlib-1.2.8/crc32.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/internal-complibs/zlib-1.2.8/deflate.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/zlib-1.2.8/deflate.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/internal-complibs/zlib-1.2.8/gzclose.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/zlib-1.2.8/gzclose.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/internal-complibs/zlib-1.2.8/gzlib.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/zlib-1.2.8/gzlib.o -DSHUFFLE_SSE2_ENABLED -msse2
          c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:256:24: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
                  state->start = LSEEK(state->fd, 0, SEEK_CUR);
                                 ^
          c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:14:17: note: expanded from macro 'LSEEK'
          #  define LSEEK lseek
                          ^
          c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:256:24: note: did you mean 'fseek'?
          c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:14:17: note: expanded from macro 'LSEEK'
          #  define LSEEK lseek
                          ^
          /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/stdio.h:162:6: note: 'fseek' declared here
          int      fseek(FILE *, long, int);
                   ^
          c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:355:9: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
              if (LSEEK(state->fd, state->start, SEEK_SET) == -1)
                  ^
          c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:14:17: note: expanded from macro 'LSEEK'
          #  define LSEEK lseek
                          ^
          c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:396:15: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
                  ret = LSEEK(state->fd, offset - state->x.have, SEEK_CUR);
                        ^
          c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:14:17: note: expanded from macro 'LSEEK'
          #  define LSEEK lseek
                          ^
          c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:492:14: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
              offset = LSEEK(state->fd, 0, SEEK_CUR);
                       ^
          c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:14:17: note: expanded from macro 'LSEEK'
          #  define LSEEK lseek
                          ^
          4 errors generated.
          error: command '/usr/bin/gcc' failed with exit code 1
          [end of output]
      
      note: This error originates from a subprocess, and is likely not a problem with pip.
      ERROR: Failed building wheel for bcolz
      Running setup.py clean for bcolz
    Failed to build bcolz
    Installing collected packages: bcolz
      Running setup.py install for bcolz ... error
      error: subprocess-exited-with-error
      
      × Running setup.py install for bcolz did not run successfully.
      │ exit code: 1
      ╰─> [158 lines of output]
          SSE2 detected
          /opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/setuptools/installer.py:30: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
            SetuptoolsDeprecationWarning,
          running install
          /opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
            setuptools.SetuptoolsDeprecationWarning,
          running build
          running build_py
          creating build
          creating build/lib.macosx-10.9-x86_64-cpython-37
          creating build/lib.macosx-10.9-x86_64-cpython-37/bcolz
          ...
          creating build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/zstd-1.3.4/deprecated
          creating build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/zstd-1.3.4/dictBuilder
          creating build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/zstd-1.3.4/legacy
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c bcolz/carray_ext.c -o build/temp.macosx-10.9-x86_64-cpython-37/bcolz/carray_ext.o -DSHUFFLE_SSE2_ENABLED -msse2
          In file included from bcolz/carray_ext.c:525:
          In file included from /opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4:
          In file included from /opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12:
          In file included from /opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1969:
          /opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: "Using deprecated NumPy API, disable it with "          "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
          #warning "Using deprecated NumPy API, disable it with " \
           ^
          1 warning generated.
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/blosc/bitshuffle-generic.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/blosc/bitshuffle-generic.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/blosc/bitshuffle-sse2.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/blosc/bitshuffle-sse2.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/blosc/blosc.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/blosc/blosc.o -DSHUFFLE_SSE2_ENABLED -msse2
          c-blosc/blosc/blosc.c:684:11: warning: variable 'ctbytes' set but not used [-Wunused-but-set-variable]
            int32_t ctbytes = 0;           /* number of compressed bytes in block */
                    ^
          c-blosc/blosc/blosc.c:874:7: warning: unused variable 'rc' [-Wunused-variable]
            int rc;
                ^
          c-blosc/blosc/blosc.c:1656:7: warning: unused variable 'rc' [-Wunused-variable]
            int rc;
                ^
          c-blosc/blosc/blosc.c:2136:7: warning: unused variable 'rc' [-Wunused-variable]
            int rc;
                ^
          4 warnings generated.
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/blosc/blosclz.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/blosc/blosclz.o -DSHUFFLE_SSE2_ENABLED -msse2
          c-blosc/blosc/blosclz.c:82:24: warning: unused function 'get_run' [-Wunused-function]
          static inline uint8_t *get_run(uint8_t *ip, const uint8_t *ip_bound, const uint8_t *ref) {
                                 ^
          1 warning generated.
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/blosc/fastcopy.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/blosc/fastcopy.o -DSHUFFLE_SSE2_ENABLED -msse2
          c-blosc/blosc/fastcopy.c:246:30: warning: unused function 'chunk_memcpy_16' [-Wunused-function]
          static inline unsigned char *chunk_memcpy_16(unsigned char *out, const unsigned char *from, unsigned len) {
                                       ^
          c-blosc/blosc/fastcopy.c:270:30: warning: unused function 'chunk_memcpy_32' [-Wunused-function]
          static inline unsigned char *chunk_memcpy_32(unsigned char *out, const unsigned char *from, unsigned len) {
                                       ^
          c-blosc/blosc/fastcopy.c:294:30: warning: unused function 'chunk_memcpy_32_unrolled' [-Wunused-function]
          static inline unsigned char *chunk_memcpy_32_unrolled(unsigned char *out, const unsigned char *from, unsigned len) {
                                       ^
          c-blosc/blosc/fastcopy.c:402:30: warning: unused function 'chunk_memcpy_aligned' [-Wunused-function]
          static inline unsigned char *chunk_memcpy_aligned(unsigned char *out, const unsigned char *from, unsigned len) {
                                       ^
          4 warnings generated.
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/blosc/shuffle-generic.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/blosc/shuffle-generic.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/blosc/shuffle-sse2.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/blosc/shuffle-sse2.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/blosc/shuffle.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/blosc/shuffle.o -DSHUFFLE_SSE2_ENABLED -msse2
          In file included from c-blosc/blosc/shuffle.c:11:
          c-blosc/blosc/shuffle-generic.h:32:13: warning: unused function 'shuffle_generic_inline' [-Wunused-function]
          static void shuffle_generic_inline(const size_t type_size,
                      ^
          c-blosc/blosc/shuffle-generic.h:61:13: warning: unused function 'unshuffle_generic_inline' [-Wunused-function]
          static void unshuffle_generic_inline(const size_t type_size,
                      ^
          2 warnings generated.
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/internal-complibs/lz4-1.8.1.2/lz4.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/lz4-1.8.1.2/lz4.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/internal-complibs/lz4-1.8.1.2/lz4hc.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/lz4-1.8.1.2/lz4hc.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/internal-complibs/snappy-1.1.1/snappy-c.cc -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/snappy-1.1.1/snappy-c.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/internal-complibs/snappy-1.1.1/snappy-sinksource.cc -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/snappy-1.1.1/snappy-sinksource.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/internal-complibs/snappy-1.1.1/snappy-stubs-internal.cc -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/snappy-1.1.1/snappy-stubs-internal.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/internal-complibs/snappy-1.1.1/snappy.cc -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/snappy-1.1.1/snappy.o -DSHUFFLE_SSE2_ENABLED -msse2
          c-blosc/internal-complibs/snappy-1.1.1/snappy.cc:1013:33: warning: comparison of integers of different signs: 'int' and 'const size_t' (aka 'const unsigned long') [-Wsign-compare]
                  if (curr_iov_index_ + 1 >= output_iov_count_) {
                      ~~~~~~~~~~~~~~~~~~~ ^  ~~~~~~~~~~~~~~~~~
          c-blosc/internal-complibs/snappy-1.1.1/snappy.cc:1095:35: warning: comparison of integers of different signs: 'int' and 'const size_t' (aka 'const unsigned long') [-Wsign-compare]
                    if (curr_iov_index_ + 1 >= output_iov_count_) {
                        ~~~~~~~~~~~~~~~~~~~ ^  ~~~~~~~~~~~~~~~~~
          2 warnings generated.
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/internal-complibs/zlib-1.2.8/adler32.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/zlib-1.2.8/adler32.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/internal-complibs/zlib-1.2.8/compress.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/zlib-1.2.8/compress.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/internal-complibs/zlib-1.2.8/crc32.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/zlib-1.2.8/crc32.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/internal-complibs/zlib-1.2.8/deflate.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/zlib-1.2.8/deflate.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/internal-complibs/zlib-1.2.8/gzclose.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/zlib-1.2.8/gzclose.o -DSHUFFLE_SSE2_ENABLED -msse2
          gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -I/opt/homebrew/anaconda3/envs/invest_research/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/opt/homebrew/anaconda3/envs/invest_research/lib/python3.7/site-packages/numpy/core/include -I/opt/homebrew/anaconda3/envs/invest_research/include/python3.7m -c c-blosc/internal-complibs/zlib-1.2.8/gzlib.c -o build/temp.macosx-10.9-x86_64-cpython-37/c-blosc/internal-complibs/zlib-1.2.8/gzlib.o -DSHUFFLE_SSE2_ENABLED -msse2
          c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:256:24: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
                  state->start = LSEEK(state->fd, 0, SEEK_CUR);
                                 ^
          c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:14:17: note: expanded from macro 'LSEEK'
          #  define LSEEK lseek
                          ^
          c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:256:24: note: did you mean 'fseek'?
          c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:14:17: note: expanded from macro 'LSEEK'
          #  define LSEEK lseek
                          ^
          /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/stdio.h:162:6: note: 'fseek' declared here
          int      fseek(FILE *, long, int);
                   ^
          c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:355:9: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
              if (LSEEK(state->fd, state->start, SEEK_SET) == -1)
                  ^
          c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:14:17: note: expanded from macro 'LSEEK'
          #  define LSEEK lseek
                          ^
          c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:396:15: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
                  ret = LSEEK(state->fd, offset - state->x.have, SEEK_CUR);
                        ^
          c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:14:17: note: expanded from macro 'LSEEK'
          #  define LSEEK lseek
                          ^
          c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:492:14: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
              offset = LSEEK(state->fd, 0, SEEK_CUR);
                       ^
          c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:14:17: note: expanded from macro 'LSEEK'
          #  define LSEEK lseek
                          ^
          4 errors generated.
          error: command '/usr/bin/gcc' failed with exit code 1
          [end of output]
      
      note: This error originates from a subprocess, and is likely not a problem with pip.
    error: legacy-install-failure
    
    × Encountered error while trying to install package.
    ╰─> bcolz
    
    note: This is an issue with the package mentioned above, not pip.
    hint: See above for output from the failure.
    (invest_research) natsu@Natsu-MacBook-Pro invest_research % 
    
    opened by Natsu-Lee 0
  • Improvements to C code that can improve performance

    Improvements to C code that can improve performance

    Hi,

    I work at Intel, and we have developed a tool that detects anomalous programming language expressions that can possibly lead to bugs. We scanned the code repository for this project as it has considerably high number of stars!

    We found a couple of places where the code/expressions are confusing and seem to implement the logic in a rather convoluted manner. We think that the expressions could be rewritten to capture the logic accurately and precisely.

    Case 1) Expression at line 839 in c-blosc/internal-complibs/zstd-1.3.4/compress/zstdmt_compress.c

    The if expression in the following code performs bitwise OR to check that the pointers are not NULL (if I'm correct). Bitwise OR operator in C does not go through short-circuit evaluation. As a result, all subexpressions (pointer derefs) are evaluated, leading to lost optimization opportunity. Boolean OR can help with such an optimization.

    if (!mtctx->factory | !mtctx->jobs | !mtctx->bufPool | !mtctx->cctxPool | !mtctx->seqPool | initError) {
            ZSTDMT_freeCCtx(mtctx);
      return NULL;
    }
    

    Case 2) Similar case for expression (offsetCode | !litLength) at line 2890 under c-blosc/internal-complibs/zstd-1.3.4/legacy/zstd_v04.c.

    Any thoughts?

    opened by nhasabni 1
  • Cannot install bcolz for python 3.8 & python 3.9

    Cannot install bcolz for python 3.8 & python 3.9

    Hi! I am using python3.8 (more specifically 3.8.5). I tried installing bcolz in a fresh virtual environment (should be no conflicts with other package versions) but I am getting errors when installing. Also, I could not install it with python 3.9

    It might be related to this issue, but let me know if I should attach a full stack trace.

    Note: it works on Python 3.7

    opened by Ranapop 8
  • how to get my carray data back afterr flush?

    how to get my carray data back afterr flush?

    i can not get my carray data back after flush, the code is below

    import numpy as np
    import bcolz
    
    bcolz.print_versions()
    arrt = np.dtype('int32,int32') 
    src_data = [(i, i * i) for i in range(10)]
    carr = bcolz.carray(src_data, dtype=arrt, rootdir='./css', mode='w')
    carr.flush()
    carr2 = bcolz.open(rootdir='./css', mode='r')
    carr2
    

    errors

    -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    bcolz version:     1.2.1
    NumPy version:     1.20.1
    Blosc version:     1.14.3 ($Date:: 2018-04-06 #$)
    Blosc compressors: ['blosclz', 'lz4', 'lz4hc', 'snappy', 'zlib', 'zstd']
    Numexpr version:   2.7.3
    Dask version:      2021.03.0
    Python version:    3.7.9 (default, Mar 10 2021, 23:54:50) 
    [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
    Platform:          linux-x86_64
    Byte-ordering:     little
    Detected cores:    4
    -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    <ipython-input-71-d6a707a69c8c> in <module>
          2 import bcolz as bz
          3 
    ----> 4 carr2 = bcolz.open(rootdir='./css', mode='r')
          5 carr2
    
    ~/.local/lib/python3.7/site-packages/bcolz/toplevel.py in open(rootdir, mode)
        138         return bcolz.ctable(rootdir=rootdir, mode=mode)
        139     else:
    --> 140         return bcolz.carray(rootdir=rootdir, mode=mode)
        141 
        142 
    
    bcolz/carray_ext.pyx in bcolz.carray_ext.carray.__cinit__()
    
    bcolz/carray_ext.pyx in bcolz.carray_ext.carray._read_meta()
    
    TypeError: data type "[('f0', '<i4'), ('f1', '<i4')]" not understood
    

    what had i done wrongly? how to get my carray data back afterr flush? #415 help, ths in advantage

    opened by kaybinwong 2
  • cant install bcolz library

    cant install bcolz library

    when i want to install bcolz library om my system(windows),my system gives this error:

    pip install bcolz Collecting bcolz Using cached bcolz-1.2.1.tar.gz (1.5 MB) Requirement already satisfied: numpy>=1.7 in c:\users\faeze\appdata\local\programs\python\python38\lib\site-packages (from bcolz) (1.19.4) Building wheels for collected packages: bcolz Building wheel for bcolz (setup.py) ... error ERROR: Command errored out with exit status 1: command: 'c:\users\faeze\appdata\local\programs\python\python38\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\faeze\AppData\Local\Temp\pip-install-_76jwhg3\bcolz_1f41c783dfc140f484cd2ff65bac521e\setup.py'"'"'; file='"'"'C:\Users\faeze\AppData\Local\Temp\pip-install-76jwhg3\bcolz_1f41c783dfc140f484cd2ff65bac521e\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\faeze\AppData\Local\Temp\pip-wheel-aqlxdtf1' cwd: C:\Users\faeze\AppData\Local\Temp\pip-install-76jwhg3\bcolz_1f41c783dfc140f484cd2ff65bac521e
    Complete output (42 lines): SSE2 detected running bdist_wheel running build running build_py creating build creating build\lib.win-amd64-3.8 creating build\lib.win-amd64-3.8\bcolz copying bcolz\arrayprint.py -> build\lib.win-amd64-3.8\bcolz copying bcolz\attrs.py -> build\lib.win-amd64-3.8\bcolz copying bcolz\chunked_eval.py -> build\lib.win-amd64-3.8\bcolz copying bcolz\ctable.py -> build\lib.win-amd64-3.8\bcolz copying bcolz\defaults.py -> build\lib.win-amd64-3.8\bcolz copying bcolz\py2help.py -> build\lib.win-amd64-3.8\bcolz copying bcolz\py2help_tests.py -> build\lib.win-amd64-3.8\bcolz copying bcolz\toplevel.py -> build\lib.win-amd64-3.8\bcolz copying bcolz\utils.py -> build\lib.win-amd64-3.8\bcolz copying bcolz\version.py -> build\lib.win-amd64-3.8\bcolz copying bcolz_init
    .py -> build\lib.win-amd64-3.8\bcolz creating build\lib.win-amd64-3.8\bcolz\tests copying bcolz\tests\all.py -> build\lib.win-amd64-3.8\bcolz\tests copying bcolz\tests\common.py -> build\lib.win-amd64-3.8\bcolz\tests copying bcolz\tests\test_attrs.py -> build\lib.win-amd64-3.8\bcolz\tests copying bcolz\tests\test_carray.py -> build\lib.win-amd64-3.8\bcolz\tests copying bcolz\tests\test_carray_objects.py -> build\lib.win-amd64-3.8\bcolz\tests copying bcolz\tests\test_ctable.py -> build\lib.win-amd64-3.8\bcolz\tests copying bcolz\tests\test_ndcarray.py -> build\lib.win-amd64-3.8\bcolz\tests copying bcolz\tests\test_queries.py -> build\lib.win-amd64-3.8\bcolz\tests copying bcolz\tests_init
    .py -> build\lib.win-amd64-3.8\bcolz\tests copying bcolz\carray_ext.pxd -> build\lib.win-amd64-3.8\bcolz running build_ext skipping 'bcolz\carray_ext.c' Cython extension (up-to-date) building 'bcolz.carray_ext' extension error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/ SSE2 detected running bdist_wheel running build running build_py copying bcolz\version.py -> build\lib.win-amd64-3.8\bcolz running build_ext skipping 'bcolz\carray_ext.c' Cython extension (up-to-date) building 'bcolz.carray_ext' extension error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/

    ERROR: Failed building wheel for bcolz Running setup.py clean for bcolz Failed to build bcolz Installing collected packages: bcolz Running setup.py install for bcolz ... error ERROR: Command errored out with exit status 1: command: 'c:\users\faeze\appdata\local\programs\python\python38\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\faeze\AppData\Local\Temp\pip-install-_76jwhg3\bcolz_1f41c783dfc140f484cd2ff65bac521e\setup.py'"'"'; file='"'"'C:\Users\faeze\AppData\Local\Temp\pip-install-76jwhg3\bcolz_1f41c783dfc140f484cd2ff65bac521e\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record 'C:\Users\faeze\AppData\Local\Temp\pip-record-vc8_dq19\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\users\faeze\appdata\local\programs\python\python38\Include\bcolz' cwd: C:\Users\faeze\AppData\Local\Temp\pip-install-76jwhg3\bcolz_1f41c783dfc140f484cd2ff65bac521e
    Complete output (42 lines): SSE2 detected running install running build running build_py creating build creating build\lib.win-amd64-3.8 creating build\lib.win-amd64-3.8\bcolz copying bcolz\arrayprint.py -> build\lib.win-amd64-3.8\bcolz copying bcolz\attrs.py -> build\lib.win-amd64-3.8\bcolz copying bcolz\chunked_eval.py -> build\lib.win-amd64-3.8\bcolz copying bcolz\ctable.py -> build\lib.win-amd64-3.8\bcolz copying bcolz\defaults.py -> build\lib.win-amd64-3.8\bcolz copying bcolz\py2help.py -> build\lib.win-amd64-3.8\bcolz copying bcolz\py2help_tests.py -> build\lib.win-amd64-3.8\bcolz copying bcolz\toplevel.py -> build\lib.win-amd64-3.8\bcolz copying bcolz\utils.py -> build\lib.win-amd64-3.8\bcolz copying bcolz\version.py -> build\lib.win-amd64-3.8\bcolz copying bcolz_init
    .py -> build\lib.win-amd64-3.8\bcolz creating build\lib.win-amd64-3.8\bcolz\tests copying bcolz\tests\all.py -> build\lib.win-amd64-3.8\bcolz\tests copying bcolz\tests\common.py -> build\lib.win-amd64-3.8\bcolz\tests copying bcolz\tests\test_attrs.py -> build\lib.win-amd64-3.8\bcolz\tests copying bcolz\tests\test_carray.py -> build\lib.win-amd64-3.8\bcolz\tests copying bcolz\tests\test_carray_objects.py -> build\lib.win-amd64-3.8\bcolz\tests copying bcolz\tests\test_ctable.py -> build\lib.win-amd64-3.8\bcolz\tests copying bcolz\tests\test_ndcarray.py -> build\lib.win-amd64-3.8\bcolz\tests copying bcolz\tests\test_queries.py -> build\lib.win-amd64-3.8\bcolz\tests copying bcolz\tests_init
    .py -> build\lib.win-amd64-3.8\bcolz\tests copying bcolz\carray_ext.pxd -> build\lib.win-amd64-3.8\bcolz running build_ext skipping 'bcolz\carray_ext.c' Cython extension (up-to-date) building 'bcolz.carray_ext' extension error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/ SSE2 detected running install running build running build_py copying bcolz\version.py -> build\lib.win-amd64-3.8\bcolz running build_ext skipping 'bcolz\carray_ext.c' Cython extension (up-to-date) building 'bcolz.carray_ext' extension error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/ ---------------------------------------- ERROR: Command errored out with exit status 1: 'c:\users\faeze\appdata\local\programs\python\python38\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\faeze\AppData\Local\Temp\pip-install-_76jwhg3\bcolz_1f41c783dfc140f484cd2ff65bac521e\setup.py'"'"'; file='"'"'C:\Users\faeze\AppData\Local\Temp\pip-install-_76jwhg3\bcolz_1f41c783dfc140f484cd2ff65bac521e\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record 'C:\Users\faeze\AppData\Local\Temp\pip-record-vc8_dq19\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\users\faeze\appdata\local\programs\python\python38\Include\bcolz' Check the logs for full command output.

    how can i fix it?

    opened by faezesdt 1
Releases(1.2.1)
  • 1.2.1(Apr 13, 2018)

    • C-Blosc internal sources updated to 1.14.3. This basically means that internal Zstd sources are bumped to 1.3.4, which may lead to noticeable improved speeds (specially for low compression ratios).

    • np.datetime64 and other scalar objects that have __getitem__() are now supported in _eval_blocks(). PR #377. Thanks to apalepu23.

    • Vendored cpuinfo.py updated to 4.0.0 (ARM aarch64 is recognized now).

    • Allow setup.py to work even if not on Intel or ARM or PPC archs are found.

    Source code(tar.gz)
    Source code(zip)
  • 1.2.0(Mar 23, 2018)

    • Support for Python <= 2.6 or Python <= 3.4 has been deprecated.

    • C-Blosc internal sources updated to 1.14.2. Using a C-Blosc library > 1.14 is important for forward compatibility. For more info see: http://blosc.org/posts/new-forward-compat-policy/

    Source code(tar.gz)
    Source code(zip)
  • 1.1.1(Feb 1, 2017)

    Changes from 1.1.0 to 1.1.1

    • Allow to delete all the columns in a ctable. Fixes #306.

    • Double-check the value of a column that is being overwritten. Fixes #307.

    • Use pkg_resources.parse_version() to test for version of packages. Fixes #322.

    • Now all the columns in a ctable are enforced to be a carray instance in order to simplify the internal logic for handling columns.

    • Now, the cparams are preserved during column replacement, e.g.:

      ct['f0'] = x + 1

      will continue to use the same cparams than the original column.

    • C-Blosc updated to 1.11.2.

    • Added a new defaults_ctx context so that users can select defaults easily without changing global behaviour. For example::

      with bcolz.defaults_ctx(vm="python", cparams=bcolz.cparams(clevel=0)): cout = bcolz.eval("(x + 1) < 0")

    • Fixed a crash occurring in ctable.todataframe() when both columns and orient='columns' were specified. PR #311. Thanks to Peter Quackenbush.

    Source code(tar.gz)
    Source code(zip)
Datashredder is a simple data corruption engine written in python. You can corrupt anything text, images and video.

Datashredder is a simple data corruption engine written in python. You can corrupt anything text, images and video. You can chose the cha

null 2 Jul 22, 2022
Data Competition: automated systems that can detect whether people are not wearing masks or are wearing masks incorrectly

Table of contents Introduction Dataset Model & Metrics How to Run Quickstart Install Training Evaluation Detection DATA COMPETITION The COVID-19 pande

Thanh Dat Vu 1 Feb 27, 2022
🧪 Panel-Chemistry - exploratory data analysis and build powerful data and viz tools within the domain of Chemistry using Python and HoloViz Panel.

???? ??. The purpose of the panel-chemistry project is to make it really easy for you to do DATA ANALYSIS and build powerful DATA AND VIZ APPLICATIONS within the domain of Chemistry using using Python and HoloViz Panel.

Marc Skov Madsen 97 Dec 8, 2022
fds is a tool for Data Scientists made by DAGsHub to version control data and code at once.

Fast Data Science, AKA fds, is a CLI for Data Scientists to version control data and code at once, by conveniently wrapping git and dvc

DAGsHub 359 Dec 22, 2022
Tuplex is a parallel big data processing framework that runs data science pipelines written in Python at the speed of compiled code

Tuplex is a parallel big data processing framework that runs data science pipelines written in Python at the speed of compiled code. Tuplex has similar Python APIs to Apache Spark or Dask, but rather than invoking the Python interpreter, Tuplex generates optimized LLVM bytecode for the given pipeline and input data set.

Tuplex 791 Jan 4, 2023
A data parser for the internal syncing data format used by Fog of World.

A data parser for the internal syncing data format used by Fog of World. The parser is not designed to be a well-coded library with good performance, it is more like a demo for showing the data structure.

Zed(Zijun) Chen 40 Dec 12, 2022
Fancy data functions that will make your life as a data scientist easier.

WhiteBox Utilities Toolkit: Tools to make your life easier Fancy data functions that will make your life as a data scientist easier. Installing To ins

WhiteBox 3 Oct 3, 2022
A Big Data ETL project in PySpark on the historical NYC Taxi Rides data

Processing NYC Taxi Data using PySpark ETL pipeline Description This is an project to extract, transform, and load large amount of data from NYC Taxi

Unnikrishnan 2 Dec 12, 2021
Created covid data pipeline using PySpark and MySQL that collected data stream from API and do some processing and store it into MYSQL database.

Created covid data pipeline using PySpark and MySQL that collected data stream from API and do some processing and store it into MYSQL database.

null 2 Nov 20, 2021
Utilize data analytics skills to solve real-world business problems using Humana’s big data

Humana-Mays-2021-HealthCare-Analytics-Case-Competition- The goal of the project is to utilize data analytics skills to solve real-world business probl

Yongxian (Caroline) Lun 1 Dec 27, 2021
Python data processing, analysis, visualization, and data operations

Python This is a Python data processing, analysis, visualization and data operations of the source code warehouse, book ISBN: 9787115527592 Descriptio

FangWei 1 Jan 16, 2022
PrimaryBid - Transform application Lifecycle Data and Design and ETL pipeline architecture for ingesting data from multiple sources to redshift

Transform application Lifecycle Data and Design and ETL pipeline architecture for ingesting data from multiple sources to redshift This project is composed of two parts: Part1 and Part2

Emmanuel Boateng Sifah 1 Jan 19, 2022
Demonstrate the breadth and depth of your data science skills by earning all of the Databricks Data Scientist credentials

Data Scientist Learning Plan Demonstrate the breadth and depth of your data science skills by earning all of the Databricks Data Scientist credentials

Trung-Duy Nguyen 27 Nov 1, 2022
PostQF is a user-friendly Postfix queue data filter which operates on data produced by postqueue -j.

PostQF Copyright © 2022 Ralph Seichter PostQF is a user-friendly Postfix queue data filter which operates on data produced by postqueue -j. See the ma

Ralph Seichter 11 Nov 24, 2022
Catalogue data - A Python Scripts to prepare catalogue data

catalogue_data Scripts to prepare catalogue data. Setup Clone this repo. Install

BigScience Workshop 3 Mar 3, 2022
A python package which can be pip installed to perform statistics and visualize binomial and gaussian distributions of the dataset

GBiStat package A python package to assist programmers with data analysis. This package could be used to plot : Binomial Distribution of the dataset p

Rishikesh S 4 Oct 17, 2022
A set of procedures that can realize covid19 virus detection based on blood.

A set of procedures that can realize covid19 virus detection based on blood.

Nuyoah-xlh 3 Mar 7, 2022
songplays datamart provide details about the musical taste of our customers and can help us to improve our recomendation system

Songplays User activity datamart The following document describes the model used to build the songplays datamart table and the respective ETL process.

Leandro Kellermann de Oliveira 1 Jul 13, 2021