High performance datastore for time series and tick data

Related tags

Data Containers python database mongodb timeseries pandas tickstore arctic timeseries-database
Overview

arctic Arctic TimeSeries and Tick store

Documentation Status Travis CI Join the chat at https://gitter.im/manahl/arctic

Arctic is a high performance datastore for numeric data. It supports Pandas, numpy arrays and pickled objects out-of-the-box, with pluggable support for other data types and optional versioning.

Arctic can query millions of rows per second per client, achieves ~10x compression on network bandwidth, ~10x compression on disk, and scales to hundreds of millions of rows per second per MongoDB instance.

Arctic has been under active development at Man AHL since 2012.

Quickstart

Install Arctic

pip install git+https://github.com/manahl/arctic.git

Run a MongoDB

mongod --dbpath <path/to/db_directory>

Using VersionStore

from arctic import Arctic
import quandl

# Connect to Local MONGODB
store = Arctic('localhost')

# Create the library - defaults to VersionStore
store.initialize_library('NASDAQ')

# Access the library
library = store['NASDAQ']

# Load some data - maybe from Quandl
aapl = quandl.get("WIKI/AAPL", authtoken="your token here")

# Store the data in the library
library.write('AAPL', aapl, metadata={'source': 'Quandl'})

# Reading the data
item = library.read('AAPL')
aapl = item.data
metadata = item.metadata

VersionStore supports much more: See the HowTo!

Adding your own storage engine

Plugging a custom class in as a library type is straightforward. This example shows how.

Documentation

You can find complete documentation at Arctic docs

Concepts

Libraries

Arctic provides namespaced libraries of data. These libraries allow bucketing data by source, user or some other metric (for example frequency: End-Of-Day; Minute Bars; etc.).

Arctic supports multiple data libraries per user. A user (or namespace) maps to a MongoDB database (the granularity of mongo authentication). The library itself is composed of a number of collections within the database. Libraries look like:

  • user.EOD
  • user.ONEMINUTE

A library is mapped to a Python class. All library databases in MongoDB are prefixed with 'arctic_'

Storage Engines

Arctic includes three storage engines:

  • VersionStore: a key-value versioned TimeSeries store. It supports:
    • Pandas data types (other Python types pickled)
    • Multiple versions of each data item. Can easily read previous versions.
    • Create point-in-time snapshots across symbols in a library
    • Soft quota support
    • Hooks for persisting other data types
    • Audited writes: API for saving metadata and data before and after a write.
    • a wide range of TimeSeries data frequencies: End-Of-Day to Minute bars
    • See the HowTo
    • Documentation
  • TickStore: Column oriented tick database. Supports dynamic fields, chunks aren't versioned. Designed for large continuously ticking data.
  • Chunkstore: A storage type that allows data to be stored in customizable chunk sizes. Chunks aren't versioned, and can be appended to and updated in place.

Arctic storage implementations are pluggable. VersionStore is the default.

Requirements

Arctic currently works with:

  • Python 2.7, 3.4, 3.5, 3.6
  • pymongo >= 3.6
  • Pandas
  • MongoDB >= 2.4.x

Operating Systems:

  • Linux
  • macOS
  • Windows 10

Acknowledgements

Arctic has been under active development at Man AHL since 2012.

It wouldn't be possible without the work of the AHL Data Engineering Team including:

Contributions welcome!

License

Arctic is licensed under the GNU LGPL v2.1. A copy of which is included in LICENSE

Comments
  • Cannot compile on Windows

    Cannot compile on Windows

    I installed the recommended C++ Compiler for Python 2.7, however I can't seem to run the installer.

    The fatal error I got is: C1083: Cannot open include file: 'stdint.h': No such file or directory

    Is it possible to release a compiled version of this?

    help wanted 
    opened by beartastic 36
  • can't install on mac os x

    can't install on mac os x 

    clang -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c src/_compress.c -o build/temp.macosx-10.10-x86_64-2.7/src/_compress.o -fopenmp
src/_compress.c:259:10: fatal error: 'omp.h' file not found
#include "omp.h"
         ^
1 error generated.
error: command 'clang' failed with exit status 1

    i installed clang-omp..

    enhancement help wanted 
    opened by ccoossdddffdd 21
  • Usage discussion: VersionStore vs TickStore, allowed options for VersionStore.write..

    Usage discussion: VersionStore vs TickStore, allowed options for VersionStore.write..

    First of all - my thanks to the maintainers. This library is exactly what I was looking for and looks very promising.

    I've been having a bit of trouble figuring how to optimally use arctic though. I've been following the examples in /howto which are... sparse. Is there somewhere else I might find examples or docs?

    Now, some dumb questions about VersionStore and TickStore:

    • I've noticed that every time I write to a VersionStore, an entirely new version is created. Are finer-grained options for versioning available? For instance, I would like to write streaming updates to a single version, only incrementing version when manually specified. I tried just passing version=1 to lib.write, but this doesn't seem to be supported.
    • In what scenarios might one want to use VersionStore vs TickStore? It's not clear to me what the differences are from the README or the code.
    • My current use case is primarily as a database for streams - for this use case TickStore is recommended? Is there a reason one might want to use VersionStore for this?
    • ~~Is TickStore appropriate for data which may have more than row for each timestamp (event data)?~~ Nope, not allowed by TickStore

    Thanks in advance for your help and patience!

    opened by rueberger 19
  • how_to_use_arctic.py fails with SyntaxError: unexpected EOF while parsing on library.read()

    how_to_use_arctic.py fails with SyntaxError: unexpected EOF while parsing on library.read()

    Attempting to run the demo as a test, and getting errors on library.read(). Also, I get an error the first time I run store.initialize_library('username.scratch'), but the second time it did works. I did have to make the following change to get import Arctic to run:

    store/_version_store_utils.py, line 66: if pd.version.startswith("0.14"):

    See paste of output included the EOF error below. I'm not sure if it's a numpy bug or arctic, but I'm not sure where to go from here.

    Python 2.7.10 |Anaconda 2.3.0 (64-bit)| (default, Oct 19 2015, 18:04:42) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. Anaconda is brought to you by Continuum Analytics. Please check out: http://continuum.io/thanks and https://anaconda.org

    from arctic import Arctic from datetime import datetime as dt import pandas as pd store = Arctic('localhost') store.list_libraries() [u'NASDAQ', u'username.scratch'] store.initialize_library('username.scratch') No handlers could be found for logger "arctic.store.version_store" store.initialize_library('username.scratch') library = store['username.scratch'] df = pd.DataFrame({'prices': [1, 2, 3]},[dt(2014, 1, 1), dt(2014, 1, 2), dt(2014, 1, 3)]) library.write('SYMBOL', df) VersionedItem(symbol=SYMBOL,library=arctic_username.scratch,data=<type 'NoneType'>,version=19,metadata=None library.read('SYMBOL') Traceback (most recent call last): File "", line 1, in File "/home/jeff/anaconda/lib/python2.7/site-packages/arctic/store/version_store.py", line 319, in read date_range=date_range, read_preference=read_preference, *_kwargs) File "/home/jeff/anaconda/lib/python2.7/site-packages/arctic/store/version_store.py", line 363, in _do_read data = handler.read(self._arctic_lib, version, symbol, from_version=from_version, *_kwargs) File "/home/jeff/anaconda/lib/python2.7/site-packages/arctic/store/_pandas_ndarray_store.py", line 279, in read item = super(PandasDataFrameStore, self).read(arctic_lib, version, symbol, *_kwargs) File "/home/jeff/anaconda/lib/python2.7/site-packages/arctic/store/_pandas_ndarray_store.py", line 193, in read date_range=date_range, *_kwargs) File "/home/jeff/anaconda/lib/python2.7/site-packages/arctic/store/_ndarray_store.py", line 180, in read return self._do_read(collection, version, symbol, index_range=index_range) File "/home/jeff/anaconda/lib/python2.7/site-packages/arctic/store/_ndarray_store.py", line 219, in _do_read dtype = self._dtype(version['dtype'], version.get('dtype_metadata', {})) File "/home/jeff/anaconda/lib/python2.7/site-packages/arctic/store/_ndarray_store.py", line 139, in _dtype return np.dtype(string, metadata=metadata) File "/home/jeff/anaconda/lib/python2.7/site-packages/numpy/core/_internal.py", line 191, in _commastring newitem = (dtype, eval(repeats)) File "", line 1 ( ^ SyntaxError: unexpected EOF while parsing

    opened by jeffneuen 19
  • ServerSelectionTimeoutError: even when can connect to mongo via CLI or Compass

    ServerSelectionTimeoutError: even when can connect to mongo via CLI or Compass

    Arctic Version

    # arctic==1.66.0

    Arctic Store

    # TICK_STORE

    Platform and version

    Ubuntu 18.04 LTS (mongodb)

    Description of problem and/or code sample that reproduces the issue

    I am trying to access a mongodb instance on the network, I have been able to connect via MongoDB CLI and Compass via the ip:port call below. But somehow when I run it I get a server timeout?

    from arctic import Arctic

# also tried full mongo uri 'mongodb://197.168.0.210:27200/'
store = Arctic('197.168.0.210:27200')

store.list_libraries()


ServerSelectionTimeoutError: 197.168.0.210:27200: timed out

    I checked the pymongo call to connect, and attempted to use the full mongo URI but it still fails any call to the database.

    opened by derekwong9 17
  • Problem with TickStore:

    Problem with TickStore: "arctic.date._mktz.TimezoneError: Timezone "UTC" can not be read"

    I'm with an issue trying to add data to a TickStore Library.

    The data is a DataFrame and it's index is a DatetimeIndex. (Pdb) data_new.index DatetimeIndex(['2014-11-21 16:56:58.534000-02:00', '2014-11-21 17:49:56.935000-02:00', '2014-11-21 18:00:01.099000-02:00', '2014-11-21 18:06:00.012000-02:00'], dtype='datetime64[ns, America/Sao_Paulo]', freq=None)

    When i try to add it to the library, i got this: (Pdb) library.write(i, data_new) arctic.date.mktz.TimezoneError: Timezone "UTC" can not be read, error: "[Errno 2] No such file or directory: '/usr/share/zoneinfo/UTC'"

    How can i deal with it?

    opened by rtadewald 17
  • Future Development? - Java API

    Future Development? - Java API

    Hi there,

    I have been trialling this out and it looks a great framework for storage retrieval, is there any plans for the Java API or are you looking for contributors to help? I'm testing this out for data storage for zipline/quantopian backtester and also for a JVM based project and was wondering what stage that was at if any?

    opened by michaeljohnbennett 17
  • arctic import error (gcc/cpython)

    arctic import error (gcc/cpython)

    Arctic Version

    1.55.0

    Arctic Store

    ChunkStore

    Platform and version

    macOS Sierra 10.12.6 anaconda python3.6

    Description of problem and/or code sample that reproduces the issue

    I am having an error when importing arctic that I believe relates to my current gcc or cython build. I think I need to install gcc --without-multilib, but I am not entirely sure. I also think it is possible that the gcc version python is using is outdated.

    Here is the full code output and error statement:

    Suryas-MacBook-Pro:~ surya$ python Python 3.6.3 |Anaconda custom (64-bit)| (default, Oct 6 2017, 12:04:38) [GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin Type "help", "copyright", "credits" or "license" for more information.

    import arctic Traceback (most recent call last): File "", line 1, in File "/Users/surya/anaconda3/lib/python3.6/site-packages/arctic/init.py", line 3, in from .arctic import Arctic, register_library_type File "/Users/surya/anaconda3/lib/python3.6/site-packages/arctic/arctic.py", line 12, in from .store import version_store, bson_store, metadata_store File "/Users/surya/anaconda3/lib/python3.6/site-packages/arctic/store/version_store.py", line 15, in from ._pickle_store import PickleStore File "/Users/surya/anaconda3/lib/python3.6/site-packages/arctic/store/_pickle_store.py", line 7, in from .._compression import decompress, compress_array File "/Users/surya/anaconda3/lib/python3.6/site-packages/arctic/_compression.py", line 3, in from . import _compress as clz4 ImportError: dlopen(/Users/surya/anaconda3/lib/python3.6/site-packages/arctic/_compress.cpython-36m-darwin.so, 2): Symbol not found: _GOMP_parallel Referenced from: /Users/surya/anaconda3/lib/python3.6/site-packages/arctic/_compress.cpython-36m-darwin.so Expected in: flat namespace in /Users/surya/anaconda3/lib/python3.6/site-packages/arctic/_compress.cpython-36m-darwin.so

    Here is my output for brew list --versions: coreutils 8.28_1 gcc 7.2.0 gmp 6.1.2_1 isl 0.18 libmpc 1.0.3_1 mongodb 3.4.10 mpfr 3.1.6 openssl 1.0.2l pkg-config 0.29.2

    opened by suryabahubalendruni 15
  • Randomly raise Exception: Error decompressing if append many times to a symbol in chunk store

    Randomly raise Exception: Error decompressing if append many times to a symbol in chunk store

    Arctic Version

    arctic (1.51.0)

    Arctic Store

    ChunkStore

    Platform and version

    Red Hat Enterprise Linux Server release 7.2 (Maipo)

    Description of problem and/or code sample that reproduces the issue

    I append daily data to one symbol (write if not exists and set chunk_size = 'A').

    The data looks like this:

    • columns of the DataFrame
    Index(['beta', 'btop', 'earnyild', 'growth', 'industry', 'leverage',
       'liquidty', 'momentum', 'resvol', 'sid', 'size', 'sizenl'],
      dtype='object')
    • head of the DataFrame (part)
                 beta   btop  earnyild  growth  industry  leverage  liquidty 
date                                                                       
2008-12-25  0.200 -0.386    -0.669  -0.432        23    -0.307     0.746   
2008-12-25  0.653  0.048     0.671   0.182        10     0.255     1.097   
2008-12-25 -1.726 -1.105    -1.042  -2.661        22    -0.732    -3.400   
2008-12-25 -0.407  2.840     2.588  -1.505        19    -0.454    -1.137   
2008-12-25  0.931  1.302    -0.946  -0.306        31     3.042    -0.429
    • the dtypes
    beta        float64
btop        float64
earnyild    float64
growth      float64
industry      int64
leverage    float64
liquidty    float64
momentum    float64
resvol      float64
sid           int64
size        float64
sizenl      float64

    it will randomly raise the following exception (2008-12-25 for example)

    [2017-08-29 14:17:00] [factor.value] [INFO] update 2008-12-25 barra exposures failed：Error decompressing
[2017-08-29 14:17:00] [factor.value] [ERROR] Traceback (most recent call last):
  File "/home/quant/newalpha/warden/warden/_update_factors.py", line 88, in _update_barra_exposures
    n = update_lib(lib_factors, 'barra_exposures', exposures)
  File "/home/quant/newalpha/warden/warden/utils.py", line 70, in update_lib
    lib.append(symbol, data_to_append, metadata=meta)
  File "/opt/anaconda3/lib/python3.5/site-packages/arctic/chunkstore/chunkstore.py", line 503, in append
    self.__update(sym, item, metadata=metadata, combine_method=SER_MAP[sym[SERIALIZER]].combine, audit=audit)
  File "/opt/anaconda3/lib/python3.5/site-packages/arctic/chunkstore/chunkstore.py", line 415, in __update
    df = self.read(symbol, chunk_range=chunker.to_range(start, end), filter_data=False)
  File "/opt/anaconda3/lib/python3.5/site-packages/arctic/chunkstore/chunkstore.py", line 268, in read
    data = SER_MAP[sym[SERIALIZER]].deserialize(chunks, **kwargs)
  File "/opt/anaconda3/lib/python3.5/site-packages/arctic/serialization/numpy_arrays.py", line 195, in deserialize
    df = pd.concat([self.converter.objify(d, columns) for d in data], ignore_index=not index)
  File "/opt/anaconda3/lib/python3.5/site-packages/arctic/serialization/numpy_arrays.py", line 195, in <listcomp>
    df = pd.concat([self.converter.objify(d, columns) for d in data], ignore_index=not index)
  File "/opt/anaconda3/lib/python3.5/site-packages/arctic/serialization/numpy_arrays.py", line 126, in objify
    d = decompress(doc[DATA][doc[METADATA][LENGTHS][col][0]: doc[METADATA][LENGTHS][col][1] + 1])
  File "/opt/anaconda3/lib/python3.5/site-packages/arctic/_compression.py", line 55, in decompress
    return clz4.decompress(_str)
  File "_compress.pyx", line 121, in _compress.decompress (src/_compress.c:2151)
Exception: Error decompressing

    It seems that the data was broken and can not be decompressed (for any date range). If I delete the document related to 2008, it can be decompressed again.

    Thx a lot!

    opened by lf-shaw 14
  • With lib_type='TickStoreV3': No field of name index - index.name and index.tzinfo not preserved - max_date returning min date (without timezone)

    With lib_type='TickStoreV3': No field of name index - index.name and index.tzinfo not preserved - max_date returning min date (without timezone)

    Hello,

    this code

    from pandas_datareader import data as pdr
symbol = "IBM"
df = pdr.DataReader(symbol, "yahoo", "2010-01-01", "2015-12-29")
df.index = df.index.tz_localize('UTC')

from arctic import Arctic
store = Arctic('localhost')
store.initialize_library('library_name', 'TickStoreV3')
library = store['library_name']
library.write(symbol, df)

    raises

    ValueError: no field of name index

    I'm using TickStoreV3 as lib_type because I'm not very interested (at least for now) by audited write, versioning...

    I noticed that

    >>> df['index']=0
>>> library.write(symbol, df)
1 buckets in 0.015091: approx 6626466 ticks/sec

    seems to fix this... but

    >>> library.read(symbol)
                           index        High   Adj Close     ...             Low       Close        Open
1970-01-01 01:00:00+01:00      0  132.970001  116.564610     ...      130.850006  132.449997  131.179993
1970-01-01 01:00:00+01:00      0  131.850006  115.156514     ...      130.100006  130.850006  131.679993
1970-01-01 01:00:00+01:00      0  131.490005  114.408453     ...      129.809998  130.000000  130.679993
1970-01-01 01:00:00+01:00      0  130.250000  114.012427     ...      128.910004  129.550003  129.869995
1970-01-01 01:00:00+01:00      0  130.919998  115.156514     ...      129.050003  130.850006  129.070007
...                          ...         ...         ...     ...             ...         ...         ...
1970-01-01 01:00:00+01:00      0  135.830002  135.500000     ...      134.020004  135.500000  135.830002
1970-01-01 01:00:00+01:00      0  138.190002  137.929993     ...      135.649994  137.929993  135.880005
1970-01-01 01:00:00+01:00      0  139.309998  138.539993     ...      138.110001  138.539993  138.300003
1970-01-01 01:00:00+01:00      0  138.880005  138.250000     ...      138.110001  138.250000  138.429993
1970-01-01 01:00:00+01:00      0  138.039993  137.610001     ...      136.539993  137.610001  137.740005

[1507 rows x 7 columns]

    It looks like as if write was looking for a DataFrame with a column named 'index'... which is quite odd.

    If I do

    df['index']=1
library.write(symbol, df)

    then

    library.write(symbol, df)

    raises

    OverflowError: Python int too large to convert to C long

    Any idea ?

    opened by femtotrader 13
  • MemoryError when saving a dataframe with large strings to TickStore

    MemoryError when saving a dataframe with large strings to TickStore

    Arctic Version

    1.79.2

    Arctic Store

    TickStore

    Platform and version

    Python 3.6.7, Linux Mint 19 Cinnamon 64-bit

    Description of problem and/or code sample that reproduces the issue

    Hi, I'm trying to save the following data: https://drive.google.com/file/d/1dWWBNvx6vjyNK4kjZTVL4-YM0fmWxT5b/view?usp=sharing

    to TickStore, code: https://pastebin.com/jEqXxq2t

    and getting a MemoryError, see the stack traces: https://pastebin.com/Uy4pYAfH

    I'm quite new to arctic so I might be doing something wrong, and I would appreciate if you could guide me with this.

    Side question: Considering the nature of my data (2 col made of a time stamp and long string/json file), what is the best way to store these using arctic?

    Thanks, Alan

    opened by alanbogossian 12
  • Missing last chunk in CHUNK_STORE

    Missing last chunk in CHUNK_STORE

    Arctic Version

    1.80.5

    Arctic Store

    # ChunkStore

    Platform and version

    Python 3.8.5

    Description of problem and/or code sample that reproduces the issue

    I noticed that if I save a dataframe where the UTC date carries over to the next day, most functions (reverse_iterator, get_chunk_ranges, get_info, ...) don't return the chunk for the new date. The following example will make this clear (jupyter notebook attached in the zip file):

    Set Up

    import pandas as pd
from arctic import Arctic, CHUNK_STORE
store = Arctic("localhost")
store.initialize_library("scratch_lib", lib_type=CHUNK_STORE)

lib = store["scratch_lib"]

    Create an Index with some times that will change dates when converted to UTC

    ind = pd.Index([pd.Timestamp("20121208T16:00", tz="US/Eastern"), pd.Timestamp("20121208T18:00", tz="US/Eastern"), 
                pd.Timestamp("20121208T20:00", tz="US/Eastern"), pd.Timestamp("20121208T22:00", tz="US/Eastern")], name="date")
print(ind)

    Output:

    DatetimeIndex(['2012-12-08 16:00:00-05:00', '2012-12-08 18:00:00-05:00', '2012-12-08 20:00:00-05:00', '2012-12-08 22:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', name='date', freq=None)

    print(ind.tz_convert("UTC"))

    Output

    DatetimeIndex(['2012-12-08 21:00:00+00:00', '2012-12-08 23:00:00+00:00', '2012-12-09 01:00:00+00:00', '2012-12-09 03:00:00+00:00'], dtype='datetime64[ns, UTC]', name='date', freq=None)

    Create dataframe, write it to the library, and read it back out

    df = pd.DataFrame([1, 2, 3, 4], index=ind, columns=["col"])
lib.write("example_df", df, chunk_size="D")
df_read = lib.read("example_df")
print(df_read)

    Output

    date col 2012-12-08 21:00:00 1 2012-12-08 23:00:00 2 2012-12-09 01:00:00 3 2012-12-09 03:00:00 4

    This is different from what I expected. Is this behavior expected?

    lib.get_info("example_df")

    Output

    {'chunk_count': 1, 'len': 4, 'appended_rows': 0, 'metadata': {'columns': ['date', 'col']}, 'chunker': 'date', 'chunk_size': 'D', 'serializer': 'FrameToArray'}

    >> expected chunk_count = 2, not 1

    list(lib.get_chunk_ranges("example_df"))

    Output

    [(b'2012-12-08 00:00:00', b'2012-12-08 23:59:59.999000')]

    >> expected [(b'2012-12-08 00:00:00', b'2012-12-08 23:59:59.999000'), (b'2012-12-09 00:00:00', b'2012-12-09 23:59:59.999000')]

    iterator = lib.reverse_iterator("example_df")
while True:
    data = next(iterator, None)
    if data is None:
        break
    print(data)

    Output

    date col 2012-12-08 21:00:00 1 2012-12-08 23:00:00 2

    **>> expected the following: date col 2012-12-09 01:00:00 3 2012-12-09 03:00:00 4

    date col 2012-12-08 21:00:00 1 2012-12-08 23:00:00 2**

    arctic_issue_example.zip

    opened by atamkapoor 1
  • best practice usage

    best practice usage

    Hello, thank you very much for making this open source

    1/ is there a optimised way to access timeseries of revisions. In the VersionStore if we have saved several versions

    version1 saved at 2022-01-04 2022-01-01 1 2022-01-02 2

    version2 saved at 2022-01-05 2022-01-01 1 2022-01-02 3

    then I would like to retrieve in an efficient manner the timeseries of change for the value as of 2022-01-02, ie: 2022-01-04 2 2022-01-05 3

    2/ is there a permission layer allowing to choose who has access to which ticker?

    opened by RockScience 0
  • Index Monotonic Sort Bug in class DateChunker

    Index Monotonic Sort Bug in class DateChunker

    Index Monotonic Sort Bug in class DateChunker (in file date_chunker.py)

    If the df's index is not monotonic increasing, arctic will sort the df by index. BUT the variable dates is still not in order.

    I suggest arctic to put the code dates = df.index.get_level_values('date') after the if sentence.

    def to_chunks(self, df, chunk_size='D', func=None, **kwargs):
    """
    chunks the dataframe/series by dates

    Parameters
    ----------
    df: pandas dataframe or series
    chunk_size: str
        any valid Pandas frequency string
    func: function
        func will be applied to each `chunk` generated by the chunker.
        This function CANNOT modify the date column of the dataframe!

    Returns
    -------
    generator that produces tuples: (start date, end date,
              chunk_size, dataframe/series)
    """
    if 'date' in df.index.names:
        dates = df.index.get_level_values('date')
        if not df.index.is_monotonic_increasing:
            df = df.sort_index()
        # TODO dates won't be sorted, which will cause data store error.
        
      # dates = df.index.get_level_values('date')

    Anyway, arctic is an excellent project !

    这是我第一次在github上留言。蹩脚的英文。

    opened by qcyfred 0
  • MongoDB 4.2 EOL April 2023 - What's Next?

    MongoDB 4.2 EOL April 2023 - What's Next?

    I recently received a email from MongoDB reminding that my 4.2 cluster will be reaching EOL in April 2023. I am sure I am not the only one who received this...

    @jamesblackburn -What's the plan here? Will we need to request an extension from Mongo to continue running 4.2? Is there any roadmap to when if at all you will support 4.4 or 5.0? Per #938 a 4.4 or 5.0 version doesn't seem to be close?

    Will we need to plan to move to the S3 version? What's going on here?

    Some clarity here would help... as the date is approaching real fast.

    Btw - Thanks for the v1.80.5

    opened by luongjames8 6
  • mongodump and mongorestore library - Blob (not pure dataframe)

    mongodump and mongorestore library - Blob (not pure dataframe)

    Arctic Version

    # 1.80.0

    Arctic Store

    # VersionStore

    Platform and version

    Spyder (Python 3.8)

    Description of problem and/or code sample that reproduces the issue

    Hi, I use mongodump and mongorestore to move libraries in between PCs (let me know if there are easier ways). So for each library (in this case, my library is called "attribution_europe_data"), it has 5 collections (from MongoDB's point of view), which are attribution_europe_data / ....ARCTIC / ....snapshots /...version_nums/...versions, and during the mongodump process, it dumps 2 files for each collection, so a total of 10 files for each library.

    I successfully manage to mongorestore those 10 files into a seperate PC. ie I can do things like below (ie I can do things like print(Arctic('localhost')['attribution_data_europe'].list_symbols())

    image

    Now, each symbol in my library represents a pandas dataframe (actually they are saved as Blob, since they contain Objects), its around 5000rows x 2000 columns. The issue is if i read it in the new PC, eg "Arctic('localhost')['attribution_europe_data'].read('20220913').data" in Spyder, it will freeze, and eventually "restarting kernel...."

    image

    It shouldn't be a memory issue reading that dataframe, as I generated a similar size dataframe randomly in the same PC and it is ok.

    As a test, I use the same mongodump and mongorestore method on a smaller / simpler library, of which the library consists of a very simple symbol of a dictionary of {'hi':1}. And the new PC (where I restore it) is is able to read this library and this symbol without any issue. Similarly I use the same method on pure dataframe, as opposed to Blob, it works as well!

    So do you think during the mongodump and mongorestore process, it corrupts Blob object?

    Also what you guys normally use to transfer arctic libraries from one PC to another? surely there is a simplier way than mongodump and mongorestore?

    ============== Just to update on more investigations:

    1. if the symbol is a dataframe (that is NOT saved as a blob), it works
    2. if the symbol is a dict say {'hi':1}, it works
    3. if the symbol is a blob, it DOES NOT work (ie it will have trouble reading that symbol from the restored library in the new PC)
    4. if the symbol is a dict wrapped around a pure dataframe, eg {'hi' : pd.DataFrame(np.random.rand(2,2))}, then it works
    5. if the symbol is a dict wrapper around a blob, DOES NOT WORK, eg {'hi': some_blob}.

    I have included what it looks like in the old PC, and what error it throws up in the new PC if the symbol is a dict wraps around a blob

    (old PC) image

    (new PC) image

    opened by fengster123 0
Releases(v1.80.5)
Owner
Man Group
Man Group
GitHub https://arctic.readthedocs.io/en/latest/
Universal 1d/2d data containers with Transformers functionality for data analysis.

XPandas (extended Pandas) implements 1D and 2D data containers for storing type-heterogeneous tabular data of any type, and encapsulates feature extra

The Alan Turing Institute 25 Mar 14, 2022
NumPy and Pandas interface to Big Data

Blaze translates a subset of modified NumPy and Pandas-like syntax to databases and other computing systems. Blaze allows Python users a familiar inte

Blaze 3.1k Jan 1, 2023
Out-of-Core DataFrames for Python, ML, visualize and explore big tabular data at a billion rows per second 🚀

What is Vaex? Vaex is a high performance Python library for lazy Out-of-Core DataFrames (similar to Pandas), to visualize and explore big tabular data

vaex io 7.7k Jan 1, 2023
A Python package for manipulating 2-dimensional tabular data structures

datatable This is a Python package for manipulating 2-dimensional tabular data structures (aka data frames). It is close in spirit to pandas or SFrame

H2O.ai 1.6k Jan 5, 2023
A pure Python implementation of Apache Spark's RDD and DStream interfaces.

pysparkling Pysparkling provides a faster, more responsive way to develop programs for PySpark. It enables code intended for Spark applications to exe

Sven Kreiss 254 Dec 6, 2022
Providing the solutions for high-frequency trading (HFT) strategies using data science approaches (Machine Learning) on Full Orderbook Tick Data.

Modeling High-Frequency Limit Order Book Dynamics Using Machine Learning Framework to capture the dynamics of high-frequency limit order books. Overvi

Chang-Shu Chung 1.3k Jan 7, 2023
A Django-style ORM idea for manipulating Google Datastore entities

No SeiQueLa ORM EM DESENVOLVIMENTO Uma ideia de ORM no estilo do Django para manipular entidades do Google Datastore. Montando seu modelo: from noseiq

Geraldo Castro 16 Nov 1, 2022
Python ts2vg package provides high-performance algorithm implementations to build visibility graphs from time series data.

ts2vg: Time series to visibility graphs The Python ts2vg package provides high-performance algorithm implementations to build visibility graphs from t

Carlos Bergillos 26 Dec 17, 2022
Analyse the limit order book in seconds. Zoom to tick level or get yourself an overview of the trading day.

Analyse the limit order book in seconds. Zoom to tick level or get yourself an overview of the trading day. Correlate the market activity with the Apple Keynote presentations.

null 2 Jan 4, 2022
Kats is a toolkit to analyze time series data, a lightweight, easy-to-use, and generalizable framework to perform time series analysis.

Kats, a kit to analyze time series data, a lightweight, easy-to-use, generalizable, and extendable framework to perform time series analysis, from understanding the key statistics and characteristics, detecting change points and anomalies, to forecasting future trends.

Facebook Research 4.1k Dec 29, 2022
Time-series-deep-learning - Developing Deep learning LSTM, BiLSTM models, and NeuralProphet for multi-step time-series forecasting of stock price.

Stock Price Prediction Using Deep Learning Univariate Time Series Predicting stock price using historical data of a company using Neural networks for

Abdultawwab Safarji 7 Nov 27, 2022
Docker image with Uvicorn managed by Gunicorn for high-performance FastAPI web applications in Python 3.6 and above with performance auto-tuning. Optionally with Alpine Linux.

Supported tags and respective Dockerfile links python3.8, latest (Dockerfile) python3.7, (Dockerfile) python3.6 (Dockerfile) python3.8-slim (Dockerfil

Sebastián Ramírez 2.1k Dec 31, 2022
Tool for producing high quality forecasts for time series data that has multiple seasonality with linear or non-linear growth.

Prophet: Automatic Forecasting Procedure Prophet is a procedure for forecasting time series data based on an additive model where non-linear trends ar

Facebook 15.4k Jan 7, 2023
Scalene: a high-performance, high-precision CPU and memory profiler for Python

scalene: a high-performance CPU and memory profiler for Python by Emery Berger 中文版本 (Chinese version) About Scalene % pip install -U scalene Scalen

Emery Berger 138 Dec 30, 2022
Fcpy: A Python package for high performance, fast convergence and high precision numerical fractional calculus computing.

Fcpy: A Python package for high performance, fast convergence and high precision numerical fractional calculus computing.

SciFracX 1 Mar 23, 2022
Mimesis is a high-performance fake data generator for Python, which provides data for a variety of purposes in a variety of languages.

Mimesis - Fake Data Generator Description Mimesis is a high-performance fake data generator for Python, which provides data for a variety of purposes

Isaak Uchakaev 3.8k Dec 29, 2022
Mimesis is a high-performance fake data generator for Python, which provides data for a variety of purposes in a variety of languages.

Mimesis - Fake Data Generator Description Mimesis is a high-performance fake data generator for Python, which provides data for a variety of purposes

Isaak Uchakaev 3.8k Jan 1, 2023
CKAN is an open-source DMS (data management system) for powering data hubs and data portals. CKAN makes it easy to publish, share and use data. It powers catalog.data.gov, open.canada.ca/data, data.humdata.org among many other sites.

CKAN: The Open Source Data Portal Software CKAN is the world’s leading open-source data portal platform. CKAN makes it easy to publish, share and work

ckan 3.6k Dec 27, 2022
A data preprocessing package for time series data. Design for machine learning and deep learning.

A data preprocessing package for time series data. Design for machine learning and deep learning.

Allen Chiang 152 Jan 7, 2023
MetaBalance: High-Performance Neural Networks for Class-Imbalanced Data

This repository is the official PyTorch implementation of Meta-Balance. Find the paper on arxiv MetaBalance: High-Performance Neural Networks for Clas

Arpit Bansal 20 Oct 18, 2021