Dear Catalyst Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

• Operating System: Windows 10
• Python Version: Python 2.7.13 :: Anaconda 4.4.0 (64-bit)
• Python Bitness: Command fails
• How did you install Catalyst: Conda
• Python packages:

alembic                   0.9.6                     <pip>
backports.functools-lru-cache 1.4                       <pip>
bcolz                     0.12.1                    <pip>
boto3                     1.5.19                    <pip>
botocore                  1.8.33                    <pip>
Bottleneck                1.2.1                     <pip>
ca-certificates           2017.08.26           h94faf87_0
ccxt                      1.10.774                  <pip>
certifi                   2016.2.28                py27_0
certifi                   2018.1.18                 <pip>
chardet                   3.0.4                     <pip>
click                     6.7                       <pip>
contextlib2               0.5.5                     <pip>
cycler                    0.10.0                    <pip>
cyordereddict             1.0.0                     <pip>
Cython                    0.27.1                    <pip>
decorator                 4.1.2                     <pip>
docutils                  0.14                      <pip>
empyrical                 0.2.1                     <pip>
enigma-catalyst           0.4.7                     <pip>
futures                   3.2.0                     <pip>
idna                      2.6                       <pip>
intervaltree              2.1.0                     <pip>
jmespath                  0.9.3                     <pip>
Logbook                   1.1.0                     <pip>
lru-dict                  1.1.6                     <pip>
Mako                      1.0.7                     <pip>
MarkupSafe                1.0                       <pip>
matplotlib                2.1.0                     <pip>
mkl                       2017.0.3                      0
multipledispatch          0.4.9                     <pip>
networkx                  2.0                       <pip>
numexpr                   2.6.4                     <pip>
numpy                     1.13.1                   py27_0
openssl                   1.0.2l            vc9hb274bc6_5  [vc9]
pandas                    0.19.2                    <pip>
pandas-datareader         0.5.0                     <pip>
patsy                     0.4.1                     <pip>
pip                       9.0.1                    py27_1
pyparsing                 2.2.0                     <pip>
python                    2.7.13              h1b6d89f_16
python-dateutil           2.6.1                     <pip>
python-editor             1.0.3                     <pip>
pytz                      2017.2                    <pip>
redo                      1.6                       <pip>
requests                  2.18.4                    <pip>
requests-file             1.4.2                     <pip>
requests-ftp              0.3.1                     <pip>
s3transfer                0.1.12                    <pip>
scikit-learn              0.19.1                    <pip>
scipy                     0.19.1              np113py27_0
setuptools                36.4.0                   py27_1
six                       1.11.0                    <pip>
sortedcontainers          1.5.7                     <pip>
SQLAlchemy                1.1.14                    <pip>
sqlite                    3.13.0                    vc9_1  [vc9]
statsmodels               0.8.0                     <pip>
subprocess32              3.2.7                     <pip>
TA-Lib                    0.4.16                    <pip>
tables                    3.4.2                     <pip>
tk                        8.5.18                    vc9_0  [vc9]
toolz                     0.8.2                     <pip>
urllib3                   1.22                      <pip>
vc                        9                    h7299396_1
vs2008_runtime            9.00.30729.1         hfaea7d5_1
wheel                     0.29.0                   py27_0
wincertstore              0.2              py27hf04cefb_0
zlib                      1.2.11               hbc2faf4_2

Now that you know a little about me, let me tell you about the issue I am having:

Description of Issue

This does not occur every time, but when I place on order on Binance live, sometimes, it does not register the order as filled. The exchange blotter continues to report the order as open, even though it clearly shows it filled on Binance. The position amount also does not update. Here is what it looks like when a buy order works:

[2018-01-22 04:18:05.033000] INFO: mean_reversion_binance: Pricerise: True, Pricerisecount: 3, pos_amount: 0, Minute Count: 1, Price: 1069.41, Trade Happened: False
[2018-01-22 04:18:05.546000] INFO: mean_reversion_binance: 2018-01-22 04:18:00+00:00: buying - price: 1069.41, rsi: 61.8281815443, reason: None
[2018-01-22 04:18:06.072000] INFO: CCXT: adjusted order amount 0.0922413095149 to 0.09224 based on lot size
[2018-01-22 04:18:06.619000] INFO: exchange_algorithm: statistics for the last 1 minutes:
......
[2018-01-22 04:19:00.817000] INFO: exchange_blotter: filled order 18336995 / eth_usdt: 0.09224, avg price: 1072.0
[2018-01-22 04:19:01.807000] INFO: exchange_algorithm: portfolio balances, cash: 2.3896038, positions: 98.88128

And here is what it looks like when the exchange_blotter doesn't recognize the order as filled:

[2018-01-22 04:58:04.836000] INFO: mean_reversion_binance: Pricerise: True, Pricerisecount: 3, pos_amount: 0, Minute Count: 1, Price: 1058.99, Trade Happened: False
[2018-01-22 04:58:05.352000] INFO: mean_reversion_binance: 2018-01-22 04:58:00+00:00: buying - price: 1058.99, rsi: 56.6059504113, reason: None
[2018-01-22 04:58:05.835000] INFO: CCXT: adjusted order amount 0.0923728494131 to 0.09237 based on lot size
[2018-01-22 04:58:06.426000] INFO: exchange_algorithm: statistics for the last 1 minutes:
[2018-01-22 04:59:00.707000] INFO: exchange_blotter: order 18345240 still open after 0 days 00:00:54.450000
[2018-01-22 04:59:01.196000] INFO: exchange_algorithm: portfolio balances, cash: 0.0, positions: 0.0
[2018-01-22 04:59:05.682000] INFO: mean_reversion_binance: Pricerise: False, Pricerisecount: 1, pos_amount: 0, Minute Count: 1, Price: 1060.79, Trade Happened: True
[2018-01-22 04:59:05.762000] INFO: exchange_algorithm: statistics for the last 1 minutes:
....
[2018-01-22 05:00:01.010000] INFO: exchange_blotter: order 18345240 still open after 0 days 00:01:54.754000
[2018-01-22 05:00:01.514000] INFO: exchange_algorithm: portfolio balances, cash: 0.0, positions: 0.0
[2018-01-22 05:00:06.189000] INFO: mean_reversion_binance: Pricerise: False, Pricerisecount: 2, pos_amount: 0, Minute Count: 2, Price: 1061.75, Trade Happened: True
[2018-01-22 05:00:06.271000] INFO: exchange_algorithm: statistics for the last 1 minutes:
...
[2018-01-22 05:01:00.524000] INFO: exchange_blotter: order 18345240 still open after 0 days 00:02:54.268000
[2018-01-22 05:01:01.025000] INFO: exchange_algorithm: portfolio balances, cash: 0.0, positions: 0.0
[2018-01-22 05:01:05.522000] INFO: mean_reversion_binance: Pricerise: True, Pricerisecount: 1, pos_amount: 0, Minute Count: 3, Price: 1061.56, Trade Happened: True
[2018-01-22 05:01:05.596000] INFO: exchange_algorithm: statistics for the last 1 minutes:
...
[2018-01-22 05:02:00.858000] INFO: exchange_blotter: order 18345240 still open after 0 days 00:03:54.602000
[2018-01-22 05:02:01.342000] INFO: exchange_algorithm: portfolio balances, cash: 0.0, positions: 0.0

Here is what I see on Binance:

As you can see, the trade for $1058.99 went through successfully on the 58 minute mark. You can also see the trade on the 18 minute mark, but actually while I posted a buy at a price of $1069.41, it bought for $1072.

After reviewing that, perhaps it's related to my buy code?

orders = get_open_orders(context.market)

        if len(orders) > 0:
            log.info(
                '{} orders open, trying to cancel.'.format(
                    len(orders))
            )
            for market, position in context.portfolio.positions.iteritems():
                order_target_percent(market, 0)
        # pos_amount == 0 pricerise and
        elif pricerise and context.pricerisecount >= 3 and pos_amount == 0:
            log.info(
                '{}: buying - price: {}, rsi: {}, reason: {}'.format(
                    data.current_dt, price, getLast(indicators, 'rsi'), context.reason
                )
            )
            # Set a style for limit orders,

            order_target_percent(
                context.market, 1, limit_price=price * 1.005
            )
            context.traded_today = True
            context.tradehappened = True
        elif pricerise == False and context.pricerisecount >= 3 and pos_amount > 0:
            log.info(
                '{}: selling - price: {}, rsi: {}, reason: {}'.format(
                    data.current_dt, price, getLast(indicators, 'rsi'), context.reason
                )
            )
            #limit_price = price * 1
            order_target_percent(
                context.market, 0, limit_price=price * .995
            )
            context.traded_today = True
            context.tradehappened = True
        else:
            log.info(
                '{}: no action - price: {}, rsi: {}'.format(
                    data.current_dt, price, getLast(indicators, 'rsi')
                )
            )
            context.tradehappened = False

As you can see, I am doing limit orders for 1.005 * price for buys. The one that failed to be recognized as filled looks like it was filled immediately for the current price of the ETH, whereas the one that worked went up from the price of 1069.41 to be purchased at 1072. Perhaps it is this instant fulfillment of a limit order that's causing Catalyst to not recognize the order as being filled? If so, any recommendations? I am trading on a minute by minute basis, so I don't want to set the limit too high and have it not be filled at all, I'm trying to just purchase for basically current price.

Lastly, you can see in my order logic that I'm trying to check for any open orders and sell them off as an effort to clear out what Catalyst thinks are unfilled ones. However that line of code is never hit, so it seems like these orders which never increase my position and are considered unfilled by the exchange_blotter are in fact being removed from my "open orders" list.

Here is how you can reproduce this issue on your machine:

Reproduction Steps

1. Hopefully the steps above are sufficient.

...

What steps have you taken to resolve this already?

Hopefully above is enough detail. ...

Anything else?

...

Sincerely, Jarlek

Dear Catalyst Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

• Operating System: ubuntu 16.04
• Python Version: 2.7
• Python Bitness: 64
• How did you install Catalyst: pip
• Python packages: alembic==0.9.6 bcolz==0.12.1 boto3==1.5.1 botocore==1.8.15 Bottleneck==1.2.1 ccxt==1.10.400 certifi==2017.11.5 chardet==3.0.4 click==6.7 contextlib2==0.5.5 cyordereddict==1.0.0 Cython==0.27.3 decorator==4.1.2 docutils==0.14 empyrical==0.2.1 enigma-catalyst==0.4.0 futures==3.2.0 idna==2.6 int-date==0.1.8 intervaltree==2.1.0 jmespath==0.9.3 Logbook==1.1.0 lru-dict==1.1.6 Mako==1.0.7 MarkupSafe==1.0 multipledispatch==0.4.9 networkx==2.0 numexpr==2.6.4 numpy==1.13.3 pandas==0.19.2 pandas-datareader==0.5.0 patsy==0.4.1 pkg-resources==0.0.0 python-dateutil==2.6.1 python-editor==1.0.3 pytz==2017.3 requests==2.18.4 requests-file==1.4.2 requests-ftp==0.3.1 s3transfer==0.1.12 scipy==1.0.0 six==1.11.0 sortedcontainers==1.5.9 SQLAlchemy==1.1.15 statsmodels==0.8.0 stockstats==0.2.0 tables==3.4.2 toolz==0.9.0 urllib3==1.22

Now that you know a little about me, let me tell you about the issue I am having:

Description of Issue

I am running a live bot on binance and evertime the bot buys something it will crash

• What did you expect to happen? The bot would continue after buying

• What happened instead?

for capital_change_packet in every_bar(dt): File "/home/catalyst/catalyst-venv/local/lib/python2.7/site-packages/catalyst/gens/tradesimulation.py", line 137, in every_bar handle_data(algo, current_data, dt_to_use) File "/home/catalyst/catalyst-venv/local/lib/python2.7/site-packages/catalyst/utils/events.py", line 216, in handle_data dt, File "/home/catalyst/catalyst-venv/local/lib/python2.7/site-packages/catalyst/utils/events.py", line 235, in handle_data self.callback(context, data) File "/home/catalyst/catalyst-venv/local/lib/python2.7/site-packages/catalyst/exchange/exchange_algorithm.py", line 641, in handle_data cash, positions_value = self.synchronize_portfolio() File "/home/catalyst/catalyst-venv/local/lib/python2.7/site-packages/catalyst/exchange/exchange_algorithm.py", line 523, in synchronize_portfolio raise ValueError('Cash on exchanges is lower than the algo.') ValueError: Cash on exchanges is lower than the algo.

Here is how you can reproduce this issue on your machine:

Reproduction Steps

1. Run the mean_reversion_simple on live mode
2. The bot will find something to buy
3. It will crash

What steps have you taken to resolve this already?

Couldnt think of anything to solvce this

Sincerely, Vincent

I am working on a nightly 'refresh' process to update data, re-train models, re-run backtests, etc. But in doing this I realized that data for the last 15 days or so is not available. I checked the status page and it appears data is only available until the 15th of Jan.

Is there any information anywhere on how this will work? Has data on the catalyst servers ceased being updated? Is it a periodic refresh? If so how often can we expect data to be updated? Is this the move towards the enigma platform, if so how do we access that data source?

Any info appreciated (sorry for the million questions, was just a bit of a surprise).

I am very confused about the handling of the final candle from data.history during backtests. Let me try to explain with code that reproduces the issue:

import pytz
from datetime import datetime, timedelta
from catalyst.api import symbol
from catalyst.utils.run_algo import run_algorithm
from utils.data import load_coin_data_from_csv

exchange_name='bitfinex'
coin = 'btc'
base_currency='usd'
n_candles = 5
resample_frequency_minutes = 5

def initialize(context):        
    context.symbol = symbol('%s_%s' % (coin, base_currency))    
    # very simple method that reads data from generated csv file
    context.all_data = load_coin_data_from_csv('btc', exchange_name, 'minute', from_date=datetime(2017, 6, 1, 0, 0, 0, 0, pytz.utc), resample='%dT' % resample_frequency_minutes)

def handle_data(context, data):    
    bundle_data = context.all_data[data.current_dt - timedelta(minutes=(n_candles - 1)*resample_frequency_minutes):][:n_candles]['close']
    backtest_data = data.history(context.symbol, 'close', bar_count=n_candles, frequency='%dT' % resample_frequency_minutes)
    print('current_time:', data.current_dt)
    
    print('bundle_data:')
    print(bundle_data)
    
    print('backtest_data:')
    print(backtest_data)
    raise Exception('fail')

run_algorithm(
            initialize=initialize,
            handle_data=handle_data,
            exchange_name=exchange_name,
            base_currency=base_currency,
            algo_namespace='investigate_mismatching_final_candle',
            live=False,
            data_frequency='minute',
            capital_base=3000,
            simulate_orders=True,
            start = datetime(2018, 2, 2, 0, 0, 0, 0, pytz.utc),
            end = datetime(2018, 2, 20, 0, 0, 0, 0, pytz.utc))

Now for btc_usd on bitfinex I get the following 2 datasets:

current_time: 2018-02-02 00:00:00+00:00

bundle_data:
2018-02-01 23:40:00    9306.2
2018-02-01 23:45:00    9270.0
2018-02-01 23:50:00    9216.9
2018-02-01 23:55:00    9181.0
2018-02-02 00:00:00    9150.0 # not matching
Freq: 5T, Name: close, dtype: float64

backtest_data:
2018-02-01 23:35:00+00:00    9269.0
2018-02-01 23:40:00+00:00    9306.2
2018-02-01 23:45:00+00:00    9270.0
2018-02-01 23:50:00+00:00    9216.9
2018-02-01 23:55:00+00:00    9181.0
2018-02-02 00:00:00+00:00    9182.1# not matching
Freq: 5T, Name: TradingPair(98871 [btc_usd]), dtype: float64

For btc_usdt on poloniex I get:

current_time: 2018-02-02 00:00:00+00:00

bundle_data:
2018-02-01 23:40:00    9297.003598
2018-02-01 23:45:00    9290.418766
2018-02-01 23:50:00    9250.762957
2018-02-01 23:55:00    9199.570000
2018-02-02 00:00:00    9168.790860 # not matching
Freq: 5T, Name: close, dtype: float64

backtest_data:
2018-02-01 23:35:00+00:00    9279.622798
2018-02-01 23:40:00+00:00    9297.003598
2018-02-01 23:45:00+00:00    9290.418766
2018-02-01 23:50:00+00:00    9250.762957
2018-02-01 23:55:00+00:00    9199.570000
2018-02-02 00:00:00+00:00    9199.570000 # not matching
Freq: 5T, Name: TradingPair(374465 [btc_usdt]), dtype: float64

Firstly data.history returns 6 candles from a request for 5. This has already been reported I think.

But more importantly: You can see the final candle of both exchanges do not match. I would expect the final close price @ time T to be the same close price as the bundle data at time T.

@fredfortier explained a few issues relating to the final candle in bug #230 but I don't think that applies here. Can someone please explain to me what is happening? Is this a bug? Are you returning a previous close price assuming that time T is slightly in the future? Is this zipline slippage/black magic?

My ultimate objective is to pre-load all my model predictions at the beginning of my backtest for performance. But I cannot do that if the data from bundles does not match what is returned by data.history.

Cheers.

Dear Catalyst Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

• Operating System: Windows 10
• Python Version: Python 2.7.13 :: Anaconda, Inc.
• Python Bitness: 64
• How did you install Catalyst: conda initiall, pip re-install to update
• Python packages:

alembic                   0.9.6                     <pip>
attrdict                  2.0.0                     <pip>
backports.functools-lru-cache 1.4                       <pip>
bcolz                     0.12.1                    <pip>
boto3                     1.5.22                    <pip>
botocore                  1.8.36                    <pip>
Bottleneck                1.2.1                     <pip>
ca-certificates           2017.08.26           h94faf87_0
ccxt                      1.10.1063                 <pip>
certifi                   2016.2.28                py27_0
certifi                   2018.1.18                 <pip>
chardet                   3.0.4                     <pip>
click                     6.7                       <pip>
contextlib2               0.5.5                     <pip>
cycler                    0.10.0                    <pip>
cyordereddict             1.0.0                     <pip>
Cython                    0.27.1                    <pip>
cytoolz                   0.9.0                     <pip>
decorator                 4.1.2                     <pip>
docutils                  0.14                      <pip>
empyrical                 0.2.1                     <pip>
enigma-catalyst           0.5.2                     <pip>
eth-abi                   1.0.0b0                   <pip>
eth-account               0.1.0a2                   <pip>
eth-keyfile               0.5.0                     <pip>
eth-keys                  0.2.0b1                   <pip>
eth-rlp                   0.1.0a2                   <pip>
eth-utils                 1.0.0b1                   <pip>
futures                   3.2.0                     <pip>
git                       2.16.1                        0
hexbytes                  0.1.0b0                   <pip>
idna                      2.6                       <pip>
intervaltree              2.1.0                     <pip>
jmespath                  0.9.3                     <pip>
Logbook                   1.1.0                     <pip>
lru-dict                  1.1.6                     <pip>
Mako                      1.0.7                     <pip>
MarkupSafe                1.0                       <pip>
matplotlib                2.1.0                     <pip>
mkl                       2017.0.3                      0
multipledispatch          0.4.9                     <pip>
networkx                  2.0                       <pip>
numexpr                   2.6.4                     <pip>
numpy                     1.13.1                   py27_0
openssl                   1.0.2l            vc9hb274bc6_5  [vc9]
pandas                    0.19.2                    <pip>
pandas-datareader         0.5.0                     <pip>
patsy                     0.4.1                     <pip>
pip                       9.0.1                    py27_1
pycryptodome              3.4.11                    <pip>
pyparsing                 2.2.0                     <pip>
pysha3                    1.0.2                     <pip>
python                    2.7.13              h1b6d89f_16
python-dateutil           2.6.1                     <pip>
python-editor             1.0.3                     <pip>
pytz                      2017.2                    <pip>
redo                      1.6                       <pip>
requests                  2.18.4                    <pip>
requests-file             1.4.2                     <pip>
requests-ftp              0.3.1                     <pip>
requests-toolbelt         0.8.0                     <pip>
rlp                       0.6.0                     <pip>
s3transfer                0.1.12                    <pip>
scikit-learn              0.19.1                    <pip>
scipy                     0.19.1              np113py27_0
setuptools                36.4.0                   py27_1
six                       1.11.0                    <pip>
sortedcontainers          1.5.7                     <pip>
SQLAlchemy                1.1.14                    <pip>
sqlite                    3.13.0                    vc9_1  [vc9]
statsmodels               0.8.0                     <pip>
subprocess32              3.2.7                     <pip>
TA-Lib                    0.4.16                    <pip>
tables                    3.4.2                     <pip>
tk                        8.5.18                    vc9_0  [vc9]
toolz                     0.8.2                     <pip>
urllib3                   1.22                      <pip>
vc                        9                    h7299396_1
vs2008_runtime            9.00.30729.1         hfaea7d5_1
web3                      4.0.0b9                   <pip>
wheel                     0.29.0                   py27_0
wincertstore              0.2              py27hf04cefb_0
zlib                      1.2.11               hbc2faf4_2

Now that you know a little about me, let me tell you about the issue I am having:

Description of Issue

My catalyst algorithm has been performing very well in backtest, but horribly in live mode. I tried to isolate the issue by seeing if it was making the same trading decisions for the same time period in backtest vs live. I did this by live trading a period of time, saving the performance csv (custom_signals_stats.csv), then waiting until the backtest data was available for that period, and performing a backtest over that same day, using the exact same trading code, then saving that performance csv separately and comparing.

I've attached portions of the two csv files to compare.

EDIT: Apparently I can't attach CSVs, so here are links to my Google Drive to download them:

Backtest csv: https://goo.gl/1vq5gK Live csv: https://goo.gl/GJZsrq

I immediately noticed a few things. Price seemed to be staggered by one minute, but while it tracked generally between the two files, it was only about 50% matching. I also noticed my algorithm's trade logic (pricerise in the files) was wildly different. That led me to compare a few of the other numbers and realized nearly every calculation coming from TALib was significantly different. Most shockingly was the huge discrepancy in the trade volume. In the live trading, it was showing a consistent volume of around 13000, but in the backtest it was jumping around between 76000, 2000 and 45 within just a 10 minute time frame. the Obv indicator was also significantly different

My algorithm involves training a machine learning algorithm using a singificant amount of backtest gathered data and indicators, so this mismatch between backtest and live data being fed to my algorithm started to help make sense of why it performed so differently between the two settings.

Here are my settings for the main catalyst algorithm which I used to generate these two files:

        if live:
            run_algorithm(
                capital_base=100,
                initialize=initialize,
                handle_data=handle_data,
                analyze=analyze,
                exchange_name='poloniex',
                live=True,
                algo_namespace=NAMESPACE,
                base_currency='usdt',
                live_graph=False,
                simulate_orders=False,
                stats_output=None,
            )

        else:
            folder = os.path.join(
                tempfile.gettempdir(), 'catalyst', NAMESPACE
            )
            ensure_directory(folder)

            timestr = time.strftime('%Y%m%d-%H%M%S')
            out = os.path.join(folder, '{}.p'.format(timestr))
            # catalyst run -f catalyst/examples/mean_reversion_simple.py \
            #    -x bitfinex -s 2017-10-1 -e 2017-11-10 -c usdt -n mean-reversion \
            #   --data-frequency minute --capital-base 10000
            run_algorithm(
                capital_base=100,
                data_frequency='minute',
                initialize=initialize,
                handle_data=handle_data,
                analyze=analyze,
                exchange_name='poloniex',
                algo_namespace=NAMESPACE,
                base_currency='usdt',
                start=pd.to_datetime('2018-2-09', utc=True),
                end=pd.to_datetime('2018-2-09', utc=True),
                output=out
            )
            log.info('saved perf stats: {}'.format(out))

Here is how you can reproduce this issue on your machine:

Reproduction Steps

I'm hoping that if you were to execute a series of live actions against poloniex eth_usdt and a backtest using bundle data you should see something similar. ...

What steps have you taken to resolve this already?

Illustrated above. ...

Anything else?

...

Sincerely, Darthus

Dear Catalyst Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

• Operating System: (Linux <hostname removed> 4.13.11-1-ARCH #1 SMP PREEMPT Thu Nov 2 10:25:56 CET 2017 x86_64 GNU/Linux)
• Python Version: 2.7.14
• Python Bitness: 64
• How did you install Catalyst: pip
• Python packages: $ pip freeze:

agate==1.6.0
agate-dbf==0.2.0
agate-excel==0.2.1
agate-sql==0.5.2
alabaster==0.7.10
alembic==0.9.6
Babel==2.5.1
backports-abc==0.5
backports.functools-lru-cache==1.4
backports.shutil-get-terminal-size==1.0.0
bcolz==0.12.1
bleach==2.1.1
boto3==1.4.8
botocore==1.8.13
Bottleneck==1.2.1
ccxt==1.10.382
certifi==2017.11.5
chardet==3.0.4
click==6.7
configparser==3.5.0
contextlib2==0.5.5
csvkit==1.0.2
cycler==0.10.0
cyordereddict==1.0.0
Cython==0.27.3
dbfread==2.0.7
decorator==4.1.2
docutils==0.14
empyrical==0.2.1
enigma-catalyst==0.4.0+1.g94583c26
entrypoints==0.2.3
enum34==1.1.6
et-xmlfile==1.0.1
functools32==3.2.3.post2
future==0.16.0
futures==3.2.0
haikunator==2.1.0
html5lib==1.0b10
idna==2.6
imagesize==0.7.1
intervaltree==2.1.0
ipdb==0.10.3
ipykernel==4.6.1
ipython==5.5.0
ipython-genutils==0.2.0
ipywidgets==7.0.5
isodate==0.6.0
jdcal==1.3
Jinja2==2.10
jmespath==0.9.3
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.1.0
jupyter-console==5.2.0
jupyter-core==4.4.0
leather==0.3.3
Logbook==1.1.0
lru-dict==1.1.6
Mako==1.0.7
MarkupSafe==1.0
matplotlib==2.1.0
mistune==0.8.1
multipledispatch==0.4.9
names==0.3.0
nbconvert==5.3.1
nbformat==4.4.0
networkx==2.0
notebook==5.2.1
numexpr==2.6.4
numpy==1.13.3
openpyxl==2.4.9
pandas==0.19.2
pandas-datareader==0.5.0
pandocfilters==1.4.2
parsedatetime==2.4
pathlib2==2.3.0
patsy==0.4.1
petname==2.2
pexpect==4.3.0
pickleshare==0.7.4
prompt-toolkit==1.0.15
ptyprocess==0.5.2
pyfolio==0.8.0
Pygments==2.2.0
pyparsing==2.2.0
python-dateutil==2.6.1
python-editor==1.0.3
python-slugify==1.2.4
pytimeparse==1.1.7
pytz==2017.3
pyzmq==16.0.3
qtconsole==4.3.1
requests==2.18.4
requests-file==1.4.2
requests-ftp==0.3.1
s3transfer==0.1.12
scandir==1.6
scikit-learn==0.19.1
scipy==1.0.0
seaborn==0.8.1
simplegeneric==0.8.1
singledispatch==3.4.0.3
six==1.11.0
snowballstemmer==1.2.1
sortedcontainers==1.5.7
Sphinx==1.6.5
sphinxcontrib-websupport==1.0.1
SQLAlchemy==1.1.15
statsmodels==0.8.0
subprocess32==3.2.7
TA-Lib==0.4.10
tables==3.4.2
terminado==0.7
testpath==0.3.1
toolz==0.8.2
tornado==4.5.2
traitlets==4.3.2
transitions==0.6.2
typing==3.6.2
Unidecode==0.4.21
urllib3==1.22
wcwidth==0.1.7
webencodings==0.5.1
widgetsnbextension==3.0.8
xlrd==1.1.0

Now that you know a little about me, let me tell you about the issue I am having:

60T resampling vs 1h ccxt missmatch

When I try to run with hourly resampling using papermode it breaks. Same applies to 2h resample.

• What did you expect to happen?: I expect to be able to use various timeframes: 30min, 60min/1h, 120min/2h, etc as expressed [here]

        '1m': '1minute',
        '1h': '1hour',
        '1d': '1day',
        '1M': '1month',
        '1y': '1year',

• What happened instead?: it crashed with the following output:

[2017-12-22 15:35:52.882634] INFO: run_algo: running algo in paper-trading mode                                                                                                    
[2017-12-22 15:35:58.458045] INFO: exchange_bundle: pricing data for [u'btc_usdt'] not found in range 2015-03-01 00:00:00+00:00 to 2017-12-21 00:00:00+00:00, updating the bundles.
    [====================================]  Ingesting daily price data for btc_usdt on poloniex:  100%
[2017-12-22 15:36:00.181265] INFO: Loader: Cache at /home/av/.catalyst/data/treasury_curves.csv does not have data from 1990-01-02 00:00:00+00:00 to 2017-12-21 00:00:00+00:00.
[2017-12-22 15:36:00.181430] INFO: Loader: Downloading treasury data for 'btc_usdt'.
[2017-12-22 15:36:04.569014] WARNING: Loader: Still don't have expected data after redownload!
[2017-12-22 15:36:04.581458] INFO: exchange_algorithm: initialized trading algorithm in live mode
[2017-12-22 15:36:04.752854] INFO: abba_test:
STARTING SIMULATION.
Candles sample rate: 60
Candles frequency: 60T
Candles buffer size: 48

[2017-12-22 15:36:05.321559] INFO: exchange_algorithm: got totals from exchanges, cash: 868.027881268 positions: 91.7521953708
[2017-12-22 15:36:05.324173] WARNING: DataPortalExchange: get history attempt 0: Request failed: '60m'
[2017-12-22 15:36:10.327806] WARNING: DataPortalExchange: get history attempt 1: Request failed: '60m'
[2017-12-22 15:36:15.333984] WARNING: DataPortalExchange: get history attempt 2: Request failed: '60m'
[2017-12-22 15:36:20.335723] WARNING: DataPortalExchange: get history attempt 3: Request failed: '60m'
[2017-12-22 15:36:25.339672] WARNING: DataPortalExchange: get history attempt 4: Request failed: '60m'
[2017-12-22 15:36:30.343947] WARNING: DataPortalExchange: get history attempt 5: Request failed: '60m'
Traceback (most recent call last):
  File "d_abba_papermode.py", line 225, in <module>
    live_graph=False
  File "/home/av/repos/example/catalyst/catalyst/utils/run_algo.py", line 563, in run_algorithm
    stats_output=stats_output
  File "/home/av/repos/example/catalyst/catalyst/utils/run_algo.py", line 346, in _run
    overwrite_sim_params=False,
  File "/home/av/repos/example/catalyst/catalyst/algorithm.py", line 724, in run
    for perf in self.get_generator():
  File "/home/av/repos/example/catalyst/catalyst/gens/tradesimulation.py", line 224, in transform
    for capital_change_packet in every_bar(dt):
  File "/home/av/repos/example/catalyst/catalyst/gens/tradesimulation.py", line 137, in every_bar
    handle_data(algo, current_data, dt_to_use)
  File "/home/av/repos/example/catalyst/catalyst/utils/events.py", line 216, in handle_data
    dt,
  File "/home/av/repos/example/catalyst/catalyst/utils/events.py", line 235, in handle_data
    self.callback(context, data)
  File "/home/av/repos/example/catalyst/catalyst/exchange/exchange_algorithm.py", line 648, in handle_data
    self._handle_data(self, data)
  File "/home/av/repos/example/strategies/family/abba.py", line 154, in default_handle_data
    frequency=context.candles_frequency)
  File "catalyst/_protocol.pyx", line 120, in catalyst._protocol.check_parameters.__call__.assert_keywords_and_call
  File "catalyst/_protocol.pyx", line 646, in catalyst._protocol.BarData.history
  File "/home/av/repos/example/catalyst/catalyst/exchange/exchange_data_portal.py", line 114, in get_history_window
    ffill)
  File "/home/av/repos/example/catalyst/catalyst/exchange/exchange_data_portal.py", line 88, in _get_history_window
    attempt_index + 1)
  File "/home/av/repos/example/catalyst/catalyst/exchange/exchange_data_portal.py", line 88, in _get_history_window
    attempt_index + 1)
  File "/home/av/repos/example/catalyst/catalyst/exchange/exchange_data_portal.py", line 88, in _get_history_window
    attempt_index + 1)
  File "/home/av/repos/example/catalyst/catalyst/exchange/exchange_data_portal.py", line 88, in _get_history_window
    attempt_index + 1)
  File "/home/av/repos/example/catalyst/catalyst/exchange/exchange_data_portal.py", line 88, in _get_history_window
    attempt_index + 1)
  File "/home/av/repos/example/catalyst/catalyst/exchange/exchange_data_portal.py", line 93, in _get_history_window
    error=e
catalyst.exchange.exchange_errors.ExchangeBarDataError: Unable to retrieve bar data: history, giving up after 5 attempts: Request failed: '60m'

Here is how you can reproduce this issue on your machine:

Reproduction Steps

1. Try to run a strategy with 60T resample on paper trading mode. ...

What steps have you taken to resolve this already?

I have been looking directly into the code to find the mapping of the re-sampling string (i.e: 60T) to the ccxt frequency parameter (i.e: 1H, 1W, 1M), which has a different notation. Since my father got hospitalized recently I don't know if I will have the time to submit a PR. I will try if possible.

Sincerely, V.

Dear Catalyst Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

• Operating System: Ubunutu 16.04
• Python Version: python 2.7

Description of Issue

1. Ingest Data not being load all markets when selecting an early starting date like 2017-01-01. If I select 2017-09-01 all markets work as expected. I have manually filters pairs that are not in the current change of the backtest to ensure that data for that day in catalyst existed.

BTC Market works for both starting dates, but XMR, eth, usdt markets are not working for 2017-01-01 similar to the previous issues in v0.3.2

I have only tested in Poloniex but my script can be used with any exchange by changing context.exchange so feel free to use this script as a unit test.

Code

"""
Requires Catalyst version 0.3.0 or above
Tested on Catalyst version 0.3.2

These example aims to provide and easy way for users to learn how to collect data from the different exchanges.
You simply need to specify the exchange and the market that you want to focus on.
You will all see how to create a universe and filter it base on the exchange and the market you desire.

The example prints out the closing price of all the pairs for a given market-exchange every 30 minutes.
The example also contains the ohlcv minute data for the past seven days which could be used to create indicators
Use this as the backbone to create your own trading strategies.

Variables lookback date and date are used to ensure data for a coin existed on the lookback period specified.
"""

import numpy as np
import pandas as pd
from datetime import timedelta
from catalyst import run_algorithm
from catalyst.exchange.exchange_utils import get_exchange_symbols

from catalyst.api import (
    symbols,
)


def initialize(context):
    context.i = -1  # counts the minutes
    context.exchange = 'poloniex'  # must match the exchange specified in run_algorithm
    context.base_currency = 'eth'  # must match the base currency specified in run_algorithm


def handle_data(context, data):
    lookback = 60 * 24 * 7  # (minutes, hours, days) of how far to lookback in the data history
    context.i += 1

    # current date formatted into a string
    today = context.blotter.current_dt
    date, time = today.strftime('%Y-%m-%d %H:%M:%S').split(' ')
    lookback_date = today - timedelta(days=(lookback / (60 * 24)))  # subtract the amount of days specified in lookback
    lookback_date = lookback_date.strftime('%Y-%m-%d %H:%M:%S').split(' ')[0]  # get only the date as a string

    # update universe everyday
    new_day = 60 * 24
    if not context.i % new_day:
        context.universe = universe(context, lookback_date, date)

    # get data every 30 minutes
    minutes = 30
    if not context.i % minutes and context.universe:
        # we iterate for every pair in the current universe
        for coin in context.coins:
            pair = str(coin.symbol)

            # 30 minute interval ohlcv data (the standard data required for candlestick or indicators/signals)
            # 30T means 30 minutes re-sampling of one minute data. change to your desire time interval.
            open = fill(data.history(coin, 'open', bar_count=lookback, frequency='1m')).resample('30T').first()
            high = fill(data.history(coin, 'high', bar_count=lookback, frequency='1m')).resample('30T').max()
            low = fill(data.history(coin, 'low', bar_count=lookback, frequency='1m')).resample('30T').min()
            close = fill(data.history(coin, 'price', bar_count=lookback, frequency='1m')).resample('30T').last()
            volume = fill(data.history(coin, 'volume', bar_count=lookback, frequency='1m')).resample('30T').sum()

            # close[-1] is the equivalent to current price
            # displays the minute price for each pair every 30 minutes
            print(today, pair, open[-1], high[-1], low[-1], close[-1], volume[-1])

            # ----------------------------------------------------------------------------------------------------------
            # -------------------------------------- Insert Your Strategy Here -----------------------------------------
            # ----------------------------------------------------------------------------------------------------------


def analyze(context=None, results=None):
    pass


# Get the universe for a given exchange and a given base_currency market
# Example: Poloniex BTC Market
def universe(context, lookback_date, current_date):
    json_symbols = get_exchange_symbols(context.exchange)  # get all the pairs for the exchange
    universe_df = pd.DataFrame.from_dict(json_symbols).transpose().astype(str)  # convert into a dataframe
    universe_df['base_currency'] = universe_df.apply(lambda row: row.symbol.split('_')[1],
                                                                       axis=1)
    universe_df['market_currency'] = universe_df.apply(lambda row: row.symbol.split('_')[0],
                                                                         axis=1)
    # Filter all the exchange pairs to only the ones for a give base currency
    universe_df = universe_df[universe_df['base_currency'] == context.base_currency]

    # Filter all the pairs to ensure that pair existed in the current date range
    universe_df = universe_df[universe_df.start_date < lookback_date]
    universe_df = universe_df[universe_df.end_daily >= current_date]
    context.coins = symbols(*universe_df.symbol)  # convert all the pairs to symbols
    print(universe_df.head(), len(universe_df))
    return universe_df.symbol.tolist()


# Replace all NA, NAN or infinite values with its nearest value
def fill(series):
    if isinstance(series, pd.Series):
        return series.replace([np.inf, -np.inf], np.nan).ffill().bfill()
    elif isinstance(series, np.ndarray):
        return pd.Series(series).replace([np.inf, -np.inf], np.nan).ffill().bfill().values
    else:
        return series


if __name__ == '__main__':
    start_date = pd.to_datetime('2017-01-01', utc=True)
    end_date = pd.to_datetime('2017-10-15', utc=True)

    performance = run_algorithm(start=start_date, end=end_date,
                                capital_base=10000.0,
                                initialize=initialize,
                                handle_data=handle_data,
                                analyze=analyze,
                                exchange_name='poloniex',
                                data_frequency='minute',
                                base_currency='eth',
                                live=False,
                                live_graph=False,
                                algo_namespace='simple_universe')

"""
Run in Terminal (inside catalyst environment):
python simple_universe.py
"""

Error

[2017-10-26 23:14:30.107384] INFO: exchange_bundle: pricing data for [u'bcn_xmr'] not found in range 1989-05-28 00:00:00+00:00 to 2017-01-01 00:00:00+00:00, updating the bundles.
    [====================================]  Fetching poloniex daily candles: :  100%
Error traceback: /home/avn3r/applications/conda/envs/catalyst/lib/python2.7/site-packages/catalyst/exchange/exchange_bundle.py (line 676)
PricingDataNotLoadedError:  Pricing data open for trading pairs bcn_xmr trading on exchange poloniex since 2014-07-23 00:00:00+00:00 is unavailable. The bundle data is either out-of-date or has not been loaded yet. Please ingest data using the command `catalyst ingest-exchange -x poloniex -f daily -i bcn_xmr`. See catalyst documentation for details.
Sincerely,

avn3r

While we are adding more built-in exchange price data, users may want to backtest with their own. For example, someone may have purchased rare historical data from Coinigy. While this should work in theory with the regular ingest command, we have been focused on exchange bundles so this need to be re-validated and fully supported.

When I select 2H bars from binance I receive the following error:

Traceback (most recent call last): File "simple_universe.py", line 320, in algo_namespace=NAMESPACE) File "/Users/Francesco/anaconda3/envs/catalyst/lib/python2.7/site-packages/catalyst/utils/run_algo.py", line 549, in run_algorithm stats_output=stats_output File "/Users/Francesco/anaconda3/envs/catalyst/lib/python2.7/site-packages/catalyst/utils/run_algo.py", line 331, in _run overwrite_sim_params=False, File "/Users/Francesco/anaconda3/envs/catalyst/lib/python2.7/site-packages/catalyst/exchange/exchange_algorithm.py", line 309, in run data, overwrite_sim_params File "/Users/Francesco/anaconda3/envs/catalyst/lib/python2.7/site-packages/catalyst/algorithm.py", line 724, in run for perf in self.get_generator(): File "/Users/Francesco/anaconda3/envs/catalyst/lib/python2.7/site-packages/catalyst/gens/tradesimulation.py", line 224, in transform for capital_change_packet in every_bar(dt): File "/Users/Francesco/anaconda3/envs/catalyst/lib/python2.7/site-packages/catalyst/gens/tradesimulation.py", line 137, in every_bar handle_data(algo, current_data, dt_to_use) File "/Users/Francesco/anaconda3/envs/catalyst/lib/python2.7/site-packages/catalyst/utils/events.py", line 216, in handle_data dt, File "/Users/Francesco/anaconda3/envs/catalyst/lib/python2.7/site-packages/catalyst/utils/events.py", line 235, in handle_data self.callback(context, data) File "/Users/Francesco/anaconda3/envs/catalyst/lib/python2.7/site-packages/catalyst/exchange/exchange_algorithm.py", line 761, in handle_data self._handle_data(self, data) File "simple_universe.py", line 221, in handle_data filter_low_vol_coins=(context.i%one_day_in_minutes)) File "simple_universe.py", line 72, in get_price_data_and_filter_coins frequency=frequency))) File "catalyst/_protocol.pyx", line 120, in catalyst._protocol.check_parameters.call.assert_keywords_and_call File "catalyst/_protocol.pyx", line 646, in catalyst._protocol.BarData.history File "/Users/Francesco/anaconda3/envs/catalyst/lib/python2.7/site-packages/catalyst/exchange/exchange_data_portal.py", line 95, in get_history_window ffill)) File "/Users/Francesco/anaconda3/envs/catalyst/lib/python2.7/site-packages/redo/init.py", line 162, in retry return action(*args, **kwargs) File "/Users/Francesco/anaconda3/envs/catalyst/lib/python2.7/site-packages/catalyst/exchange/exchange_data_portal.py", line 69, in _get_history_window ffill) File "/Users/Francesco/anaconda3/envs/catalyst/lib/python2.7/site-packages/catalyst/exchange/exchange_data_portal.py", line 218, in get_exchange_history_window False) File "/Users/Francesco/anaconda3/envs/catalyst/lib/python2.7/site-packages/catalyst/exchange/exchange.py", line 505, in get_history_window frequency, data_frequency File "/Users/Francesco/anaconda3/envs/catalyst/lib/python2.7/site-packages/catalyst/exchange/utils/datetime_utils.py", line 317, in get_frequency raise InvalidHistoryFrequencyAlias(freq=freq) catalyst.exchange.exchange_errors.InvalidHistoryFrequencyAlias: Invalid frequency alias 2H. Valid suffixes are M (minute) and D (day). For example, these aliases would be valid 1M, 5M, 1D.

I can fetch minute frequency data, for example using '15T', no problem.

Hi,

See this issue for my environment.

Description of Issue

When selling positions in bch_btc and ltc_btc on Poloniex using effectively market orders (actually limit orders with limit one quarter of the current price), the following happened (according to the log below):

• The two orders were placed at 06:01:00.
• They were filled around 6:01:02.
• A few seconds later, the algorithm somehow saw the orders as still open. It mentions that they have been open for 48 seconds, which doesn't make sense to me: they were placed a few seconds earlier. Or maybe this refers to when the ordering function was called?
• The algorithm detected that the positions in bch_btc and ltc_btc were now zero, and updated itself without adding to the cash the proceeds of the sale.
• This results in an apparent loss corresponding to the amount of the sold positions. See the output of the algorithm below. The algorithm uses order_target() to do the ordering.

I should emphasize this behavior is rather random. Numerous positions were sold previously without problem. Maybe the problem lies in the way that Catalyst deals with imperfect network connections.

[2018-04-20 06:01:04.910546] INFO: exchange_blotter: poloniex order 205328425212 for etc_btc still open after 0 days 00:00:48.910343
[2018-04-20 06:01:06.870621] INFO: exchange_blotter: poloniex order 118498224337 for zec_btc still open after 0 days 00:00:48.870499
[2018-04-20 06:01:07.968919] WARNING: Exchange: detected lower balance for etc_btc on poloniex: 1e-08 < 1.21984671, updating position amount
[2018-04-20 06:01:07.969499] WARNING: Exchange: detected lower balance for zec_btc on poloniex: 0.0 < 0.08515753, updating position amount

Here is how you can reproduce this issue on your machine:

Reproduction Steps

1. Let an algorithm buy and sell positions on Poloniex with market orders for some time. Sorry not to be able to be more specific.

Thanks,

Samuel

Hi, I wasn't able to install catalyst.

Here's my environment info:

• Operating System: macOS
• Python Version: Python 3.6.3
• Python Bitness: 64
• How did you install Catalyst: I tried: pip install enigma-catalyst matplotlib

then: pip install -r requirements.txt

and both gave me a similar error (below).

• Python packages: aiodns==1.1.1 aiohttp==2.3.6 alembic==0.7.7 async-timeout==2.0.0 bcolz==0.12.1 boto3==1.4.8 botocore==1.8.15 Bottleneck==1.0.0 cchardet==2.1.1 ccxt==1.10.283 certifi==2017.11.5 chardet==3.0.4 click==4.0 contextlib2==0.4.0 cycler==0.10.0 cyordereddict==0.2.2 Cython==0.25.2 decorator==4.0.0 docutils==0.14 empyrical==0.2.1 enigma-catalyst==0.4.0 idna==2.6 intervaltree==2.1.0 jmespath==0.9.3 Logbook==0.12.5 lru-dict==1.1.4 Mako==1.0.1 MarkupSafe==0.23 matplotlib==2.1.1 multidict==3.3.2 multipledispatch==0.4.8 networkx==1.9.1 numexpr==2.6.1 numpy==1.13.1 pandas==0.19.2 pandas-datareader==0.2.1 patsy==0.4.0 pycares==2.3.0 pyparsing==2.2.0 python-dateutil==2.4.2 python-editor==1.0.3 pytz==2016.4 requests==2.9.1 requests-file==1.4.1 requests-ftp==0.3.1 s3transfer==0.1.12 scipy==0.17.1 six==1.10.0 sortedcontainers==1.4.4 SQLAlchemy==1.0.8 statsmodels==0.6.1 tables==3.3.0 toolz==0.8.2 urllib3==1.22 yarl==0.16.0

I tried to execute catalyst --version to check installation was successful and got: RuntimeError: module compiled against API version 0xc but this version of numpy is 0xb Traceback (most recent call last): File "/Users/Quentin/Documents/Projects/Catalyst/catalyst-venv/bin/catalyst", line 7, in <module> from catalyst.__main__ import main File "/Users/Quentin/Documents/Projects/Catalyst/catalyst-venv/lib/python3.6/site-packages/catalyst/__init__.py", line 25, in <module> from .utils.run_algo import run_algorithm File "/Users/Quentin/Documents/Projects/Catalyst/catalyst-venv/lib/python3.6/site-packages/catalyst/utils/run_algo.py", line 13, in <module> from catalyst.data.bundles import load File "/Users/Quentin/Documents/Projects/Catalyst/catalyst-venv/lib/python3.6/site-packages/catalyst/data/bundles/__init__.py", line 2, in <module> from . import quandl # noqa File "/Users/Quentin/Documents/Projects/Catalyst/catalyst-venv/lib/python3.6/site-packages/catalyst/data/bundles/quandl.py", line 21, in <module> from catalyst.data.bundles.core import register_bundle File "/Users/Quentin/Documents/Projects/Catalyst/catalyst-venv/lib/python3.6/site-packages/catalyst/data/bundles/core.py", line 14, in <module> from ..us_equity_pricing import ( File "/Users/Quentin/Documents/Projects/Catalyst/catalyst-venv/lib/python3.6/site-packages/catalyst/data/us_equity_pricing.py", line 23, in <module> from bcolz import ( File "/Users/Quentin/Documents/Projects/Catalyst/catalyst-venv/lib/python3.6/site-packages/bcolz/__init__.py", line 55, in <module> from bcolz.carray_ext import ( ImportError: numpy.core.multiarray failed to import

I tried with numpy 1.13.3 (latest) and 1.13.1 (from requirements.txt) but nothing worked...

thanks

Patching CVE-2007-4559

Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

When I enter the command 'catalyst --version' in PyCharm's terminal, then it shows the following:

catalyst : The item 'catalyst' was not recognized as the name of a cmdlet, function, script file, or runnable program. Please check the spelling of the name and if you include a path make sure the path correct and try again. Location Line: 1 Character: 1

• catalyst --version +~~~~~~~~~
  • CategoryInfo : ObjectNotFound: (catalyst:String) [], CommandNotFoundException
  • FullyQualifiedErrorId : CommandNotFoundException

Dear Catalyst Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

• Operating System: MacOS Big SUR 11.2.2 (Macbook Air M1)
• Python Version: Python 3.6.13 :: Anaconda, Inc.
• Python Bitness: 64
• How did you install Catalyst: trying with conda but cant achieve it
• Python packages:

aiodns==2.0.0 aiohttp==3.7.4.post0 async-timeout==3.0.1 attrs==21.2.0 ccxt==1.51.73 certifi==2021.5.30 cffi==1.14.5 chardet==4.0.0 cryptography==3.4.7 cvxpy==1.1.13 ecos==2.0.7.post1 idna==2.10 idna-ssl==1.1.0 multidict==5.1.0 numpy==1.19.5 osqp==0.6.2.post0 pycares==4.0.0 pycparser==2.20 python-snappy==0.6.0 qdldl==0.1.5.post0 requests==2.25.1 scipy==1.5.4 scs==2.1.4 typing==3.7.4.3 typing-extensions==3.10.0.0 urllib3==1.26.5 yarl==1.6.3

Now that you know a little about me, let me tell you about the issue I am having:

When I run 'conda env create -f environment.yml' with this env file: https://github.com/enigmampc/catalyst/blob/master/etc/python3.6-environment.yml

First, it says 'ccxt version not found' Then I change the version to 1.18.1 This time, it gives the following error:

'gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/bahadirtasdemir/opt/anaconda3/envs/catalyst/include -arch x86_64 -I/Users/bahadirtasdemir/opt/anaconda3/envs/catalyst/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ibcolz -Ic-blosc/blosc -Ic-blosc/internal-complibs/zstd-1.3.4 -Ic-blosc/internal-complibs/lz4-1.8.1.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.3.4/compress -Ic-blosc/internal-complibs/zstd-1.3.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.3.4/decompress -Ic-blosc/internal-complibs/zstd-1.3.4/legacy -Ic-blosc/internal-complibs/zstd-1.3.4/common -Ic-blosc/internal-complibs/zstd-1.3.4/dll -Ic-blosc/internal-complibs/zstd-1.3.4/deprecated -I/Users/bahadirtasdemir/opt/anaconda3/envs/catalyst/lib/python3.6/site-packages/numpy/core/include -I/Users/bahadirtasdemir/opt/anaconda3/envs/catalyst/include/python3.6m -c c-blosc/internal-complibs/snappy-1.1.1/snappy-stubs-internal.cc -o build/temp.macosx-10.7-x86_64-3.6/c-blosc/internal-complibs/snappy-1.1.1/snappy-stubs-internal.o -DSHUFFLE_SSE2_ENABLED -msse2 clang: warning: include path for libstdc++ headers not found; pass '-stdlib=libc++' on the command line to use the libc++ standard library instead [-Wstdlibcxx-not-found] c-blosc/internal-complibs/snappy-1.1.1/snappy-stubs-internal.cc:29:10: fatal error: 'algorithm' file not found #include ^~~~~~~~~~~ 1 error generated. error: command 'gcc' failed with exit status 1'

I tried to install xcode, change the version of the numpy but this time all pip packages start to conflict with each other. Could you please help me, I just couldn't successfully create the environment. Thanks in advance.

Sincerely, Bahadir Tasdemir btasdemir.com [email protected]

Dear Catalyst Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

• Operating System: (Mac OS 10.14.6)
• Python Version: $ python 3.6.13
• Python Bitness: 64'
• How did you install Catalyst: (conda)
• Python packages: $ conda list

Now that you know a little about me, let me tell you about the issue I am having:

after I have set up the env conda in anaconda3 and used following code:

conda env create -f python3.6-environment.yml source activate catalyst pip install enigma-catalyst matplotlib (by troubleshooting)

I can still not properly download catalyst in my env. When I tip this code "pip install enigma-catalyst matplotlib" in terminal, it shows me: "Building wheel for bcolz (setup.py) ... error......; Building wheel for cyordereddict (setup.py) ... error.....; Building wheel for pandas (setup.py) ... error....; Running setup.py install for pandas ... error......;"

And if I want to check the version of catalyst with code "catalyst --version", it shows me: "......from numpy.testing.decorators import slow # noqa ModuleNotFoundError: No module named 'numpy.testing.decorators'"

• What did you expect to happen?

of course I can set up catalyst successfully.

• What happened instead? please see above

What steps have you taken to resolve this already?

I have repeated the code "pip install enigma-catalyst matplotlib" in catalyst env, but still doesn't work

Sincerely, Rubbermen

Dear Catalyst Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

• Operating System: (Windows Version or $ uname --all)
• Python Version: $ python --version
• Python Bitness: $ python -c 'import math, sys;print(int(math.log(sys.maxsize + 1, 2) + 1))'
• How did you install Catalyst: (pip, conda, or other (please explain))
• Python packages: $ pip freeze or $ conda list

Now that you know a little about me, let me tell you about the issue I am having:

Description of Issue

• What did you expect to happen?
• What happened instead?

Here is how you can reproduce this issue on your machine:

Reproduction Steps

...

What steps have you taken to resolve this already?

...

Anything else?

...

Sincerely, $ whoami