Describe the bug I use Jesse with default Docker Compose from GitHub. When I run "Optimization" I get always a value error at some step of the process. It happens with every strategy, timeframe or coin

Same error happens also with "Import Candles" sometimes

Traceback Message:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.9/site-packages/timeloop/job.py", line 19, in run
    self.execute(*self.args, **self.kwargs)
  File "/jesse-docker/jesse/modes/optimize_mode/Optimize.py", line 71, in check_for_termination
    if process_status() != 'started':
  File "/jesse-docker/jesse/services/redis.py", line 64, in process_status
    raise ValueError(f'No value exists in Redis for process ID of: {pid}')
ValueError: No value exists in Redis for process ID of: 4110

To Reproduce Just run "Optimization" or "Import Candles" from GUI Webinterface

Screenshot

Enviroment (please complete the following information):

• Host OS: Ubuntu 20.04.3 LTS
• Docker Version: 20.10.12
• Jesse Version: 0.33.3

Describe the bug

ERROR:
Traceback (most recent call last):
  File "jesse_live\exchanges\LiveExchange.py", line 367, in jesse_live.exchanges.LiveExchange.LiveExchange._connect_ws._on_message
  File "jesse_live\exchanges\Binance\BinancePerpetualFuturesMain.py", line 380, in jesse_live.exchanges.Binance.BinancePerpetualFuturesMain.BinancePerpetualFuturesMain.on_ws_message
  File "jesse_live\exchanges\Binance\BinancePerpetualFuturesMain.py", line 482, in jesse_live.exchanges.Binance.BinancePerpetualFuturesMain.BinancePerpetualFuturesMain.prepare_candle_stream
  File "jesse_live\exchanges\LiveExchange.py", line 1080, in jesse_live.exchanges.LiveExchange.LiveExchange.add_candle
  File "C:\Users\xxxxx\AppData\Local\Programs\Python\Python39\lib\site-packages\jesse\store\state_candles.py", line 187, in add_candle
    f"Could not find the candle with timestamp {jh.timestamp_to_time(candle[0])} in the storage. Last candle's timestamp: {jh.timestamp_to_time(arr[-1])}. timeframe: {timeframe}, exchange: {exchange}, symbol: {symbol}"
  File "C:\Users\xxxxx\AppData\Local\Programs\Python\Python39\lib\site-packages\jesse\helpers.py", line 776, in timestamp_to_time
    return str(arrow.get(timestamp / 1000))
  File "C:\Users\xxxxx\AppData\Local\Programs\Python\Python39\lib\site-packages\arrow\api.py", line 91, in get
    return _factory.get(*args, **kwargs)
  File "C:\Users\xxxxx\AppData\Local\Programs\Python\Python39\lib\site-packages\arrow\factory.py", line 267, in get
    raise TypeError(f"Cannot parse single argument of type {type(arg)!r}.")
TypeError: Cannot parse single argument of type <class 'numpy.ndarray'>.

To Reproduce live trading on Binance Perpetual Futures 2022/09/16 18:52 that led to the following message simultaneously in multiple instances (BTC-USDT, and ETH-USDT) being raised to the discord channels at 30x per minute each, until I got home to stop it around 21:00 UTC

Expected behavior Single error message, then recovery preferably without interruption otherwise restarting the strategy, or perhaps stopping.

Screenshots N/A

Enviroment (please complete the following information):

• OS: Windows 11
• Version jesse, version 0.39.4 and jesse_live v0.39.1

Additional context System had been running for 3 days on a 1min timeframe. Previously, same setup had been running for 2 weeks.

[INFO | 2022-09-14T11:58:40] Debug mode is ENABLED
[INFO | 2022-09-14T11:58:40] Local candle generation mode is DISABLED
[INFO | 2022-09-14T11:58:40] Trading 2 routes with 0 extra routes:
Binance Perpetual Futures-DEFI-USDT-1h
Binance Perpetual Futures-ATOM-USDT-1h

[INFO | 2022-09-14T11:58:47] Persistency setting is disabled, will close all existing positions and orders on the exchange before starting the new session
[INFO | 2022-09-14T11:58:47] Canceling all existing orders (if any) for DEFI-USDT on Binance Perpetual Futures
[INFO | 2022-09-14T11:58:47] Canceling all existing orders (if any) for ATOM-USDT on Binance Perpetual Futures
[INFO | 2022-09-14T11:58:47] Updated leverage for DEFI-USDT on Binance Perpetual Futures to 2x
[INFO | 2022-09-14T11:58:47] Updated leverage for ATOM-USDT  on Binance Perpetual Futures to 2x

[INFO | 2022-09-14T11:58:47] Authenticated to Binance Perpetual Futures successfully.
[INFO | 2022-09-14T11:58:47] Successfully opened WS connection to "wss://fstream.binance.com/stream?streams=xxxxxxxxxxxxxxxxxxxx"

[INFO | 2022-09-14T14:00:00] Executing  SlowTrendFollowing-Binance Perpetual Futures-ATOM-USDT-1h
[INFO | 2022-09-14T14:00:00] SUBMITTED order: ATOM-USDT, MARKET, sell, -0.67, $14.18
[INFO | 2022-09-14T14:00:00] EXECUTED order: ATOM-USDT, MARKET, sell, 0.67, $14.18
[INFO | 2022-09-14T14:00:00] OPENED short position for ATOM-USDT: qty: -0.67, entry_price: 14.178
[INFO | 2022-09-14T14:00:00] QUEUED order: ATOM-USDT, LIMIT, buy, 0.67, $11.28
[INFO | 2022-09-14T14:00:00] SUBMITTED order: ATOM-USDT, STOP, buy, 0.67, $14.7

[INFO | 2022-09-15T07:47:49] Authenticated to Binance Perpetual Futures successfully.
[INFO | 2022-09-15T08:16:49] Authenticated to Binance Perpetual Futures successfully.
[INFO | 2022-09-15T08:45:49] Authenticated to Binance Perpetual Futures successfully.

Problem 1: 2022-09-15 07:48:14 listenKeyExpired (websocket pushed Event: User Data Stream Expired)

2022-09-15 08:25:54 ATOM-USDT, STOP order has been EXECUTED in exchange,But Jesse get nothing from websocket...

so the position is still there in Jesse (and forever) ,but in fact, no position in exchange anymore

[INFO | 2022-09-15T09:00:00] Executing  SlowTrendFollowing-Binance Perpetual Futures-DEFI-USDT-1h
[INFO | 2022-09-15T09:00:00] SUBMITTED order: DEFI-USDT, MARKET, sell, -0.012, $723.2

[INFO | 2022-09-15T10:00:00] Executing  SlowTrendFollowing-Binance Perpetual Futures-DEFI-USDT-1h
[INFO | 2022-09-15T10:00:00] cancel all remaining orders to prepare for a fresh start...
[INFO | 2022-09-15T10:00:00] CANCELED order: DEFI-USDT, MARKET, sell, 0.012, $723.2
[INFO | 2022-09-15T10:00:00] SUBMITTED order: DEFI-USDT, MARKET, sell, -0.013, $722.1

Problem 2: 2022-09-15 09:00:00 DEFI-USDT, MARKET, sell has been EXECUTED in exchange,But Jesse get nothing from websocket... 2022-09-15 09:00:00 DEFI-USDT, MARKET, sell has been EXECUTED in exchange,But Jesse get nothing from websocket...

at last, in Jesse,there is no any DEFI-USDT position, but in exchange,there is 0.025 DEFI position...

I am using Linux. Used the script to install everything from a fresh installation. Configured and ran the psql service and db. No luck running Jesse. Stuck on import-candles. Following is the error:

jesse import-candles Bitfinex BTCUSD 2019-01-01

Traceback (most recent call last): File "/usr/local/bin/jesse", line 5, in from jesse.init import cli File "/usr/local/lib/python3.8/dist-packages/jesse/init.py", line 60, in inject_local_routes() File "/usr/local/lib/python3.8/dist-packages/jesse/init.py", line 53, in inject_local_routes router.set_routes(local_router.routes) AttributeError: module 'routes' has no attribute 'routes'

import_candles() when called via jesse.research exits early when the start_date of the candles is before the first available date. This is due to the "return" returning nothing and calling run() on a separate line instead of returning the result of run().

I have used the existing running_via_dashboard if statement to run the current version of the code, if not running_via_dashboard then instead the recursive run() is called, passed the relevant parameters, and then returned.

This results in the expected behavior of receiving the "Successfully imported candles for..." message regardless of whether the start_date that was passed to import_candles() via jesse.research was before the first available date or not.

N.B The "No candle exists in the market for..." message is not known to the user when they call import_candles(), They only ever receive the success message. Considering the return statements in run() are only returned when not using the dashboard, i.e. via research, a named tuple could be returned to return the "No candle exists in the market for..." if there is one, and the success message. Not a deal breaker but a nice to know for the user if they are logging the candle downloading. Currently the only way to know if that occurred is to compare the start_date that was passed to import_candles() and to look at the success message and see if the dates match.

Hey, I got this error when I try to launch a Backtest.

root@797704255f30:/home/mybot# jesse backtest 2020-01-01 2020-11-10

loading candles...
============================== EXCEPTION TRACEBACK:
  File "/usr/local/bin/jesse", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/jesse/__init__.py", line 322, in backtest
    backtest_mode.run(start_date, finish_date, chart=chart, tradingview=tradingview, csv=csv, json=json)
  File "/usr/local/lib/python3.8/dist-packages/jesse/modes/backtest_mode/__init__.py", line 41, in run
    candles = load_candles(start_date, finish_date)
  File "/usr/local/lib/python3.8/dist-packages/jesse/modes/backtest_mode/__init__.py", line 105, in load_candles
    cached_value = cache.get_value(cache_key)
  File "/usr/local/lib/python3.8/dist-packages/jesse/services/cache.py", line 69, in get_value
    return pickle.load(f)
=========================================================================

 Uncaught Exception: UnpicklingError: invalid load key, '\x00'.

Is your feature request related to a problem? Please describe. Can't go short on spot type backtests

Describe the solution you'd like If we have access to the current assets we can use them to filter out orders that we can't afford and avoid InsufficientMargin errors

Describe alternatives you've considered We could keep track of that manually on the strategy but I think it makes more sense to just pass it down from the SpotExchange model

Additional context Does this make sense or am I missing something here?

Everytime I run a backtest with the --chart option I run into this problem:

 File "/Applications/anaconda3/envs/crypto/lib/python3.8/site-packages/jesse/services/charts.py", line 84, in portfolio_vs_asset_returns
    price_dict[key]['prices'][price_dict[key]['indexes'][str(int(t.closed_at))]]
=========================================================================
 Uncaught Exception: KeyError: '1603929600000'

Using the csv output works fine and so does the json (I even used the graphical tool and it showed it fine)

I tried the --chart parameter and got this as an exception whilst --tradingview is working fine.

` File "c:\programdata\anaconda3\lib\site-packages\jesse_init_.py", line 286, in backtest backtest_mode.run(start_date, finish_date, chart=chart, tradingview=tradingview) File "c:\programdata\anaconda3\lib\site-packages\jesse\modes\backtest_mode_init_.py", line 71, in run charts.portfolio_vs_asset_returns() File "c:\programdata\anaconda3\lib\site-packages\jesse\services\charts.py", line 111, in portfolio_vs_asset_returns plt.savefig(file_path) File "c:\programdata\anaconda3\lib\site-packages\matplotlib\pyplot.py", line 722, in savefig res = fig.savefig(*args, **kwargs) File "c:\programdata\anaconda3\lib\site-packages\matplotlib\figure.py", line 2180, in savefig self.canvas.print_figure(fname, **kwargs) File "c:\programdata\anaconda3\lib\site-packages\matplotlib\backend_bases.py", line 2089, in print_figure **kwargs) File "c:\programdata\anaconda3\lib\site-packages\matplotlib\backends\backend_agg.py", line 530, in print_png cbook.open_file_cm(filename_or_obj, "wb") as fh: File "c:\programdata\anaconda3\lib\contextlib.py", line 112, in enter return next(self.gen) File "c:\programdata\anaconda3\lib\site-packages\matplotlib\cbook_init_.py", line 447, in open_file_cm fh, opened = to_filehandle(path_or_file, mode, True, encoding) File "c:\programdata\anaconda3\lib\site-packages\matplotlib\cbook_init_.py", line 432, in to_filehandle fh = open(fname, flag, encoding=encoding)

Uncaught Exception: OSError: [Errno 22] Invalid argument: 'storage/charts/BT-2020-04-18T16:56:53.png'`

Also, what value is the qty exactly, in the case of BTCUSDT, does 1 qty mean opening a trade for 1 BTC or 1$?

I apologize for the number of opened issues but I'm migrating my algorithm to this framework and finding issues / questions as I go along but other than that quite impressed by how it all works :)

Thanks.

Currently terminate() is called before closing out the final backtest trades. The backtest is complete once all trades have been closed, metrics updated etc, not before. Even Jesse's documentation states the purpose of the function. "Examples of such a task would be to log a value or save a machine learning model.". Depending what values are saved they will probably be wrong because the final trades would alter them.

Imagine the most basic use of the terminate() function, for e.g. writing all trades to a log file. The current order of operation would mean there would be trades missing...

This is my attempt at this task

The cached decorator can then be used like this:

    @property
    @cached
    def atr(self):
        return atr(self.candles, self.vars['atr_period'])

However, when running the TurtleRules strategy, I had no noticeable performance gains. It may be that this strategy in particular is not that computationally expensive, though

I tried these three options and got more or less the same results:

# Using functools.cache
def cached(method):
    def decorated(self, *args, **kwargs):
        cached_method = self._cached_methods.get(method)
        if cached_method is None:
            cached_method = cache(method)
            self._cached_methods[method] = cached_method
        return cached_method(self, *args, **kwargs)
    return decorated


# Using functools.lru_cache
def cached(method):
    def decorated(self, *args, **kwargs):
        cached_method = self._cached_methods.get(method)
        if cached_method is None:
            cached_method = lru_cache()(method)
            self._cached_methods[method] = cached_method
        return cached_method(self, *args, **kwargs)
    return decorated


# Using functools.lru_cache with maxsize = 1
def cached(*dargs, maxsize=1, **dkwargs):
    def decorator(method):
        def decorated(self, *args, **kwargs):
            cached_method = self._cached_methods.get(method)
            if cached_method is None:
                dkwargs['maxsize'] = maxsize
                cached_method = lru_cache(*dargs, **dkwargs)(method)
                self._cached_methods[method] = cached_method
            return cached_method(self, *args, **kwargs)
        return decorated
    return decorator

Also, I didn't find any methods in Strategy.py worth caching, but I still haven't gone through all of them

Let me know what you think and if I should change something

Upon starting the docker on ARM and attempting to download the live-trading plugin, I get the following error:

is_64_bit True                                                                                                                                                            │
│ is_arm False                                                                                                                                                              │
│ Downloading the latest version of the live-trade plugin...                                                                                                                │
│ Installing jesse_live-0.41.4-cp39-cp39-manylinux2010_x86_64.whl...                                                                                                        │
│ ERROR: jesse_live-0.41.4-cp39-cp39-manylinux2010_x86_64.whl is not a supported wheel on this platform.                                                                    │
│ Traceback (most recent call last):                                                                                                                                        │
│   File "/usr/local/bin/jesse", line 33, in <module>                                                                                                                       │
│     sys.exit(load_entry_point('jesse', 'console_scripts', 'jesse')())                                                                                                     │
│   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1128, in __call__                                                                                     │
│     return self.main(*args, **kwargs)                                                                                                                                     │
│   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1053, in main                                                                                         │
│     rv = self.invoke(ctx)                                                                                                                                                 │
│   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1659, in invoke                                                                                       │
│     return _process_result(sub_ctx.command.invoke(sub_ctx))                                                                                                               │
│   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1395, in invoke                                                                                       │
│     return ctx.invoke(self.callback, **ctx.params)                                                                                                                        │
│   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 754, in invoke                                                                                        │
│     return __callback(*args, **kwargs)                                                                                                                                    │
│   File "/jesse-docker/jesse/__init__.py", line 195, in install_live                                                                                                       │
│     install(HAS_LIVE_TRADE_PLUGIN, strict)                                                                                                                                │
│   File "/jesse-docker/jesse/services/installer.py", line 89, in install                                                                                                   │
│     _pip_install(filepath)                                                                                                                                                │
│   File "/jesse-docker/jesse/services/installer.py", line 11, in _pip_install                                                                                              │
│     subprocess.check_call([sys.executable, "-m", "pip", "install", package])                                                                                              │
│   File "/usr/local/lib/python3.9/subprocess.py", line 373, in check_call                                                                                                  │
│     raise CalledProcessError(retcode, cmd)                                                                                                                                │
│ subprocess.CalledProcessError: Command '['/usr/local/bin/python', '-m', 'pip', 'install', 'storage/jesse_live-0.41.4-cp39-cp39-manylinux2010_x86_64.whl']' returned non-z │
│ Stream closed EOF for jesse/jesse-78bb7856d8-84q77 (jesse)

Could the ARM be supported as well, please? Happy to help with the efforts.

Fixed issue with the docker build for ARM platforms. The root cause is with outdated config:

#0 1.622 checking build system type... ./config.guess: unable to guess system type
#0 1.687
#0 1.687 This script, last modified 2006-07-02, has failed to recognize
#0 1.687 the operating system you are using. It is advised that you
#0 1.687 download the most up to date version of the config scripts from
#0 1.687
#0 1.687   http://savannah.gnu.org/cgi-bin/viewcvs/*checkout*/config/config/config.guess
#0 1.687 and
#0 1.687   http://savannah.gnu.org/cgi-bin/viewcvs/*checkout*/config/config/config.sub
#0 1.687
#0 1.687 If the version you run (./config.guess) is already up to date, please
#0 1.687 send the following data and any information you think might be
#0 1.687 pertinent to <[email protected]> in order to provide the needed
#0 1.687 information to handle your system.
#0 1.687
#0 1.687 config.guess timestamp = 2006-07-02
#0 1.687
#0 1.687 uname -m = aarch64
#0 1.687 uname -r = 5.10.124-linuxkit
#0 1.687 uname -s = Linux
#0 1.687 uname -v = #1 SMP PREEMPT Thu Jun 30 08:18:26 UTC 2022
#0 1.687
#0 1.687 /usr/bin/uname -p =
#0 1.687 /bin/uname -X     =
#0 1.687
#0 1.687 hostinfo               =
#0 1.687 /bin/universe          =
#0 1.687 /usr/bin/arch -k       =
#0 1.687 /bin/arch              =
#0 1.687 /usr/bin/oslevel       =
#0 1.687 /usr/convex/getsysinfo =
#0 1.687
#0 1.687 UNAME_MACHINE = aarch64
#0 1.687 UNAME_RELEASE = 5.10.124-linuxkit
#0 1.687 UNAME_SYSTEM  = Linux
#0 1.687 UNAME_VERSION = #1 SMP PREEMPT Thu Jun 30 08:18:26 UTC 2022
#0 1.690 configure: error: cannot guess build type; you must specify one
#0 1.707 make: *** No rule to make target 'install'.  Stop.
------
ERROR: failed to solve: executor failed running [/bin/sh -c cd /tmp && /tmp/install_ta-lib.sh && rm -r /tmp/*ta-lib*]: exit code: 2

After adding the proposed changed, the build passes fine.

docker buildx build --platform linux/arm64 -t petrmac/jesse:latest .
[+] Building 75.5s (17/17) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                    0.0s
 => => transferring dockerfile: 32B                                                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                                                       0.0s
 => => transferring context: 35B                                                                                                                                        0.0s
 => [internal] load metadata for docker.io/library/python:3.9-slim                                                                                                      0.5s
 => [internal] load build context                                                                                                                                       0.2s
 => => transferring context: 38.12kB                                                                                                                                    0.2s
 => [jesse_basic_env  1/11] FROM docker.io/library/python:3.9-slim@sha256:9e0b4391fc41bc35c16caef4740736b6b349f6626fd14eba32793ae3c7b01908                              0.0s
 => CACHED [jesse_basic_env  2/11] RUN apt-get update     && apt-get -y install git build-essential libssl-dev wget     && apt-get clean     && pip install --upgrade   0.0s
 => CACHED [jesse_basic_env  3/11] RUN pip3 install Cython numpy                                                                                                        0.0s
 => CACHED [jesse_basic_env  4/11] RUN mkdir /jesse-docker                                                                                                              0.0s
 => CACHED [jesse_basic_env  5/11] WORKDIR /jesse-docker                                                                                                                0.0s
 => CACHED [jesse_basic_env  6/11] COPY docker_build_helpers/* /tmp/                                                                                                    0.0s
 => CACHED [jesse_basic_env  7/11] RUN cd /tmp && /tmp/install_ta-lib.sh && rm -r /tmp/*ta-lib*                                                                         0.0s
 => [jesse_basic_env  8/11] COPY requirements.txt /jesse-docker                                                                                                         0.0s
 => [jesse_basic_env  9/11] RUN pip3 install -r requirements.txt                                                                                                       71.3s
 => [jesse_basic_env 10/11] COPY . /jesse-docker                                                                                                                        0.1s
 => [jesse_basic_env 11/11] RUN pip3 install -e .                                                                                                                       1.4s
 => [jesse_with_test_0 1/1] WORKDIR /home                                                                                                                               0.0s
 => exporting to image                                                                                                                                                  1.9s
 => => exporting layers                                                                                                                                                 1.9s
 => => writing image sha256:a363e88349794fa750f8497c1d3bc99cb093f95a5b0a494bba5ec9abbd8725a5                                                                            0.0s
 => => naming to docker.io/petrmac/jesse:latest                                                                                                                         0.0s
❯ docker push docker.io/petrmac/jesse:latest

The ARM image can be built with platform specification, e.g.:

docker buildx build --platform linux/arm64 -t petrmac/jesse:latest .

Change the Docker file in order to run the process as jesse.

Also the host user can be able to bind his user_id and group_id using the user option i.e docker-compose.yml file

services:
  jesse-test:
    image: salehmir/jesse:latest
    tty: true
    # Optional bind your current user
    # id & gid with the container's user
    # copy paste the output below
    # echo "$(id -u):$(id -g)"
    user: "1000:1000"
    env_file:
      - ../.env

Hi, when I run docker image on my mac m1, I'm getting warning:

jesse The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

It's not blocking me, but I'm pretty sure it has performance impact, so would be awesome to have linux/arm64 images, which also make it possible to run on ARM Linux hosting.

And it doesn't look very hard: https://itnext.io/building-multi-cpu-architecture-docker-images-for-arm-and-x86-1-the-basics-2fa97869a99b

It would be great if I can reattach the optimization process after closing the browser tab or a lost network connection. At the moment Jesse is working in the background but its not possible to get any results or to see the current progress after closing a browser tab or a lost network connection. I always have to kill the processes at OS level when something like this happens, because they are using a lot of CPU and the results are lost. It would also be very helpful for long running optimization sessions.