EXPERIMENT - proof of internal orderbook inconsistency

Relates to Issue https://github.com/bmoscon/cryptofeed/issues/604#issuecomment-899036978

Tested on Binance DASH-BUSD. Ask limit orders added into book: $195.66 (0.06 DASH) $195.67 (0.06 DASH) - added AFTER starting cryptofeed $195.68 (0.06 DASH) $195.69 (0.06 DASH)

Snap at 1629025913.852 (truncated book from 26 to 20):
194.18|13.024, 194.19|1.000, 194.21|2.014, 194.24|2.766, 194.26|3.307, 194.27|5.196, 194.36|3.756, 194.37|5.377, 194.42|3.600, 194.43|37.133, 194.46|10.753, 194.62|0.497, 194.77|9.027, 194.81|2.436, 194.87|3.756, 195.16|53.100, 195.67|0.060, 195.69|164.660, 195.70|0.060, 196.00|67.144
ERROR. 195.66 not in book but should be!!
ERROR. 195.68 not in book but should be!!

^ internal OB top 20 ask levels clearly go up to $196.00 so we would expect all our orders to be in the book.

The limit orders I added at 195.66, 195.68 BEFORE starting cryptofeed cannot be found (was outside of the first 20 levels in cryptofeed's initial book snapshot collection).

The limit order at 195.67 added after starting cryptofeed can be found (because binance sent a delta over the WS for this which cryptofeed processed)

The limit order at 195.69 had more size added to the px level by another participant since starting cryptofeed, so it can be found.

What this means: Any orders outside of the initial book snap will NOT be in cryptofeed internal book if: a) the market moves in that direction, and b) those price levels never receive a size change.

I know that this problem is NOT a cryptofeed/cryptostore problem, and I can remove post if it's considered too offtopic! I tried asking for this in all the related githubs and forums and have received no answers, so I'm trying here since I expect a lot of people who use cryptofeed to use Influx too, and for the same exact application as me.

I’m querying BitMEX tick data to process it and aggregate it. Those were my query times (number of rows on the left, query time in hours, minutes and seconds on the right), were every query is 1 month of data:

9480222, 2:07:46

12839124, 3:06:02

17256737, 4:19:54

13716707, 3:28:37

12671435, 2:35:27

11112483, 2:15:53

17055181, 3:34:21

21232810, 6:29:42

16935780, 4:47:56

Those numbers seem a bit off. The average is around 60-70k rows per minute, 1k rows per second. Since this is my first experience with TS Databases and with Influx, would you consider this performance normal? Do you guys have roughly the same query times? I’m running InfluxDB 1.7.9, Influx Python client 5.2.3, Python 3.7, this was ran from Pycharm, on a MacBook Pro with 16GB Ram. I don’t use any addon like Telegraph or Kapacitor.

Using the demo in the examples, Kibana does not recognize the time field as a timestamp. Do I need to change how it is indexed? Or do I need to create mappings before starting the demo_elastic.py?

Any help or direction would be much appreciated!

(This is an amazing project btw!)

Thank you!

Fixes https://github.com/bmoscon/cryptofeed/issues/671

• [X] - Tested
• [X] - Changelog updated
• [X] - Tests run and pass
• [X] - Flake8 run and all errors/warnings resolved
• [X] - Contributors file updated (optional)

Describe the bug Binance futures now has 200 stream limit (2020-10-27) https://binance-docs.github.io/apidocs/futures/en/#change-log

To Reproduce Use pairs=binance_futures_pairs() the app will freeze without getting any update. I have to make an array of SYMBOLS and remove some from the full list. Full list has 216 symbols.

Describe the bug From time to time, I get this error message.

2020-12-05 20:48:42,032 : ERROR : KRAKEN: encountered an exception, reconnecting
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/cryptofeed-1.6.2-py3.8.egg/cryptofeed/feedhandler.py", line 271, in _connect
    await self._handler(websocket, feed.message_handler, feed.uuid)
  File "/usr/local/lib/python3.8/dist-packages/cryptofeed-1.6.2-py3.8.egg/cryptofeed/feedhandler.py", line 300, in _handler
    await handler(message, self.last_msg[feed_id])
  File "/usr/local/lib/python3.8/dist-packages/cryptofeed-1.6.2-py3.8.egg/cryptofeed/exchange/kraken.py", line 150, in message_handler
    if self.channel_map[msg[0]][0] == 'trade':
KeyError: 400

To Reproduce Extract of the config file.

    KRAKEN:
        channel_timeouts:
            trades: 90
            l2_book: 90
        retries: -1
        trades: [BCH-BTC,BTC-USD,LTC-BTC]
        l2_book:
            symbols: [BCH-BTC,BTC-USD,LTC-BTC]
            book_delta: true
            book_interval: 100000

storage_interval: 2M

Expected behavior I have no idea what a KeyError: 400 refers to, but it seems that if a key is not existing, it is masking another trouble that might be worth to catch.

Operating System: Ubuntu 18.04

Cryptofeed Version 1.6.2

Describe the bug When installing via git clone and pipenv, the user must also run python3 -m pip install ., running python3 -m pipenv install is not enough (with Python 3.7.5), but the INSTALL.md documention does not mention this.

The error appears to be that the pipenv install does not result in uvloop (or cryptofeed itself ) being installed.

To Reproduce

user@Ubuntu18:~/cryptofeed$ git log | head -n 1
commit 6eafd56b98a6a895d7485d46841e908992082e92

user@Ubuntu18:~/cryptofeed$ python3.7 -m pipenv install
Creating a virtualenv for this project...
Pipfile: /home/user/cryptofeed/Pipfile
Using /usr/bin/python3.7 (3.7.5) to create virtualenv...
⠇ Creating virtual environment...created virtual environment CPython3.7.5.final.0-64 in 433ms
  creator CPython3Posix(dest=/home/user/.local/share/virtualenvs/cryptofeed-XkEPfgbK, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/user/.local/share/virtualenv)
    added seed packages: pip==20.3.1, setuptools==51.0.0, wheel==0.36.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

✔ Successfully created virtual environment! 
Virtualenv location: /home/user/.local/share/virtualenvs/cryptofeed-XkEPfgbK
Installing dependencies from Pipfile.lock (ba6a78)...
Ignoring idna-ssl: markers 'python_version < "3.7"' don't match your environment
Ignoring typing: markers 'python_version < "3.7"' don't match your environment
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 64/64 — 00:00:54
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

user@Ubuntu18:~/cryptofeed$ pipenv shell
Launching subshell in virtual environment...
user@Ubuntu18:~/cryptofeed$  . /home/user/.local/share/virtualenvs/cryptofeed-XkEPfgbK/bin/activate

(cryptofeed) user@Ubuntu18:~/cryptofeed$ python examples/demo.py 
Traceback (most recent call last):
  File "examples/demo.py", line 9, in <module>
    from cryptofeed.pairs import binance_pairs
ModuleNotFoundError: No module named 'cryptofeed'

(cryptofeed) user@Ubuntu18:~/cryptofeed$ python -m pip install .
[... log entries omitted ...]
Installing collected packages: uvloop, cryptofeed
Successfully installed cryptofeed-1.7.0 uvloop-0.14.0

(cryptofeed) user@Ubuntu18:~/cryptofeed$ python examples/demo.py                                                                              
Timestamp: 1.609108 Feed: BLOCKCHAIN Pair: BTC-USD Book Bid Size is 189 Ask Size is 113
Timestamp: 1.609108 Feed: BLOCKCHAIN Pair: ETH-USD Book Bid Size is 58 Ask Size is 28   

Operating System:

• Ubuntu 18

Cryptofeed Version

• New connection design. Handles rest and websocket connections seamlessly for websocket and REST endpoint exchanges. Will allow exchanges to use more than 1 websocket connection seamlessly.
• Update BitMax to use their new pro API

• [X] - Tested
• [X] - Tests run and pass
• [X] - Flake8 run and all errors/warnings resolved

When inserting Trades into influxDB the ID field is of type float for Binance and Coinbase while for Poloniex is type string.

When the program is runing it seems to be doing OK insetring data for all three exchanges but periodically it will produce this error.

2021-02-24 14:17:08,143 : ERROR : POST to http://localhost:8086/write?db=CryptoDataV2 failed: 400 - {"error":"partial write: field type conflict: input field "id" on measurement "trades-POLONIEX" is type float, already exists as type string dropped=1"}

To Reproduce f.add_feed(Poloniex(subscription={'TUSD-USDT'], }, callbacks={TRADES: TradeInflux(db_addr, db = db_name , numeric_type=float )}))

Expected behavior Trade ID type should be consistent for all exchanges. Dropping values because of wrong ID type should be avoided.

Operating System:

• rpi4 linux

During the pairs retrieval at startup time, some exchanges may reply an erroneous response. Currently Cryptofeed does not always handle the exception, and crashes.

Some users think this is a bug in Cryptofeed: https://github.com/bmoscon/cryptofeed/issues/371#issuecomment-751899389

This PR improves the user experience by providing a clear explanation of the failure.

Hello,

Could you advice me how I access Binance top of the book quotes instead of L2? Top of the book updates real time and L2 each 100ms only.

Regards, Eugene.

Describe the bug

by using the follow : f.add_feed(OKX(checksum_validation=True, symbols=['BTC-USDT-PERP'], channels=[TRADES, TICKER, FUNDING, OPEN_INTEREST, LIQUIDATIONS, L2_BOOK], callbacks={L2_BOOK: book, TICKER: ticker, LIQUIDATIONS: liquidations, FUNDING: funding, OPEN_INTEREST: oi, TRADES: trade}))

I try to compare the L2_book to the orderbook on OKX . the price is match, however, the volume attach with the price is incorrect. different coin has different scale up or down.

To Reproduce just using the demo.py, and just catch one symbol. such as BTC-USDT-PERP. will notice the volume is big differntthan the OKX's website

Operating System:

• linux,

Cryptofeed Version -Dec 21,2022's latest version (v2.3.1)

Issue/Feature request: Missing funding rate for dydx. I am able to use dydx and call other functions: for example

pairs = ['ETH-USD-PERP'] f.add_feed(dYdX(symbols=pairs, channels = [TRADES], callbacks= {TRADES: trade}))

generates ETH/USD data --> exchange: DYDX symbol: ETH-USD-PERP side: sell amount: 2.996 price: 1273.8 id: None type: None timestamp: 1670913339.171

However funding generates: cryptofeed.exceptions.UnsupportedDataFeed: funding is not supported on DYDX

Looking at dydx client there is an option to query funding rate using client.get_perpetual_market so I was wondering if this will be included. Thanks!

TICKER sometimes means 'bookTicker' (real-time best bid and ask updates), but sometimes 'ticker' (slow, usualy ever second updates with a bunch of extra info like volume in last 24h).

E.g. on binance it correponds to 'bookTicker':

    websocket_channels = {
    L2_BOOK: 'depth',
    TRADES: 'aggTrade',
    TICKER: 'bookTicker',
    CANDLES: 'kline_',
    BALANCES: BALANCES,
    ORDER_INFO: ORDER_INFO
}

But on gateio to 'ticker', despite that the 'bookTicker' endpoint is also available on gateio:

websocket_channels = {
     L2_BOOK: 'spot.order_book_update',
     TRADES: 'spot.trades',
     TICKER: 'spot.tickers',
     CANDLES: 'spot.candlesticks'
}

Are there plans to make TICKER, or to maybe have L1_BOOK correspond to 'bookTicker', while TICKER to 'ticker' endpoints?

Description of code - what bug does this fix / what feature does this add?

Added support for spot websocket endpoints on Bybit. Current implementation includes trade and orderbook channels.

Testing

To connect to a spot endpoint, specify a standardised spot symbol. See example code below, connecting to both spot and perpetual endpoints for trades and orderbook.

from decimal import Decimal
from cryptofeed import FeedHandler
from cryptofeed.exchanges import Bybit
from cryptofeed.defines import TRADES, L2_BOOK, BID, ASK


async def book(book, receipt_timestamp):
    print(f'Book received at {receipt_timestamp} for {book.exchange} - {book.symbol}, with {len(book.book)} entries. Top of book prices: {book.book.asks.index(0)[0]} - {book.book.bids.index(0)[0]}')
    if book.delta:
        print(f"Delta from last book contains {len(book.delta[BID]) + len(book.delta[ASK])} entries.")
    if book.sequence_number:
        assert isinstance(book.sequence_number, int)


async def trade(t, receipt_timestamp):
    assert isinstance(t.timestamp, float)
    assert isinstance(t.side, str)
    assert isinstance(t.amount, Decimal)
    assert isinstance(t.price, Decimal)
    assert isinstance(t.exchange, str)
    print(f"Trade received at {receipt_timestamp}: {t}")


def main():
    f = FeedHandler()
    f.add_feed(Bybit(symbols=["ETH-USDT", "ETH-USDT-PERP"], channels=[TRADES, L2_BOOK], callbacks={TRADES: trade, L2_BOOK: book}))
    f.run()


if __name__ == '__main__':
    main()

• [x] - Tested
• [x] - Changelog updated
• [x] - Tests run and pass
• [x] - Flake8 run and all errors/warnings resolved
• [x] - Contributors file updated (optional)

• Support REST API only
• Support on OrderBook, Ticker

Web: https://inlock.io Public API Doc: https://app.swaggerhub.com/apis/IncomeLocker/Inlock_Public_Tokenmarket_API/ Private API Doc: https://app.swaggerhub.com/apis-docs/IncomeLocker/inlock_retail_api/

Description of code - what bug does this fix / what feature does this add?

• [x] - Tested
• [ ] - Changelog updated
• [ ] - Tests run and pass
• [ ] - Flake8 run and all errors/warnings resolved
• [x] - Contributors file updated (optional)