Hello. I recently bought a book and am reading it well. Thank you.

I'm going to practice the code.

Using 'conda env create -f environment.yml'

I was trying to install my environment, but the following error occurred:

What should I do? (I am a Windows os user.)

`ResolvePackageNotFound:

• binutils_impl_linux-64=2.28.1
• gxx_impl_linux-64=7.2.0
• gxx_linux-64=7.2.0
• libgcc-ng=8.2.0
• libstdcxx-ng=8.2.0
• readline=7.0
• gcc_linux-64=7.2.0
• gmp=6.1.2
• libuuid=1.0.3
• gstreamer=1.14.0
• graphviz=2.40.1
• dbus=1.13.2
• binutils_linux-64=7.2.0
• expat=2.2.6
• libgfortran-ng=7.3.0
• gcc_impl_linux-64=7.2.0
• ncurses=6.1
• gst-plugins-base=1.14.0
• libedit=3.1.20170329`

Hello I have tried to create an environment using the environment.yml file provided but I get the following error:

Solving environment: failed

ResolvePackageNotFound:

• gcc_linux-64=7.2.0
• binutils_impl_linux-64=2.28.1
• gxx_linux-64=7.2.0
• gst-plugins-base=1.14.0
• gstreamer=1.14.0
• gmp=6.1.2
• pango=1.42.4
• dbus=1.13.2
• gcc_impl_linux-64=7.2.0
• binutils_linux-64=7.2.0
• gxx_impl_linux-64=7.2.0
• ncurses=6.1
• libgcc-ng=8.2.0
• libstdcxx-ng=8.2.0
• libuuid=1.0.3
• readline=7.0
• expat=2.2.6
• fribidi=1.0.5
• libgfortran-ng=7.3.0
• graphviz=2.40.1
• libedit=3.1.20170329

is there any way I can fix this? thanks for the help

The section "Buy-Sell Order Distribution" uses "sns.distplot" which fails due to a missing 'import seaborn as sns' up at the very top under "Imports". Once I added this and re-ran both it passes without error...

Hey!

Love the work so far. I've noticed a couple of changes to make though. In the first Jupyter Notebook in Chapter 2 ("01_build_itch_order_book.ipynb"), Seaborn was not imported so I ran into an error in the "Buy-Sell Order Distribution" section, so add:

import seaborn as sns

...to the top. Also, I was having an issue in the Download/Unzip section at the beginning. Even though I had already downloaded and unzipped the sample file, when I went to run the book again, it was starting to unzip the .gz file all over again, so I also changed the line:

unzipped = data_path / (filename.stem + '.bin')\n",

...to:

unzipped = data_path / (os.path.splitext(SOURCE_FILE)[0] + '.bin')

...and also need to add:

import os

...to the top. This will then properly see that there's already an unzipped file there and won't start to unzip the .gz file again.

Now, I'm running into a memory error and I'm wondering if an external hard drive is actually the solution, or if Windows is just running out of memory from trying to work with such a huge file? My traceback leading up to the memory error looks like this:

Empty DataFrame Columns: [Message Type, # Trades] Index: [] <class 'pandas.core.frame.DataFrame'> RangeIndex: 2010099 entries, 0 to 2010098 Data columns (total 9 columns): timestamp 2010099 non-null datetime64[ns] buy_sell_indicator 1873082 non-null float64 shares 1995581 non-null float64 price 1995581 non-null float64 type 2010099 non-null object executed_shares 54956 non-null float64 execution_price 500 non-null float64 shares_replaced 14159 non-null float64 price_replaced 14159 non-null float64 dtypes: datetime64ns, float64(7), object(1) memory usage: 138.0+ MB <class 'pandas.io.pytables.HDFStore'> File path: data\order_book.h5 /AAPL/buy frame_table (typ->appendable,nrows->177108242,ncols->2,indexers->[index],dc->[]) /AAPL/messages frame (shape->[2010099,9]) /AAPL/sell frame_table (typ->appendable,nrows->183264614,ncols->2,indexers->[index],dc->[]) /AAPL/trades frame (shape->[59796,3]) <class 'pandas.core.frame.DataFrame'> DatetimeIndex: 59796 entries, 2019-03-27 04:00:56.459428646 to 2019-03-27 19:54:05.600648466 Data columns (total 3 columns): shares 59796 non-null int32 price 59796 non-null int32 cross 59796 non-null int32 dtypes: int32(3) memory usage: 1.1 MB None 100,000 0:01:02.811120 200,000 0:01:27.826051 300,000 0:01:28.564991 400,000 0:01:27.096437 500,000 0:01:29.712940 600,000 0:01:29.556783 700,000 0:01:31.360356 800,000 0:01:35.391212 900,000 0:01:35.037438 1,000,000 0:01:56.868902 1,100,000 0:02:12.290003 1,200,000 0:01:42.614183 1,300,000 0:01:51.042871 1,400,000 0:01:41.064018 1,500,000 0:01:48.513128 1,600,000 0:01:41.310528 1,700,000 0:01:52.142803 1,800,000 0:01:44.541379 1,900,000 0:01:48.800505 2,000,000 0:01:48.460462 A 924117 D 869968 X 2789 E 52299 P 6995 F 2282 U 14159 C 473 dtype: int64 <class 'pandas.io.pytables.HDFStore'> File path: data\order_book.h5 /AAPL/buy frame_table (typ->appendable,nrows->265662363,ncols->2,indexers->[index],dc->[]) /AAPL/messages frame (shape->[2010099,9]) /AAPL/sell frame_table (typ->appendable,nrows->274896921,ncols->2,indexers->[index],dc->[]) /AAPL/trades frame (shape->[59796,3]) Traceback (most recent call last): File "01_build_itch_order_book.py", line 464, in sell = store['{}/sell'.format(stock)].reset_index().drop_duplicates() File "C:\Users\windowshopr\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\core\frame.py", line 4630, in drop_duplicates duplicated = self.duplicated(subset, keep=keep) File "C:\Users\windowshopr\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\core\frame.py", line 4687, in duplicated labels, shape = map(list, zip(*map(f, vals))) File "C:\Users\windowshopr\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\core\frame.py", line 4668, in f vals, size_hint=min(len(self), _SIZE_HINT_LIMIT)) File "C:\Users\windowshopr\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\util_decorators.py", line 188, in wrapper return func(*args, **kwargs) File "C:\Users\windowshopr\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\core\algorithms.py", line 613, in factorize na_value=na_value) File "C:\Users\windowshopr\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\core\algorithms.py", line 460, in _factorize_array na_value=na_value) File "pandas_libs\hashtable_class_helper.pxi", line 1209, in pandas._libs.hashtable.Int64HashTable.factorize File "pandas_libs\hashtable_class_helper.pxi", line 1104, in pandas._libs.hashtable.Int64HashTable._unique MemoryError

So I'm assuming it's because every time I try to run the program, it has to chew up a lot of working memory, so is there a way to work around this/work with a smaller ITCH sample/is there anything else you'd recommend? I'd love to be able to continue on working with this. My laptop has plenty of Hard Drive space, RAM is 8GB though.

Thanks for your input!

I have installed all packages from an updated version from packtpublishing and I am getting an erorr:

Downloading... ftp://emi.nasdaq.com/ITCH/Nasdaq_ITCH/03272019.NASDAQ_ITCH50.gz
---------------------------------------------------------------------------
error_perm                                Traceback (most recent call last)
~\anaconda3\envs\ml4trading\lib\urllib\request.py in ftp_open(self, req)
   1564         try:
-> 1565             fw = self.connect_ftp(user, passwd, host, port, dirs, req.timeout)
   1566             type = file and 'I' or 'D'

~\anaconda3\envs\ml4trading\lib\urllib\request.py in connect_ftp(self, user, passwd, host, port, dirs, timeout)
   1586         return ftpwrapper(user, passwd, host, port, dirs, timeout,
-> 1587                           persistent=False)
   1588 

~\anaconda3\envs\ml4trading\lib\urllib\request.py in __init__(self, user, passwd, host, port, dirs, timeout, persistent)
   2407         try:
-> 2408             self.init()
   2409         except:

~\anaconda3\envs\ml4trading\lib\urllib\request.py in init(self)
   2419         _target = '/'.join(self.dirs)
-> 2420         self.ftp.cwd(_target)
   2421 

~\anaconda3\envs\ml4trading\lib\ftplib.py in cwd(self, dirname)
    630         cmd = 'CWD ' + dirname
--> 631         return self.voidcmd(cmd)
    632 

~\anaconda3\envs\ml4trading\lib\ftplib.py in voidcmd(self, cmd)
    277         self.putcmd(cmd)
--> 278         return self.voidresp()
    279 

~\anaconda3\envs\ml4trading\lib\ftplib.py in voidresp(self)
    250         """Expect a response beginning with '2'."""
--> 251         resp = self.getresp()
    252         if resp[:1] != '2':

~\anaconda3\envs\ml4trading\lib\ftplib.py in getresp(self)
    245         if c == '5':
--> 246             raise error_perm(resp)
    247         raise error_proto(resp)

error_perm: 550 The system cannot find the file specified. 

During handling of the above exception, another exception occurred:

URLError                                  Traceback (most recent call last)
<ipython-input-6-8ab12e3a4ec2> in <module>
----> 1 file_name = may_be_download(urljoin(FTP_URL, SOURCE_FILE))
      2 date = file_name.name.split('.')[0]

<ipython-input-4-4c71a35e1865> in may_be_download(url)
      7     if not filename.exists():
      8         print('Downloading...', url)
----> 9         urlretrieve(url, filename)
     10     unzipped = data_path / (filename.stem + '.bin')
     11     if not (data_path / unzipped).exists():

~\anaconda3\envs\ml4trading\lib\urllib\request.py in urlretrieve(url, filename, reporthook, data)
    246     url_type, path = splittype(url)
    247 
--> 248     with contextlib.closing(urlopen(url, data)) as fp:
    249         headers = fp.info()
    250 

~\anaconda3\envs\ml4trading\lib\urllib\request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)
    221     else:
    222         opener = _opener
--> 223     return opener.open(url, data, timeout)
    224 
    225 def install_opener(opener):

~\anaconda3\envs\ml4trading\lib\urllib\request.py in open(self, fullurl, data, timeout)
    524             req = meth(req)
    525 
--> 526         response = self._open(req, data)
    527 
    528         # post-process response

~\anaconda3\envs\ml4trading\lib\urllib\request.py in _open(self, req, data)
    542         protocol = req.type
    543         result = self._call_chain(self.handle_open, protocol, protocol +
--> 544                                   '_open', req)
    545         if result:
    546             return result

~\anaconda3\envs\ml4trading\lib\urllib\request.py in _call_chain(self, chain, kind, meth_name, *args)
    502         for handler in handlers:
    503             func = getattr(handler, meth_name)
--> 504             result = func(*args)
    505             if result is not None:
    506                 return result

~\anaconda3\envs\ml4trading\lib\urllib\request.py in ftp_open(self, req)
   1581         except ftplib.all_errors as exp:
   1582             exc = URLError('ftp error: %r' % exp)
-> 1583             raise exc.with_traceback(sys.exc_info()[2])
   1584 
   1585     def connect_ftp(self, user, passwd, host, port, dirs, timeout):

~\anaconda3\envs\ml4trading\lib\urllib\request.py in ftp_open(self, req)
   1563             dirs = dirs[1:]
   1564         try:
-> 1565             fw = self.connect_ftp(user, passwd, host, port, dirs, req.timeout)
   1566             type = file and 'I' or 'D'
   1567             for attr in attrs:

~\anaconda3\envs\ml4trading\lib\urllib\request.py in connect_ftp(self, user, passwd, host, port, dirs, timeout)
   1585     def connect_ftp(self, user, passwd, host, port, dirs, timeout):
   1586         return ftpwrapper(user, passwd, host, port, dirs, timeout,
-> 1587                           persistent=False)
   1588 
   1589 class CacheFTPHandler(FTPHandler):

~\anaconda3\envs\ml4trading\lib\urllib\request.py in __init__(self, user, passwd, host, port, dirs, timeout, persistent)
   2406         self.keepalive = persistent
   2407         try:
-> 2408             self.init()
   2409         except:
   2410             self.close()

~\anaconda3\envs\ml4trading\lib\urllib\request.py in init(self)
   2418         self.ftp.login(self.user, self.passwd)
   2419         _target = '/'.join(self.dirs)
-> 2420         self.ftp.cwd(_target)
   2421 
   2422     def retrfile(self, file, type):

~\anaconda3\envs\ml4trading\lib\ftplib.py in cwd(self, dirname)
    629             dirname = '.'  # does nothing, but could return error
    630         cmd = 'CWD ' + dirname
--> 631         return self.voidcmd(cmd)
    632 
    633     def size(self, filename):

~\anaconda3\envs\ml4trading\lib\ftplib.py in voidcmd(self, cmd)
    276         """Send a command and expect a response beginning with '2'."""
    277         self.putcmd(cmd)
--> 278         return self.voidresp()
    279 
    280     def sendport(self, host, port):

~\anaconda3\envs\ml4trading\lib\ftplib.py in voidresp(self)
    249     def voidresp(self):
    250         """Expect a response beginning with '2'."""
--> 251         resp = self.getresp()
    252         if resp[:1] != '2':
    253             raise error_reply(resp)

~\anaconda3\envs\ml4trading\lib\ftplib.py in getresp(self)
    244             raise error_temp(resp)
    245         if c == '5':
--> 246             raise error_perm(resp)
    247         raise error_proto(resp)
    248 

URLError: <urlopen error ftp error: error_perm('550 The system cannot find the file specified. ',)>

seems like the url changed but I am not sure where I can get the updated one from.

The chapter 2 01_build_itch_order_book notebook, attempts to read message type definitions from an xlsx file using Pandas read_excel method.

One of the parameters passed is the file's encoding. However the current version of pandas doesn't support the encoding argument, causing it to throw an error.

Happily simply removing the parameter allows the file to be loaded without problem, though I don't know if there might be backwards compatibility issues for people running older versions of Pandas.

Following the directions for creating the environment in the installation.md I get conflicts.

conda env create -f environment_linux.yml

I figured I would try and update a few of these and give a pull request back with updates, however I failed miserably. I tried the suggestion from @TheStoneMX in #9 and followed the directions in the post he gave. I appended '37' to the name to signify python 3.7 as I surmised this might be the issue:

conda create --name ml4t37 python=3.7
conda activate ml4t37
conda env update --file environment_linux.yml

In either case, I get an amazing amount of package conflicts (9,266 lines of conflict errors which I am attaching).

error.log

This enourmous amount of conflicts leads me to believe that I have done something wrong as it seems excessive for this. I have duplicated this on both an arch linux machine and a bionic beaver machine. Any suggestions on what I might have done wrong?

https://github.com/PacktPublishing/Hands-On-Machine-Learning-for-Algorithmic-Trading/tree/master/Chapter02/02_data_providers

Some 'U' in titles are missing